Pretty much ‘yes’ to all of that. You need appropriate rights to both the library and object to do anything. For most tasks that only requires *READ rights to the library, but you do need *ADD / *DLT to add or remove an object (plus *EXIST to the object for delete).
And you also need rights to the object appropriate to the task you are performing. eg: *USE rights to run a command, appropriate data rights to open a file, etc.