The 'write' access of a component (ex: modifying it's name, owner, visibility) is governed by the component owner.
If the component is owned by a contributor, then only that contributor can change these properties. If it is owned by a project or team area, then this can be controlled by permissions in the project area itself.
Ex: "Process Configuration" tab --> Team Configuration --> Permissions --> Source Control / Save Component (server)
(in particular "Remove a component from a project/team area" or "Add a component to a project/team area").

The visibility field of a component (i.e it's read access) will control who can see the component and it's change sets, but will not grant 'write' access to that component (as explained above).