How to set state transitions to only the work item owner "owned by"

As far as I can tell, you would have to create a custom work item save advisor (operational behavior) that would detect and block the save of the state change if the user doing it is not the owner. Some links to material for training and to enable yourself:

https://jazz.net/library/article/1000
https://rsjazz.wordpress.com/2013/02/28/setting-up-rational-team-concert-for-api-development/
https://rsjazz.wordpress.com/2012/11/01/restrict-delivery-of-changesets-to-workitem-types-advisordelivery-of-changesets-associated-to-wrong-work-item-types-advisor/