Force files to be in UTF-8 Format for checking in the code
Accepted answer
It looks like there is a precondition (Restrict Check-in Based on Specified MIME Types and Encodings) that you can set to restrict check-in of files to only those files that have a specific encoding. You can set the scope of the precondition to all components in the project area or to specific components. See Team process preconditions for Rational Team Concert source control for details.
2 other answers
RTC does not have the control of the file encoding. The encoding should be set in Eclipse IDE or on Eclipse project level.
Unless you wait for 6.0.4 and the availability of the check in operation you would have to prevent delivery similar to https://rsjazz.wordpress.com/2012/11/01/restrict-delivery-of-changesets-to-workitem-types-advisordelivery-of-changesets-associated-to-wrong-work-item-types-advisor/