Can RTC handle merging of WTX maps ?

I am not aware of a supported integration with WTX maps, and doubt that there will be documentation on it.

Assuming that a WTX map is stored in a file, then the RTC merge machinery will invoke whatever merge tool is registered in your IDE for a WTX map file.   So the real question is whether a WTX merge tool is registered to your IDE.   If you are using the Eclipse client, RTC will invoke whatever merge tool is registered with Eclipse for that type of file, or you can tell RTC to use some non-Eclipse merge tool.   But in either case, you will need a WTX merge tool for RTC to invoke.