If a promotion fails, does RTC-EE automatically roll back to the original state or just report an error?

There is component based and work item based promotion. With component based promotion we promote the binaries first and then the source. For Work Item based promotion we promote the source and then the binaries. In both cases we do not have a roll back mechanism if the second step fails. In the case of component based promotion the reported problem can be fixed and the promotion rerun or you can manually sync the 'from' and the 'to' streams yourself. For Work Item based promotion if the source promote step works but the binaries promote step fails then when you try the promotion again we will attempt to promote the binaries again.

So although there is not rollback between the two steps you do have the opportunity to fix the issue and try again.

If you feel that it is important to role the first step back if the second step fails then please open an enhancement request up.