I had the same problem with installing /rm to an existing installation. I found that Liberty is not copying over the war file and directory from the liberty template directory.
stop your server, go to .../server/liberty/clmserverTemplate/apps copy the dm.war.zip AND the dm.war directory over to: .../server/liberty/servers/clm/apps now restart your server and login to jts/admin you should now be able to run setup to configure /rm and register it. NOTE if you are using an enterprise database (DB), you have to set up the database first, so you can enter its configuration parameters during setup.