I'm in search of the same information. What I did find is the following possibilities but I'm reluctant to make the change without getting some input from the Jazz team first.

One possibility is to remove the RQM service (if you have set that up already) and then adjust the memory setting in the service creation batch file, default max is 700 MB I believe.

The other way might be to set an environment variable:
CATALINA_OPTS and JAVA_OPTS
with..... -server -Xmx256m .... or whatever value you desire for the limit.....

On my Windows server I do not have either of those variables and it seems that it is probably somewhere in the registry, which would be an easier way to make the change without setting a server env and rebooting the server to make sure its picked up by the tomcat app.