In RQM 4.0.4, how to filter cases belonging to function and not belonging to a specific test plan?

I think "Function" and "Test Plan" should be part of the Column Display Setting.
Then you can set the filter criteria for "Function".
For test plan you can either mark "not in any test plan" or mark all available test plans but not your specific test plan.

For me it is not clear if you talk about two different testplans (add to TP A if not in TP B) or only one Test plan is meant. In later case if you add within a test plan test cases, only test cases are offered which are not already part of the test plan.

I'm using RQM 4.0.5 and hope that nevertheless this answer is correct for your environment.:-)

    Regards, Jörg