How to avoid work items to be mixed between two build definitions that use different workspaces but common stream?

I have a code stream "Main Development Stream" which has components say:-

• component 1
• common component
• component 2

Now I have got two RTC Build definitions, namely "Main Build Definition" and "Common Build Definition" using two separate RTC build workspaces "Main Build Definition Workspace" and "Common Build Definition Workspace".

Main Build Definition needs to build code in component 1 and component 2 whereas Common Build Definition needs code only from common component.

Below are my expectations:-

1. Work Items populated in "Main Build Definition" build result as well as the build comparison should show the work items for which the code has been delivered to component 1 and component 2 only, and NOT common component.
2. Work Items populated in "Common Build Definition" build result as well as the build comparison should show the work items for which the code has been delivered to common component only and NOT component 1 and component 2.

What I actually observe is, the work items populated in the build result and build comparison are for all components mixed.

Please note I want to make the two builds completely independent of each other, and should not mix work items .

I tried the 'Components to exclude' option to exclude components in the build definitions, but it did not seem to have any effect.
I also tried creating "Common Build Definition Workspace" by including just the common component, but the RTC refreshes and accepts other components automatically when we request a new build.

Any pointers to achieve what I want, would be appreciated!

Thanks!