How to restrict permission to access project area and sprint planning in RTC?

There is no simple/automated solution for that as far as I can tell. RTC is for collaboration and not for preventing it.

You can find a small description of what you can do with RTC 6.x here: https://rsjazz.wordpress.com/2016/01/27/manage-access-control-permissions-for-work-items-and-versionables/ section The Rules – Work Items.

1. You can limit access on project area level. This is on a general level, not per work item type. You can use one project area for the defects and another one for the rest and manage the visibility that way
   
2. You can use categories to limit access to work items by the team areas associated to the category. Setting the category is done by hand and is not work item type specific.
3. You can use access groups - this is again not automated and not type dependent

I think the solution I would look into is number 3 with a follow up action setting/clearing the access context to the desired access group value if needed (creation, type change, type is now a defect or the type is changed away from defect) in a follow up action. E.g. clear the access context for defects so anyone has access. The way access groups work should prevent the not privileged users also from type changes.

The API's involved are described in https://rsjazz.wordpress.com/2016/01/27/manage-access-control-permissions-for-work-items-and-versionables/ and the related posts.  

Please note that the user can still look at plans, queries, reports etc. but they run with their read access so they would only be ale to see the data for the work items they are allowed to see. E.g. queries/plans for the externals would only show defects because they are not member of the access groups needed to look at the other work items.

Category based access control (item 2) works a bit different than what users expect. A follow up action as described above could also be created.

In addition to the links that Ralph provided, this tutorial does a good job of providing an overview of the various access control mechanisms:  Control access to project areas and their artifacts.