Is it possible to set the mandatory field for Links in RTC?

If you look at https://jazz.net/wiki/bin/view/Main/AttributeCustomization which is the only "official" documentation for script based extension, you might realize that work item links are not mentioned - because that API does not provide access to them as they are actually not attribute based.

Now, you might be able to debug and find some internal DOJO API that you can use.

The Java API provides access to the links of a work item. You can create a work item save advisor that prevnts from saving if there are some kinds of links missing. See https://rsjazz.wordpress.com/?s=advisor for examples for advisors.