How can be set "IF priority is set to high then story point field should be mandatory" in work item type.
One answer
This can be achieved by writing a JavaScript attribute customization of type condition and configuring the "Required Attributes for condition" advisor with the condition. See Process Enactment Workshop for the Rational solution for Collaborative Lifecycle Management where one of the examples shows this in Lab 5.
Also see Customization of Work Items in Rational Team Concert and the Jazz.net wiki page on Attribute Customization
Also see Customization of Work Items in Rational Team Concert and the Jazz.net wiki page on Attribute Customization