I've extended a new link type for the work item. And the target end
point object is also a work item. Now we are trying to find a way to
give a customized dialog to allow user to select the target work item
in different way when users try to add this link type to a work item.

But we didn't find any extension point to do that. Is there a possible
way to do this extension? And is there a way to change the trigger
point to add the link for the work item? For example, when user try
to add our customized link type, a dialog provided by us will be
pop-up to enable user to select the target endpoint object instead of
the current dialog.

There is a mechanism to contribute picker modules as documented in the
OSLC-CM specification
see e.g. https://jazz.net/wiki/bin/view/Main/CALMPicker

Your problem sounds a bit different, though, as you are trying to extend
RTC itself.
Unfortunately, there is currently no extension point for this, but I have
created

https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/95421

for this request.

Another question related with this topic is that can we add an
extended model item as a target endpoint for a link type? For
example, if we have a customized item which extends the basic
artifact item in RTC, can we link a work item to this type of
customized item?

Yes, that shouldn't be a problem. If you also implement the
StandardLabelProvider mechanism for these items, the label in the work
item editor is under your control. Otherwise, we will show the link
comment. In order to open the item, you will also need to participate in
the HyperlinkHandler infrastructure. The missing part is again the picker,
as described above.

See also:
com.ibm.team.jface.labelProviders.StandardLabelProvider
com.ibm.team.foundation.rcp.core.hyperlinks.HyperlinkHandler


--
Regards,
Patrick
Jazz Work Item Team