How to recover missed email notification as SMTP server down
The SMTP server used by our RTC server went down for more then two days and we have many email notifications from work item updates failed to be sent out.
After we restored the SMTP server, we found that we can only receive the email notifications from new updates, but can't recover theose missed and supposedly accumulated during the SMTP server down.
The error we saw from the jazz.log when the SMTP server down is as below:
ERROR com.ibm.team.repository - Could not connect to SMTP host: xx.xxxx.ibm.com, port: 25;
This means that the failed emails won't be stored on the SMTP server.
How RTC email notification handle the pending emails?
How do they do the retry when a connection failure occur?
Most important of all, how do we recover the missed emails now? Are they lost forever?
Please share your experience and help
After we restored the SMTP server, we found that we can only receive the email notifications from new updates, but can't recover theose missed and supposedly accumulated during the SMTP server down.
The error we saw from the jazz.log when the SMTP server down is as below:
ERROR com.ibm.team.repository - Could not connect to SMTP host: xx.xxxx.ibm.com, port: 25;
This means that the failed emails won't be stored on the SMTP server.
How RTC email notification handle the pending emails?
How do they do the retry when a connection failure occur?
Most important of all, how do we recover the missed emails now? Are they lost forever?
Please share your experience and help
6 answers
The work item e-mail task is able to recover and send out missing
e-mails for a short period. Did you see any messages in the log related
to ChangeEventMailNotifier?
As long as SMTP is down, feeds can be used to track work item changes
(its the preferred way of tracking changes actually) from the "My Work
Item Changes" feed.
e-mails for a short period. Did you see any messages in the log related
to ChangeEventMailNotifier?
As long as SMTP is down, feeds can be used to track work item changes
(its the preferred way of tracking changes actually) from the "My Work
Item Changes" feed.
The work item e-mail task is able to recover and send out missing
e-mails for a short period. Did you see any messages in the log related
to ChangeEventMailNotifier?
As long as SMTP is down, feeds can be used to track work item changes
(its the preferred way of tracking changes actually) from the "My Work
Item Changes" feed.
How long is the short period? Our SMTP server was down for over two and half days. Is it still possible to recover them and how?
It sounds like RTC is handling emails differently than other apps, which will hand off to a local process to send/queue mail.
If I understand right, the email notification is for communication collaoration so all subscribers will be notified with the work item change. But feed is only for each individual unless others RSS it?
I do see messages related to ChangeEventMailNotifier recorded during the SMTP server down, which are nested withing the message about the connection to the SMTP server was refused:
at $Proxy187.sendMail(Unknown Source)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.sendIndividualMails(ChangeEventMailNotifier.java:405)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.processOneChange(ChangeEventMailNotifier.java:362)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.runTask(ChangeEventMailNotifier.java:202)
Here is the whole message:
2010-01-18 02:46:39,323 ERROR com.ibm.team.repository - Could not connect to SMTP host: xxxx.swg.usma.ibm.com, port: 25;
nested exception is:
java.net.ConnectException: Connection refused
javax.mail.MessagingException: Could not connect to SMTP host: xxxx.swg.usma.ibm.com, port: 25;
nested exception is:
java.net.ConnectException: Connection refused
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1213)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:311)
at javax.mail.Service.connect(Service.java:233)
at javax.mail.Service.connect(Service.java:134)
at javax.mail.Service.connect(Service.java:86)
at com.sun.mail.smtp.SMTPTransport.connect(SMTPTransport.java:144)
at com.ibm.team.repository.service.internal.MailerService.getConnectedTransport(MailerService.java:464)
at com.ibm.team.repository.service.internal.MailerService.sendMessage(MailerService.java:417)
at com.ibm.team.repository.service.internal.MailerService.sendMail(MailerService.java:391)
at com.ibm.team.repository.service.internal.MailerService.sendMail(MailerService.java:330)
at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:378)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:364)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy187.sendMail(Unknown Source)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.sendIndividualMails(ChangeEventMailNotifier.java:405)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.processOneChange(ChangeEventMailNotifier.java:362)
at com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier.runTask(ChangeEventMailNotifier.java:202)
at com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:88)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:378)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:364)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy132.executeTask(Unknown Source)
at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:118)
at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:92)
at java.lang.Thread.run(Thread.java:810)
Please CC to 47010: Are lost notifications being resent?, its related.
The interval until notifications are lost is 16 hours at the moment.
It turns out that our RTC server already has an local SMTP server running, accepting only local connections. I then configured RTC to use localhost as its mail server, then it seems working right away.
This way, our IT experts explained to me, the system will accept, queue, and forward e-mails instead of RTC doing so. In the end the mail will still go through the remote cooperate SMTP server . The benefit of this is that if the remote SMTP server is unreachable for whatever reason, localhost will very very likely still be reachable and able to queue mail unless the system itself is down (then RTC would be too :)). The system's sendmail is configured to queue mail for 5 days (the default) before giving up and we changed it to 10 days. Also, this would generate system log entries an admin could read if there are problems.
We'll need to wait until the next downtime to verify the queuing and resending part.
Please CC to 47010: Are lost notifications being resent?, its related.
The interval until notifications are lost is 16 hours at the moment.
It turns out that our RTC server already has an local SMTP server running, accepting only local connections. I then configured RTC to use localhost as its mail server, then it seems working right away.
This way, our IT experts explained to me, the system will accept, queue, and forward e-mails instead of RTC doing so. In the end the mail will still go through the remote cooperate SMTP server . The benefit of this is that if the remote SMTP server is unreachable for whatever reason, localhost will very very likely still be reachable and able to queue mail unless the system itself is down (then RTC would be too :)). The system's sendmail is configured to queue mail for 5 days (the default) before giving up and we changed it to 10 days. Also, this would generate system log entries an admin could read if there are problems.
We'll need to wait until the next downtime to verify the queuing and resending part.
Having a local SMTP server sounds like an excellent idea. Not heard that one before
anthony
Hi Guowei,
What would the port be if we are using the localhost as SMTP? Will it still be 25?
Regards,
Sunita
It turns out that our RTC server already has an local SMTP server running, accepting only local connections. I then configured RTC to use localhost as its mail server, then it seems working right away.
This way, our IT experts explained to me, the system will accept, queue, and forward e-mails instead of RTC doing so. In the end the mail will still go through the remote cooperate SMTP server . The benefit of this is that if the remote SMTP server is unreachable for whatever reason, localhost will very very likely still be reachable and able to queue mail unless the system itself is down (then RTC would be too :)). The system's sendmail is configured to queue mail for 5 days (the default) before giving up and we changed it to 10 days. Also, this would generate system log entries an admin could read if there are problems.
We'll need to wait until the next downtime to verify the queuing and resending part.
What would the port be if we are using the localhost as SMTP? Will it still be 25?
Regards,
Sunita
Please CC to 47010: Are lost notifications being resent?, its related.
The interval until notifications are lost is 16 hours at the moment.
It turns out that our RTC server already has an local SMTP server running, accepting only local connections. I then configured RTC to use localhost as its mail server, then it seems working right away.
This way, our IT experts explained to me, the system will accept, queue, and forward e-mails instead of RTC doing so. In the end the mail will still go through the remote cooperate SMTP server . The benefit of this is that if the remote SMTP server is unreachable for whatever reason, localhost will very very likely still be reachable and able to queue mail unless the system itself is down (then RTC would be too :)). The system's sendmail is configured to queue mail for 5 days (the default) before giving up and we changed it to 10 days. Also, this would generate system log entries an admin could read if there are problems.
We'll need to wait until the next downtime to verify the queuing and resending part.