Reporting incorrect test result
We are using RQM to run Rational Functional Tester Java scripts and seeing problems with tests that actually fail being reported as passing.
Initially we had a problem where we were calling RationalTestScript.logTestInfo() and passing a log string containing control characters - this was resulting in XML files which did not parse. Somehow or other this meant that when we accessed the log files from RQM (even though the last thing the Java code did was call logTestResult(false)) the RQM result was shown as a pass (possibly it's down to failing to parse back the incorrect XML). We're now stripping control characters from log statements but still seeing fails reported as passes. When this happens we're also unable to see the RQM logs (they download from the server as zero size files). This is causing us major issues as we can no longer trust the RQM results. This is with RQM 2.0.1.1 iFix 3 and Rational Functional Tester 8.2.0.2. Is anyone else seeing this behaviour ? |
2 answers
|
The problem was caused by logging (i.e. calling RationalTestScript.logInfo()) in separate threads. It appears the logging API is not thread safe. It's easily reproducible with the RFT code below:
public void testMain (Object []args) { int a = 100; ThreadLogger[] threadLoggers = new ThreadLogger ; Object notificationObject = new Object(); for (int i = 0; i < (threadLoggers.length - 1); i++) { threadLoggers = new ThreadLogger ("Logging from thread number " + (i + 1), null); threadLoggers .start (); } threadLoggers = new ThreadLogger("Logging from last thread", notificationObject); threadLoggers .start (); synchronized (notificationObject) { try { notificationObject.wait (); } catch (InterruptedException ie) { } } // Order threads run not deterministic (can't rely just on completion of final thread above). Make sure everything // is complete before logging failure try { Thread.sleep (2000); } catch (InterruptedException ie) { } System.out.println("Logging the fail now"); logTestResult("Should be a fail", false); } protected class ThreadLogger extends Thread { private String m_msg; private Object m_notificationObject; public ThreadLogger(String msg, Object notificationObject) { m_msg = msg; m_notificationObject = notificationObject; } public void run () { RationalTestScript.logInfo(m_msg); System.out.println(m_msg); if (m_notificationObject != null) { synchronized(m_notificationObject) { m_notificationObject.notify(); } } } } When this happens the XML log is corrupt. Failure to parse it subsequently causes the test to be reported as a pass in RQM despite calling RationalTestScript.logTestResult() passing false for fail. |
Hi Paul,
we are facing a similar issue with following versions: RQM 2.0.1 iFix5 and RFT 8.2. Do you know if the development team is aware of this and if it is resolved or planned to work on it. If yes, do you know which versions should fix this issue? Have you found any workarounds on this? Thanks in advance. |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.