It's all about the answers!

Ask a question

DNG and ETM (7.0.2) How do people handle Analysis and Calculation Evidence?

Glyn Costello (14044) | asked Jan 05, 12:11 p.m.
edited Jan 05, 12:14 p.m. by Fariz Saracevic (919613)

 Dear all, 

An open question here, looking for best practice and experience of others. 

How have people found it best to handle traceability of requriements to V&V evidence such as design inspection, calculations, simulations and analysis (i.e. not tests)? This is more for non-software systems and product engineering. 

Do you manage this in DNG with a custom link type? Or in ETM with the "Validates" link type? We have customers who want to see documented evidence anmd traceability to requirements. 

We've previously tried writing up the analyses in ETM as a test case, the execution result then becomes the result of any associated team-review of the analysis. But this is a bit cumbersome. Especially when it comes to publishing and sending the evidence to the customer (i.e. a well formatted report with images, attachments, etc.). 

What are other's experience?


Accepted answer

permanent link
Davyd Norris (2.4k217) | answered Jan 07, 7:42 p.m.
edited Jan 07, 8:57 p.m.
I do a lot of work with clients in the Civil and Construction space, and in the Defence and System Engineering space where there is a lot of Hardware and Mechanical V&V required.

There are two approaches people take in these areas, one of which is the easy way out to an extent, and the other is a much more rigorous approach and works well but which requires an actual effort. Most contractors want the easiest way out!

Easy way: 

Just use DNG and define Verification (or Verification Action) artefacts and Verified By link types. The Verification artefact Primary Text contains the acceptance criteria that needs to be met, and then there are attributes for Verification Method, Project Phase, Verification Result, Verification Evidence (usually a URL to the transmittal system being used), Evidence Document, Section, and Client Acceptance. Workflow is used to follow maturity of the Verification itself, not the result

During early analysis, the Verifications are created and the criteria and method are filled in. Verifications are linked to requirements and it's a many to many relationship - if you need more than one method then you create multiple Verifications, and if one Verification can tick off several requirements you link them both to the one artefact. 

As the project progresses, the phase and status are updated, and the evidence is supplied via formal transmittal. A link or ID for the evidence is supplied, and the specific document or file and section/datum is added, along with the contractor's analysis of the result. The client can then assess the progress and then run acceptance reviews - sorting by document and section makes it much easier to go through things like design reports or analysis reports by page turning.

More rigorous way:

Define your verification criteria as an ETM Test Case and use all the capability of ETM to define phase plans, etc. If your Verification requires an ITP, an inspection, or some sort of analysis, then create it as a Test Script so that you have a check list to follow as a Test Procedure. Use Execution Records to contain the Results and either record it directly, and attach evidence or a link to the transmittal data similar to the above, or run the procedure and record the results against the steps. I have found inspections work really well when you can execute the procedure on a mobile device and take photos of welds or joints and attach to the steps.

In this case you use the Execution Record result to record the contractor analysis of the evidence, and the TER state to record the client acceptance of the result.

I have several clients who then also take advantage of the Test Environments and Test Cell definitions to define their test beds - this requires pretty much completely gutting the existing out of the box values, which are all very software centric, but it works very well, especially when they have several testing facilities around the world and then can book the test cells ahead of time
Glyn Costello selected this answer as the correct answer

Glyn Costello commented Jan 08, 4:35 a.m.

 See below resopnse, too long for a comment

One other answer

permanent link
Glyn Costello (14044) | answered Jan 08, 4:35 a.m.

 (To long for a comment)

 Hi, thanks for the excellent response. You have pretty much described our journey in exploring how we could implement this. We started the other way round, by trying to use ETM for all V&V evidence, however, we were trying to make the TCER and Result the "Evidence" by writing up the results of analyses and inspections directly in the tool. when people struggled with this approach, we allowed people to upload reports written in Word to ETM, but then managing and retrieving the "evidence" to send to a customer then became much more difficult. We also (as you indicated) gutted the built-in environments in ETM (frustrating that it's so software oriented). 

All of which has led us to think that doing it all in DNG might be more approachable to people, especially when it comes to producing VVRMs and supplying/exporting evidence to customers. 

I have also considered making the TCER just the generic "checklist" of the analysis or calculation and have all the details of the analysis in the test case. 

Do you have a process for "sizing" calcs and anaylsis that inform the specs/design? i.e. how big should this weld be to withstand the defined load requirements?

Davyd Norris commented Jan 08, 6:29 p.m.
My overwhelming preference would be to use ETM - it's actually a really cool way to approach the whole quality process, but there's a lot of pushback in the Civil and Construction space for this, where they typically get some poor lackey to upload everything as spreadsheets.

If you are working in a contracting environment then typically they will have some sort of document management or transmittal system that has been mandated for all official communication, including the submission of all evidence, and so this will force your hand and you'll end up using custom fields to point to the formal submission.

In one large project I'm consulting on, the entire project uses ELM, and everything is linked up, and the contractor uses ETM to do all their testing and quality work on a daily basis, but is then required to print the results as a PDF using ELO Publisher and submit it via the transmittal system!

This is despite the stakeholders watching the quality process in real time in ELM!!!

Davyd Norris commented Jan 08, 6:36 p.m.
To answer some of your points:
 - make the Test Case the criteria for acceptance, specifically the Test Case Design
 - make the Test Script the steps for the Test Procedure if there is an actual set of things they have to do. If there's no actual procedure and there is a report or certificate or other that has to be supplied then don't write a Test Script at all. We use the Expected Results to document what the expected evidence will be
 - use the TCER to document and schedule the actual Verification work
 - use the Test Results to document the evidence and where it is, or to capture the Test Procedure script results
 - use the TCER final result to document the contractor's view of the Verification result
 - use the TCER status to document the client's view of the Verification result

Your answer

Register or to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.