DCC not collecting all requirements / missing data?

June Boston (194●19●38) | asked Sep 13 '16, 2:01 p.m.

Hello,

DCC v5.0.2 iFix 014, Oracle db, DCC and JRS on their own WAS/RHEL server, RTC/DNG/RQM each on their own RHEL/WAS servers.

Client was having some serious data issues, with items duplicated and missing and all kinds of corruption. To remedy with a fresh start, we dropped the entire Data Warehouse from Oracle and set DCC to do a Full job. This had a few results:

a) The IDs for projects (and likely other things) changed, meaning all JRS reports had to have the projects re-selected and any manually run SQL data warehouse pulls had to have their IDs updated. I expected this sort of behaviour from prior experience with Insight, just noting it for others. Any ID that isn't directly from CLM is not guaranteed to be static during a refresh operation.

b) Some larger jobs like History were failing; we think this has been corrected by increasing the temp space in Oracle. Perhaps this impacted other jobs in a non-failing error?

c) There are lots of artifacts missing from the data warehouse now. A DNG query that returns ~1500 requirements only returns a couple hundred from the data warehouse. If I just pull from VW_REQUIREMENTS where the ID < 100, I get only a handful instead of our first hundred artifacts. I'm requesting DB access now to see if the ROIDS.REQUIREMENTS table has all our data and it somehow isn't making it into the view (I've seen issues with the soft_deleted value being incorrectly set to 1 before, though I thought that was just passed to VW_REQUIREMENTS anyways).

C is the real problem here. CCM seems fine, I'll check QM, DNG is just wildly under-collected. I am considering setting DNG to a Full ETL and running them all again; perhaps now that the temp space is expanded it won't encounter such issues? However, I have fears that maybe it's improperly processing some custom attributes or something like that. There's usually more delivered than processed in the DCC result overview for DNG jobs, but I don't know how significant that is. I am aware v6 improves DCC but unfortunately we cannot quite yet upgrade, it'll be a month or two.

4 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Francesco Chiossi (5.7k●1●9●19) | answered Sep 14 '16, 4:32 a.m.

Hello June,

one thing that can explain the difference in C could be the presence of archived project areas / artifacts.

For example if you have a project A and you archive it, it will be marked as soft deleted in the data warehouse.
However if you drop the data warehouse and recreate it from scratch the data from the archived elements won't be loaded again.

So I would recommend to check what you have in RIODS.REQUIREMENT against what you have in DNG active projects, to understand if everything is there.

Best Regards,

Francesco Chiossi

permanent link

James Freeman (16●1) | answered Sep 23 '16, 1:53 p.m.

Hi,

I'm having a similar issue. DNG requirements are present in the RIODS.REQUIREMENT table, but RRDI does not see the requirements or project area. JRS shows the active project, but nothing returns when running the report (1942 known requirements). RRDI query studio and report studio do not list the active project (Reporting Data Model (DW) -> Operational Data Store -> Requirement Area -> Requirement -> Project Name).

Setup: DCC v5.0.2 iFix 015, Oracle DB (11.2.0.4), DCC and RRDI are on their own WAS (8.5.5) / RHEL (6.7) server, RTC/DNG/RQM on one RHEL/WAS server (JTS).

History: Upgraded from version 3.0.1.1 to 5.0.2 on 15 June 2016.

Any help is greatly appreciated.

v/r

James Freeman

Comments

James Freeman commented Sep 26 '16, 7:47 a.m.

Thank you Francesco,

I double checked the ISSOFTDELETED column and all requirements are set to 0.

I'm the user with all of the DNG roles are assigned; Administrator, Author, Commenter, Project Snapshot Administrator. The access control is set to Members of the project area hierarchy. Our email addresses did change before the upgrade. I'm able to see the CCM data, but not the DNG requirements. The only Project/requirements that are showing up in JRS/RRDI are from a prototype version during our initial testing in 2011.

Thank you,

James

Francesco Chiossi commented Sep 27 '16, 10:13 a.m.

Hello James,

to test if the problem is with the Project Access Control you can try to disable it.
To do so:
1) In JRS, go to the Data Sources page
2) open the Data Warehouse data source
3) Uncheck the Enable project access control option
4) Save
5) Run the report again

Do you see more data?

Best Regards
Francesco Chiossi

James Freeman commented Sep 28 '16, 7:44 a.m.

Francesco,

I'm seeing the data now in JRS. This will work for my purpose for now. Do you know why I don’t see the data/project in RRDI?

Thank you very much,

v/r

James

Francesco Chiossi commented Sep 28 '16, 8:58 a.m.

Hello James,

if you can see the data it means that the Project Access Control data for your user is not correctly populated in the data warehouse.

Project Access Control, is the feature that allows a user to only see data in the reports from the projects he is member of in CLM.

If you are not interested in using it, you can disable Project Access Control also in RRDI:
see Switching to the unrestricted data model

Best Regards
Francesco Chiossi

James Freeman commented Sep 28 '16, 9:54 a.m.

Francesco,

How do I fix the Project Access Control in the data warehouse? I have another problem that was discovered after the upgrade. Some of my project members display a warning that they are not members of the project/team area. I removed everyone and added them back to the project/team area. Now, only about 50 out of 108 members belong to the team area, a different set of people from before.

Thank you again for helping. I'll take a look at switching to the unrestricted data model for a temporary work around for now.

v/r,

James

James Freeman commented Sep 30 '16, 3:02 p.m.

I was able to switch to the unrestricted data model in RRDI, but not with the directions provided above. I’m able to see the requirements now. Thank you.
Steps taken to switch to unrestricted data model:
1. From RTC’s menu, select Reports -> Using Cognos
2. Select “Manage resource in Cognos Connection”, log into cognos if required.
3. Select Launch -> IBM Cognos Administration
4. Configuration tab
5. Content Administration
6. Folder list – IBM Content for RRDI 5.0.2
7. Rational Reporting Data Model (DW+CALM, select “More…”
8. Set properties
9. Import tab
10. Deployment archive: “Change the deployment archive…”
11. Selected “Rational Reporting Data Model (DW+CALM)-no-security
12. Default options – replace existing entries, include data sources and connections, do not include references to external namespaces, owner form the source, and new and existing entries.
13. OK.
14. I had to setup the connection to the data warehouse for RIDW with Oracle settings (signon).

showing 5 of 6 show 1 more comments

permanent link

Francesco Chiossi (5.7k●1●9●19) | answered Sep 26 '16, 4:26 a.m.

Hello James,

if the requirements are present in the RIODS.REQUIREMENT table but not visible in the reports (JRS or RRDI) there are 2 possible causes that comes to my mind:

1) The requirements are masked as deleted. Check what is the value of the ISSOFTDELETED column in RIODS.REQUIREMENT; if it's 1 then the requirement is marked as deleted and will not show up in the reports.

2) Your user don't have the project access control permissions to see the data.
This can happen if you have project access control enabled and your user:
- is not a member of the project area
- does not have a valid email address in CLM

Best Regards
Francesco Chiossi

Comments

June Boston commented Sep 28 '16, 1:05 p.m.

We had a PMR on this matter. We had queried the RIODS tables and determined that many artifacts were marked ISSOFTDELETED, despite not actually being deleted. On IBM's advice we ran a query to manually update everything to ISSOFTDELETED, then set the ETL mode for DNG to Full and ran a full ETL. Our data became good again.

Of course, now some time later we are having our ODS RM ETL fail entirely and generally lock up the RM server while doing so. I think v5 DCC has some fundamental issues with the ETL programming. We will upgrade to 6 in November.

permanent link

Norman Dignard (356●6●81●142) | answered Sep 27 '16, 2:43 p.m.

We have similar problems after upgrade from 502 ix14 to 602 ifix 3. Single CLM install on RH w/Oracle backend.
In 2 of our projects, we have missing/duplicate data or no data at all. These projects were not deleted/archived (although we have a number of archived RM projects.) We had to wait for Ifix 2 to address migration problems due to archived projects.

I have a PMR in support on our RM problems post upgrade but have yet to hear back from them.

One suspicion I have is that there was some data corruption in 502 usage resulting from stalled doc imports. One problem we had as causing stalled imports was lack of disk space for the index dir. The only recourse to get this working again was to restart JAZZ. I think this is were the data corruption occurred. The app however appeared to function without issue. Support also had us run a bunch of utilities on rm (verify, validate module etc..) just prior to the upgrade and noted no issues.

Not much help in resolving your issue but your problem does not appear to be unique.

Norm

Comments

Norman Dignard commented Sep 27 '16, 2:44 p.m.

I should have noted that DCC is also failing in its RM job which i believe is due to RM I problems noted

June Boston commented Sep 28 '16, 1:18 p.m.

I would try an offline re-index with repotools-rm, and make sure your database's temp space has enough room to expand during ETLs. I think that's what is doing this, I'll update if we fix it.

Norman Dignard commented Sep 28 '16, 2:16 p.m.

Already tried that and its still a no go. Index space is not a problem.

June Boston commented Sep 28 '16, 4:03 p.m.

How's the database server doing? Tomorrow we are going to run a data collection while actively watching the Oracle server, to see if it bottlenecks on CPU or swap space or something.

Norman Dignard commented Sep 29 '16, 10:55 a.m.

The db side was fine. The DCC job was always failing at the same spot in trying to find a specific reference in running its rm job.

It's all about the answers!

Related questions

DCC not collecting all requirements / missing data?

4 answers

Your answer