EDC5133I No space left on device during Dependency build

Alex Akilov

(121●19●24) edited

Mar 09 '15, 6:19 p.m.

We have a few components under source control. We created two build definitions that use the same language definitions/translators but a different build subset (in both cases we had to keep the subsets small since we were getting IOExceptions if the subset contained more than a few dozen source members). The problem is that the first build worked but the second one is failing fairly consistently.

Here's an excerpt from the build log:

* [(internal) logpublishertask] com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device.

* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager$StreamLengthUtility.run(ContentManager.java:263)

* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager.storeContent(ContentManager.java:403)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.storeContent(ContentPublisher.java:180)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.getFileContent(ContentPublisher.java:94)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractFilePublisher.initializeContribution(AbstractFilePublisher.java:49)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractContributionPublisher.publish(AbstractContributionPublisher.java:120)

* [(internal) logpublishertask] at com.ibm.team.build.internal.ant.AbstractContentPublisherTask.updateBuildResult(AbstractContentPublisherTask.java:174)

* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractPublisherTask.doExecute(AbstractPublisherTask.java:105)

* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:666)

Snipped the rest of the stack trace. After that we see the following:

* [antz:compile] 2015-03-09 12:48:04,709 CRHTC1501E Failed: MSP FM/Batch COBOL compilation and link-edit : SPKP.BLZBLD.CP05.DEV.BT.P.COBOL(PMSP314L) BuildException : The following error occurred while executing this line:

* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..

* IKJ56246I FILE SYSADATA NOT ALLOCATED, FILE IN USE

And so on (i.e. same error for the rest of the source members in the subset).

Is there a way to find out what device it's talking about?

Some more details about this issue. I narrowed it down to a single program that seems to be the one that RTC is struggling with. It's about 12K lines long (100Kb in size). The impact analysis for it contains 123 files it depends on. The buildable files list that gets generated for it is 105K in size. Are we hitting some size limit that RTC doesn't seem to be able to handle? If so, this could be a showstopper for us as we have a lot of "large" programs that could cause a similar problem for RTC.

0 votes

Comments

Ralph Schoon

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Mar 10 '15, 3:28 a.m.

Have you checked the disk size? That is what the error message clearly hints to.

Alex Akilov

Mar 12 '15, 1:52 p.m.

I'm not sure which folder it's complaining about. Here's the output of df -kP (load directory is in /var/jazz502 which has 46% available). Granted, some of these allocations are low and we've asked for additional space but it would help if the error message was a little more helpful in telling us where is the space that we're running out of (e.g. is it tmp space? work/load directory? What file was it trying to write to? etc.). df -kP Filesystem 1024-blocks Used Available Capacity Mounted on OEMP.RTC.V5R0M2.SYSS.ZFS 874080 406713 467367 47% /usr/lpp/jazz/v5.0.2 OMVS.SYSS.CICSTS42.ZFS 28800 26409 2391 92% /CICSTZ42/usr/lpp/cicsts/cicsts42 OMVS.DB2.DAS1.DB2MQL 1920 1723 197 90% /DB2/DAG1/mql OMVS.DB2.DAS1.DB2WORF 1440 1352 88 94% /DB2/DAG1/worf OMVS.DB2.DAS1.DB2JDBC 11520 11039 481 96% /DB2/DAG1/jdbc OMVS.DB2.DAS1.DB2BASE 4320 4146 174 96% /DB2/DAG1 OMVS.DB2.DDS1.DB2MQL 1920 1723 197 90% /DB2/DDG1/mql OMVS.DB2.DDS1.DB2WORF 1440 1352 88 94% /DB2/DDG1/worf OMVS.DB2.DDS1.DB2JDBC 11520 11039 481 96% /DB2/DDG1/jdbc OMVS.DB2.DDS1.DB2BASE 4320 4146 174 96% /DB2/DDG1 OMVS.DB2.D2SA.DB2MQL 1920 1723 197 90% /DB2/D2D0/mql OMVS.DB2.D2SA.DB2WORF 1440 1352 88 94% /DB2/D2D0/worf OMVS.DB2.D2SA.DB2JDBC 11520 11039 481 96% /DB2/D2D0/jdbc OMVS.DB2.D2SA.DB2BASE 4320 4146 174 96% /DB2/D2D0 UNIX.SYSS.SSERS2.SIGYROOT 768 326 442 43% /usr/lpp/cobol UNIX.SYSS.SSERS2.SCFZHFS2 108720 29952 78768 28% /SYSTEM/var/wbem UNIX.SYSS.SSERS2.JV390 576000 323081 252919 57% /usr/lpp/java/J7.0_64 UNIX.SYSS.SSERS2.ROOT 2348000 2314573 33427 99% / /TMP 10240 216 10024 3% /SYSTEM/tmp *AMD/u 4 4 0 100% /u SPKP.SYSS.SP0309.HFS 3600 52 3480 2% /u/sp0309 TSNW.SYSS.TS0620.HFS 720 60 592 10% /u/ts0620 SPAC.SYSS.SP5572.HFS 49680 32304 17256 66% /u/sp5572 OMVS.SYSS.ZENA.AGENT.HFS 5760 4464 1176 80% /SYSTEM/etc/Zena/agent OEMP.RTC.V5R0M2.SYSS.VAR.HFS 28800 12960 15724 46% /SYSTEM/var/jazz502 OEMP.RTC.V5R0M2.SYSS.ETC.HFS 3600 72 3460 3% /SYSTEM/etc/jazz502 OEMP.RTC.V5R0M3.SYSS.VAR.HFS 43200 56 43024 1% /SYSTEM/var/jazz50 OEMP.RTC.V5R0M3.SYSS.ETC.HFS 3600 76 3456 3% /SYSTEM/etc/jazz50 OEMP.RTC.V5R0M3.SYSS.HFS 504000 375588 128344 75% /usr/lpp/jazz/v5.0 OMVS.SYSS.IBMRAA.HFS 18000 280 17596 2% /usr/lpp/dmh OMVS.SYSS.INTERNET.LOGS.HFS 108000 70772 37108 66% /usr/lpp/internet/server_root/logs OEMP.RDZ.V9R1M0.SYSS.VAR.HFS 1785600 815952 969444 46% /SYSTEM/var/rdz OEMP.RDZ.V9R1M0.SYSS.CFG.HFS 720 76 576 12% /SYSTEM/etc/rdz OEMP.RDZ.V9R1M0.SYSS.HFS 259200 182636 76448 71% /usr/lpp/rdz OMVS.DB2.DDS1.HFS 144000 48968 94916 35% /DB2V9/DDG1 OMVS.DB2.D2SA.HFS 144000 48968 94916 35% /DB2V9/D2D0 OMVS.DB2.DAS1.HFS 144000 48968 94916 35% /DB2V9/DAG1 OMVS.SYSS.SYSLOGD.LOG.HFS 3600000 297156 3302748 9% /tcpip UNIX.SYSS.ZOSV1R8.CRON 720 32 620 5% /usr/lib/cron UNIX.SYSS.ZOSV1R8.SPOOL 1440 884 432 68% /usr/spool OMVS.SYSS.X509.CERTIFS.HFS 1440 412 960 31% /SYSTEM/etc/x509 UNIX.SYSS.SSERS2.NETVHFS 7200 5516 1564 78% /usr/lpp/netview/v6r2 UNIX.SYSS.JAVA60.SR5.JV390 344160 343312 772 100% /usr/lpp/java/J6.0.1_64 UNIX.SYSS.ZOSV2R1.DEV 720 100 504 17% /SYSTEM/dev UNIX.SYSS.ZOSV2R1.VAR 2160 904 1136 45% /SYSTEM/var UNIX.SYSS.ZOSV2R1.ETC 2880 2224 548 81% /SYSTEM/etc

5 answers

10,336 views

0 votes

Accepted answer

Permanent link

Liam Doherty

JAZZ DEVELOPER (231●2) Mar 27 '15, 8:46 a.m.

I wonder if it is /tmp that has the issue. USS services seem to be not so good to put a good message out when it has a space issue. Is there anything in the system log? Also I see your /u/ has no space. I presume this is because your user set up a mount point at /u. Is there a chance that the userid running the build has not got a directory set up?

Alex Akilov selected this answer as the correct answer

0 votes

Comments

Alex Akilov

Apr 15 '15, 4:38 p.m.

Looks like you guys were right. Our /tmp was in memory structure for performance which is why it was allocated so small (10M). We increased it to about 32M with the ability to grow 4 times by 4M each time and that seems to have alleviated the problem. Still, would be good to be able to override the location of the scratchpad. As I indicated, setting the tmpdir value in bfagent.conf and the gateway shell script didn't seem to help.

Ralph Schoon

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Apr 16 '15, 2:43 a.m.

Please provide an enhancement request, if you think this would be useful.

4 other answers

Permanent link

Liam Doherty

JAZZ DEVELOPER (231●2) Mar 24 '15, 5:15 a.m.

Alex, I believe the message tell you the file it is having a problem with:

* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..

In ISPF 3.17 can you navigate to /var/jazz502/build/MSP_PLEVEL. Then next to the MSP_PLEVEL directory enter the FS line command. This will tell you how much space is available similar to this:

Block size . . . . : 1024
Total blocks . . . : 720000
Available blocks . : 419655
Blocks in use . . : 300345

Let me know what it says.

0 votes

Comments

Alex Akilov

Mar 24 '15, 5:34 p.m.

Liam, Thanks, here's what it shows: Block size . . . . : 4096 Total blocks . . . : 7200 Available blocks . : 3931 Blocks in use . . : 3242

Permanent link

Liam Doherty

JAZZ DEVELOPER (231●2) Mar 24 '15, 5:53 p.m.

Ok, then I suspect that is the issue. That is not a lot of space, especially when you have a large file with a lot of dependencies. In the same information screen it ill tell you the data set being used. Is there any way you can extend that? For example, our build directory has:

Block size . . . . : 1024
Total blocks . . . : 720000
Available blocks . : 531056
Blocks in use . . : 188944

Which is 100 times bigger than yours. Remember all of the build files, metadata etc are loaded into that directory prior the build.

0 votes

Comments

Alex Akilov

Mar 24 '15, 5:59 p.m.

Thanks Liam, that's what I suspected but I needed ammunition to justify my request to increase that storage :-). Working late I see :-).

Alex Akilov

Mar 25 '15, 2:44 p.m.

Well, sad news. We increased the space to 180K blocks (176K available) but still failed with the no space left on device. Any other ideas?

Permanent link

Nicolas Dangeville

JAZZ DEVELOPER (316●3●25●25) Mar 27 '15, 10:28 a.m.

Have you tried to expand tmp ? It looks like this is happening when publishing a log, and there's a conversion of SYSPRINT to a file and I believe it happens in tha java tmp folder (which is probably /tmp).

your tmp looked 99% full.

0 votes

Permanent link

Nicolas Dangeville

JAZZ DEVELOPER (316●3●25●25) Mar 27 '15, 10:29 a.m.

So now I see that Liam & I seem to think the same thing ...

0 votes

Comments

Alex Akilov

Mar 30 '15, 5:53 p.m.

We initially suspected tmpdir and tried to change the tmpdir setting in bfagent.conf. Not sure if that works or not since the product documentation skips that entry but documents all of the others. I think we also tried to set a build property -Djava.io.tmpdir=<new temp folder> but that didn't work either. I'm not sure if there is a way to set a different temp folder for RTC to use rather than increasing the system allocation for temp (which would require sys admin involvement, approvals, etc. and, therefore, not ideal). As far as a home folder for the build user (Liam's suggestion e.g. /u/blduser), in our shop service accounts do not get a home folder (again, a policy that may be difficult to fight/change). If there is a way to make all RTC related output go to the same mount point where the load directory is (/var/jazz502 in our case) that would be ideal for us.

Your answer

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers

Answers and Comments

Question details

rational-team-concert

× 43,076

usage

× 12,144

enterprise-extensions

× 60

× 38

Question asked: Mar 09 '15, 5:53 p.m.

Question was seen: 10,336 times

Last updated: Apr 16 '15, 2:43 a.m.

EDC5133I No space left on device during Dependency build

Comments

5 answers

10,336 views

0 votes

Accepted answer

Comments

4 other answers

Comments

Comments

Comments

Your answer

Follow this question

Question details

Related questions