EDC5133I No space left on device during Dependency build

Alex Akilov (121●14●24) | asked Mar 09 '15, 5:53 p.m.
edited Mar 09 '15, 6:19 p.m.

We have a few components under source control. We created two build definitions that use the same language definitions/translators but a different build subset (in both cases we had to keep the subsets small since we were getting IOExceptions if the subset contained more than a few dozen source members). The problem is that the first build worked but the second one is failing fairly consistently.

Here's an excerpt from the build log:

* [(internal) logpublishertask] com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device.

* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager$StreamLengthUtility.run(ContentManager.java:263)

* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager.storeContent(ContentManager.java:403)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.storeContent(ContentPublisher.java:180)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.getFileContent(ContentPublisher.java:94)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractFilePublisher.initializeContribution(AbstractFilePublisher.java:49)

* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractContributionPublisher.publish(AbstractContributionPublisher.java:120)

* [(internal) logpublishertask] at com.ibm.team.build.internal.ant.AbstractContentPublisherTask.updateBuildResult(AbstractContentPublisherTask.java:174)

* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractPublisherTask.doExecute(AbstractPublisherTask.java:105)

* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:666)

Snipped the rest of the stack trace. After that we see the following:

* [antz:compile] 2015-03-09 12:48:04,709 CRHTC1501E Failed: MSP FM/Batch COBOL compilation and link-edit : SPKP.BLZBLD.CP05.DEV.BT.P.COBOL(PMSP314L) BuildException : The following error occurred while executing this line:

* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..

* IKJ56246I FILE SYSADATA NOT ALLOCATED, FILE IN USE

And so on (i.e. same error for the rest of the source members in the subset).

Is there a way to find out what device it's talking about?

Some more details about this issue. I narrowed it down to a single program that seems to be the one that RTC is struggling with. It's about 12K lines long (100Kb in size). The impact analysis for it contains 123 files it depends on. The buildable files list that gets generated for it is 105K in size. Are we hitting some size limit that RTC doesn't seem to be able to handle? If so, this could be a showstopper for us as we have a lot of "large" programs that could cause a similar problem for RTC.

Comments

Ralph Schoon commented Mar 10 '15, 3:28 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Have you checked the disk size? That is what the error message clearly hints to.

Alex Akilov commented Mar 10 '15, 10:55 a.m. | edited Mar 12 '15, 1:52 p.m.

I'm not sure which folder it's complaining about. Here's the output of df -kP (load directory is in /var/jazz502 which has 46% available). Granted, some of these allocations are low and we've asked for additional space but it would help if the error message was a little more helpful in telling us where is the space that we're running out of (e.g. is it tmp space? work/load directory? What file was it trying to write to? etc.).

df -kP

Filesystem 1024-blocks Used Available Capacity Mounted on

OEMP.RTC.V5R0M2.SYSS.ZFS 874080 406713 467367 47% /usr/lpp/jazz/v5.0.2

OMVS.SYSS.CICSTS42.ZFS 28800 26409 2391 92% /CICSTZ42/usr/lpp/cicsts/cicsts42

OMVS.DB2.DAS1.DB2MQL 1920 1723 197 90% /DB2/DAG1/mql

OMVS.DB2.DAS1.DB2WORF 1440 1352 88 94% /DB2/DAG1/worf

OMVS.DB2.DAS1.DB2JDBC 11520 11039 481 96% /DB2/DAG1/jdbc

OMVS.DB2.DAS1.DB2BASE 4320 4146 174 96% /DB2/DAG1

OMVS.DB2.DDS1.DB2MQL 1920 1723 197 90% /DB2/DDG1/mql

OMVS.DB2.DDS1.DB2WORF 1440 1352 88 94% /DB2/DDG1/worf

OMVS.DB2.DDS1.DB2JDBC 11520 11039 481 96% /DB2/DDG1/jdbc

OMVS.DB2.DDS1.DB2BASE 4320 4146 174 96% /DB2/DDG1

OMVS.DB2.D2SA.DB2MQL 1920 1723 197 90% /DB2/D2D0/mql

OMVS.DB2.D2SA.DB2WORF 1440 1352 88 94% /DB2/D2D0/worf

OMVS.DB2.D2SA.DB2JDBC 11520 11039 481 96% /DB2/D2D0/jdbc

OMVS.DB2.D2SA.DB2BASE 4320 4146 174 96% /DB2/D2D0

UNIX.SYSS.SSERS2.SIGYROOT 768 326 442 43% /usr/lpp/cobol

UNIX.SYSS.SSERS2.SCFZHFS2 108720 29952 78768 28% /SYSTEM/var/wbem

UNIX.SYSS.SSERS2.JV390 576000 323081 252919 57% /usr/lpp/java/J7.0_64

UNIX.SYSS.SSERS2.ROOT 2348000 2314573 33427 99% /

/TMP 10240 216 10024 3% /SYSTEM/tmp

*AMD/u 4 4 0 100% /u

SPKP.SYSS.SP0309.HFS 3600 52 3480 2% /u/sp0309

TSNW.SYSS.TS0620.HFS 720 60 592 10% /u/ts0620

SPAC.SYSS.SP5572.HFS 49680 32304 17256 66% /u/sp5572

OMVS.SYSS.ZENA.AGENT.HFS 5760 4464 1176 80% /SYSTEM/etc/Zena/agent

OEMP.RTC.V5R0M2.SYSS.VAR.HFS 28800 12960 15724 46% /SYSTEM/var/jazz502

OEMP.RTC.V5R0M2.SYSS.ETC.HFS 3600 72 3460 3% /SYSTEM/etc/jazz502

OEMP.RTC.V5R0M3.SYSS.VAR.HFS 43200 56 43024 1% /SYSTEM/var/jazz50

OEMP.RTC.V5R0M3.SYSS.ETC.HFS 3600 76 3456 3% /SYSTEM/etc/jazz50

OEMP.RTC.V5R0M3.SYSS.HFS 504000 375588 128344 75% /usr/lpp/jazz/v5.0

OMVS.SYSS.IBMRAA.HFS 18000 280 17596 2% /usr/lpp/dmh

OMVS.SYSS.INTERNET.LOGS.HFS 108000 70772 37108 66% /usr/lpp/internet/server_root/logs

OEMP.RDZ.V9R1M0.SYSS.VAR.HFS 1785600 815952 969444 46% /SYSTEM/var/rdz

OEMP.RDZ.V9R1M0.SYSS.CFG.HFS 720 76 576 12% /SYSTEM/etc/rdz

OEMP.RDZ.V9R1M0.SYSS.HFS 259200 182636 76448 71% /usr/lpp/rdz

OMVS.DB2.DDS1.HFS 144000 48968 94916 35% /DB2V9/DDG1

OMVS.DB2.D2SA.HFS 144000 48968 94916 35% /DB2V9/D2D0

OMVS.DB2.DAS1.HFS 144000 48968 94916 35% /DB2V9/DAG1

OMVS.SYSS.SYSLOGD.LOG.HFS 3600000 297156 3302748 9% /tcpip

UNIX.SYSS.ZOSV1R8.CRON 720 32 620 5% /usr/lib/cron

UNIX.SYSS.ZOSV1R8.SPOOL 1440 884 432 68% /usr/spool

OMVS.SYSS.X509.CERTIFS.HFS 1440 412 960 31% /SYSTEM/etc/x509

UNIX.SYSS.SSERS2.NETVHFS 7200 5516 1564 78% /usr/lpp/netview/v6r2

UNIX.SYSS.JAVA60.SR5.JV390 344160 343312 772 100% /usr/lpp/java/J6.0.1_64

UNIX.SYSS.ZOSV2R1.DEV 720 100 504 17% /SYSTEM/dev

UNIX.SYSS.ZOSV2R1.VAR 2160 904 1136 45% /SYSTEM/var

UNIX.SYSS.ZOSV2R1.ETC 2880 2224 548 81% /SYSTEM/etc

Accepted answer

permanent link

Liam Doherty (231●2) | answered Mar 27 '15, 8:46 a.m.
JAZZ DEVELOPER

I wonder if it is /tmp that has the issue. USS services seem to be not so good to put a good message out when it has a space issue. Is there anything in the system log? Also I see your /u/ has no space. I presume this is because your user set up a mount point at /u. Is there a chance that the userid running the build has not got a directory set up?

Alex Akilov selected this answer as the correct answer

Comments

Alex Akilov commented Apr 15 '15, 4:38 p.m.

Looks like you guys were right. Our /tmp was in memory structure for performance which is why it was allocated so small (10M). We increased it to about 32M with the ability to grow 4 times by 4M each time and that seems to have alleviated the problem. Still, would be good to be able to override the location of the scratchpad. As I indicated, setting the tmpdir value in bfagent.conf and the gateway shell script didn't seem to help.

Ralph Schoon commented Apr 16 '15, 2:43 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Please provide an enhancement request, if you think this would be useful.

4 other answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Liam Doherty (231●2) | answered Mar 24 '15, 5:15 a.m.
JAZZ DEVELOPER

Alex, I believe the message tell you the file it is having a problem with:

* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..

In ISPF 3.17 can you navigate to /var/jazz502/build/MSP_PLEVEL. Then next to the MSP_PLEVEL directory enter the FS line command. This will tell you how much space is available similar to this:

Block size . . . . : 1024
Total blocks . . . : 720000
Available blocks . : 419655
Blocks in use . . : 300345

Let me know what it says.

Comments

Alex Akilov commented Mar 24 '15, 5:33 p.m. | edited Mar 24 '15, 5:34 p.m.

Liam,

Thanks, here's what it shows:

Block size . . . . : 4096

Total blocks . . . : 7200

Available blocks . : 3931

Blocks in use . . : 3242

permanent link

Liam Doherty (231●2) | answered Mar 24 '15, 5:53 p.m.
JAZZ DEVELOPER

Ok, then I suspect that is the issue. That is not a lot of space, especially when you have a large file with a lot of dependencies. In the same information screen it ill tell you the data set being used. Is there any way you can extend that? For example, our build directory has:

Block size . . . . : 1024
Total blocks . . . : 720000
Available blocks . : 531056
Blocks in use . . : 188944

Which is 100 times bigger than yours. Remember all of the build files, metadata etc are loaded into that directory prior the build.

Comments

Alex Akilov commented Mar 24 '15, 5:59 p.m.

Thanks Liam, that's what I suspected but I needed ammunition to justify my request to increase that storage :-). Working late I see :-).

Alex Akilov commented Mar 25 '15, 2:44 p.m.

Well, sad news. We increased the space to 180K blocks (176K available) but still failed with the no space left on device. Any other ideas?

permanent link

Nicolas Dangeville (306●1●20●25) | answered Mar 27 '15, 10:28 a.m.
JAZZ DEVELOPER

Have you tried to expand tmp ? It looks like this is happening when publishing a log, and there's a conversion of SYSPRINT to a file and I believe it happens in tha java tmp folder (which is probably /tmp).

your tmp looked 99% full.

permanent link

Nicolas Dangeville (306●1●20●25) | answered Mar 27 '15, 10:29 a.m.
JAZZ DEVELOPER

So now I see that Liam & I seem to think the same thing ...

Comments

Alex Akilov commented Mar 30 '15, 5:53 p.m.

We initially suspected tmpdir and tried to change the tmpdir setting in bfagent.conf. Not sure if that works or not since the product documentation skips that entry but documents all of the others. I think we also tried to set a build property -Djava.io.tmpdir=<new temp folder> but that didn't work either. I'm not sure if there is a way to set a different temp folder for RTC to use rather than increasing the system allocation for temp (which would require sys admin involvement, approvals, etc. and, therefore, not ideal). As far as a home folder for the build user (Liam's suggestion e.g. /u/blduser), in our shop service accounts do not get a home folder (again, a policy that may be difficult to fight/change). If there is a way to make all RTC related output go to the same mount point where the load directory is (/var/jazz502 in our case) that would be ideal for us.

It's all about the answers!

Related questions

EDC5133I No space left on device during Dependency build

Accepted answer

4 other answers

Your answer