It's all about the answers!

Ask a question

EDC5133I No space left on device during Dependency build


Alex Akilov (1211224) | asked Mar 09 '15, 5:53 p.m.
edited Mar 09 '15, 6:19 p.m.
We have a few components under source control.  We created two build definitions that use the same language definitions/translators but a different build subset (in both cases we had to keep the subsets small since we were getting IOExceptions if the subset contained more than a few dozen source members).  The problem is that the first build worked but the second one is failing fairly consistently.

Here's an excerpt from the build log:

* [(internal) logpublishertask] com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device.
* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager$StreamLengthUtility.run(ContentManager.java:263)
* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager.storeContent(ContentManager.java:403)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.storeContent(ContentPublisher.java:180)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.getFileContent(ContentPublisher.java:94)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractFilePublisher.initializeContribution(AbstractFilePublisher.java:49)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractContributionPublisher.publish(AbstractContributionPublisher.java:120)
* [(internal) logpublishertask] at com.ibm.team.build.internal.ant.AbstractContentPublisherTask.updateBuildResult(AbstractContentPublisherTask.java:174)
* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractPublisherTask.doExecute(AbstractPublisherTask.java:105)
* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:666)

Snipped the rest of the stack trace.  After that we see the following:

 * [antz:compile] 2015-03-09 12:48:04,709 CRHTC1501E Failed: MSP FM/Batch COBOL compilation and link-edit : SPKP.BLZBLD.CP05.DEV.BT.P.COBOL(PMSP314L) BuildException : The following error occurred while executing this line:
* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..
* IKJ56246I FILE SYSADATA NOT ALLOCATED, FILE IN USE 

And so on (i.e. same error for the rest of the source members in the subset).

Is there a way to find out what device it's talking about?

Some more details about this issue.  I narrowed it down to a single program that seems to be the one that RTC is struggling with.  It's about 12K lines long (100Kb in size).  The impact analysis for it contains 123 files it depends on.  The buildable files list that gets generated for it is 105K in size.  Are we hitting some size limit that RTC doesn't seem to be able to handle?  If so, this could be a showstopper for us as we have a lot of "large" programs that could cause a similar problem for RTC.

Comments
Ralph Schoon commented Mar 10 '15, 3:28 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Have you checked the disk size? That is what the error message clearly hints to.


Alex Akilov commented Mar 10 '15, 10:55 a.m. | edited Mar 12 '15, 1:52 p.m.

 I'm not sure which folder it's complaining about.  Here's the output of df -kP (load directory is in /var/jazz502 which has 46% available).   Granted, some of these allocations are low and we've asked for additional space but it would help if the error message was a little more helpful in telling us where is the space that we're running out of (e.g. is it tmp space?  work/load directory?  What file was it trying to write to? etc.).


df -kP
Filesystem         1024-blocks        Used  Available  Capacity Mounted on     
OEMP.RTC.V5R0M2.SYSS.ZFS 874080      406713     467367       47% /usr/lpp/jazz/v5.0.2
OMVS.SYSS.CICSTS42.ZFS   28800       26409       2391       92% /CICSTZ42/usr/lpp/cicsts/cicsts42
OMVS.DB2.DAS1.DB2MQL      1920        1723        197       90% /DB2/DAG1/mql
OMVS.DB2.DAS1.DB2WORF     1440        1352         88       94% /DB2/DAG1/worf
OMVS.DB2.DAS1.DB2JDBC    11520       11039        481       96% /DB2/DAG1/jdbc
OMVS.DB2.DAS1.DB2BASE     4320        4146        174       96% /DB2/DAG1
OMVS.DB2.DDS1.DB2MQL      1920        1723        197       90% /DB2/DDG1/mql
OMVS.DB2.DDS1.DB2WORF     1440        1352         88       94% /DB2/DDG1/worf
OMVS.DB2.DDS1.DB2JDBC    11520       11039        481       96% /DB2/DDG1/jdbc
OMVS.DB2.DDS1.DB2BASE     4320        4146        174       96% /DB2/DDG1
OMVS.DB2.D2SA.DB2MQL      1920        1723        197       90% /DB2/D2D0/mql
OMVS.DB2.D2SA.DB2WORF     1440        1352         88       94% /DB2/D2D0/worf
OMVS.DB2.D2SA.DB2JDBC    11520       11039        481       96% /DB2/D2D0/jdbc
OMVS.DB2.D2SA.DB2BASE     4320        4146        174       96% /DB2/D2D0
UNIX.SYSS.SSERS2.SIGYROOT  768         326        442       43% /usr/lpp/cobol
UNIX.SYSS.SSERS2.SCFZHFS2 108720       29952      78768       28% /SYSTEM/var/wbem
UNIX.SYSS.SSERS2.JV390  576000      323081     252919       57% /usr/lpp/java/J7.0_64
UNIX.SYSS.SSERS2.ROOT  2348000     2314573      33427       99% /
/TMP                     10240         216      10024        3% /SYSTEM/tmp
*AMD/u                       4           4          0      100% /u
SPKP.SYSS.SP0309.HFS      3600          52       3480        2% /u/sp0309
TSNW.SYSS.TS0620.HFS       720          60        592       10% /u/ts0620
SPAC.SYSS.SP5572.HFS     49680       32304      17256       66% /u/sp5572
OMVS.SYSS.ZENA.AGENT.HFS  5760        4464       1176       80% /SYSTEM/etc/Zena/agent
OEMP.RTC.V5R0M2.SYSS.VAR.HFS 28800       12960      15724       46% /SYSTEM/var/jazz502
OEMP.RTC.V5R0M2.SYSS.ETC.HFS 3600          72       3460        3% /SYSTEM/etc/jazz502
OEMP.RTC.V5R0M3.SYSS.VAR.HFS 43200          56      43024        1% /SYSTEM/var/jazz50
OEMP.RTC.V5R0M3.SYSS.ETC.HFS 3600          76       3456        3% /SYSTEM/etc/jazz50
OEMP.RTC.V5R0M3.SYSS.HFS 504000      375588     128344       75% /usr/lpp/jazz/v5.0
OMVS.SYSS.IBMRAA.HFS     18000         280      17596        2% /usr/lpp/dmh
OMVS.SYSS.INTERNET.LOGS.HFS 108000       70772      37108       66% /usr/lpp/internet/server_root/logs
OEMP.RDZ.V9R1M0.SYSS.VAR.HFS 1785600      815952     969444       46% /SYSTEM/var/rdz
OEMP.RDZ.V9R1M0.SYSS.CFG.HFS 720          76        576       12% /SYSTEM/etc/rdz
OEMP.RDZ.V9R1M0.SYSS.HFS 259200      182636      76448       71% /usr/lpp/rdz
OMVS.DB2.DDS1.HFS       144000       48968      94916       35% /DB2V9/DDG1
OMVS.DB2.D2SA.HFS       144000       48968      94916       35% /DB2V9/D2D0
OMVS.DB2.DAS1.HFS       144000       48968      94916       35% /DB2V9/DAG1
OMVS.SYSS.SYSLOGD.LOG.HFS 3600000      297156    3302748        9% /tcpip
UNIX.SYSS.ZOSV1R8.CRON     720          32        620        5% /usr/lib/cron
UNIX.SYSS.ZOSV1R8.SPOOL   1440         884        432       68% /usr/spool
OMVS.SYSS.X509.CERTIFS.HFS 1440         412        960       31% /SYSTEM/etc/x509
UNIX.SYSS.SSERS2.NETVHFS  7200        5516       1564       78% /usr/lpp/netview/v6r2
UNIX.SYSS.JAVA60.SR5.JV390 344160      343312        772      100% /usr/lpp/java/J6.0.1_64
UNIX.SYSS.ZOSV2R1.DEV      720         100        504       17% /SYSTEM/dev
UNIX.SYSS.ZOSV2R1.VAR     2160         904       1136       45% /SYSTEM/var
UNIX.SYSS.ZOSV2R1.ETC     2880        2224        548       81% /SYSTEM/etc

Accepted answer


permanent link
Liam Doherty (2312) | answered Mar 27 '15, 8:46 a.m.
JAZZ DEVELOPER
I wonder if it is /tmp that has the issue. USS services seem to be not so good to put a good message out when it has a space issue. Is there anything in the system log? Also I see your /u/ has no space. I presume this is because your user set up a mount point at /u. Is there a chance that the userid running the build has not got a directory set up?
Alex Akilov selected this answer as the correct answer

Comments
Alex Akilov commented Apr 15 '15, 4:38 p.m.

 Looks like you guys were right.  Our /tmp was in memory structure for performance which is why it was allocated so small (10M).  We increased it to about 32M with the ability to grow 4 times by 4M each time and that seems to have alleviated the problem.  Still, would be good to be able to override the location of the scratchpad.  As I indicated, setting the tmpdir value in bfagent.conf and the gateway shell script didn't seem to help.


Ralph Schoon commented Apr 16 '15, 2:43 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Please provide an enhancement request, if you think this would be useful.

4 other answers



permanent link
Liam Doherty (2312) | answered Mar 24 '15, 5:15 a.m.
JAZZ DEVELOPER
Alex, I believe the message tell you the file it is having a problem with:

* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..

In ISPF 3.17 can you navigate to /var/jazz502/build/MSP_PLEVEL. Then next to the MSP_PLEVEL directory enter the FS line command. This will tell you how much space is available similar to this:

Block size . . . . : 1024  
Total blocks . . . : 720000
Available blocks . : 419655
Blocks in use  . . : 300345

Let me know what it says.

Comments
Alex Akilov commented Mar 24 '15, 5:33 p.m. | edited Mar 24 '15, 5:34 p.m.

 Liam,


Thanks, here's what it shows:

 Block size . . . . : 4096  
 Total blocks . . . : 7200  
 Available blocks . : 3931  
 Blocks in use  . . : 3242  


permanent link
Liam Doherty (2312) | answered Mar 24 '15, 5:53 p.m.
JAZZ DEVELOPER
Ok, then I suspect that is the issue. That is not a lot of space, especially when you have a large file with a lot of dependencies. In the same information screen it ill tell you the data set being used. Is there any way you can extend that? For example, our build directory has:

Block size . . . . : 1024   
Total blocks . . . : 720000 
Available blocks . : 531056 
Blocks in use  . . : 188944 

Which is 100 times bigger than yours. Remember all of the build files, metadata etc are loaded into that directory prior the build.

Comments
Alex Akilov commented Mar 24 '15, 5:59 p.m.

 Thanks Liam, that's what I suspected but I needed ammunition to justify my request to increase that storage :-).  Working late I see :-).


Alex Akilov commented Mar 25 '15, 2:44 p.m.

Well, sad news.  We increased the space to 180K blocks (176K available) but still failed with the no space left on device.  Any other ideas?


permanent link
Nicolas Dangeville (30612025) | answered Mar 27 '15, 10:28 a.m.
JAZZ DEVELOPER
Have you tried to expand tmp ? It looks like this is happening when publishing a log, and there's a conversion of SYSPRINT to a file and I believe it happens in tha java tmp folder (which is probably /tmp).

your tmp looked 99% full.

permanent link
Nicolas Dangeville (30612025) | answered Mar 27 '15, 10:29 a.m.
JAZZ DEVELOPER
So now I see that Liam & I seem to think the same thing ...

Comments
Alex Akilov commented Mar 30 '15, 5:53 p.m.

We initially suspected tmpdir and tried to change the tmpdir setting in bfagent.conf. Not sure if that works or not since the product documentation skips that entry but documents all of the others.  I think we also tried to set a build property -Djava.io.tmpdir=<new temp folder> but that didn't work either.  I'm not sure if there is a way to set a different temp folder for RTC to use rather than increasing the system allocation for temp (which would require sys admin involvement, approvals, etc. and, therefore, not ideal).  As far as a home folder for the build user (Liam's suggestion e.g. /u/blduser), in our shop service accounts do not get a home folder (again, a policy that may be difficult to fight/change).  If there is a way to make all RTC related output go to the same mount point where the load directory is (/var/jazz502  in our case) that would be ideal for us.


Your answer


Register or to post your answer.