"System memory is exhausted" error while writing a buffer to a stream

Some of you with long memories might remember that I posted a question in the old forum concerning an RTF exporter I've written for DOORS - that time around it was about DOORS apparently crashing while "show"ing the dialogue box. Well I fixed that (it was to do with initialising the array size to -1, which DOORS didn't like). And now I have a user who's managed to run out of memory - apparently.

The user gets a DOORS report saying "System memory is exhausted." But the total memory consumed (for all processes, according to Task Manager) is a whisker over 2GB, on a machine with 3GB physical memory. The message pops up while writing a buffer out to a stream (according to the reported line number in the DXL Interaction window). The stream has already had two other buffers written out to it. It hasn't been flushed (that will be my first attempt to fix it).
The problem can be worked around by chopping up the module for export into smaller chunks, which indicates to me that it is indeed a memory problem.
Has anyone seen anything similar? Has anyone any quick fix recommendations for this problem? Or do I need to do some work to fix it? :(
We're using DOORS 8.3 on Windows XP Pro 2002 SP3 (client -- not sure on the server, they don't let me near that!).
Tippers - Tue Aug 11 03:50:09 EDT 2009

Re: "System memory is exhausted" error while writing a buffer to a stream
llandale - Tue Aug 11 10:34:30 EDT 2009

• Louie

Re: "System memory is exhausted" error while writing a buffer to a stream
Tippers - Tue Aug 11 10:49:17 EDT 2009

Thanks for the reply Louie - I knew I could rely on you.

The bad news is, you're way off target! My exporter only uses Word at the last moment. It writes the content of the module (or modules) for export to an RTF file (via a Stream), closes the file, then opens that file in Word. So there is no speed-up available in Word. In fact the exporter hasn't yet started Word when it falls over.

When it comes to virtual memory, I don't think there's a problem. The screen dumps I've seen show the following, which all seems in order to me:

Physical Memory:
Total 3000M
Avail  909M
Cache  699M
 
Commit Charge:
Total 2129M
Limit 5074M
Peak  2129M
 
Kernel Memory:
Total   160M
Paged   126M
Nonpaged 35M

I'm increasingly of the opinion that the problem lies somewhere within DOORS, either as a buffer size limit or a stream size limit or some ghastly interaction between the two.

Re: "System memory is exhausted" error while writing a buffer to a stream
llandale - Tue Aug 11 13:01:23 EDT 2009

Physical Memory:
Total 3000M
Avail  909M
Cache  699M
 
Commit Charge:
Total 2129M
Limit 5074M
Peak  2129M
 
Kernel Memory:
Total   160M
Paged   126M
Nonpaged 35M

Don't know. But if is a DOORS buffer-to-stream limitation, the following function may help:

void    WriteBufferToStream(Buffer buf, Stream out)
{     // Write the contents of the buffer to a stream,
        //      Breaking up the individual writes in order
        //      to overcome some unknown Buffer-to-Stream
        //      Limitation.
 
                        //  Calibration, play with this:
        const int       cMaxBuffWriteSize = 10000 // Max bytes to write to Stream
 
        if (null buf or null out) return()
 
        
        int     LenBuf          = length(buf)
        int     NumWritten      = 0,
                NumToWrite      = 0
        string  ErrMess         = ""
        Buffer  bufTemp = create()
 
 
        while(NumWritten <= LenBuf)
        {  bufTemp    = ""  // Erase temp buffer
 
           if (LenBuf < NumWritten + cMaxBufWriteSize) //-
           then NumToWrite = LenBuf - NumWritten
           else NumToWrite = cMaxBufWriteSize
 
           combine(bufTemp, buf, NumWritten, NumWritten + NumToWrite -1)
           noError()
           out << bufTemp
           ErrMess = lastError()
           if (!null ErrMess) golly, punt I guess.
 
           NumWritten += NumToWrite
        }
        delete(bufTemp)
        return()
}     // end WriteBufferToStream()

I didn't run it nor debug it. Its likely I have a "<" to "<=" error, or a "+1" or "-1" error. Its also possible that the buffer write command will write the entire buffer and not just the current contents; in which case either a "setempty(bufTemp)" command, or delete and recreate the bufTemp (inside the loop) will work.

I don't know if the noError()/lastError() will actually trap errors or not. Perhaps try it with a Stream opened for Read.

I'd get the function to work by by setting the calibration to some small number like 4 and testing that. Once the function works, play with big calibrations, settling on one well below the theoretical max.

If you get it to work and do some testing on a reasonable Calibration, please post it.

• Louie

Advanced testing would involve doing repeated timing tests as well as increasing memory useage tests, in order to derive an 'optimum' calibration.

Re: "System memory is exhausted" error while writing a buffer to a stream
jsarkic1 - Mon Aug 24 11:50:14 EDT 2009

Re: "System memory is exhausted" error while writing a buffer to a stream
Doug.Zawacki - Mon Aug 31 11:11:16 EDT 2009

Re: "System memory is exhausted" error while writing a buffer to a stream
vimo - Fri Sep 04 14:46:53 EDT 2009

• remove all that is unnecessary on C:, delete or move it on another device, say D:
• stop Indexation, Restauration, File Replication service, Hibernation
• set pagefile to 0
• defragment, many times with windows tool or with another tool like MyDefrag (ex JkDefrag)
• set pagefile to previous value so it becomes contiguous

Re: "System memory is exhausted" error while writing a buffer to a stream
lnk369 - Wed Nov 04 11:45:04 EST 2015

I do facing "memory Exhausted "  errors while streaming the data.

I'm using Win7 with 8GBRAM still I'm getting very frequently.

Any suggestions.

Advance thanks

Re: "System memory is exhausted" error while writing a buffer to a stream
DHAM - Thu Nov 05 13:53:24 EST 2015

stream the data to a file and avoid using up memory

Re: "System memory is exhausted" error while writing a buffer to a stream
Mathias Mamsch - Sat Nov 07 07:54:53 EST 2015

Usually memory exhaustion errors are the result of a programming error. Show us the DXL code, that produces the error - I am pretty sure, that you simply might be leaking a lot of buffers filling up your memory in notime. Regards, Mathias

Re: "System memory is exhausted" error while writing a buffer to a stream
lnk369 - Fri Nov 13 09:21:50 EST 2015

Thanks for getting back to me.

I'm pulling the requirements from various modules to a single module depending upon the user selection this means depending on features.

I tried to increase the DOORS 9.3 RAM limitation to 2GB but problem still same. I do not have customized dxl since it was developed by some other team.

if you can advice I'll look into the code and let you know if it rectified.

Any advice is welcome and thanks in advance

Lnk

Re: "System memory is exhausted" error while writing a buffer to a stream
Mathias Mamsch - Fri Nov 13 16:01:39 EST 2015

First of all, check if your program leaves open modules. The easiest way to fill up memory from the code is to leave modules open. If you made sure, that no modules stay opened, then

take the following code from this post:

pragma runLim, 0
int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *getCurrentDXLContextPtr () {
    DB x = create ""
        int *ptr = addr_ x
        int *result = ptr @ 48
        destroy x
        return result
}
 
int *getMemoryBlockNodes (int *cc) { return cc @ 0x74 }
int *nextNode      (int *memNode) { return memNode @ 8 }
 
int countAllocatedObjects() {
        int *memBlocks = getMemoryBlockNodes getCurrentDXLContextPtr() 
        int count = 0
        while (!null memBlocks) {
 
                memBlocks = nextNode memBlocks
                count++
        }
        return count
}

and put it before the code. Then inside the main loop (after every module where you pulled requirements from) of your code put the following code:

print "Allocations: " countAllocatedObjects() "\n"

Check how fast the allocations will rise. In my experience DXL will raise memory errors at the most unusual places, because the point where the memory runs out is almost never the point where the memory is wasted.

Regards, Mathias

Re: "System memory is exhausted" error while writing a buffer to a stream
lnk369 - Wed Nov 18 17:22:28 EST 2015

Thanks for the inputs.

Meanwhile I'd like to add requirement number before the requirement text(see ex below) to every object which is tagged as requirement

Req100v1:The feature shall support all features supported in xxxx 2.0, unless noted here

I would like add Req100v1 before the requirement object text and also increment numbers from 100 to end of the documents which are tagged as requirement.

Pls. any advice to write a script.

Thanks in advance