DXL function that returns current DOORS RAM usage?

I have written a rather massive program that generates specifications in MS Word using data stored in various Modules and Projects within DOORS. The specifications can sometimes be well over 1,000 pages long and include data from more than 100 different Modules.

The problem I am having is that, on some of the documents I generate that are extraordinarily long, my customer is running out of memory during execution. When this happens, DOORS crashes, which may cause corruption in their Modules. What I would like to do is keep a global variable that is constantly being set to the number of MB of RAM DOORS is using. In my main loop, I would like to check that variable against a certain threshold, and handle the case that globalRamTracker >= ramThreshold.

I've looked through the DXL reference manual, and I have not found anything useful. I've also googled this issue to little avail. There is one option that is available to DOORS users with 9.4 (or 9.6, maybe) or higher. Unfortunately, I and my customer are using DOORS version 9.2 and 9.3. Does anyone know of some function that is similar to the following defintion:

int memUsage()//returns the amount of RAM currently being used by DOORS in MB

Thanks in advance for your help.

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Wed Mar 04 17:51:13 EST 2015

I have actually located an undocumented function called isMemoryLow(), which returns bool. I'm working on creating the most inefficient code possible to try to test when this thing returns true, but I haven't gotten quite inefficient enough yet. Does anyone know anything about this function?

Re: DXL function that returns current DOORS RAM usage?
kourosh - Wed Mar 04 19:58:59 EST 2015

Hi, which version of 9.3 are you using? This was added in DOORS 9.3.0.7:

getMemoryUsage

Declaration

int getMemoryUsage()

Operation

Returns the Rational DOORS client memory usage in MB.

Re: DXL function that returns current DOORS RAM usage?
SudarshanRao - Thu Mar 05 02:49:01 EST 2015

See Vinayn's response in this thread, I think it has exactly what you're looking for:

https://www.ibm.com/developerworks/community/forums/html/topic?id=c5d4cc33-9986-463e-a73b-36523e6add7a&ps=25

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Mon Mar 09 11:15:22 EDT 2015

Thank you for your responses. Unfortunately, I am running 9.2 and the customer is running 9.3.0.0. Both getMemoryUsage and the response alluded to by SudarshanRao are not valid for us. My customer is a massive corporation, and there is a great deal of red tape to go through in order to update to a newer version. Is there anything undocumented for 9.2 that I could use for this? Thanks again.

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Mon Mar 09 15:27:48 EDT 2015

Well, I've tried some testing with function memoryIsLow, which returns bool. I would have thought (given the name of the function) that this function simply returns whether the DOORS client is using too much RAM. I will post the code I wrote below, but first let me give the odd results. After a while, the RAM usage of DOORS gets to around 1.7 GB. That's a lot! Then, it is suddenly back down around 1GB. As you can see in my code, I am not dealocating any memory. This thing will run in an infinite loop, though, because the memoryIsLow function never seems to return true. Any ideas?

int i = 0;
string str = "test";

while(i < 1)
{
    string s = "test" str;
    str = s;
    Skip s1 = create();
    put(s1, 1, str);
    Buffer b = create();
    b = str;
    if(memoryIsLow())
    {
        ack("Memory is low");
        halt();
    }
}

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Mon Mar 09 23:27:56 EDT 2015

int i = 0;
string str = "test";

while(i < 1)
{
    string s = "test" str;
    str = s;
    Skip s1 = create();
    put(s1, 1, str);
    Buffer b = create();
    b = str;
    if(memoryIsLow())
    {
        ack("Memory is low");
        halt();
    }
}

Well, I have done some more testing to try and figure out the behavior of the memoryIsLow function. I added 65,000 as an integer argument to the buffer creation on line ten of the code above, which makes me reach 1.7 GB of ram a great deal more quickly. The memoryIsLow function never does return anything other than false, and even with it commented out, the behavior is the same. This last time, after it reaches the apparent cap, the RAM usage declines back to where it was before the program was executed. Then, the program stops executing and gives me an 'undefined error,' on the line where I create the buffer, which I assume just means it tried to allocate RAM and was unable. I'm running out of ideas, I've exhausted my internet and DXL reference manual search abilities, and I'm about ready to try and simply implement a workaround. If any of you guys can help, I would really appreciate it so much. Thanks.

Re: DXL function that returns current DOORS RAM usage?
Mathias Mamsch - Tue Mar 10 05:17:25 EDT 2015

For older DOORS versions I suggest you use the following approach. I made an small C tool, that will determine the memory usage for a process. You can find the tool and the source (in case you want to compile for yourself) in the attached ZIP.  The executable has been compiled with Visual Studio and therefore needs msvcrt.dll (which should be installed everywhere). The tool can be called: printMemory.exe doors.exe or printMemory.exe <pid> where <pid> is the process ID.

The DXL interface is simple and simply executes the file and reads the output. If you need to be able to handle multiple DOORS processes, I can also help you with some code the get the PID of the current DOORS process.

In case of questions post back. Hope this helps. Regards, Mathias

 

// Put the absolute path of the tool, put the tool in the path, or implement another
// mechanism to find the executable file. 
string gsMemoryToolFile = (findFileRelative "../lib/tools/memory/printMemory.exe") " doors.exe" 
Regexp gRegexpMemoryValues = regexp "([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+)" 

bool doorsMemLoad( int &Private, int &Working_Set, int &Paged_Pool, 
                                 int &Non_Paged_Pool, int &Peak_Working_Set, int &Peak_Paged_Pool, 
                                         int &Peak_Non_Paged, int &Peak_Pagefile ) {

        

        string sReturn = ""; 
        win32SystemWait_ ( gsMemoryToolFile , sReturn, -1)
        // print gsMemoryToolFile ":" sReturn
        
        if (gRegexpMemoryValues sReturn) {
                Private          = intOf realOf sReturn[match 1]; 
                Working_Set      = intOf realOf sReturn[match 2]; 
                Paged_Pool       = intOf realOf sReturn[match 3]; 
                Non_Paged_Pool   = intOf realOf sReturn[match 4]; 
                Peak_Working_Set = intOf realOf sReturn[match 5]; 
                Peak_Paged_Pool  = intOf realOf sReturn[match 6]; 
                Peak_Non_Paged   = intOf realOf sReturn[match 7]; 
                Peak_Pagefile    = intOf realOf sReturn[match 8];
                return true;
        } else {
                Private          = 0
                Working_Set      = 0
                Paged_Pool       = 0
                Non_Paged_Pool   = 0
                Peak_Working_Set = 0
                Peak_Paged_Pool  = 0
                Peak_Non_Paged   = 0
                Peak_Pagefile    = 0
                return false;
        }              
}

int doorsPrivateMemory () {
        int Private, Working_Set, Paged_Pool, Non_Paged_Pool, Peak_Working_Set, Peak_Paged_Pool
        int  Peak_Non_Paged, Peak_Pagefile
        
        if ( doorsMemLoad(Private, Working_Set, Paged_Pool,  Non_Paged_Pool, 
                                          Peak_Working_Set, Peak_Paged_Pool, Peak_Non_Paged, Peak_Pagefile  )) return  Private
                                          
        return 0
}

Re: DXL function that returns current DOORS RAM usage?
Wolfgang Uhr - Tue Mar 10 06:25:05 EDT 2015

 

// Put the absolute path of the tool, put the tool in the path, or implement another
// mechanism to find the executable file. 
string gsMemoryToolFile = (findFileRelative "../lib/tools/memory/printMemory.exe") " doors.exe" 
Regexp gRegexpMemoryValues = regexp "([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+)" 

bool doorsMemLoad( int &Private, int &Working_Set, int &Paged_Pool, 
                                 int &Non_Paged_Pool, int &Peak_Working_Set, int &Peak_Paged_Pool, 
                                         int &Peak_Non_Paged, int &Peak_Pagefile ) {

        

        string sReturn = ""; 
        win32SystemWait_ ( gsMemoryToolFile , sReturn, -1)
        // print gsMemoryToolFile ":" sReturn
        
        if (gRegexpMemoryValues sReturn) {
                Private          = intOf realOf sReturn[match 1]; 
                Working_Set      = intOf realOf sReturn[match 2]; 
                Paged_Pool       = intOf realOf sReturn[match 3]; 
                Non_Paged_Pool   = intOf realOf sReturn[match 4]; 
                Peak_Working_Set = intOf realOf sReturn[match 5]; 
                Peak_Paged_Pool  = intOf realOf sReturn[match 6]; 
                Peak_Non_Paged   = intOf realOf sReturn[match 7]; 
                Peak_Pagefile    = intOf realOf sReturn[match 8];
                return true;
        } else {
                Private          = 0
                Working_Set      = 0
                Paged_Pool       = 0
                Non_Paged_Pool   = 0
                Peak_Working_Set = 0
                Peak_Paged_Pool  = 0
                Peak_Non_Paged   = 0
                Peak_Pagefile    = 0
                return false;
        }              
}

int doorsPrivateMemory () {
        int Private, Working_Set, Paged_Pool, Non_Paged_Pool, Peak_Working_Set, Peak_Paged_Pool
        int  Peak_Non_Paged, Peak_Pagefile
        
        if ( doorsMemLoad(Private, Working_Set, Paged_Pool,  Non_Paged_Pool, 
                                          Peak_Working_Set, Peak_Paged_Pool, Peak_Non_Paged, Peak_Pagefile  )) return  Private
                                          
        return 0
}

Hi Mathias

The doors versions from 9.6 uses 64 bit technology. The use of this routines is limited up to this version, isn't it?

Best regards

Wolfgang

Re: DXL function that returns current DOORS RAM usage?
Mathias Mamsch - Tue Mar 10 08:29:46 EDT 2015

Probably. I wondered why I get no response for certain processes from the printMemory.exe tool (e.g. printMemory.exe cmd.exe) ... This is probably because the processes are 64 bits. If this is of general interest I can research on how to read the memory of 64 bit applications.

On the other hand I figure, memory problems are not so important for 64 Bit DOORS, since you got no limit.

Regards, Mathias

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Tue Mar 10 19:05:06 EDT 2015

Thanks Mathias. I revised my code as follows:

#include </Dxl/include/DOORSMemUsage.inc>

int i = 0;
int mem = 0;
string str = "test";

while(i < 1)
{
    string s = "test" str;
    str = s;
    Skip s1 = create();
    put(s1, 1, str);
    Buffer b = create(65000);
    
    b = str;
    if(length(b) == 0)
    {
        print("buffer memory allocation failed. exiting loop.\n");
        halt();
    }
    mem = doorsPrivateMemory();
    if(0 < mem)
        print(mem "\n");
}

DoorsMemUsage.inc is the name I gave the file I created where I pasted your DXL code.

The doorsPrivateMemory function often returns 0, so I added a test for it so I didn't have too much printing going on. DOORS crashed when (in task manager) it got up to 178,000 KB. The output from the above code at that time was '1819896.' I may be able to use this to make some sort of cap where I can safely halt my DXL program before DOORS crashes.

If you wouldn't mind, could you please give me a detailed description of what exactly the two DXL functions do? I know it may be a lot to ask, but I have a hard time turning in code to the customer not knowing exactly what it does, and I must admit, I don't entirely understand this code. Thank you so much for your help.

woodPryan

Re: DXL function that returns current DOORS RAM usage?
Mathias Mamsch - Wed Mar 11 03:47:50 EDT 2015

#include </Dxl/include/DOORSMemUsage.inc>

int i = 0;
int mem = 0;
string str = "test";

while(i < 1)
{
    string s = "test" str;
    str = s;
    Skip s1 = create();
    put(s1, 1, str);
    Buffer b = create(65000);
    
    b = str;
    if(length(b) == 0)
    {
        print("buffer memory allocation failed. exiting loop.\n");
        halt();
    }
    mem = doorsPrivateMemory();
    if(0 < mem)
        print(mem "\n");
}

First of all there is the commandline tool "printMemory.exe" ... This tool takes one parameter - a PID or a process name, and it will print the current memory usage of the process (comma separated). So the first thing you need to do is run the tool over the command line and check its output, e.g.

printMemory.exe doors.exe

The DXL functions for this are pretty straight forward. They simply run the above command over the shell, and get the resulting output using the "win32SystemWait_" function. From the output it uses a regular expression to parse the integer values. The doorsMemLoad function will return all those memory columns using references. If the parsing fails, the function will return 0 for all values .... In this case you need to check the output of the printMemory tool, e.g. insert:

print sReturn

in the code, when the value cannot be parsed to check what is going wrong (either the printMemory tool did not return a value or it returned an invalid format where the regular expression does not apply).

Regards, Mathias

Re: DXL function that returns current DOORS RAM usage?
Wolfgang Uhr - Thu Mar 12 04:33:18 EDT 2015

Not only the 64 bit systems are not so important, the 32 bit values are not so important either. That is because doors handles its on different datastructures and to get a crash or a problem you have to fill up only one of them.

For me in case of a 64 bit system it is more interesting to get answers on the following questions:

1. Do we have the same memory leak situation? Can we slow down the script by creating memory leaks? (It makes to sense to be able to create 10 Millions Skips if you need 100 years to do it)
2. Do we have the same string table structure and can we fill them up in the same way as now?
3. Which new problems do we get?

Re: DXL function that returns current DOORS RAM usage?
woodpryan - Fri Mar 13 17:36:33 EDT 2015

Mathias, and anyone else looking for a solution to a similar problem. I made some modifications to the code (some of them just formatting, some of them more substantial). I'll post the code first, then explain my solution.

MemoryPrint C program

/*
    Purpose:       This program prints the memory usage of the process given to
                   int main to a file in the local machine's /DXL directory.
    Preconditions: The /DXL directory must already exist. This program will
                   not create it.
    Usage:         execute the program, passing the name of the process for which
                   to print memory use as a string
    Authors:       Mathias, DeveloperWorks community forums,
                   Ryan Wood, iroquoiSystems, inc.
    Created On:    3/11/15
    Last Modified: 3/13/15
*/

#include <windows.h>
#include <stdio.h>
#include <psapi.h>

/*  Function:   printMemoryInfo(HANDLE)
    Purpose:    Creates a file in the /DXL directory and outputs the 
                current RAM usage of the process to which the given handle
                corresponds. Closes the file and the handle when it is finished.
    Parameters: A windows process handle of type HANDLE included from Psapi.h
*/
void printMemoryInfo(HANDLE hProcess)
{
    FILE                       *fp;
    PROCESS_MEMORY_COUNTERS_EX  pmc;

    //open the file for write
    fp = fopen("/DXL/memprint.txt", "wt");
    if(NULL == fp)
        return;

    //Print the each type of memory usage, then the actual numbers
    if(GetProcessMemoryInfo(hProcess, &pmc, sizeof(pmc)))
    {
        fprintf(fp, "Private, Working Set, Paged Pool, Non Paged Pool, Peak Working Set, Peak Paged Pool, Peak Non Paged, Peak Pagefile (all in KB)\n"); 
        fprintf(fp, "%d, ", pmc.PrivateUsage / 1024);
        fprintf(fp, "%d, ", pmc.WorkingSetSize  / 1024);
        fprintf(fp, "%d, ", pmc.QuotaPagedPoolUsage  / 1024);
        fprintf(fp, "%d, ", pmc.QuotaNonPagedPoolUsage / 1024);
        fprintf(fp, "%d, ", pmc.PagefileUsage / 1024);
        fprintf(fp, "%d, ", pmc.PeakWorkingSetSize  / 1024);
        fprintf(fp, "%d, ", pmc.QuotaPeakPagedPoolUsage  / 1024);
        fprintf(fp, "%d, ", pmc.QuotaPeakNonPagedPoolUsage  / 1024);
        fprintf(fp, "%d\n", pmc.PeakPagefileUsage  / 1024);
    }

    //close the handle and the file
    CloseHandle(hProcess);
    fclose(fp);
}

int main(int argc, char *argv[])
{
    unsigned int i = 0;
    int          s = 0;

    TCHAR  buffer[MAX_PATH];
    DWORD  aProcesses[1024];
    DWORD  cbNeeded;
    DWORD  cProcesses;
    HANDLE hProcess;
    
    if(argc != 2) 
    {
         printf("Usage %s <pid|processname>\n"); 
         return 2; 
    }

    //initialize the aProcesses array
    if(!EnumProcesses(aProcesses, sizeof(aProcesses), &cbNeeded))
    {
        return 1;
    }

    // Calculate how many process identifiers were returned.
    cProcesses = cbNeeded / sizeof(DWORD);

    //loop through the Processes to find the one given in argv[1]
    for(i = 0; i < cProcesses; i++)
    {
        //Retrieve a handle to the process at i
        hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, aProcesses[i]);
        if(NULL == hProcess)//if we got nothing, go to the next
            continue;
        
        //get the name of the process, and store it in the TCHAR character buffer
        if(GetModuleFileNameExA(hProcess, 0, buffer, MAX_PATH))
        {
            //if the argument passed to main is not contained in the buffer continue
            if(strstr(buffer, argv[1]) == NULL && atoi(argv[1]) != aProcesses[i]) 
                continue;

            //print the memory info of the process to the file
            printMemoryInfo(hProcess);
        }
    }
    return 0;
}

Now DOORSMemUsage.inc

/* Authors:       Mathias, DeveloperWorks community,
                  Ryan Wood, iroquoiSystems, inc.
 * Creation date: March 11, 2015
 * DXL Editor is provided by SODIUS (www.sodius.com)
*/
Regexp gRegexpMemoryValues = regexp "([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+), ([0-9]+)";
string gsMemoryToolFile    = "/DXL/C/MemoryPrint.exe";
string param               = "doors.exe";
string winCommandExecTool  = gsMemoryToolFile " " param;

int memPrivate        = 0;
int memWorkingSet     = 0;
int memPagedPool      = 0;
int memNonPagedPool   = 0;
int memPeakWorkingSet = 0;
int memPeakPagedPool  = 0;
int memPeakNonPaged   = 0;
int memPeakPageFile   = 0;

/*  Function:   doorsMemLoad()
    Purpose:    executes the MemoryPrint.exe file and reads the output file
                it generates to obtain the current DOORS memory load.
                sets all the global memory integers so they may be retrieved.
*/
bool doorsMemLoad() 
{
    string sReturn   = null;
    string sInStream = null;
    Buffer buf       = null;
    
    //execute the MemoryPrint.exe file
    system(winCommandExecTool);
    sleep_(100);//sleep for a tenth of a second to make sure the file has written
    
    //open up the memprint.txt file that MemoryPrint writes
    Stream inStream = read("/DXL/memprint.txt");
    
    if(null == inStream)
        return false;
        
    
    buf = create();
    
    //read the lines of the file into the buffer
    while(!end(of(inStream)))
    {
        inStream >> sInStream;
        buf+= sInStream;
    }
    sReturn = tempStringOf(buf);
    close(inStream);
    
    
    if(gRegexpMemoryValues sReturn)//cast string return to regular expression
    {
        //get each value and store in the globals. This code parses the commas
        memPrivate       = intOf realOf sReturn[match 1]; 
        memWorkingSet    = intOf realOf sReturn[match 2]; 
        memPagedPool     = intOf realOf sReturn[match 3]; 
        memNonPagedPool  = intOf realOf sReturn[match 4]; 
        memWorkingSet    = intOf realOf sReturn[match 5]; 
        memPeakPagedPool = intOf realOf sReturn[match 6]; 
        memPeakNonPaged  = intOf realOf sReturn[match 7]; 
        memPeakPageFile  = intOf realOf sReturn[match 8];
        
        delete(buf);
        return true;
    }
    
    delete(buf);
    return false;
}

/*  Function:   getDoorsPrivateMemory()
    Purpose:    retreives the private memory usage of DOORS
    Return:     private memory Windows statistic
*/
int getDoorsPrivateMemory()
{
    if(doorsMemLoad())
        return memPrivate;

    return 0;
}

So, the win32SystemWait_ function has a problem. It doesn't wait. No matter what I give it as the third parameter, the function does not wait for any length of time. This caused the getDoorsPrivateMemory function to return 0 most of the time, making it far too unreliable to use. Therefore, I had to replace it with the system(string) function. Since there is no way (none that is documented) to retrieve any sort of return value from that function, I had to come up with another way for the C program to give me the memory usage. So, I changed the printing to go to a file, and modified the DXL code to read the generated file. In my larger program that is running into a memory problem, I periodically do a test to see if the RAM usage has exceeded 1,500,000 (1.5GB) by calling the getDoorsPrivateMemory function. If it is above that number, I attempt to close any Module that are not needed by the program and are not currently visible.

Thank you for your help, Mathias. I never would have thought of this without you, and the code you provided was a great start. Let me know if you have any questions or comments.

woodpryan