Need help with Skip of Arrays and maybe indirections

Hi,

I would like your help in identifying what I'm doing wrong. I have pruned down the original code to what I think is the bare minimum to re-create the symptom. I assume there is memory corruption somewhere, but I can't find it.  I would appreciate your help in identifying where I am going wrong.

The symptom:   If a module has been baselined in the session, closing that module results in DOORS completely shutting down without even a diagnostic window. any other modules that are open will also be closed without any changes being saved.

Intended purpose:  This is part of the code used to track which objects are selected in open modules to allow users to go back to previous objects that they have worked on. When a module is opened for display, a new array is created to track the selected objects in that module. When a module is closed the array is deleted.  Based on commenting things out it seems that one of the delete statements is what causes this behavior. I cannot see what is wrong with what is done.

I have a debugging info box so I can see what is happening. The info boxes show the correct sequence of addresses and data elements. Your assistance, please?

startup_file.inc:

// Global Variables. This is in a file that runs whenever DOORS is started
// (located in C:\Program Files\IBM\Rational\DOORS\9.6\lib\dxl\startupfiles).

int aSize = 50 // Default Array size
Skip allArrays = createString // skip list with all the arrays
string s // used in lots of places as a temporary string variable

module_open.inc:

// Trigger creation. This is in a file that runs whenever a module is opened.
// (located in C:\Program Files\IBM\Rational\DOORS\9.6\lib\dxl\config\formalfiles)/

#include <addins/includes/test skip list of arrays 2.inc>

Module localMod = current Module
string currModName = fullName(localMod)

if (!isBaseline(localMod)) {

        Trigger selectObjUtil = trigger(module->object, sync, 4, do_Obj_Select_Util)
        do_Obj_Select_Util(selectObjUtil, true) // accommodate that the module opens with an object already selected
        moduleCloseTrigger = trigger(module->currModName, close, 10, moduleCloseAction)

}

addins/includes/test skip list of arrays 2.inc

//
// addins/includes/test skip list of arrays 2.inc
//

void print_array (string where, Array arr) {

        int i0
        int i1
        int i2
        int i3
        Addr64_ das1 = (addr_ arr) Addr64_
 
        i0 = (int get (arr, 0, 0))
        i1 = (int get (arr,1, 0))
        i2=(int get (arr, 2, 0))
        i3 = (int get (arr,3, 0))
 
        infoBox(where ": Address = " das1 ", data = " i0 ", " i1 ", " i2 ", " i3 "")

}

void initEmptyArray(Array arr)
{
        if (null arr) {
                return
        }
 
        print_array ("Before initialization", arr)

        int i
        for (i = 0; i < aSize; i++) {
                put(arr, i, i, 0)
        }
 
        print_array ("After initialization", arr)
}

bool getArray(Skip allArrays, string modNameKey, Array localArray)
{
        // get the array. 
        bool found = find(allArrays, modNameKey, localArray)
        if (found) print_array("Found", localArray)
        return found
}

// This function is called every time an object is selected within a formal module
void do_Obj_Select_Util(Trigger t, bool eraseExisting)
{
        Module localMod = current Module
  
        if (!null(t)) {
                //localMod = module(t, 0) // this resulted in DOORS errors in 9.6, so replaced it
                localMod = module(t)    
        }
  
        if (null(localMod)) {
                localMod = current Module
        }
  
        // Object o = current Object
        Object o = current(localMod)

        if (null(localMod) || null(o)) {
                return
        }
 
        // Set the key for storing the arrays. Use global variable s
        s = fullName(localMod)
 
        // Do not process back and forward for baselines!!!!
        // (superfluous; the trigger is not set on baseline modules)
        if (isBaseline(localMod)) {
                return
        }
 
        if ((table(o) && tableContents(localMod)) || row(o)) {
                return
        }
        
        // There is a problem with creating the initial object in an empty
        // module. Need to force firstTime = true in this case.
        if (first(localMod) == last(localMod)) {
                firstTime = true
        }

        Array localArray = null
        bool result
 
        bool foundExisting = getArray(allArrays, s, localArray)
 
        if (eraseExisting || !foundExisting) {
                Array localTmpArray = null
                if (!null(allArrays) && find(allArrays, s, localTmpArray)) {
                        result = delete(allArrays, s)
                        if (!null(localTmpArray)) {
                                print_array("Deleting in do_Obj_Select_Util", localTmpArray)
                                delete(localTmpArray) //commenting this out solves the problem
                        }
                }

                // This must be handled here instead of in getArray or the variable scope
                // will be wrong
                localArray = create(aSize, 1)
                initEmptyArray(localArray)
                print_array("Creating in do_Obj_Select_Util", localArray)
                put(allArrays, s, localArray) 
        }
 
        // do other stuff with localArray
}

void do_Obj_Select_Util(Trigger t)
{
        do_Obj_Select_Util(t, false)
}

void moduleCloseAction(Module localMod)
{
        bool result

        if (null(localMod)) {
                localMod = current Module
                if (null(localMod)) {
                        return
                }
        }
 
        Array localTmpArray = null
        if (!null(allArrays) && find(allArrays, fullName(localMod), localTmpArray)) {
                result = delete(allArrays, fullName(localMod))
                if (!null(localTmpArray)) {
                        print_array("Deleting in moduleCloseAction", localTmpArray)
                        delete(localTmpArray)
                }
        } 
}

bool moduleCloseAction (Trigger t)
{
        Module localMod = null
        if (null(t)) {
                localMod = current Module
        } else {
                localMod = module(t, 0) 
        }

        if (isBaseline(localMod)) {
                return(true)
        }
   
        if (null(localMod)) {
                localMod = current Module
        }
 
        moduleCloseAction(localMod)
 
        return(true)
}

Re: Need help with Skip of Arrays and maybe indirections
Mathias Mamsch - Tue Oct 23 03:44:08 EDT 2018

I think the approach that you wanted to choose does not work in DXL. Every object (array, skip) you create in DXL will only live inside the current DXL context. A Trigger is executed inside its own context.

That means when you create an array here:

// This function is called every time an object is selected within a formal module
void do_Obj_Select_Util(Trigger t, bool eraseExisting)
{
      ....
                // This must be handled here instead of in getArray or the variable scope
                // will be wrong
                localArray = create(aSize, 1)
      ....
}

the localArray object will only live inside the trigger and afterwards be marked as freed by DOORS. Now every DXL script is free to reuse the memory although you are still storing a reference to the array in the global allArrays. When you now execute the delete(localTmpArray the array might already been freed and overwritten, which will cause the delete call to crash.

I just realize the above might not be 100% correct, because DOORS should keep the array alive as long as the trigger context is alive. But maybe there is some mechanics, that on closing the module the select triggers are disposed (freeing the array) which then will result in the same thing.

So long story short:

- Did you try to just remove the "delete" call in the onClose trigger (make sure you do not touch the local array there, just remove it from the skip) and trust that DXL will perform the cleanup of the array contents and the trigger itself?

Maybe this helps, regards, Mathias

Re: Need help with Skip of Arrays and maybe indirections
Mathias Mamsch - Tue Oct 23 17:00:09 EDT 2018

// This function is called every time an object is selected within a formal module
void do_Obj_Select_Util(Trigger t, bool eraseExisting)
{
      ....
                // This must be handled here instead of in getArray or the variable scope
                // will be wrong
                localArray = create(aSize, 1)
      ....
}

Just realized that you wrote already in your post, that commenting out "delete" resolves the problem. So then it is as I expected. When you close the module, the sync trigger DXL context is terminated, automatically freeing the array. Freeing it again inside the close trigger brings your exception. Therefore just remove the delete and you should be fine. Regards, Mathias

Re: Need help with Skip of Arrays and maybe indirections
anotherDaniel - Tue Oct 23 21:07:16 EDT 2018

Thank you for your answers. The structure is a little different from what you have described but the basic concept may still be correct.

The initial creation of the array is not from a Trigger but from an include that is in the dxl/config/formalfiles directory. That is also where we put the include that creates the additional toolbar buttons. Perhaps the scope of the array is the same as the toolbar buttons, and it gets destroyed when the module is closed for some reason  An example of this is when a baseline is performed and everything in formalfiles gets executed. Interestingly, the module close trigger does not react to a baseline, but a module open trigger and the files in formalfiles directory do get triggered.

As you noted, I did have it working with that commented out, but I never understood why. I really appreciate you introducing the scope to the discussion because I had not considered that. If the scope might be trigger-related (under some circumstances it might be a trigger rather than the formalfiles  include that creates the array) then I probably need to make it a global array.

Unfortunately, when I transferred the pruned down code to my real code, the problem did not get fixed. The real equivalent of the "getArray()" function in my example correctly retrieves the array, but the caller "do_Obj_Select_Util()" receives a null. I'm starting to like that global idea…

Thank you again for your help, Mathias

Re: Need help with Skip of Arrays and maybe indirections
Mathias Mamsch - Thu Oct 25 04:00:55 EDT 2018

You have to be very careful when you start putting stuff from a local DXL context to the global one. You cannot rely on anything to survive when the local DXL context ends, not even strings (that you use inside a Skip as a key). Any allocated Object like Skip, Array, etc. will also be freed. The problem is that you will not realize it, until the memory is overwritten which will give you very hard to find errors.

To make it clear: When you write a string from a local DXL context inside a Skip list in the global DXL context, then after the local DXL ends, the string memory will be freed (except if its persistet in the string table). It is not that the global DXL contexts "takes over" the stuff.

As an alternative: Why don't you put the whole functionality in one script, e.g. inside a small GUI with two navigation buttons? I successfully used GUIs to keep data alive in a controlled local DXL context to use it from other contexts. Using "timers" you can implement background behaviours. There is alot of stuff you can do this way, usually with a good user experience.

For your example about just tracking objects, I think you are doing a complete overkill. You could just have a sync trigger write to a (config area / user profile) file and have a GUI read from it. Performance will NOT be of any relevance on this one and filesystem caching will make this a fast one.

Regards, Mathias

Re: Need help with Skip of Arrays and maybe indirections
anotherDaniel - Fri Oct 26 17:08:10 EDT 2018

Thank you for your great suggestions.  There is actually a lot of processing that goes on that I didn't describe because I was able to isolate the source of the error to the code that I provided. For example, there are toolbar buttons with appropriate graying out (menuAvailable_ and menuAvailable_) in addition to implementing functionality that is all based on the tracking information. The functionality is best implemented in an array, which means it is best done with a dxl Array.

I inherited this code, and based on what you've told me I can see how my predecessors tried to deal with the memory issues. In DOORS 9.5 the functionality mostly worked and after a few hours there would be a diagnostic log that didn't seem to cause much other trouble. In DOORS 9.6 that same code fails in a much worse way, which is why I am working through it.

It seems to me that the following design might address all of the issues and allow me to keep the data in an Array for ease of use.

1. Create a single global Array to store all of the data.
2. Other than the full name of the module, all of the rest of the data being stored are integers.
3. Define a global string variable for use when putting the module name into the first column of the global array. This will ensure that it does not get corrupted. (Use this instead of the skip list of the original design.)
4. Every time something needs to be done, the first column is searched to see which Array row contains the full name of the module. In the loop that searches for which row contains the data for a given module, I am hoping that using the following construct will keep the interim string out of the string table: if ((string get(backForwardArray, i, 0)) == modNameKey) {"

By enforcing this discipline in the code, will the scope issue go away because everything that goes into the global array will either be an integer or will be coming from a global string variable?

Re: Need help with Skip of Arrays and maybe indirections
Mathias Mamsch - Sat Oct 27 09:13:59 EDT 2018

The global string variable in 3. will not help you. It is not the string variable (which is just a pointer to the string memory) that defines how long the data will live, but the string memory which is allocated inside a DXL context. Trying to make a string variable persist (e.g. by putting it into a buffer and call 'stringOf') might be stretching the "hackiness" of your code to a level where nobody will understand it anymore and it relies on a lot of undocumented DXL features (which can change at any time). This  will lead to problems on every DOORS update and hard to debug crash problems. So If I were you, you should find a way of achieving what you need with a minimum of hacks and as much documented DXL as you can.

Unfortunately I did not understand exactly what kind of information you are trying to persist between which DXL executions. From your example and your posts it seemed you wanted to store the navigation history of the user inside the session (between starting and ending the DOORS client) and you want to use the information for dynamically adapting menu / symbol bars when opening the modules. I don't understand the use case, but to me it seems the amount of data you need to store would fit nicely to config area / temp area. Unless you do not want to store the complete contents of a module you should also not worry about string table pollution or memory leaks and unless you are not planning to fetch part of the information from a Layout DXL you should also not worry about performance.

So I would suggest you the following:

- Store the information to one or more files in a clever way. Test the performance - you might be surprised. All your memory hacks and crashes, incompatibilities will go away.

- If you (or anyone reading this) is interested I will assist you in putting out an open source performance optimized storage library for DXL that gives the user the possibility to access a persistent client storage (per computer), a session storage (while DOORS lives) and a global storage (across computers).

Just thinking, regards, Mathias

Re: Need help with Skip of Arrays and maybe indirections
PekkaMakinen - Sat Oct 27 09:34:06 EDT 2018

Some examples of using DOORS configuration files are available by SmartDXL at http://www.smartdxl.com/content/?cat=38

Re: Need help with Skip of Arrays and maybe indirections
anotherDaniel - Sat Oct 27 14:38:06 EDT 2018

My company has rules about sharing anything and I'm trying to live by them and ask for help at the same time. For this reason I haven't said exactly why we are doing this, even though that could lead to a brilliant resolution. Also, all the code I have provided is completely rewritten from what I am actually working with. I really appreciate all of your help and suggestions given how cagey I have been.

I find this whole discussion very ironic. In the years I have been looking things up on this forum, I have read a lot about how to conserve memory so strings don't persist and why that is desirable. I've also seen your excellent string table tutorial, Mathias. In this case I actually want something to persist rather than have its memory recovered and there is no way to reliably ensure that happens that isn't a hack. It seems to me that assigning it to global variable should be a very good trigger to force it to persist, but we don't know if IBM has that in the interpreter and, even if they do, they will keep it that way. I believe that was your point about what I was trying.

It seems like the uniqueID would be a great filename in the config area, if those are legal for that. A file for each module in use, using the uniqueID as the filename and whose only content is the index into the global Array (which now will only contain integers) is my first thought. Part of the initialization when DOORS is first started is to delete any of those files that may have been left over from a previous session. A second approach is using a hash function on the uniqueID to convert it to an integer to allow the global Array to store everything (I know the hazards of that - clashing) but may take more processing time than the config. I may have other ideas as the weekend progresses. I will look at SmartDXL and the code I am working with, which also uses the config area.

Thank you both again for your help.

Re: Need help with Skip of Arrays and maybe indirections
Mathias Mamsch - Mon Oct 29 05:54:37 EDT 2018

Every company has rules about "sharing" information I guess. Unless your company does not forbid the usage of open source technology (which no company can probably afford nowadays), you should be fine to use and create pull requests on a DXL library for accessing storage from DXL which contains 0% business logic or anything company specific. The idea of not contributing to open source efforts also makes not a lot of sense. In the end having you merge your features into the open source upstream on every release rather than making a pull request once and have them integrated in the official release with no effort for the future should also be more than worth the effort.

Anyway - I will not be able to change the DXL community ;-) And this all goes to much off-topic.

Why again would you need a global array? Why can't you just use Config Area? Or any other storage (like a File, a COM Object, a database, whatever) Regards, Mathias

Re: Need help with Skip of Arrays and maybe indirections
anotherDaniel - Mon Oct 29 08:15:22 EDT 2018

My company has blocked access to the common sites where code is freely shared.

For each module, I need to store one string and a lot of integers. I could use the config area to store everything, but then each time I need it I would have to read it into an Array (or locate the relevant portion of the data and put it in a one-dimensional array). When a single integer needs to change I would have to rebuild the the config file. There is no good reason to write all that code. The only thing that a global Array doesn't work for is the string. So store the string in the config area and all the rest in the Array. Even if the performance of all that string processing is negligible, I see no advantage to warrant writing more code than necessary.

I was just thinking about the original code. My predecessors stored the fullName of the module instead of the Module in the original skip list. If the Module can be used as a key in a global Skip and it remains valid in a global sense without needing a dialog box to remain open, the way that integers do and strings do not, that is another way to solve this problem. I may try that.