Thoughts on evalTop_ of a DXL Library

Over time, I've developed a library of DXL components.

Since conditional inclusion is not available in DXL, as the library grew and the inter-dependencies got ever more complex, I created a single include file that brings in the entire library and goes some way to documenting inter-dependencies in the library.

Its convenient to only need to include 1 file to gain access to the components, but the problem with this approach is that the interpretation time for all of the components is getting longer as the library grows.

If performance is key, I need to manually include the required components and dependencies, which is a bit of a chore.

I had the thought that I could modify our clients to evalTop_ the library upon startup, with the effect that it is interpreted once per DOORS session, but that would entail the update of all scripts to remove any library inclusions (as they would otherwise fail due to redeclaration errors). That's a pretty huge change and I'm not convinced its the best solution.

Does anyone have any thoughts/approaches for DXL library management/inclusion?

Cheers,

Antonio.

Re: Thoughts on evalTop_ of a DXL Library
Mathias Mamsch - Thu Nov 17 08:00:27 EST 2016

I always stayed with a manual system for include management on the libraries, which works for any complexity. The idea is, to include a comment and a global at the start of each library: 

//!-- This is my cool Library (myCoolLibrary.inc)

/* ------------------- Dependencies ---------------------
    #include <lib/dependencyA.inc>
    #include <lib/dependencyB.inc>
    #include <lib/myCoolLibrary.inc>
*/

int gIncludeMyCoolLibraryInc = 0

The global variable prevents double inclusion (can produce nasty errors). Now regarding the comment you can see: 

• it lists the necessary include statements for using the library. 
• You can easily check if the include statement is correct. Simply copy paste the comment in the Run/DXL and see if it parses correctly

The most important rule for this system is: 

• A library must never include another library (just have it listed as a dependency inside the comments)

We automated the process of scanning the include files of the library and "checkDXL" all comments, to check if the include statements for all libraries work. If you want to do a thourough check (for unnecessary dependencies), you can take the comments and remove each line once, and eval_ to see if there are unnecessary includes. 

Now if you have a DXL using 8 libraries, you will have to take the include statements from all of them and put them in your DXL file. Then you will remove duplicates includes. 

Developers need to be trained, to work with new include depdendencies the following way: 

- Whenever one adds an include dependency, he will do a search through the library for the name of the library (e.g. myCoolLibrary.inc). He will find all depdendent libraries through the comments (the have an include statement for mycoolLibrary.inc). Now the developer needs to add the include depdency for all dependent libraries (if not already existing). 

This approach works very well for manual include management and I never found the necessity for optimizing. 

Another thing to think about: 

- You can "build" a DXL program by simply inlining all include files. This will give you a standalone file, that has no include dependencies at all. You do not need to care about updates of libraries breaking a DXL file again, until you decide to "rebuild" and test it for a new version of your library. Of course this build process can also be automated. 

Regards, Mathias

Another note: You could also put a checkin trigger on your version control or build server, to test the integrity of your dependency comments.

Re: Thoughts on evalTop_ of a DXL Library
Antonio__Norkus - Thu Nov 17 09:00:43 EST 2016

Ah. So you have the global just to error-out with a simple error, in case of double inclusion? Clever!

Thanks for your wisdom, Mathias.

I was wondering, what are your thoughts on evalTop_ of the library? I imagine that is what is done by the DOORS client for the standard DXL API functions. 

Re: Thoughts on evalTop_ of a DXL Library
Mathias Mamsch - Thu Nov 17 15:34:32 EST 2016

Well you do not necessarily need to use evalTop_ ... you can also include it from the client startup files yourself (which are also executed on global level). But I would not advise it.

The problems is that a library on a global level might not behave as you would expect, especially when it comes to global variables. Data will stay inside the global variables, even after the DXL ends, but it might contain references to memory from the local data context pointing to the freed memory Example:

Buffer buf = null; 

void myFunction () {
    if (null buf) buf = create(); 
    buf += "Hallo"; 
    print buf "\n";
}

When you put this code on the global context, and execute the function from local context, it will work the first time. The second time you execute the script, then buf will point to the freed Buffer from the first run and DOORS will crash on you.

Therefore I would NEVER recommend putting a library on top level. You are inviting crashes on places, where you would not expect them (where in your development everything works perfect).

Regards, Mathias

Re: Thoughts on evalTop_ of a DXL Library
Antonio_Norkus - Sun Nov 20 04:48:18 EST 2016

Thanks again for sharing your insights, Mathias. As you allude to, any library that exists in the top context must be proven to be bulletproof.

I try to apply the following rules to my basic component functions:

- assumes that the caller of the function is responsible for memory management of interface parameters

- encapsulates its algorithm to only its input parameters

For the example you give, would the following be OK in the top context, where the caller is responsible for the declaration of buf?

void myFunction (Buffer &buf) {
    if (null buf) buf = create(); 
    buf += "Hallo"; 
    print buf "\n";
}

Re: Thoughts on evalTop_ of a DXL Library
Mathias Mamsch - Sun Nov 20 06:21:56 EST 2016

void myFunction (Buffer &buf) {
    if (null buf) buf = create(); 
    buf += "Hallo"; 
    print buf "\n";
}

That case would be fine. Also that case would probably be fine: 

Buffer buf = create();

void myFunction () {
    buf += "Hallo"; 
    print buf "\n";
}

Note that the above is a common approach for string functions, that will use a buffer for internal processing (to avoid the time for allocating one on every call. 

I guess the general rule would be to never store an allocated object to a global variable (with the exception of int, bool, real, char) that was passed as a function parameter or created inside a library function outside the libraries startup code (i.e. on non global DXL context level). Not even strings! This restriction would be too much for me, since for optimizing performance and memory management I personally rely a lot on global variables (custom data types, etc.). 

Another thing you should keep in mind: 

You need some strategy to avoid those nasty DXL errors, when people do not have the library installed. Normally you ship a library with the corresponding scripts, so you can make sure, that all scripts have their libraries available. For a global library you would need to either install the library "on the fly" with some kind of wrapper script: 

if (!checkDXL "myLibraryFunction") {
    alert "Please run this script again!";
    evalTop_ "#include <myLibrary>"
} else {
    eval_ "myScript.dxl"
}

or you would need to make sure that all DOORS clients have the library included in their startup scripts. 

Regards, Mathias

Re: Thoughts on evalTop_ of a DXL Library
Antonio__Norkus - Thu Dec 01 07:34:46 EST 2016

I just did a little test and it seems that there are no re-declaration errors between top-context and normal context. Running:

evalTop_("int gGlobal = 0")

then running

int gGlobal = 1

doesn't result in a re-declaration error, in the same way the following doesn't result in a re-declaration error:

int gGlobal = 1

if (true)
{
        int gGlobal = 2
        print "local version = " gGlobal "\n"
}

print "true global = "gGlobal ""

The ramification is that deploying a top-context library doesn't necessitate the refactoring of code that uses the library. Since the interpretation time for my library is around 0.5 seconds, it would result in significant performance improvements.

Re: Thoughts on evalTop_ of a DXL Library
Mathias Mamsch - Thu Dec 01 09:20:48 EST 2016

evalTop_("int gGlobal = 0")

int gGlobal = 1

int gGlobal = 1

if (true)
{
        int gGlobal = 2
        print "local version = " gGlobal "\n"
}

print "true global = "gGlobal ""

0.5 seconds of parse time bother you? Why? Take the attached file. It has a parse time of 2.5 seconds depending on CPU/Memory speed. If you now put that file in a layout attribute, it will only be slow ONCE when loading the view (or creating the layout). The DXL code is 'compiled' only once and executed several times. So I figure by local library inclusion, you will loose 0.5 seconds when starting a DXL script from the menu. Seems not such a big deal? I can imagine a couple of use cases though, where the library will slow you down significantly: 

- Lots of eval_ with the whole library

- Lots of DXL attributes of views, each one including the library

Are people loading the library from a network share or from a local hard disk? 

Regards, Mathias

Re: Thoughts on evalTop_ of a DXL Library
Antonio__Norkus - Thu Dec 01 10:07:53 EST 2016

The lib is loaded from a network share, which is probably where most of the time is being wasted. Seems to be between 0.5s and 2s, depending on environment. I don't think I want to go down the path of locally installed dxl.

I have a lot of occurrences of "Lots of DXL attributes of views, each one including the library".

Also a lot of my menu tools use the library...they all have a sluggish startup, which becomes near-instantaneous when using the evalTop_ library.

Just feels much better from a user-experience perspective, akin to application startup improvement after upgrading from a HDD to a SSD.