Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)

Hi all, 

*** NOTE: THIS IS NOT APPLICABLE FOR DOORS 9.6.1.7 . For DOORS 9.6.1.7 and upwards look here ***

finally I had some time to investigate DOORS 64 bit in terms of all those bad addr_, eval_ tricks I used to post earlier. A lot of you already noticed that code like this

string s = "Hallo"; 
eval_  "string s = addr_ " ((addr_ s) int) "; print s\n";

which was used to work on DOORS 32 bit, does not work reliably anymore on DOORS 64 bit ... To explain this, we need to take a look, on how 64 bit affects DXL. For this we need to take a look at the memory. For this we can use this small function: 

string arhex[] = { "00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "0a", "0b", "0c", "0d", "0e", "0f", 
               "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "1a", "1b", "1c", "1d", "1e", "1f", 
               "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "2a", "2b", "2c", "2d", "2e", "2f", 
               "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "3a", "3b", "3c", "3d", "3e", "3f", 
               "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "4a", "4b", "4c", "4d", "4e", "4f", 
               "50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "5a", "5b", "5c", "5d", "5e", "5f", 
               "60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "6a", "6b", "6c", "6d", "6e", "6f", 
               "70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "7a", "7b", "7c", "7d", "7e", "7f", 
               "80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "8a", "8b", "8c", "8d", "8e", "8f", 
               "90", "91", "92", "93", "94", "95", "96", "97", "98", "99", "9a", "9b", "9c", "9d", "9e", "9f", 
               "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", "a8", "a9", "aa", "ab", "ac", "ad", "ae", "af", 
               "b0", "b1", "b2", "b3", "b4", "b5", "b6", "b7", "b8", "b9", "ba", "bb", "bc", "bd", "be", "bf", 
               "c0", "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "ca", "cb", "cc", "cd", "ce", "cf", 
               "d0", "d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "da", "db", "dc", "dd", "de", "df", 
               "e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7", "e8", "e9", "ea", "eb", "ec", "ed", "ee", "ef", 
               "f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "fa", "fb", "fc", "fd", "fe", "ff"
}
Buffer buf = create(); 
string dumpmemory (int &a, int r) {
     buf = "";
     int *ptr; 
     ptr = &a; 
     int i; for (i = 0; i < r; i++) {     
             buf += arhex[ (*ptr) & 255 ];
             if (i < r-1) ptr += 1;
             if (i % 4 == 3) buf += " "; 
     }
     return stringOf buf
}

Using this function we can now inspect our memory and investigate how DXL does memory handling on 64 bit. 

So take the following code: 

int test2 = 0x05060708;
int test = 0x01020304; 
// prints: 08070605 00000000 04030201 00000000 
print dumpmemory(test2,16) "\n"

As you can see DOORS will store the two integer variables with 8 Bytes each. The memory layout shows that integers are stored in "little endian" format, which means, that the lowest byte has the smallest memory address. Note that this depends on the hardware of your machine, but its likely that you get the same result running the above code. 

Ok, so we see, that integers are stored with 64 bit, the higher 4 Bytes of the integer are empty. Now lets test what happens when we perform arithmetics: 

int test2 = 0x05060708 * 256;
int test = 0x01020304; 
// prints: 00080706 05000000 04030201 00000000 
print dumpmemory(test2,16) "\n"

Wow. You can see now when multiplying with 2^8, the bytes of the integer have now shifted right in memory by one byte. This shows that DXL will perform full 64 bit arithmetics. But what happens if we print it ? 

print (0x05060708)
print (0x05060708 * 256) ""  // prints 101124096! Should be: 21575960576

What happened? Is DXL calculating wrong? No. The int to string conversion function (::.. (int, string) ) treats the passed integer as a 32 bit signed integer. That means, the result you was generated only from the lower 4 bytes of the integer, i.e.   0x06070800 ... You can verify by printing it: 

print (0x06070800 ) // prints 101124096

You should also note that in DXL literals (i.e. numbers that you write inside your code) are only interpreted as 32 Bit signed integers, that is, the maximum numbers you can specify inside your code is -0x7FFFFFFF ... 0x7FFFFFFF or -2147483647 to +2147483647 ... Note that this is not the full 32 bit range. There is exactly one 32 bit number missing:  -2147483648 or 0x80000000 ... We addressed this problem on another post. So this one 32 bit number cannot be expressed with one DXL Literal. The easy workaround is to perform a calculation (-2147483647 - 1). 

Alright. So now we know that 
a) DXL Literals can express 32 bit numbers as signed integers with the one exception of the number 0x80000000
b) DXL performs full 64 bit arithmetics. 

So how can we now construct a 64 bit integer, e.g. hex(FE ED DC CB BA A9 98 87)? 

Since DXL does full 64 bit arithmetics, lets split the memory in three parts:   FE ED DC  , CB BA A9,  98 87 ... 
Now we can easily create these three numbers in little endian:  0xDCEDFE, 0xA9BACB, 0x879800

Now the only thing we need to do, is perform a calculation to build up our memory: 

int SHIFT3 = (256*256*256);
int SHIFT6 = SHIFT3 * SHIFT3; 
int val = 0xDCEDFE + 0xA9BACB * SHIFT3 + 0x8798 * SHIFT6; 
print dumpmemory(val,8) "\n"  // prints "feeddccb baa99887"

Voila. Exactly the 64 bit number that we wanted. So the question is now, how can we 'reverse' that, from a 64 bit integer variable (e.g. memory address) to get the DXL code that will produce that value? Easy, again arithmetics produces the right result: 

int SHIFT3 = 256 * 256 * 256; 
int SHIFT6 = 256 * 256 * 256 * 256 * 256 * 256
int val = 0xDCEDFE + 0xA9BACB * SHIFT3 + 0x8798 * SHIFT6

int SHIFT4 = 256*256*256*256; 
int LOWMASK = 0xFFFFFF*256 + 255; // 0xFFFFFFFF

int getLowInt (int &ref) { return ref & LOWMASK; }
int getHighInt (int &ref) { return (ref / SHIFT4) & LOWMASK; }

int lowval =  getLowInt(val);
int highval = getHighInt(val); 

print lowval "\n";  // prints -874713602
print highval "\n"; // prints -2020038213
print dumpmemory(lowval, 16) // prints feeddccb 00000000 bba99887 00000000

Alright. So here we got those two interesting numbers: -874713602 and  -2020038213. Note that these are full 32 bit numbers representing the high and low integers of the 64 bit number. Now we know the simplest way to reconstruct our value: 

int SHIFT4 = 256*256*256*256; 
int LOWMASK = 0xFFFFFF*256 + 255; // 0xFFFFFFFF

int makeInt64 (int low, int high) {
           return (low & LOWMASK) + (high & LOWMASK) * SHIFT4; 
}

int val = makeInt64 ( -874713602, -2020038213); 
print dumpmemory(val, 8)   // prints  feeddccb bba99887

Ok. So now we can: 

1. Create a 64 bit Integer from to 32 bit integers
2. Get two 32 Bit integers from a 64 bit integers. 

And that means, now we can enable our addr_ eval hacks on DOORS 64 bit! But first we need to check if addr_ will really work with 64 bit values (it should!): 

int val = 16909060 * 256 * 256 * 256 * 256; 
print val "/" (val / 256 / 256 / 256 / 256) "\n" // prints 0/16909060
int newval = 0; newval += (addr_ val) int;
print newval "/" (newval / 256 / 256 / 256 / 256) "\n" // prints 0/16909060

So here we see, that addr_ works as expected. It will convert full 64 bit values to any type. So it is safe to use "addr_ strVar" or similar to get the contents / address of variables. 

So finally we can come up with the following functions: 

int SHIFT4 = 256*256*256*256; 
int LOWMASK = 0xFFFFFF*256 + 255; // 0xFFFFFFFF
int PROBLEMVAL = 0x7FFFFFFF + 1; 

int makeInt64 (int low, int high) {
           return (low & LOWMASK) + (high & LOWMASK) * SHIFT4; 
}

int getLowInt (int &ref) { return ref & LOWMASK; }
int getHighInt (int &ref) { return (ref / SHIFT4) & LOWMASK; }

Buffer bufLit = create();
string save32BitLiteral (int val) {
      setempty(bufLit); 
      int l = getLowInt val; 
      int h = getHighInt val; 
      if (l == PROBLEMVAL) bufLit += "((-0x7FFFFFFF - 1)" else bufLit += "((" l "&(0xFFFFFF*256+255))";
      if (h == PROBLEMVAL) bufLit += "+(-0x7FFFFFFF - 1))" else bufLit += " + (" h "*(0xFFFFFF*256+256)))"; 
      return stringOf bufLit; 
}

Now lets try this: 

int val = makeInt64 ( -874713602, -2020038213); 
print val "/" (val / 256 / 256 / 256 / 256) "\n"  // prints -874713602/-2020038212

string lit = save32BitLiteral(val);
string evalcode = "int val = " lit "\nprint val \"/\" (val / 256 / 256 / 256 / 256) \"\\n\""; 

print evalcode "\n"; 
eval_ evalcode;     // prints -874713602/-2020038212

Ok. The fact that both prints (the one inside the eval and the one outside the eval) get the same result, show that the conversion of the 64 bit to the eval was successful! This is my proposed way to migrate that legacy  eval_ statements to DOORS 64 bits. 

Now lets try the problematic integer 0x80000000 that can not be expressed as a literal. Replace in the above code the following line: 

int val = makeInt64 ( -0x7FFFFFFF - 1, -0x7FFFFFFF - 1);

Works. Prints are the same. That was all folks. Making code that is compatible with 32 bit and 64 bit is left as an exercise for the reader. 

Regards, Mathias

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
Mathias Mamsch - Fri Jul 01 04:47:49 EDT 2016

In case anyone tried the code already and experienced crashes, I just noticed there was a problem with the calculation, which I now fixed on the code. Inside the save32BitLiteral function the low DWORD needs to be masked.

For anyone wondering how this code helps for passing variables to eval_ code on 64 bit machines. 

bool b = false;

string sCode = "
bool &b = addr_ " save32NitLiteral( (addr_ (&b)) int ) "
b = true; 
"
eval_ sCode
print b "\n" // prints true.

Regards, Mathias

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
Mike.Scharnow - Fri Jul 01 06:29:56 EDT 2016

bool b = false;

string sCode = "
bool &b = addr_ " save32NitLiteral( (addr_ (&b)) int ) "
b = true; 
"
eval_ sCode
print b "\n" // prints true.

Mathias, that's great!

I certainly know some guys who will be adopting their code real soon :-)

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
oaklodge - Fri Jul 01 15:33:50 EDT 2016

Warms my heart to get a DOORS under-the-hood lecture from Mathias.  :)

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
SebastienBoucard - Thu Jan 26 09:25:51 EST 2017

Unfortunately this trick doesn't seem to work anymore starting with DOORS 9.6.1.7.

Using the to32BitLiteral() function on a 64-bit int:

int internal32bit_SHIFT4 = 256*256*256*256; 
int internal32bit_LOWMASK = 0xFFFFFF*256 + 255; /* 0xFFFFFFFF */ 
int internal32bit_PROBLEMVAL = 0x7FFFFFFF + 1; 
Buffer internal32bit_buffer = create(); 
string to32BitLiteral(int val) {
    if (internal32bit_SHIFT4 == 0) return "" val "";
    setempty(internal32bit_buffer);
    int l = val & internal32bit_LOWMASK; 
    int h = (val / internal32bit_SHIFT4) & internal32bit_LOWMASK; 
    if (l == internal32bit_PROBLEMVAL) internal32bit_buffer += "((-0x7FFFFFFF - 1)" else internal32bit_buffer += "((" l " & (0xFFFFFF*256+255))"; 
    if (h == internal32bit_PROBLEMVAL) internal32bit_buffer += " + (-0x7FFFFFFF - 1))" else internal32bit_buffer += " + (" h "*(0xFFFFFF*256+256)))"; 
    return stringOf internal32bit_buffer; 
}

int i = 0xFFFFFFFF + 1;
print "" to32BitLiteral(i) ""

On DOORS 9.6.1.6, I get the following:

((0 & (0xFFFFFF*256+255)) + (1*(0xFFFFFF*256+256)))

On DOORS 9.6.1.7, I get:

0

I'm anything but a 64-bit memory expert and might have done something wrong. But I'm a bit afraid something has changed on DOORS side, notably seeing this king of fix:

http://www-01.ibm.com/support/docview.wss?uid=swg1PI68894

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
Mathias Mamsch - Thu Jan 26 11:46:31 EST 2017

int internal32bit_SHIFT4 = 256*256*256*256; 
int internal32bit_LOWMASK = 0xFFFFFF*256 + 255; /* 0xFFFFFFFF */ 
int internal32bit_PROBLEMVAL = 0x7FFFFFFF + 1; 
Buffer internal32bit_buffer = create(); 
string to32BitLiteral(int val) {
    if (internal32bit_SHIFT4 == 0) return "" val "";
    setempty(internal32bit_buffer);
    int l = val & internal32bit_LOWMASK; 
    int h = (val / internal32bit_SHIFT4) & internal32bit_LOWMASK; 
    if (l == internal32bit_PROBLEMVAL) internal32bit_buffer += "((-0x7FFFFFFF - 1)" else internal32bit_buffer += "((" l " & (0xFFFFFF*256+255))"; 
    if (h == internal32bit_PROBLEMVAL) internal32bit_buffer += " + (-0x7FFFFFFF - 1))" else internal32bit_buffer += " + (" h "*(0xFFFFFF*256+256)))"; 
    return stringOf internal32bit_buffer; 
}

int i = 0xFFFFFFFF + 1;
print "" to32BitLiteral(i) ""

Hi Sebastien, 

Mike already warned that IBM would do this. This is what they wrote regarding a APAR: 

...
Thank you for contacting IBM support. I will be helping you with this.


This is a known issue that will be fixed in DOORS Client 9.6.1.7 (due before
the end of 2016), and there is also a workaround in current versions of
DOORS.

The problem lays in the conversion of integers into strings. It works well
for values below 2^31, but not for greater values: the string value gets
capped to 2^31-1=2147483647, as shown in the example below:
print intOf "111112222233333" // prints 2147483647

In DXL integers are meant to be (signed) 32bit long, but they can actually
store 64bit addresses anyway. Unfortunately several operations only work for
32bit long values (among which the conversion to string). Problems only
occur when an address is greater than what a 32bit long integer can hold.
This can occur in any 64bit versions of the DOORS client, and any version of
Windows, but it is much more likely to occur with version 9.6.1.6 and on
other OS than windows 7.

DOORS 9.6.1.7 will fix this problem by creating a new type (Addr64_) to hold
addresses, while integers will be enforced to be really 32bit long. So you
will have to modify your script for DOORS 9.6.1.7 accordingly (use a
variable of the new type to record the address of variables).

...

So it seems we need to lookout for the new Addr64_ perm. If you can find the solution, can you post? In the meantime I will add a fat note above. 

Regards, Mathias

Re: Tutorial: DOORS 64bit memory and hacks (addr, eval_, etc)
Mathias Mamsch - Fri Jan 27 09:36:37 EST 2017

int internal32bit_SHIFT4 = 256*256*256*256; 
int internal32bit_LOWMASK = 0xFFFFFF*256 + 255; /* 0xFFFFFFFF */ 
int internal32bit_PROBLEMVAL = 0x7FFFFFFF + 1; 
Buffer internal32bit_buffer = create(); 
string to32BitLiteral(int val) {
    if (internal32bit_SHIFT4 == 0) return "" val "";
    setempty(internal32bit_buffer);
    int l = val & internal32bit_LOWMASK; 
    int h = (val / internal32bit_SHIFT4) & internal32bit_LOWMASK; 
    if (l == internal32bit_PROBLEMVAL) internal32bit_buffer += "((-0x7FFFFFFF - 1)" else internal32bit_buffer += "((" l " & (0xFFFFFF*256+255))"; 
    if (h == internal32bit_PROBLEMVAL) internal32bit_buffer += " + (-0x7FFFFFFF - 1))" else internal32bit_buffer += " + (" h "*(0xFFFFFF*256+256)))"; 
    return stringOf internal32bit_buffer; 
}

int i = 0xFFFFFFFF + 1;
print "" to32BitLiteral(i) ""

Solution for DOORS 9.6.1.7 upwards  -  see here: 

https://www.ibm.com/developerworks/community/forums/html/topic?id=d7274c76-77e4-4932-ab29-829cb132dd4e

Hope it helps, regards, Mathias