DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES

I've probably committed a cardinal forum sin here by borrowing a colleague's forum log-in, but I will correct that later once I have worked out what my original account was....  And of course apologize to Peter when the sun rises again in his part of the world tomorrow.  My name is Alex, by the way.

So..

This subtle but amazingly fundamental bug certainly exists in DOORS version 9.5.1.0.  I wonder whether pushing through an update may solve it, since I know the bug also existed in a previous version.  I haven't seen anything posted about it - but I suspect some of the posts I've read about mystery bugs may actually be due to this bug and no-one has realised it!.
 

Anyways, consider the code below where I use a reference variable (&MyRef) to represent a variable passed by reference as a subroutine parameter - it happens in that scenario too, but writing it the way I have makes for a simpler example and has the same behaviour.

int ConstA = 1
int ConstB = 2


int MyA=ConstA
int &MyARef=MyA

I then have created a series of one-liner tests to demonstrate the bug:

print "Test 1 (A==ConstA should all be true)   => " (MyA==ConstA) " " (ConstA==MyA) " " (MyARef==ConstA) " " (ConstA==MyARef) "\n" 
print "Test 2 (A==ConstB should all be false)  => " (MyA==ConstB) " " (ConstB==MyA) " " (MyARef==ConstB) " " (ConstB==MyARef) "\n" 
print "Test 3 (A!=ConstA should all be false)  => " (MyA!=ConstA) " " (ConstA!=MyA) " " (MyARef!=ConstA) " " (ConstA!=MyARef) "\n" 
print "Test 4 (A!=ConstB should all be true)   => " (MyA!=ConstB) " " (ConstB!=MyA) " " (MyARef!=ConstB) " " (ConstB!=MyARef) "\n" 
print "Test 5 (!(A==ConstA) should all be false) => " (!(MyA==ConstA)) " " (!(ConstA==MyA)) " " (!(MyARef==ConstA)) " " (!(ConstA==MyARef)) "\n" 
print "Test 6 (!(A==ConstB) should all be true)  => " (!(MyA==ConstB)) " " (!(ConstB==MyA)) " " (!(MyARef==ConstB)) " " (!(ConstB==MyARef)) "\n" 
print "Test 7 (!(A!=ConstA) should all be true)  => " (!(MyA!=ConstA)) " " (!(ConstA!=MyA)) " " (!(MyARef!=ConstA)) " " (!(ConstA!=MyARef)) "\n" 
print "Test 8 (!(A!=ConstB) should all be false) => " (!(MyA!=ConstB)) " " (!(ConstB!=MyA)) " " (!(MyARef!=ConstB)) " " (!(ConstB!=MyARef)) "\n"

The output from this test series produces:

Test 1 (A==ConstA should all be true) => true true false true

Test 2 (A==ConstB should all be false) => false false false false

Test 3 (A!=ConstA should all be false) => false false true false

Test 4 (A!=ConstB should all be true) => true true true true

Test 5 (!(A==ConstA) should all be false) => false false true false

Test 6 (!(A==ConstB) should all be true) => true true true true

Test 7 (!(A!=ConstA) should all be true) => true true false true

Test 8 (!(A!=ConstB) should all be false) => false false false false

In other words, test numbers 1, 3, 5 and 7 demonstrate that there is a bug in the evaluation of the '==' and '!=' operators (not just for int - it happens for strings, structs, anything!)  It occurs when the LHS operand is a reference variable!

Does anyone have a fix for this extremely dangerous (development-cost-wise) and ubiquitous bug that can sometimes (according to situation) appear to be working 99.9% of the time but then give false outcomes that are virtually untraceable in a complex program?

With a fault as fundamental as this, I'm surprised that DXL is even considered usable!

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Sun Oct 26 17:03:45 EDT 2014

Have established that this bug exists in 9.5.1.0, 9.5.1.1, 9.5.1.2, 9.5.1.3 and 9.5.2.2 of the Windows client.

How is it surviving this long?

Can someone test for it in 9.6.X.X ?  (I can't upgrade further because my customer isn't ready to upgrade to 9.6 yet).

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Sun Oct 26 17:05:32 EDT 2014

Just to demonstrate how fundamental to the DXL interpreter this bug is, consider: 

int MyA = 1
int *MyARef=&MyA

print ((*MyARef)==MyA) "\n"  // produces false
print (MyA==(*MyARef)) "\n"  // produces true

So I suspect it is the interpreter passing a wrong argument to the '==' operator, which simply cannot be undone by overriding the operator (I tried and failed).  Incidentally, I tried this for '<=' and '>=' and the bug is not there then.  This eventually led to a conceptual "workaround" which, whilst ugly, solves the cause- but the implications of implementing it are a bit untenable in my mind: (a) you'd have to override all instances of the '==' and '!=' operator with code that shouts (say) error("This operator is defunct: use EQ or NEQ"), and then (b) replace '==' and '!=' in ALL your code, never to be used again!

So here's that supposed work-around: 

int ConstA = 1
int ConstB = 2


int MyA=ConstA
int &MyARef=MyA

bool EQ(int a,int b) { return (a)==(b) }
bool ::==(int a, b) {error("The '==' operator is unsafe! Use EQ instead")}

bool NEQ(int a,int b) { return (a)!=(b) }
bool ::!=(int a, b) {error("The '!=' operator is unsafe! Use NEQ instead")}

print "Test 1 (A==ConstA should all be true)   => " EQ(MyA,ConstA) " "EQ(ConstA,MyA) " " EQ(MyARef,ConstA) " " EQ(ConstA,MyARef) "\n"  
print "Test 2 (A==ConstB should all be false)  => " EQ(MyA,ConstB) " " EQ(ConstB,MyA) " " EQ(MyARef,ConstB) " " EQ(ConstB,MyARef) "\n"  
print "Test 3 (A!=ConstA should all be false)  => " NEQ(MyA,ConstA) " " NEQ(ConstA,MyA) " " NEQ(MyARef,ConstA) " " NEQ(ConstA,MyARef) "\n"  
print "Test 4 (A!=ConstB should all be true)   => " NEQ(MyA,ConstB) " " NEQ(ConstB,MyA) " " NEQ(MyARef,ConstB) " " NEQ(ConstB,MyARef) "\n"  
print "Test 5 (!(A==ConstA) should all be false) => " (!EQ(MyA,ConstA)) " " (!EQ(ConstA,MyA)) " " (!EQ(MyARef,ConstA)) " " (!EQ(ConstA,MyARef)) "\n"  
print "Test 6 (!(A==ConstB) should all be true)  => " (!EQ(MyA,ConstB)) " " (!EQ(ConstB,MyA)) " " (!EQ(MyARef,ConstB)) " " (!EQ(ConstB,MyARef)) "\n"  
print "Test 7 (!(A!=ConstA) should all be true)  => " (!NEQ(MyA,ConstA)) " " (!NEQ(ConstA,MyA)) " " (!NEQ(MyARef,ConstA)) " " (!NEQ(ConstA,MyARef)) "\n"  
print "Test 8 (!(A!=ConstB) should all be false) => " (!NEQ(MyA,ConstB)) " " (!NEQ(ConstB,MyA)) " " (!NEQ(MyARef,ConstB)) " " (!NEQ(ConstB,MyARef)) "\n"  

print (ConstA==ConstA) "" // Generates a runtime error

RESULT

Test 1 (A==ConstA should all be true) => true true true true

Test 2 (A==ConstB should all be false) => false false false false

Test 3 (A!=ConstA should all be false) => false false false false

Test 4 (A!=ConstB should all be true) => true true true true

Test 5 (!(A==ConstA) should all be false) => false false false false

Test 6 (!(A==ConstB) should all be true) => true true true true

Test 7 (!(A!=ConstA) should all be true) => true true true true

Test 8 (!(A!=ConstB) should all be false) => false false false false

-R-E- DXL: <Line:11> The '==' operator is unsafe! Use EQ instead

Backtrace: <Line:25>

-I- DXL: execution halted

The solution works, because the EQ and NEQ internally use '==' and '!=' with dereferenced values - plus, the interpreter doesn't exhibit the bug when calling routines in general, only when specifically calling '==' and '!=' and even then, only when with an LHS variable reference.

But of course, this somewhat radical "workaround" is also ultimately doomed, since if the code is included a second time, the solutions (EQ and NEQ) will then raise errors because they use '==' and '!=' themselves!  Hah - Snookered!  So the overrides that prevent people from using '==' and '!=' would eventually have to be removed, which means the bug could then still find its mark in future unsuspecting code!

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Sun Oct 26 17:06:47 EDT 2014

int MyA = 1
int *MyARef=&MyA

print ((*MyARef)==MyA) "\n"  // produces false
print (MyA==(*MyARef)) "\n"  // produces true

int ConstA = 1
int ConstB = 2


int MyA=ConstA
int &MyARef=MyA

bool EQ(int a,int b) { return (a)==(b) }
bool ::==(int a, b) {error("The '==' operator is unsafe! Use EQ instead")}

bool NEQ(int a,int b) { return (a)!=(b) }
bool ::!=(int a, b) {error("The '!=' operator is unsafe! Use NEQ instead")}

print "Test 1 (A==ConstA should all be true)   => " EQ(MyA,ConstA) " "EQ(ConstA,MyA) " " EQ(MyARef,ConstA) " " EQ(ConstA,MyARef) "\n"  
print "Test 2 (A==ConstB should all be false)  => " EQ(MyA,ConstB) " " EQ(ConstB,MyA) " " EQ(MyARef,ConstB) " " EQ(ConstB,MyARef) "\n"  
print "Test 3 (A!=ConstA should all be false)  => " NEQ(MyA,ConstA) " " NEQ(ConstA,MyA) " " NEQ(MyARef,ConstA) " " NEQ(ConstA,MyARef) "\n"  
print "Test 4 (A!=ConstB should all be true)   => " NEQ(MyA,ConstB) " " NEQ(ConstB,MyA) " " NEQ(MyARef,ConstB) " " NEQ(ConstB,MyARef) "\n"  
print "Test 5 (!(A==ConstA) should all be false) => " (!EQ(MyA,ConstA)) " " (!EQ(ConstA,MyA)) " " (!EQ(MyARef,ConstA)) " " (!EQ(ConstA,MyARef)) "\n"  
print "Test 6 (!(A==ConstB) should all be true)  => " (!EQ(MyA,ConstB)) " " (!EQ(ConstB,MyA)) " " (!EQ(MyARef,ConstB)) " " (!EQ(ConstB,MyARef)) "\n"  
print "Test 7 (!(A!=ConstA) should all be true)  => " (!NEQ(MyA,ConstA)) " " (!NEQ(ConstA,MyA)) " " (!NEQ(MyARef,ConstA)) " " (!NEQ(ConstA,MyARef)) "\n"  
print "Test 8 (!(A!=ConstB) should all be false) => " (!NEQ(MyA,ConstB)) " " (!NEQ(ConstB,MyA)) " " (!NEQ(MyARef,ConstB)) " " (!NEQ(ConstB,MyARef)) "\n"  

print (ConstA==ConstA) "" // Generates a runtime error

Some further experimentation discovered an interesting effect of overloading functions by differences in parameter passing mechanism alone.  Consider the following test code:

//-----------------------------------------------------------
// CHOOSE ONE ONLY OF THE FOUR ROUTINES BELOW AND IT WORKS
// CHOOSE MORE THAN ONE AND IT FAILS IN INTERESTING WAYS
// ONE OF THESE MANY WAYS IS IDENTICAL TO THE '==' BUG!
//-----------------------------------------------------------
bool EQ(int a,int b)    { print "E1>"; return  (a-b>=0)&&(b-a>=0)  }
// bool EQ(int &a,int b)   { print "E2>"; return  (*(addr_ a)-b>=0)&&(b-*(addr_ a)>=0)  }
// bool EQ(int a,int &b)   { print "E3>"; return  (*(addr_ b)-a>=0)&&(a-*(addr_ b)>=0)  }
// bool EQ(int &a,int &b)  { print "E4>"; return  (*(addr_ b)-*(addr_ a)>=0)&&(*(addr_ a)-*(addr_ b)>=0)  }



Although not relevant, the code in these routines seek to carefully "emulate" an '==' operation without actually using '=='.  Instead they carefully use '<=' and '>=', because these do not exhibit the bug in themselves.  If you select just any single one of these routines, then it works in all respects.  HOWEVER, selecting two or more has varying effects, from getting the wrong answer all the time, through exhibiting behaviour like the '==' bug, right up to actually crashing DOORS!   Studying these behaviours leaves me to believe that the interpreter "prepares data" for the CORRECTLY MATCHING routine if it is defined, but then actually CALLS the last routine defined (every time - irrespective of the referencing mechanism of the variables you send to it) without then coercing the mechanism to match the routine it actually calls.

Experimenting with various combinations and declaration orders of these routines appears to suggest that, somewhere in the depths of the DXL code base, there could be exactly two such overloaded routines (differing only in their parameter passing mechanism) for '==' and '!=' and the coder just got lucky that it works for three-out--of-four actual parameter passing mechanism combinations without crashing.  I can emulate exactly that outcome by (tediously) experimenting with the above routines.   However, the bottom line is that ANY ONE of the routines would work perfectly on its own and the other three are redundant - dangerous even.  Individually they work for all source variables because of the automatic translation perms (for *X and &X), but I suspect it is that very automatic translation that is fouling up here, because it fights with (and wins against) the overloaded routine selection mechanism.

The follow-on conclusion might be that one should NEVER, in one's own code, create overloaded routines that differ only in their parameter passing mechanism but do not differ in actual parameter types.  DOORS allows it, but it breaks things in all kinds of new, diverse and interesting ways.  

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Sun Oct 26 17:08:06 EDT 2014

//-----------------------------------------------------------
// CHOOSE ONE ONLY OF THE FOUR ROUTINES BELOW AND IT WORKS
// CHOOSE MORE THAN ONE AND IT FAILS IN INTERESTING WAYS
// ONE OF THESE MANY WAYS IS IDENTICAL TO THE '==' BUG!
//-----------------------------------------------------------
bool EQ(int a,int b)    { print "E1>"; return  (a-b>=0)&&(b-a>=0)  }
// bool EQ(int &a,int b)   { print "E2>"; return  (*(addr_ a)-b>=0)&&(b-*(addr_ a)>=0)  }
// bool EQ(int a,int &b)   { print "E3>"; return  (*(addr_ b)-a>=0)&&(a-*(addr_ b)>=0)  }
// bool EQ(int &a,int &b)  { print "E4>"; return  (*(addr_ b)-*(addr_ a)>=0)&&(*(addr_ a)-*(addr_ b)>=0)  }

SUMMARY CONCLUSION

Let me be clear on this:

IMHO the '==' and '!=' behaviour is not the bug - it is the symptom.

The bug, I believe, is that the pointer-coercion perms (&) and (*) work independently of the interpreter's selection of an overload and produce a mismatch between the selected overload and the coercion that they produce.  This occurs when the overloads are only distinguishable by passing mechanism, not by the underlying type (e.g:.  (int &) versus (int).  The outcome is either that an address is used as a value (can be hard to detect), or that a value is used as an address (usually causing an unhelpful stack-dumping crash).  Changing the order of declaration of the overloads changes the outcome, and can appear to "fix" things - but never for ALL combinations of the parameter's own original source nature (be that ref or value). 

You cannot 'fix' this problem by defining yet another overload - it will only produce other (differing) faulty outcomes - because the bug lies in the fact that the interpreter generates call-preparation code after regarding (int &) and (int) parameters to be different types and thus assuming a particular overload will be called.  It then prepares for that call.  But then it completely ignores the impact of what it has just done and always calls the last defined overloaded routine because it then does not treat (int &) and (int) parameters as different types and so chooses the wrong subroutine to actually call.  This disjoint behaviour can only be due to a bug in the code of the interpreter and IBM will therefore need to fix this to make the problems go away.  They need to decide whether to regard references (int &), pointers (int *) and values (int) as different types, or as same types with differing passing mechanisms, and then make sure that all parts of the interpreter play consistently by the same rules.  Furthermore, if they decide upon the latter (which is how most sane languages behave) they should then make it an error to create overloads that differ only in passing mechanism and not in type.  This is called "joined up" behaviour.

Whilst all the perms-lists I've looked at don't appear to show any '==' or ':=' with reference parameters, they DO show that there are definitions for EACH TYPE and then ALSO for the universal types: (_xx, _xx) - and this combination is probably just as likely to fall foul of the above bug.  It is probably quite telling that there are NO universal type overloads for '<=' and '=>', and that these operators do not share the problems that '==' and '!=' have.  However, if IBM simply removed the universal type overloads for '==' and '!=', then you'd probably lose the ability to compare struct, DxlObject, etc. But like I said before... Symptom, not cause.  It would be the wrong way to fix this!

To "prove" the bug, you can construct identical symptoms with ANY routine by creating overloads, with ref-params / value-params of the same type in varying combinations.  Juggling the order of declaration of these routines will (in a certain combination) emulate the symptoms exhibited by '==' and ':='.  Some combinations will always result in addresses being sent as values.  Other combinations will always crash DOORS with that unhelpful stack dump.  This is ultimately why it is a bug - not a feature.

Until IBM sort this out, it will always be unsafe to define overloads that differ only in whether its parameters are passed by value or by reference and not by type as well - and adding another overload that uses the universal type (as occurs in the perms) clearly does exactly this!

Finally, using a variable, that is sourced as a reference, on the LHS of either '==' or '!=' will presently always fail to work, without producing any "compile error" or "runtime error" reports in itself.  Thus you will never be really sure just how many undetected bugs you have in your DXL program.   But like I said, I believe this to be merely a symptom of the above common bug.

If IBM could address and solve this bug, most of the mysterious behaviours in DXL could quite possibly go away and we could all live easier and less stressful lives.

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
Mathias Mamsch - Mon Oct 27 12:49:06 EDT 2014

Ok, after this very detailed demonstration of what you call a bug, we should clear up the mechanisms behind that (also for the user that might not have a deep insight as you). The first thing people need to know, is that in DOORS all operators are functions (perms, build into the interpreter):

Looking at one of the available perms list, you can see that DXL knows the above variants of the comparison operator, the bold one taking a special role as we will see. So the first thing to notice here is that DXL knows special comparison operators for: Buffers, ModuleVersions, Dates, Strings, etc. We will come to those in a second.

The next thing that you need to know about is how exactly DXL handles reference variables. This can be confusing because it is different from C/C++. DXL knows about two reference types, the normal reference (declared by an & operator in front of the variable name) and a pointer (declared by *). 

Pointers behave very straight forward, because there are no mechanisms attached to them. References behave in a very certain way, and lets clear them up:

1. A reference can be converted to a pointer and a pointer to a reference by the following perms.
   

   _k* ::(&) (_k&)

   _k& ::(*) (_k*)

   
   So you need to realize what you are doing, when you do:
   
   int *a = 10;  int x = *a;
   
   Is that by using *a you are making a reference from a (and NOT an integer value) and assigning the reference to the variable x. This will be important shortly.
    
2. DXL will convert references to non references, when calling a function.
   
   When you do:
   
   

   void testme (int a) { print a }
   
   int a = 10; 
   int &b = a; 
   
   testme(b)
   

   Then the DXL interpreter will know to pass the VALUE of b to the function testme and not the reference. Of course the same works backwards too. When you do:
   
    

   void testme (int &a) { print a }
   
   int b = 10; testme(b)
   

   The DXL interpreter will know to pass a reference to b to the function testme.

Knowing these two things clears up the behaviour. Take as a first example:

int a = 10
int b = 10

int &x = a
int &y = b

// TRUE, because DXL will use ::== (int, int) and pass Values 
print (x == y) "\n"

So you need to understand, that DXL will NOT compare references! If you want to compare references (i.e. memory addresses), you need to cast to a pointer:

int a = 10
int b = 10

int &x = a
int &y = b
int &z = a

// FALSE, because DXL will compare pointers
print (&x == &y) "\n"

// TRUE, because pointers are equal and DOORS will use bool ::== (_xx, _xx)
print (&x == &z) "\n"

So lets take your example.

int MyA = 1
int *MyARef=&MyA

print ((*MyARef)==MyA) "\n"  
print (MyA==(*MyARef)) "\n" 

We simplify (remove pointers) and see:

int a = 1
int &b = a

print (b == a) "\n"  // False, uses the ::==(_k, _k) but DXL does not correctly dereference the first parameter! 

print (a == b) "\n"  // True, uses the ::==(int, int), correct dereference
print (b == a+0) "\n"  // True, force the parser back to the ::==(int, int), correct dereference

print (a == b+0) "\n"  // True, like above

print (b+0 == a) "\n"  // True, like above
print (a+0 == b) "\n"  // True, like above

Why does the first case above yield false? Because DOORS is using the 'wrong'  perm for comparison: bool ::== (_xx, _xx)

It is a known issue, that DOORS does not handle parsing of those "underscore" types not correctly. This can bring misfit and crashes in a lot of situations. The general problems with the underscore parameters is that the parser simply does only work for the first parameter correctly ;-) We even had some problems around code being interpretable the first time and the second time fail with an error ...

The general rule of thumb is, to never use references in expressions and dereference properly. In a case, where you want to compare "references" make sure, that you are using pointers. And of course not to use to much "magic" in DXL code, that will not only make your code fail badly but might also crash the client from time to time.

Regards, Mathias

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
Mathias Mamsch - Mon Oct 27 13:09:56 EDT 2014

So just to state this clearly. This is NOT a problem about references. Reference handling in DXL is fine. It is NOT a problem of the DXL interpreter passing references incorrectly to functions in general.

It is a combined problems of two things that every skilled DXL programmer should know of:

- Be careful when you call ANY perm using "underscore" types (usually they are ambiguous to another perm) and never pass references or any expressions to them.
   --> and since ::== has an "underscore" variant --> Never use references in comparisons ...  --> always tell the interpreter exactly which perm to call!

- Declare functions with underscore types only when you know VERY much what you are doing.

- Never declare functions with reference underscore parameters (it simply does not work, although you can do it)

In my opinion by the way it required no skill to make DOORS crash - so therefore I would not see this example of what does not work as so important to pull 3-4 threads from the past and reference it :-) Making something that works in DXL is harder. And making it in an easy, understandable and compatible way (and use only documented DXL functionality) is an art that some people take not serious enough.

Regards, Mathias

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Tue Oct 28 04:13:20 EDT 2014

Thanks for your reply Mathias (somehow I knew that you'd be the first responder!)  After all no-one exhibits such a clear and deep understanding of DXL as you do. And you are somewhat better at writing it down more clearly than I am.

Much code is re-written "after the fact", and therefore it is all to easy for even experienced DXL programmers to take a subroutine like:

bool Process(int &a, int b)
{
   ... do stuff with a and b, updating a as an 'out' parameter
   ... do other stuff
   ... return a boolean flag
}

and then edit it (to solve some new requirement or bug) to get:

bool Process(int &a, int b)
{
    ... do stuff with a and b, updating a as an 'out' parameter
    ... do other stuff
    return (a==b)    
}

And worse, if 90% of the time 'a' really does not equal 'b', you will probably not even notice something is wrong until much, much, later.

If you then take a "straw poll" of the average programmers who fall foul of this - unaware of the need to take care with undocumented underscore-perms, or which ones they are - then I would respectfully suggest these programmers would call this a bug (and do so with gritted teeth).   Especially because if the previous unedited code had used one of the perms '>=' '>', <=' or '<', then there would not have been any anomalous behaviour, nor any hint that changing it to '==' would be doomed to failure. 

When I say "average programmer", I am counting myself in.  When I say "gritted teeth", I actually meant lots of shouting.  Why such a reaction?  Because this anomalous behaviour can be so hard to detect - it is the kind of bug that typically shows itself by its side-effects somewhere else far away.  That makes it expensive to find (time is money).   And remember that underscore perms are undocumented unless you are willing (like we both would be) to scour the forums instead of just spending that time bug-hunting the code.

But I still think IBM could fix this by having the interpreter do some additional work when it encounters a universal type (_x) in an overload, and by generally preventing (via an error) the creation of overloads that differ only in the presence or absence of the '&' passing mechanism (the latter being achievable via a bolt-on check).

The latter part of the fix would also make sense, since other languages that allow overloads do generally disallow overloading of routines that differ by parameter passing mechanism alone (think of Delphi (extended Pascal) routines with and without the 'var' in their parameters).  From looking at the perms list, I suspect that this would not harm the existing perms.  From the point of view of existing programs that are out there, I suspect that it would lead to a discovery of many 'bugs' rather than 'breaking' them.

Or, as a short-term alternative, they could provide something like the old 'lint' tool, that you could use to analyse your DXL scripts for known faux-pas. 

(Lint?  my God - remember 'lint' and you are really showing your age!)

But until such a time as IBM might tackle this, it seems that we agree that the following code examples will always lead to bugs in your own program:

void Test(int &a, int b)
{
   if (a==b)...  will always return false unless b happens to equal the address of a
   if (b==a)...  will work
   if (a!=b)...  will always return true unless b happens to equal the address of a
   if (b=!a)...  will always work
   if (a<=b)...  will always work, as would (b>=a)
   if (a>=b)...  will always work, as would (b<=a)
   if (a<b)...  will always work, as would (b<a)
   if (a>b)... will always work, as would (b>a)
}

void Test(int a, int b)  // DANGEROUS - Differs only in parameter passing mech.
{
  // This routine, by being too similar an overload, can lead to a stack-dump
  // crash when it accesses 'a', because the passing mechanism has been 
  // confused by the existence of an overload using '&a' rather than 'a'.
  // As a result, 'a' may possibly be an address rather than a value
  // and accessing it can therefore crash DOORS.
  // However, this depends upon the order in which the overloads have been
  // defined!
}

I still think both behaviours described in the snippet above are symptoms of one and the same underlying problem - or as you appear to be saying, a combination of behaviours that result in an undesirable behaviour.

Ultimately its a behaviour that easily leads to bugs that are sometimes very hard to find. 

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Tue Oct 28 05:16:44 EDT 2014

On reflection (and discussion over the phone with Peter), I also realise that my long posts and examples are disguising the fact that this problem is not the outcome of doing "clever" stuff.  The examples and test code I provided were to try analyse and identify cause of the problem.  This is entirely my fault - like I said, you are far better at making a clear understandable post!

In fact, writing extremely simple code on a "clean" DOORS installation suffers from the problem.  I was merely trying to identify its cause, reach and possible solution.  I'm sorry if that wasn't clear (Peter, and probably many others).  I'll have to work on that.

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
Mathias Mamsch - Tue Oct 28 06:58:05 EDT 2014

I do agree, that this is a bug - and although I cannot remember that exactly this problem bit me in the past, I can imagine that this will give you hours of ripping out your hair before finding it ... And I also think it should be fixed ...

But: All my experience with IBM in the past I am pretty sure, that they will NEVER change such a fundamental behaviour of the interpreter :-) Still I would welcome someone putting the examples in as a problem report.

The whole symbol lookup of the interpreter could need some rework.

• It should be forbidden to declare functions that only differ in return type (crashes)
   
• It should be forbidden to declare functions that only differ in the fact that some parameters are references (crashes)
   
• In general they should give a warning whenever an ambigous statement is encountered (i.e. a statement that can be parsed using different functions / perms) like
  

  string func ()         { print "A"; return "" }
  string func (string s) { print "B"; return "" }
  
  func   ""  // calls B
  func() ""  // calls A
  

  In a large program, where someone would add a function to one library that exists in another library stuff like this can break your code in bad ways...
   
• They should fix the problem, that defining a function with a 'name' will hide a global function 'name' without parameters, i.e. defining void read(int a, int b) will hide the declartion of the global function
  

  void read (int x, int y) { }
  
  // Permission p = read | write   // does not work, read hidden
  Permission p = read() | write()  // does work
  

• They should fix the stack overwrite problems for Range_ parameters
  

  void func (Range_ r, int a, int b) {
      print a " " b "\n"
  }
  
  func ( (1:5), 9, 10)
  

   

• They should fix the problems with function names beginning with underscores ...

• ... you name it ...

So lets face it ... The DXL interpreter is flawed ... still it works reasonably enough for these bugs to come out most of the times in "clever" approaches - but, and here I agree very much - also in simple normal code that makes the average developer pull out their hair... And the problem you stated here can definitly appear in normal code and has not been stated before that clearly in this forum as far as I know. So thanks for sharing! And feel free to file a problem report or multiple to IBM ...

Regards, Mathias

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Tue Oct 28 07:26:39 EDT 2014

Wow - thanks for all these other examples!

Maybe now it's worth buying that wig (to replace my hair) and carry on programming!

A list like the one you gave here is worth its weight in gold  (or hair)!

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
GerhardS - Thu Nov 06 08:52:05 EST 2014

Just to complete the list of short stories: A few years ago I had a situation, where a boolean comparison failed in a trigger context. The boolean variable was "true", it was compared against "true", but the result was "false". Definitely a bug in DOORS.

I solved the problem by simply changing the comparison to a string comparison:

if (boolVar "" == true "") {...}

Just a hack for extreme situations like the one I experienced ;-)

- Hardy

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Fri Nov 07 06:32:07 EST 2014

if (boolVar "" == true "") {...}

Yes, this should usually work.  This is because a type-conversion such as this stops the interpreter from offering a ref-variable directly to the operator.  Not quite so sure that a type-coercion (casting) would work, by the way, but what you have here is an actual conversion, and that will do the trick.

Of course, it still depends on you discovering the source of the bug in the first place - which can be quite difficult unless you first suspend your capacity for disbelief when debugging (like when watching sci-fi horror movies).

Another work around is to swap the order of variables so that the ref-variable is on the RHS.  This doesn't help of course if both sides are ref-variables, in which case a solution like yours would have to be used to stop the interpreter from offering a ref-variable directly to the '==' or '!=' operator.

Another defensive technique (given that the bug will continue to exist for the foreseeable future) is to get into the habit of always putting any constant used in a comparison on the LHS - however unnatural this may feel.  This is on the basis that it is better not to be potentially subjected to the bug rather than have to spend hours tracking it down if it occurs.   Either way, vigilance is required whenever you write '==' or '!=' in your code:  make sure you know where the variables came from!

For the really paranoid:

    int LocalX = MyRefVariable

    if (LocalX == aValue) ....

This being of course, another (cumbersome) workaround - albeit more explicit.

Best regards

Alex

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
Wolfgang Uhr - Fri Nov 14 02:46:55 EST 2014

> So thanks for sharing! And feel free to file a problem report or multiple to IBM ...

Some years ago, I have messaged an exchange-error to ibm. Exchange for itself could detect of an embedded ole object has changed. But the change itself is not reported to the DeXI-Status-Attribute. So I've to start a separate script to detect this.

1. They know about the error
2. They have the proper information in their software
3. They do not change anything

The only thing they was interested in was to get information about the workaround I've used and I think, they was interested for it in order to be able to tell it to the other users.

The only chance to change anything inside of ibm is

1. Escalate the problem in your firm up to the ceo
2. Let the ceo speek whith the ceo at ibm

That's the reason for: IBM allways works, until you found how.

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
llandale - Thu Nov 20 14:51:57 EST 2014

bool Process(int &a, int b)
{
   ... do stuff with a and b, updating a as an 'out' parameter
   ... do other stuff
   ... return a boolean flag
}

bool Process(int &a, int b)
{
    ... do stuff with a and b, updating a as an 'out' parameter
    ... do other stuff
    return (a==b)    
}

void Test(int &a, int b)
{
   if (a==b)...  will always return false unless b happens to equal the address of a
   if (b==a)...  will work
   if (a!=b)...  will always return true unless b happens to equal the address of a
   if (b=!a)...  will always work
   if (a<=b)...  will always work, as would (b>=a)
   if (a>=b)...  will always work, as would (b<=a)
   if (a<b)...  will always work, as would (b<a)
   if (a>b)... will always work, as would (b>a)
}

void Test(int a, int b)  // DANGEROUS - Differs only in parameter passing mech.
{
  // This routine, by being too similar an overload, can lead to a stack-dump
  // crash when it accesses 'a', because the passing mechanism has been 
  // confused by the existence of an overload using '&a' rather than 'a'.
  // As a result, 'a' may possibly be an address rather than a value
  // and accessing it can therefore crash DOORS.
  // However, this depends upon the order in which the overloads have been
  // defined!
}

This is a different problem.  I have noticed that when a variable is defined as a reference input parameter to a function, there was some difficulty in using this parameter as both an input and an output parameter several times within the function.

Instead of this:

• void fn(int &a)
• {  use and adjust the value of a throughout
• }

I found I had to do this:

• void fn(int &a)
• {  int aStaged = a
•    use and adjust the value of aStaged throughtout
•    a = aStaged
• }

Having said that, I now see MM's post above suggesting it would fail on the 2nd use of the reference, which seems to have some relevance to my problem above.

-Louie

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
llandale - Thu Nov 20 15:05:26 EST 2014

• "..not to use to much "magic" in DXL code.."
• "And making it in an easy, understandable and compatible way (and use only documented DXL functionality) is an art that some people take not serious enough."

Who are you really and what have you done with the REAL MM?  lol

You are clearly the best person in the world at writing DXL code that violates those two tenants (meaning your code is extremely trustworthy), but you also appear to be the most prolific at violating them.  You rarely post code here that has no "magic" and contains only understandable functionality in the DXL manual.  For example, DXL manual has no "pointers".

But again yes, you generally let others resolve the more mundane questions and consern yourself with providing functionality with, well, your Voodoo because few other folks can.

-Louie

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
llandale - Thu Nov 20 15:28:13 EST 2014

There is something fundamental bothering me here.  While I've copied some of MMs code to provide some low level functionality (in the library, invisible to normal scripts), I can safely say that besides using "Reference" variables as output parameters of functions, I've never had any need whatsoever to use Reference and Pointer variables in DXL.  And I've written a lot of DXL.

Now surely someone will slap me upside the head and enlighten me on some need, but I'm pretty sure most of the time folks are being "tricky" because they can.  In the original example, what is the point of having two identifiers pointing to the same memory?  It makes the code very difficult to understand.

-Louie

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
Mathias Mamsch - Thu Nov 20 15:45:28 EST 2014

Who are you really and what have you done with the REAL MM?  lol

I had my ears open to humble DXL masters, that have a much longer DXL experience than me ;-)

... I can safely say that besides using "Reference" variables as output parameters of functions, I've never had any need whatsoever to use Reference...

Have you ever used references for IN/OUT parameters to functions? I am pretty sure, that bug also applies to this non tricky kind of DXL. This is about a == comparison not working correctly on a reference parameter, which could of course screw up also very non tricky DXL. Regards, Mathias

Re: DXL "EQUAL" AND "NOT EQUAL" OPERATORS FAILURE ON REF VARIABLES
AlexTidmarsh - Fri Nov 21 05:13:13 EST 2014

Yup - that's it in one!

I only used "explicit" reference pointers to illustrate the problem (because those didn't need as much code to demonstrate).    I'm sorry if my original code simplification caused a confusion of purpose.

But the REAL problem lies with values that were passed by reference as arguments (parameters) to a function.  Its just the mechanism is basically the same. 

Once you are in that subroutine, the DXL interpreter busily works behind the scenes automatically "dereferencing" those reference variables!

99% of the time, it gets this right.

EXCEPT when the dereference occurs on the left-hand-side of any '==' or '!=' test.

THEN the DXL interpreter gets its knickers in a twist: every time.  Consistently.  Irrespective of variable type.

Ok - now that really is about as succinct as I can put it!  :-)

Do you really not use (&x) style parameters in your functions Louie?  That I find very hard to believe!