ColorWorks V2 vs. GRADD

An Adventure in Reverse Engineering

by Michal Necasek, January 2002


Download patches - if you're not interested in my revealing scientific analysis, jump straight to the download links. But you're missing a lot!

It is a well known fact that the venerable OS/2 graphics program ColorWorks does not work very well on OS/2 systems that run display drivers based on the GRADD architecture. Users inevitably blame that on IBM or GRADD driver vendors but reality is not so black and white. I knew about these problems for a long time but didn't really know what to do about them.

Then I started working on the OS/2 history articles for the VOICE Newsletter. When I was writing about OS/2 1.3, I simply used an existing program called PMCamera to take screenshots. When I went back to OS/2 1.2 and 1.1 however, PMCamera failed me -  so I decided to whip up my own simple little utility to take screenshots. And I did. The utility worked fine on the VGA drivers provided in OS/2 1.1, 1.2, 1.3, 2.0 as well as SDD on OS/2 Warp 4 and eCS. But then I needed to take pictures of OS/2 2.1 and OS/2 Warp and to my great surprise, my utility failed. More precisely, the headers of saved screenshots (in OS/2 BMP format) claimed that the images only had one single scanline. Needless to say, I was rather surprised after using the exact same binary on so many other versions of OS/2.

It did not take me long to track the problem to this bit of code:

  ....
  /* Read just one scanline to get the bitmap header */
  GpiQueryBitmapBits(hpsMemory, 0, 1, pchBits, &bmap.bmi);
  ....

On the failing systems, the bitmap header returned by GpiQueryBitmapBits() had the cy member (bitmap height) set to one, apparently because I only queried one scanline. So I opened the trusty PM Programmer's Reference manual to find out which version of OS/2 was right. I was not enlightened. The PM Reference is not clear on this point and the returned value might be really either. It obviously depends on the display driver which value is returned (not good if you ask me).

I easily worked around the problem in my screenshot program but it got me thinking. The symptoms for some reason reminded me of the ColorWorks problems on GRADD. And I thought, what if ColorWorks is running against some kind of similar problem? And if it does, could it be fixed? Because in many effects previews of ColorWorks, if checking the "Stack Previews" option, ColorWorks would only display the leftmost column of the preview window and the rest would be black. Moreover, in some cases the application would lock up completely.

So I pulled out IDA (Interactive DisAssembler) and set out to explore the ColorWorks executable. I quickly identified few important Gpi calls and was shocked and horrified to discover the cause of lockups seen with ColorWorks on GRADD drivers. It was ColorWorks' "ingenious" error handling. Their code looks approximately like this:

  do {
      rc = GpiXXX();
      if (rc != GPI_ALTERROR)
          DosSleep(0);
  } while (rc != GPI_ALTERROR);

It is immediately obvious that if the Gpi call fails for any reason, the application will lock up. It is entirely beyond me what the programmers were trying to achieve there. Whatever it was, it didn't work.

But what I really wanted to find out was why the Gpi call in question (GpiSetBitmapBits()) was failing in the first place. I patched a debug interrupt (INT 3) into the loop and ran ColorWorks under good old IPMD. I discovered that the bitmap header passed to GpiSetBitmapBits() contained garbage in the cx and cy structure members. Because the call in question was only copying a single scanline, I can imagine that some display drivers did not care. GRADD however does - and rightly so in my opinion.

Now that I knew what was going on, the next question was how to fix it. Writing code that would initialize the bitmap header values properly would be easy but I did not know how to add it to the existing executable. Adding code to OS/2 LX executables is not trivial. The executable header would have to be modified and perhaps a new page of code added, a messy business with too many things that could go wrong. But then I looked closer at the code that I needed to modify and realization dawned. I didn't have to expand the executable at all!

Because ColorWorks is written in C (and built with IBM's VisualAge C++ compiler or perhaps the older CSet++), the generated code is not quite as tight as it could be. I found out that I could easily compress the code by optimizing it a little (basically by making better use of registers) and I would have enough space to insert my patch and still have some extra bytes left.

So I did just that and made sure the cx and cy members were initialized to reasonable values, derived from the other parameters passed to GpiSetBitmapBits(). And guess what, it worked! ColorWorks was no longer locking up and the stacked previews were displaying properly.

The last task was creating a small patch that could be easily distributed to users. I decided to take advantage of existing tools and use the PATCH utility that has been part of OS/2 since version 1.0. I could not find any utility that would automate creation of patches (I'm almost sure there is one) but the difference between patched and unpatched versions was so small that I simply edited the patch file by hand. And here are the results:
 

Download

Patches should be applied by running the OS/2 PATCH command in the ColorWorks directory. The patch files look for CW.EXE and attempt to verify that the expected (unpatched) file is present to minimize the danger of messing up your data. It you have PATCH.EXE (a DOS program) in your ColorWorks directory, make especially sure you're running the OS/2 PATCH utility.

Please choose the appropriate patch file depending on which version of ColorWorks you use:

ColorWorks V1+
ColorWorks V2 - unpatched
ColorWorks V2 - with previous patch from SPG applied

What these patches fix:

If you think that the patches for ColorWorks V1+ and for previously unpatched ColorWorks V2 are the same, that's because they are. The executables bear uncanny resemblance to each other. Imagine my surprise when I was working on a patch for V1+, only to discover that I already had it!
 

Conclusion

The fact that a software vendor is not interested in fixing their product doesn't mean it can't be done. It can be even fun! And educational to boot! As usual, feel free to flame me at <MichalN@prodigy.net>.