Skip to content

The dangers of self-modifying code

One of my coding stories recently showed up on reddit and spawned a giant thread. Having it brought up again reminded me of another fun bug from the early days, in this case 1991.

I had just graduated from college and joined Blue Sky Productions, soon to be renamed Looking Glass Technologies, where we were working on the PC game Ultima Underworld, a first-person 3D dungeon crawl. I needed a machine so I was given an 33 MHz 80486 PC. Everybody else was jealous because not only was it immediately the fastest PC in the office, it was also the only one with a floating point unit. The downside was that everybody wanted to run their tools requiring floating point on my machine.

Anyway, of course the first thing to do as soon as the machine showed up was to run the in-progress version of Underworld on it to see how zippy it would be—perhaps it could get all the way up to 15 frames a second! Unfortunately, the game completely failed on my machine. I can’t remember whether it just crashed, or everything on screen was just black, or what, but it just did not work at all.

After much hair-tearing, we figured out the problem. The texture mapper (in software, not hardware, of course), which had been written by an outside guy (Chris Green—I see he’s now at Valve) and handed to us, was using self-modifying code in an attempt to squeeze as much performance as possible out of the system. Where a normal program might have a loop that accesses a variable each time through the loop, our texture mapper’s loop would just refer to a constant value. Before entering the loop, it would do the variable lookup once, poke that value into its own code, replacing the constant, and then run the loop a bunch of times without having to waste time looking up the value again. Brilliant! Impossible to debug, but brilliant.

Unfortunately, the 486 had its own optimization—an instruction cache. So we were dutifully poking new values into program memory that had already been read and was never reread, which meant that all of the updates were completely ignored. Oops.

I see that the Wikipedia article on self-modifying code has a section on exactly this problem.