[Ed: I’m changing things up today and writing about Windows. Obviously, the day job is involved.]
I’ve been consistently running into a problem with signtool.exe refusing to work at random points throughout my day. It would complain that it requires capicom.dll v. 126.96.36.199 or higher. Despite the fact that I had several capicom.dll files on my system (no, I don’t know why) and they’re all v 188.8.131.52, signtool.exe would refuse to use them.
I believe (and here I really want to stress that I’m not speaking from a position of authority!) that the problem stemmed from the dll registration system. When I tried to register capicom.dll (any of them) manually, regsvr32 would fail and report “invalid access to memory location”.
Some poking around on Google revealed the cause: Windows XP SP2 added support for the modern processor’s No Execute Bit (which prevents the execution of code in data segments of memory and provides some much needed security from buffer overrun attacks). Apparantly, Microsoft considers regsvr32 an attack vector or something and it won’t work correctly with this extra protection turned on.
So the answer was to disable it. This Microsoft Knowledgebase article describes the functionality (called Data Execution Prevention, or DEP, in Microspeak) and how to control it. I went with changing my boot.ini’s noexecute parameter to AlwaysOff. And signtool.exe magically works!
This is obviously not a “correct” solution as my system is suddenly more vulnerable to machine-pwning attacks. But it works and it lets me get back to doing my job. So I’m not going to spend any more time on it, despite the fact that I’m somewhat unhappy about how it all turned out.
Also: “capicom.dll” is possibly the most ridiculous file name I’ve ever seen. Every time I say it out loud, I want to punch myself in the face.
[Added 2008-08-27]: So, the symptoms are happening again. My fix didn’t actually fix it. It’s such an intermittent problem, that it’s hard to diagnose. I hate intermittent problems. Anyway. Bear this in mind before you go an disable the DEP support on your system.