Since I was unemployed, I was playing games I always wanted to ever since I built a PC with my adult money (and thankfully before the RAM crisis). And in doing so, I ended up trying to play Burnout Paradise. Just like with many of the games I played in the past like Assassin's Creed Black Flag, and well, mostly Black Flag, Burnout Paradise also had a remastered version. Just like with most of the games I own, I started with the pirated version to know if owning the game was worth it. So, I downloaded a repack from FitGirl.It crashed.Not dramatically. There was no blue screen, no smoke, no heroic final whine from the fans. The game would load, I would get into the city, I would take a corner at a hundred and forty and introduce myself to the side of a bus, and then the window would simply cease to be. Windows would put up its small apologetic box, in the tone of a butler informing you that your house has burned down but that he has taken the liberty of laying out breakfast. I would relaunch. Ninety seconds later, the same performance.My first assumption was the obvious one, and I suspect it was yours as well. Pirated build. Bad repack. Somebody's compression had gone off in the night. This is precisely the situation that buying the game is invented to resolve.So I bought it. Twenty dollars, Steam, official, verified files, clean install, the full sacrament.It crashed at the same address.There is something almost admirable about a bug that survives a purchase. It has principles. It cannot be bribed. Twenty dollars is a very small sum to pay for the discovery that you were confidently wrong about something, and I have certainly paid a great deal more for a great deal less.
Two Registers That Should Have Been The Same Thing
Windows keeps crash dumps, and almost nobody opens them, which is a pity, because a dump is not a diagnostic tool so much as a photograph. It is the machine at the exact instant of its death, pockets still full, and the first thing you want out of its pockets is the registers.
Exception 0xC0000005 (access violation, read)
Faulting module BurnoutPR.exe (itself, not a driver, not a DLL)
Fault offset 0x0038c094
Faulting address 0xFF34F984
Edx = 0xFF34F980 <- dereferenced, faulted
Edi = 0x0134F980 <- same pointer, intact
Sit with those last two lines a moment. Edi holds 0x0134F980. Edx holds 0xFF34F980. They are the same pointer, except that one of them has had its top byte painted over. 0x01 has become 0xFF.The program asked the operating system what lived at 0xFF34F984. The operating system replied that nothing lived there, that nothing had ever lived there, and that it would be terminating the conversation and the program together.A second dump, from a different session on a different day, showed exactly the same thing at a different address: Edx = 0xFF34F590 against Ebx = 0x0134F580. Same corruption, same single byte, same 0xFF.That is a very good clue, and it is good precisely because of what it is not. A wild pointer, the sort you get from reading a variable before you have troubled to set it, looks like refuse. This did not look like refuse. This looked like an entirely correct pointer that somebody had vandalised with great precision, in one place, twice. And the timing carried its own signature: the game expired anywhere between 1.2 seconds and 1431 seconds after launch, which is the calling card of two threads racing rather than one program being reliably stupid.A 2008 Xbox 360 port, written for a console with six hardware threads, running on a twelve-thread Ryzen. That was the shape of the thing. What I did not have was the mechanism, and the mechanism cost me two days.I should say plainly how the work was done, since it matters later. I did not disassemble anything. I ran this the way I run most technical work now, which is to say sitting with Claude, deciding what to test next, arguing about what the evidence meant, and delegating the reading of bytes to something with more patience for bytes than I have. The mistakes in this post are therefore mostly ours rather than mine, and one or two of them are the particular sort you only make when your collaborator is fast enough to let you be careless.
Seventeen Things It Was Not
Most of debugging is subtraction, and subtraction is as tedious to read as it is to perform, so allow me to compress two days into a paragraph.I reinstalled. I verified. I installed the game a second time, elsewhere, out of spite. I wiped saves, config, the GameData folder and the shader cache. I found and removed a genuinely disgraceful compatibility shim that Windows had silently strapped to the executable, 640X480 WIN7RTM DISABLEDXMAXIMIZEDWINDOWEDMODE, which is not a thing that should be anywhere near a 2018 remaster. I forced windowed mode. I forced 1080p. I forced 1440p. I confined the game to two cores. I gave it all twelve. I renamed the Steam overlay DLL out of existence and then went into the crash dump's module list to confirm that it had taken the hint. I checked for NVIDIA overlays, Discord overlays, EA overlays, RivaTuner, Afterburner and OBS. I checked the C++ redistributables, which is the technical equivalent of looking under the sofa.The community's canonical fix list for this game (unplug your webcam, kill your virtual camera, install Bo98's Core-Bugfixes mod, disable the EA overlay) was either already satisfied or did not apply. There are no cameras attached to this machine. There is one GPU. The mod was already installed.Every one of those was a dead end, and their collective deadness was the most useful result of the first day. This was not one of the documented crashes. Everybody else's Burnout crash has a known cause and a known cure. Mine had neither, which is either very bad news or a compliment, depending on temperament.I should confess to a rot in that list, because I only noticed it afterwards.My elimination table had a line declaring the Core-Bugfixes mod innocent. What I had actually established was that the mod loaded; its loader appears in ten dumps out of ten. I had confirmed that a suspect was present at the scene and then released him without questioning. The mod was never once removed. Two days later I found a forum comment stating flatly that installing it causes crashes on some machines, and realised my table had been lying to me with a straight face the entire time.
An elimination table is only ever as honest as its weakest row.
That row should have read untested, in capitals, for forty-eight hours.
Three Occasions On Which I Was Confidently Wrong
The dead ends cost time, which is cheap. These cost credibility, which is not, and they are the portion of this story I would most like a younger version of myself to read while wincing.The shim. Discovering that compatibility layer felt like the end of the investigation. It was a genuine offence. Removing it was unambiguously correct. And the very next run crashed at precisely the same offset. I had come about a murder and found a parking violation, and for about four minutes I could not tell the difference, because finding a problem and finding the problem produce an identical sensation and only one of them is any use.The p-value that meant nothing at all. At one point a CPU affinity change produced a statistically significant improvement against a baseline of a hundred and twenty-two runs. Significant! p = 0.86%! There was, admittedly, the small matter that the same session had also changed the resolution, the quality settings and the display mode. Four things moved and I handed the medal to whichever one I had been thinking about. The arithmetic was impeccable and the conclusion was fiction. It is entirely possible to be rigorous and wrong simultaneously, and that combination is far more dangerous than being merely wrong, because it arrives with a certificate.The windowed-mode breakthrough. One run in windowed mode lasted 539 seconds against a baseline maximum of 330, which looked very much like an answer. It had two holes in it. The first was that the run had also had restricted CPU affinity, so I was back to crediting the wrong variable. The second is funnier and more educational: the game was quietly rewriting WindowMode=1 back to 0 in its own configuration file every time it closed. Every subsequent "windowed" test was not windowed. The game had been marking my homework, disagreeing with it, and correcting it after I left the room.Making config.ini read-only settled that argument, and once the setting was actually enforced, windowed mode produced entirely unremarkable runs.What made all three possible is the same thing. The baseline distribution is enormous: survival times spanning 1.2 seconds to 330, mean 89.7, median 72.2. When your data covers three orders of magnitude, three runs will cheerfully support whatever you walked in believing. I executed two of my own significant findings in a single afternoon, and I only managed it because I had the poor manners to look at them twice.
The Executable Is Encrypted, Which Was Rude
At some point one stops speculating and goes to read the offending instruction. I had the offset, BurnoutPR.exe+0x38C094. All that remains is to map that address back to a position in the file on disk and look at what is written there.
--- 48 bytes at the fault site, on disk ---
45 1A 7B A1 7D 0F 1B 58 2D 20 BC 56 28 C6 80 15
E6 09 AC 04 80 4F BD E4 CB F8 6F 02 FA 99 B2 79
65 DB 63 A7 7D 49 FB BA 3A 54 2E 0F 9E F7 8D 8A
That is not x86. That is not anything. That is the sound of a filing cabinet being pushed down a stairwell.So rather than have an opinion about it, we measured it.
Denuvo string hits in the exe 20
Shannon entropy of .code 7.998 bits/byte (normal x86 is around 6.0 to 6.5)
Zero bytes in first 200KB 0.37% (real code is full of them)
Section layout .code .link .sdata .trace .ooa
.trace 113 MB of a 130 MB binary
Entropy of 7.998 out of a possible 8.0 means the bytes are statistically indistinguishable from noise. Compiled code is a tremendously repetitive thing, full of zeroes and habits and small nervous tics; this had almost none. The game is wrapped in Denuvo, and the whole code section sits encrypted on disk, becoming instructions only in memory, at runtime, after the protection has consented to unwrap it.Which is a peculiar place to arrive. The crash occurs at a stable, reproducible offset, the same one in a hundred and forty-five consecutive runs, inside code that cannot be looked at. I had a body, a time of death and an address, and the address was in a building with no doors.It did answer one question I had been carrying since the beginning. Was the copy protection itself the villain? No. I had already run that experiment by accident on day one, when the repack, which has the protection removed entirely, crashed at exactly the same fault offset. Denuvo was not corrupting the pointer. Whatever was broken belonged to Criterion, and had been there long before anyone thought to encrypt it.
The Run That Lasted Twenty-Four Minutes
Everything clustered. Seventy seconds, ninety seconds, a hundred and ten. And then one run went for 1431 seconds and I sat up like a spaniel.
baseline median (n=122) 72.2s
baseline mean 89.7s
music volume 0 98.3s
OS audio muted 105.3s
2 cores + windowed 160.0s
baseline maximum 330.4s
the unexplained run 1431.3s
I remembered something about that run. The music had not loaded. The game had been silent apart from engine noise, and I had thought at the time only that it was a shame.That felt enormous. Burnout is a game built around its soundtrack, streaming audio continuously on background threads while everything else happens. A crash that vanishes when the audio streaming stops is a crash that lives in the audio streaming. I was thoroughly convinced, and I went looking for confirmation, which is the point in any investigation at which you should start being frightened of yourself.Here is the worst mistake of the two days.To test "the music did not load", I moved seventy-three soundtrack files out of the game directory. The game crashed thirty seconds later at 0x004dc903, an address I had never seen, before the menu had even appeared.I had not reproduced anything. I had commissioned a new bug, at some expense, by removing files from a program that had every reason to expect them. "The music failed to load" and "the music files are absent" are two entirely different states of the world, and I had substituted the one that was convenient to arrange for the one I had actually witnessed. Restoring the files put the crash obediently back at 0x0038c094.The theory died properly a little later, and it died on the stack. Crash dumps are seventeen megabytes and contain the memory of the thread that fell over, which means you can walk backwards through its return addresses and see what it had been doing with its life. Two things fell out. The stack bottoms out in kernel32!BaseThreadInitThunk, which confirms the crash happens on a spawned worker thread rather than the main one. And there is no dsound.dll, no AudioSes.dll, no audio module anywhere on it whatsoever.I had spent an afternoon building an elegant and detailed case against a suspect who had an alibi, was not in the country, and did not exist.
The Instrument Kept Breaking
By the middle of the second day the bottleneck was no longer the game but my record-keeping. I was reconstructing each run's conditions afterwards, from memory and file timestamps, which is exactly how the confounded results had happened in the first place.So we wrote a harness that records every variable at launch, before the run begins, and logs what actually survived.
Two of those columns contain the whole lesson. mask_req is the CPU affinity I requested. mask_actual is the affinity the process turned out to have. Logging both caught a run in which the affinity call had failed silently and which I would otherwise have filed away as perfectly good evidence.And then the harness developed a bug of its own, because of course it did. It attached itself to the first process named BurnoutPR.exe that it saw, which is a short-lived launcher shim rather than the game, and dutifully recorded a 6.3-second "clean exit" that had never occurred. It had to be taught to wait for a process that sticks around and has real threads in it, which is more or less what one has to teach a junior employee.Twice in one day the measuring instrument was the broken thing. If you take one practical item from all of this, let it be that you should log what actually happened rather than what you asked for. They part company far more often than is comfortable.
The Part That Actually Cost Something
Steam will refund a game you have played for under two hours, on the charitable assumption that nobody could form a settled opinion in less time. I had formed several.When the harness began running, the account showed sixty-nine minutes. By the time anybody thought to look at that number again it showed a hundred and seventeen. Forty-eight minutes of refund eligibility had been fed into automated crash runs, leaving three minutes on the clock, and I had not glanced at the counter once.Every launch was being treated as free, on the grounds that CPU time is free and my time was already spent. It was not free. There was a clock running on a resource that had nothing whatever to do with the machine, and an automated loop consumed nearly all of it while both of us were absorbed in hexadecimal.That is the part I keep returning to. The wrong theories were not really losses; they were the work. But when you hand a task to something that can run experiments faster than you can supervise them, the cost that eventually finds you is never the one you are watching. It is the one with a deadline attached that nobody thought to put in the log file.
Five Instructions
The breakthrough arrived sideways, as they tend to.The DRM-free build of the same executable has its code section unencrypted. Same 130 MB binary, same sections, the same twenty Denuvo strings still loitering inside it, and an entropy of 3.521 rather than 7.998. Which meant that the bytes at the fault site could at last be read as what they were.
0078C08E 0F 7E E2 movd edx, mm4 ; pull an index out of an MMX register
0078C091 03 D7 add edx, edi ; edx = base + index
0078C094 8B 42 04 mov eax, [edx+4] ; <<< FAULT
0078C097 8B 12 mov edx, [edx]
0078C099 0F 73 D4 20 psrlq mm4, 32 ; shift down, fetch the next index
Five instructions. A hand-optimised loop walking a list of pointers, keeping a pair of array indices packed together inside an MMX register called mm4, pulling one out, adding it to a base pointer in edi, dereferencing the result, then shifting the register down thirty-two bits and doing the whole thing again for the second index. It is a lovely piece of work. Somebody was proud of it in 2008, and they were entitled to be.And now the arithmetic closes, because I have the corrupt value, the clean base pointer, and the knowledge that the instruction just before this block scales the index by four.
0xFF34F980 - 0x0134F980 = 0xFE000000 the corruption, isolated
0xFE000000 >> 2 = 0x3F800000 the scaling, undone
= 1.0f
0x3F800000 is the bit pattern of the floating-point number 1.0. The second crash resolves the same way, to 0x3F800004.The register that was meant to be holding a modest integer index into an array was holding the number one, as a float. And that is the entire bug.
What Intel Did In 1997
MMX was Intel's first attempt at SIMD, and it required eight new registers. Adding eight new registers to x86 is not a modest undertaking, because every operating system in existence contains code that saves and restores the processor's state when it switches between programs. New registers mean every one of those operating systems must be rewritten before the feature can be used at all, and there is no polite way to ask the world to do that.So Intel declined to add any.MM0 through MM7 are not registers. They are the mantissa fields of the eight x87 floating-point registers that already existed, ST(0) through ST(7), wearing a false moustache. An operating system that had never heard of MMX would save and restore MMX state flawlessly, because as far as it was concerned it was saving floating-point state, which it had been doing for years.It is a beautiful solution to a real problem, and like most beautiful solutions to real problems it billed its costs to a future that had not yet arrived. An x87 register is eighty bits wide: one sign bit, fifteen exponent bits, and sixty-four bits of mantissa. MMX writes only the mantissa. So whenever an MMX instruction touches one of these registers, the hardware forces all sixteen of the remaining bits to 1, so that anything reading the register as a floating-point number sees a NaN and knows to be suspicious.Those forced bits are my corruption. 0xFF sitting in the top byte of a pointer is not a buffer overrun and not a race on a shared variable. It is an architectural decision from 1997 doing exactly and only what it was designed to do, surfacing in a routine that carries integer indices in MMX while doing floating-point mathematics all around them, at the moment when floating-point state and integer state find themselves sharing a register file.I find this genuinely delightful. Nobody wrote this bug. Criterion wrote a fast loop. Intel made a compatibility decision while Criterion's developers were still doing their homework. The bug lives in the seam between the two, and it lay there in perfect silence for seventeen years, waiting for the world to change around it.
So why now, and not in 2008?
Three things moved underneath a binary that did not move at all.The instruction that preserves processor state during a context switch changed. FXSAVE became XSAVE, which is cleverer, and which may skip or reinitialise portions of that state depending on various flags, one of which is the FPU tag word, which MMX obligingly marks as "all valid".Six hardware threads became twelve. Every time the operating system migrates a thread from one core to another, the entire floating-point state is saved off one physical register file and restored onto a different one.And scheduling grew denser. More interrupts, more preemption, more opportunities for a context switch to land inside the ten-instruction window between writing that register and reading it back.
The engine did not rot. The window simply started getting hit.
That sentence is the reason I wanted to write any of this down. We speak of old software "breaking" as though the code were fruit, but the bytes on that disk are identical to the bytes Criterion shipped. Every single thing that changed, changed underneath them. The program is standing precisely where it was left. It is the floor that has been moving.I wrote something adjacent to this a few weeks ago about how errors lie about their own location, and this is the same idea one storey further down. The access violation insists that the pointer is bad. The pointer is impeccable. The pointer is downstream of a register, which is downstream of a decision made before the developers could drive, and not one of those floors will ever appear in an error message. The message is a witness, and witnesses do not lie so much as report, with great sincerity, where they happened to be standing when the building came down.
The Fix Is One Core
Here is what makes the ending satisfying, and I say that as someone who spent two days earning it.Everything up to this point had been subtraction: a hundred and forty runs of ruling things out, three false positives, and one entire afternoon prosecuting an innocent subsystem. But the disassembly does not merely explain. It predicts. If the poisoning occurs because floating-point state is being moved between physical register files, then a process that never migrates should never be poisoned. That is a claim, it is falsifiable, and it costs one run.
0xFFF, all 12 logical CPUs 99.0s
0xFFF, repeat 107.7s
0x5, CPUs 0+2, two physical cores 160.0s
0x1, CPU 0 only, no migration 2386.8s
Thirty-nine minutes. Clean exit. No crash, no dump, no access violation, no apologetic box. The first run in roughly a hundred and forty-five attempts that ended because I decided it should.I will state the caveat plainly rather than hide it behind the good news, since hiding things behind good news is the exact error I committed three times already. Running on a single logical CPU also ruins the framerate, which means the loop executes rather fewer times per second. "Never migrates" and "does less work" are both entirely consistent with this result, and I cannot yet separate them.The experiment that separates them is a pretty one. Affinity mask 0x3 is CPUs 0 and 1, which on a Ryzen are SMT siblings: two threads sharing a single physical core. Two threads restores most of the framerate. One physical core means the floating-point state never crosses register files. If 0x3 survives as 0x1 did, migration is the mechanism and the framerate objection is dead. If 0x3 dies at around a hundred and sixty seconds as 0x5 did, then migration is not it at all; it is any context switch whatsoever landing in that ten-instruction window, and single-core was merely buying time by running the loop less often.Three runs per mask at absolute minimum, and I would want considerably more. Against a baseline spanning 1.2 seconds to 330, anything less will politely confirm whichever story I have grown fond of. I have been on the wrong end of that twice this week already.
What I Am Taking Away From It
The proper fix is not affinity at all. It is a binary patch: a guard on the value before it is used as an index, checking for a high byte that a legitimate index can never possess. Somebody in the modding community will write it eventually and it will be four instructions long, which is the usual ratio between the size of a fix and the size of the investigation that produced it. Until then the choice is one core or an unplayable game, and those are the only two honest options on the table.The workaround is not the part I keep turning over, though.I bought a racing game in order to relax between jobs. I received instead a two-day archaeological dig terminating in a compatibility decision from 1997, and the answer had been sitting in a register the entire time, wearing the number one as a disguise.The lessons that generalise, in roughly the order I paid for them:Enforce the variable you claim to be testing. The game rewrote its own configuration on exit and an entire hypothesis was tested against a setting that had already reverted. Making one file read-only was the difference between measuring something and measuring nothing at all.A wide baseline devours small samples. Two significant-looking results died in a single session, and both had felt like the end of the investigation on arrival.Verify the absence, not the setting. The Steam overlay was disabled in Steam's own configuration and injecting into every single run regardless. The checkbox governs whether it draws, not whether it turns up. Only the module list would settle it.Reproduce the report, not your paraphrase of it. "The music did not load" became "delete the music files" and produced a fresh crash at a fresh address. The observation was precise. My restatement of it was merely convenient.Name the variable, not your summary of it. For two days my notes said "2 cores". 0x5 and 0x3 are both two cores and they test opposite hypotheses. A label that quietly collapses the thing you are manipulating will cost you a day, and it will do so without ever raising its voice.Watch the clock that is not the CPU. Refund windows, trial periods, warranties. Automated loops consume them in perfect silence and nobody notices until it is very nearly too late.That last one is the one I would underline twice. Every other mistake in this post cost time, and time spent on a bug this good is not really lost. That one cost money, and it was entirely recoverable right up until the moment it was not.
