Ever since I decided to take a break from employment, I started playing games I had wanted to play for a long time. I had built a PC with my adult money (thankfully before the RAM crisis) but I had seldom played many games on it. That machine has gone through a lot. It is my experimental box: the place I run hypervisors, the place I banged my head against the table installing Arch (I use Arch, by the way), and the place where anything and everything experimental under the sun gets tried first. If something on it is going to misbehave, that is where I want it to misbehave. And in doing so, I ended up trying to play Burnout Paradise. Just like with many of the games I played in the past like Assassin's Creed Black Flag, and well, mostly Black Flag, Burnout Paradise also had a remastered version. Just like with most of the games I own, I started with the pirated version to know if owning the game was worth it. So, I downloaded a repack from FitGirl.It crashed.Not dramatically. There was no blue screen of death, no smoke, no traces of what went wrong. The game would load, I would get into the city, take a corner at a hundred and forty and introduce myself to the side of a bus, and then the window would simply cease to be. No this, no that, no mention of what happened. No logs, no error box. Just a black screen back to desktop. I would relaunch. Ninety seconds later, the same pattern repeated.My first assumption was the obvious one, and I suspect it was yours as well. Pirated build. Bad repack. Somebody's compression had gone off in the night. This is precisely the situation that buying the game is invented to resolve.So I bought it. Twenty dollars, Steam, official, verified files, clean install, the full sacrament.It crashed at the same time. Around the 90 seconds mark. At this point, I had started keeping a log, cause this was the only thing I had going on.I had tried all the bug fixes from Reddit and other forums. All of them had various reasons but all of them did not solve the problem for me nor for many of the other members out there. This had me intrigued. My interest piqued. And my gears grinding. I had to figure this out.
Seventeen Suspects
Most of debugging is subtraction, and subtraction is as tedious to read as it is to perform, so allow me to compress the first day into a paragraph.I reinstalled. I verified. I installed the game a second time, elsewhere, out of spite. I wiped saves, config, the GameData folder and the shader cache. We found and removed a genuinely disgraceful compatibility shim that Windows had silently strapped to the executable, 640X480 WIN7RTM DISABLEDXMAXIMIZEDWINDOWEDMODE, which is not a thing that should be anywhere near a 2018 remaster. I forced windowed mode. I forced 1080p. I forced 1440p. I confined the game to two cores. I gave it all twelve. I renamed the Steam overlay DLL out of existence and later made sure it had actually taken the hint. I checked for NVIDIA overlays, Discord overlays, EA overlays, RivaTuner, Afterburner and OBS. I checked the C++ redistributables, which is the technical equivalent of looking under the sofa.The community's canonical fix list for this game (unplug your webcam, kill your virtual camera, install Bo98's Core-Bugfixes mod, disable the EA overlay) was either already satisfied or did not apply. There are no cameras attached to this machine. There is one GPU. The mod was already installed.Every one of those was a dead end, and their collective deadness was the most useful result of the first day. This was not one of the documented crashes. Everybody else's Burnout crash has a known cause and a known cure. Mine had neither. I closed the forum tabs. Nobody on them had been where I was going.I should confess to a rot in that list, because I only noticed it afterwards.My elimination table had a line declaring the Core-Bugfixes mod innocent. What I had actually established was that the mod loaded; its loader turned up in every crash dump I later opened, ten out of ten. I had confirmed that a suspect was present at the scene and then released him without questioning. The mod was never once removed. Two days later I found a forum comment stating flatly that installing it causes crashes on some machines, and realised my table had been lying to me with a straight face the entire time.
An elimination table is only ever as honest as its weakest row.
That row should have read untested, in capitals, for forty-eight hours.That was where the forums ran out. So I opened Claude Code.I should say plainly how the work was done from here, since it matters later. I did not disassemble anything. I ran this the way I run most technical work now, which is to say sitting with Claude, deciding what to test next, arguing about what the evidence meant, and delegating the reading of bytes to something with more patience for bytes than I have. The mistakes in this post are therefore mostly ours rather than mine, and one or two of them are the particular sort you only make when your collaborator is fast enough to let you be careless.
Rigorous and Wrong
The dead ends cost time, which is cheap. These cost credibility, which is not, and they are the portion of this story I would most like a younger version of myself to read while wincing. Not understanding the machinery was never the problem. Believing I understood the result was.The shim. Discovering that compatibility layer felt like the end of the investigation. It was a genuine offence. Removing it was unambiguously correct. And the very next run crashed at precisely the same offset. I had come about a murder and found a parking violation, and for about four minutes I could not tell the difference, because finding a problem and finding the problem produce an identical sensation and only one of them is any use.The p-value that meant nothing at all. At one point a CPU affinity change produced a statistically significant improvement against a baseline of a hundred and twenty-two runs. Significant! p = 0.86%! There was, admittedly, the small matter that the same session had also changed the resolution, the quality settings and the display mode. Four things moved and I handed the medal to whichever one I had been thinking about. The arithmetic was impeccable and the conclusion was fiction. It is entirely possible to be rigorous and wrong simultaneously, and that combination is far more dangerous than being merely wrong, because it arrives with a certificate.The windowed-mode breakthrough. One run in windowed mode lasted 539 seconds against a baseline maximum of 330, which looked very much like an answer. It had two holes in it. The first was that the run had also had restricted CPU affinity, so I was back to crediting the wrong variable. The second is funnier and more educational: the game was quietly rewriting WindowMode=1 back to 0 in its own configuration file every time it closed. Every subsequent "windowed" test was not windowed. The game had been marking my homework, disagreeing with it, and correcting it after I left the room.Making config.ini read-only settled that argument, and once the setting was actually enforced, windowed mode produced entirely unremarkable runs.What made all three possible is the same thing. The baseline distribution is enormous: survival times spanning 1.2 seconds to 330, mean 89.7, median 72.2. When your data covers three orders of magnitude, three runs will cheerfully support whatever you walked in believing. I executed two of my own significant findings in a single afternoon, and I only managed it because I had the poor manners to look at them twice.The instrument. By the middle of the second day the bottleneck was no longer the game but my record-keeping. I was reconstructing each run's conditions afterwards, from memory and file timestamps, which is exactly how the confounded results had happened in the first place. So we built a harness that records every variable at launch, before the run begins, and logs what actually survived.
Two of those columns contain the whole lesson. mask_req is the CPU affinity I requested. mask_actual is the affinity the process turned out to have. Logging both caught a run in which the affinity call had failed silently and which I would otherwise have filed away as perfectly good evidence.And then the harness developed a bug of its own, because of course it did. It attached itself to the first process named BurnoutPR.exe that it saw, which is a short-lived launcher shim rather than the game, and dutifully recorded a 6.3-second "clean exit" that had never occurred. It had to be taught to wait for a process that sticks around and has real threads in it, which is more or less what one has to teach a junior employee.Twice in one day the measuring instrument was the broken thing. If you take one practical item from all of this, let it be that you should log what actually happened rather than what you asked for. They part company far more often than is comfortable.The clock nobody was watching. Steam will refund a game you have played for under two hours, on the charitable assumption that nobody could form a settled opinion in less time. I had formed several.When the harness began running, the account showed sixty-nine minutes. By the time anybody thought to look at that number again it showed a hundred and seventeen. Forty-eight minutes of refund eligibility had been fed into automated crash runs, leaving three minutes on the clock, and I had not glanced at the counter once.Every launch was being treated as free, on the grounds that CPU time is free and my time was already spent. It was not free. There was a clock running on a resource that had nothing whatever to do with the machine, and an automated loop consumed nearly all of it while both of us were absorbed in hexadecimal. When you hand a task to something that can run experiments faster than you can supervise them, the cost that eventually finds you is never the one you are watching.
Twenty-Four Minutes of Silence
Everything clustered. Seventy seconds, ninety seconds, a hundred and ten. And then one run went for 1431 seconds and I sat up like a spaniel.
baseline median (n=122) 72.2s
baseline mean 89.7s
music volume 0 98.3s
OS audio muted 105.3s
2 cores + windowed 160.0s
baseline maximum 330.4s
the unexplained run 1431.3s
I remembered something about that run. The music had not loaded. The game had been silent apart from engine noise, and I had thought at the time only that it was a shame.Then it landed. If the game could survive twenty-four minutes with the sound off, then something that happened while the sound was on was killing it. For two days I had been poking the game from the outside. This was the first time it had told me something about itself.Archimedes, the story goes, lowered himself into the bath, watched the water climb over the rim, understood that a body displaces its own volume, and ran through Syracuse naked shouting eureka. I kept my clothes on. Only just.Burnout is a game built around its soundtrack, streaming audio continuously on background threads while everything else happens. A crash that vanishes when the audio streaming stops is a crash that lives in the audio streaming. I was thoroughly convinced, and I went looking for confirmation, which is the point in any investigation at which you should start being frightened of yourself.Here is the worst mistake of the two days.To test "the music did not load", I moved seventy-three soundtrack files out of the game directory. The game crashed thirty seconds later at 0x004dc903, an address I had never seen, before the menu had even appeared.I had not reproduced anything. I had commissioned a new bug, at some expense, by removing files from a program that had every reason to expect them. "The music failed to load" and "the music files are absent" are two entirely different states of the world, and I had substituted the one that was convenient to arrange for the one I had actually witnessed. Restoring the files put the crash obediently back at 0x0038c094.
Debugging Where I Know Nothing
The silence had to be pointing at something, and the only place left to look was the crash itself.Windows keeps crash dumps, and almost nobody opens them, which is a pity, because a dump is not a diagnostic tool so much as a photograph. It is the machine at the exact instant of its death, pockets still full, and the first thing you want out of its pockets is the registers. It would have been a good thing if I understood what it meant. I couldn't in the past. But in the modern world of fables and mythos, I found my Opus to help me.
Exception 0xC0000005 (access violation, read)
Faulting module BurnoutPR.exe (itself, not a driver, not a DLL)
Fault offset 0x0038c094
Faulting address 0xFF34F984
Edx = 0xFF34F980 <- dereferenced, faulted
Edi = 0x0134F980 <- same pointer, intact
I know, right. I don't understand it either.The program asked the operating system what lived at 0xFF34F984. The operating system replied that nothing lived there, that nothing had ever lived there, and that it would be terminating the conversation and the program together.A second dump, from a different session on a different day, showed exactly the same thing at a different address: Edx = 0xFF34F590 against Ebx = 0x0134F580. Same corruption, same single byte, same 0xFF.That is a very good clue, and it is good precisely because of what it is not. A wild pointer, the sort you get from reading a variable before you have troubled to set it, looks like refuse. This did not look like refuse. This looked like an entirely correct pointer that somebody had vandalised with great precision, in one place, twice. And the timing carried its own signature: the game expired anywhere between 1.2 seconds and 1431 seconds after launch, which is the calling card of two threads racing rather than one program being reliably stupid.A 2008 Xbox 360 port, written for a console with six hardware threads, running on a twelve-thread Ryzen. That was the shape of the thing. What I did not have was the mechanism, and the mechanism cost me two days.The audio theory died here, and it died on the stack. Crash dumps are seventeen megabytes and contain the memory of the thread that fell over, which means you can walk backwards through its return addresses and see what it had been doing with its life. Two things fell out. The stack bottoms out in kernel32!BaseThreadInitThunk, which confirms the crash happens on a spawned worker thread rather than the main one. And there is no dsound.dll, no AudioSes.dll, no audio module anywhere on it whatsoever.I had spent an afternoon building an elegant and detailed case against a suspect who had an alibi, was not in the country, and did not exist.And yet the water had still come over the rim.A theory, a run, a dead theory, another theory, the same crash at the same address. For a day and a half every suspect had lived on the outside of the game: the overlays, the resolution, the shim, the affinity. Twenty-four minutes of engine noise was the first thing that said anything about the inside. Something running in the background was killing it. "Audio" was the wrong name for that something. The question underneath it was right.It was chasing the wrong answer that made us read the stack in the first place, and it was the stack that said the crash lived on a spawned worker thread rather than the main one. I didn't know what a worker thread was in any useful sense. I knew what the data was pointing at, and that was enough to choose the next step. From there the road ran straight: worker threads, a twelve-thread CPU, an engine written for six, and eventually a register that kept being moved from one core to another. In hindsight, my best reading of that run is that a quieter game simply gave the scheduler fewer reasons to shuffle threads around, which is the same mechanism the fix exploits from the other side. I can't prove that from one run. But it is the run that turned me around, and I would not have found the register without it.
A Building With No Doors
At some point one stops speculating and goes to read the offending instruction. We had the offset, BurnoutPR.exe+0x38C094. All that remained was to map that address back to a position in the file on disk and look at what is written there.
--- 48 bytes at the fault site, on disk ---
45 1A 7B A1 7D 0F 1B 58 2D 20 BC 56 28 C6 80 15
E6 09 AC 04 80 4F BD E4 CB F8 6F 02 FA 99 B2 79
65 DB 63 A7 7D 49 FB BA 3A 54 2E 0F 9E F7 8D 8A
That, Claude told me, is not x86. That is not anything. That is the sound of a filing cabinet being pushed down a stairwell.So rather than have an opinion about it, we measured it.
Denuvo string hits in the exe 20
Shannon entropy of .code 7.998 bits/byte (normal x86 is around 6.0 to 6.5)
Zero bytes in first 200KB 0.37% (real code is full of them)
Section layout .code .link .sdata .trace .ooa
.trace 113 MB of a 130 MB binary
I didn't need to know how to calculate entropy. I needed to know what the number meant, and the mentor's job was to tell me. 7.998 out of a possible 8.0 means the bytes are statistically indistinguishable from noise. Compiled code is a tremendously repetitive thing, full of zeroes and habits and small nervous tics; this had almost none. The game is wrapped in Denuvo, and the whole code section sits encrypted on disk, becoming instructions only in memory, at runtime, after the protection has consented to unwrap it.Which is a peculiar place to arrive. The crash occurs at a stable, reproducible offset, the same one in a hundred and forty-five consecutive runs, inside code that cannot be looked at. I had a body, a time of death and an address, and the address was in a building with no doors.It did answer one question I had been carrying since the beginning. Was the copy protection itself the villain? No. I had already run that experiment by accident on day one, when the repack, which has the protection removed entirely, crashed at exactly the same fault offset. Denuvo was not corrupting the pointer. Whatever was broken belonged to Criterion, and had been there long before anyone thought to encrypt it.
Five Instructions
The breakthrough arrived sideways, as they tend to.The DRM-free build of the same executable has its code section unencrypted. Same 130 MB binary, same sections, the same twenty Denuvo strings still loitering inside it, and an entropy of 3.521 rather than 7.998. Which meant that the bytes at the fault site could at last be read as what they were.
0078C08E 0F 7E E2 movd edx, mm4 ; pull an index out of an MMX register
0078C091 03 D7 add edx, edi ; edx = base + index
0078C094 8B 42 04 mov eax, [edx+4] ; <<< FAULT
0078C097 8B 12 mov edx, [edx]
0078C099 0F 73 D4 20 psrlq mm4, 32 ; shift down, fetch the next index
Five instructions. Claude walked me through them one at a time: a hand-optimised loop walking a list of pointers, keeping a pair of array indices packed together inside an MMX register called mm4, pulling one out, adding it to a base pointer in edi, dereferencing the result, then shifting the register down thirty-two bits and doing the whole thing again for the second index. It is a lovely piece of work. Somebody was proud of it in 2008, and they were entitled to be.And now the arithmetic closes, because we had the corrupt value, the clean base pointer, and the knowledge that the instruction just before this block scales the index by four.
0xFF34F980 - 0x0134F980 = 0xFE000000 the corruption, isolated
0xFE000000 >> 2 = 0x3F800000 the scaling, undone
= 1.0f
0x3F800000 is the bit pattern of the floating-point number 1.0. The second crash resolves the same way, to 0x3F800004.The register that was meant to be holding a modest integer index into an array was holding the number one, as a float. And that is the entire bug. I could not have done that subtraction unprompted. I could check it, though, and I did, and it came out the same both times.
1997
MMX was Intel's first attempt at SIMD, and it required eight new registers. Adding eight new registers to x86 is not a modest undertaking, because every operating system in existence contains code that saves and restores the processor's state when it switches between programs. New registers mean every one of those operating systems must be rewritten before the feature can be used at all, and there is no polite way to ask the world to do that.So Intel declined to add any.MM0 through MM7 are not registers. They are the mantissa fields of the eight x87 floating-point registers that already existed, ST(0) through ST(7), wearing a false moustache. An operating system that had never heard of MMX would save and restore MMX state flawlessly, because as far as it was concerned it was saving floating-point state, which it had been doing for years.It is a beautiful solution to a real problem, and like most beautiful solutions to real problems it billed its costs to a future that had not yet arrived. An x87 register is eighty bits wide: one sign bit, fifteen exponent bits, and sixty-four bits of mantissa. MMX writes only the mantissa. So whenever an MMX instruction touches one of these registers, the hardware forces all sixteen of the remaining bits to 1, so that anything reading the register as a floating-point number sees a NaN and knows to be suspicious.Those forced bits are my corruption. 0xFF sitting in the top byte of a pointer is not a buffer overrun and not a race on a shared variable. It is an architectural decision from 1997 doing exactly and only what it was designed to do, surfacing in a routine that carries integer indices in MMX while doing floating-point mathematics all around them, at the moment when floating-point state and integer state find themselves sharing a register file.I find this genuinely delightful. Nobody wrote this bug. Criterion wrote a fast loop. Intel made a compatibility decision while Criterion's developers were still doing their homework. The bug lives in the seam between the two, and it lay there in perfect silence for seventeen years, waiting for the world to change around it.
The Floor Moved
Three things moved underneath a binary that did not move at all.The instruction that preserves processor state during a context switch changed. FXSAVE became XSAVE, which is cleverer, and which may skip or reinitialise portions of that state depending on various flags, one of which is the FPU tag word, which MMX obligingly marks as "all valid".Six hardware threads became twelve. Every time the operating system migrates a thread from one core to another, the entire floating-point state is saved off one physical register file and restored onto a different one.And scheduling grew denser. More interrupts, more preemption, more opportunities for a context switch to land inside the ten-instruction window between writing that register and reading it back.
The engine did not rot. The window simply started getting hit.
That sentence is the reason I wanted to write any of this down. We speak of old software "breaking" as though the code were fruit, but the bytes on that disk are identical to the bytes Criterion shipped. Every single thing that changed, changed underneath them. The program is standing precisely where it was left. It is the floor that has been moving.I wrote something adjacent to this a few weeks ago about how errors lie about their own location, and this is the same idea one storey further down. The access violation insists that the pointer is bad. The pointer is impeccable. The pointer is downstream of a register, which is downstream of a decision made before the developers could drive, and not one of those floors will ever appear in an error message. The message is a witness, and witnesses do not lie so much as report, with great sincerity, where they happened to be standing when the building came down.
One Core
Everything up to this point had been subtraction: a hundred and forty runs of ruling things out, three false positives, and one entire afternoon prosecuting an innocent subsystem. But the disassembly does not merely explain. It predicts. If the poisoning occurs because floating-point state is being moved between physical register files, then a process that never migrates should never be poisoned. That is a claim, it is falsifiable, and it costs one run.
0xFFF, all 12 logical CPUs 99.0s
0xFFF, repeat 107.7s
0x5, CPUs 0+2, two physical cores 160.0s
0x1, CPU 0 only, no migration 2386.8s
Thirty-nine minutes. Clean exit. No black screen. No desktop. The first run in roughly a hundred and forty-five attempts that ended because I decided it should.It was also slow. Running on a single logical CPU ruins the framerate, which means the loop executes rather fewer times per second. "Never migrates" and "does less work" are both entirely consistent with this result, and that one run could not separate them.The experiment that separates them is a pretty one. Affinity mask 0x3 is CPUs 0 and 1, which on a Ryzen are SMT siblings: two threads sharing a single physical core. Two threads restores most of the framerate. One physical core means the floating-point state never crosses register files. If 0x3 survives as 0x1 did, migration is the mechanism and the framerate objection is dead. If 0x3 dies at around a hundred and sixty seconds as 0x5 did, then migration is not it at all; it is any context switch whatsoever landing in that ten-instruction window, and single-core was merely buying time by running the loop less often.So I ran 0x3.It held. The loading screens took a little longer than they had any right to, but the game ran, the framerate came back, and nothing fell over. I played a few rounds, and the window stayed exactly where it was.I should be honest about what that is and is not. It is one session per mask, not the three or more I would want against a baseline this wide. It points at migration; it does not prove it. A properly paranoid version of me would have run each mask a dozen times and drawn the distributions.The proper fix is not affinity at all. It is a binary patch: a guard on the value before it is used as an index, checking for a high byte that a legitimate index can never possess. Somebody in the modding community will write it eventually and it will be four instructions long, which is the usual ratio between the size of a fix and the size of the investigation that produced it. Until then, pinning the game to a single physical core with mask 0x3 is the workaround that let me play: slightly longer loading screens in exchange for a game that stays alive.
Closing the Game
I didn't run the dozen sessions.I played a few rounds. Then I closed the game, and I have not opened it since.I still can't read a crash dump on my own. I couldn't disassemble a binary if you paid me. But I can point at the register that held the number one, and I know which of my hundred and forty-five runs lied to me and why. Somebody else read the bytes. I decided what to ask, what to test next, and when a result had earned my belief.The lessons that generalise, in roughly the order I paid for them:Enforce the variable you claim to be testing. The game rewrote its own configuration on exit and an entire hypothesis was tested against a setting that had already reverted. Making one file read-only was the difference between measuring something and measuring nothing at all.A wide baseline devours small samples. Two significant-looking results died in a single session, and both had felt like the end of the investigation on arrival.Verify the absence, not the setting. The Steam overlay was disabled in Steam's own configuration and injecting into every single run regardless. The checkbox governs whether it draws, not whether it turns up. Only the module list would settle it.Reproduce the report, not your paraphrase of it. "The music did not load" became "delete the music files" and produced a fresh crash at a fresh address. The observation was precise. My restatement of it was merely convenient.Name the variable, not your summary of it. For two days my notes said "2 cores". 0x5 and 0x3 are both two cores and they test opposite hypotheses. A label that quietly collapses the thing you are manipulating will cost you a day, and it will do so without ever raising its voice.Watch the clock that is not the CPU. Refund windows, trial periods, warranties. Automated loops consume them in perfect silence and nobody notices until it is very nearly too late.I bought a racing game to relax. I came back with a bug from 1997 and no further interest in the racing game. I wanted to play. I needed to understand.
