The Arch Linux Rite of Passage: Nine Errors That Lied

I installed Arch Linux this week, and it took two days to work. Not because the install is hard. Because every single failure lied to me about where it lived. I've written before that abstraction is forgetting on purpose, and that the danger isn't the forgetting but inheriting a layer you can't see through. This week I got to live inside that argument. What follows is the trace of a system failing one layer at a time, and me chasing each failure down to the floor below the one it complained about. There's a single sentence underneath the whole thing. I'll give it to you now so you can watch it hold:

The error is never where it says it is. It only tells you where it was standing when it fell.

Could Not Resolve Host

The first message was a DNS error. Could not resolve host. The machine couldn't turn a name into a number. If you've inherited your networking stack instead of understanding it, you stop here. You edit resolver settings. You paste in a nameserver. You treat the words on the screen as the location of the problem. I pinged a bare IP instead. 1.1.1.1. No names involved. Network is unreachable. So it was never DNS. The machine couldn't reach a number either. The resolver was just the outermost complaint on a stack of failures, and it had confidently pointed me at the wrong floor. That's the pattern in its smallest form: the symptom tells the truth about what hurts and lies about where. One layer down: no IP address, no route. The wired interface had no cable. The wireless one was up, awake, and connected to nothing.

Name: could not resolve hosterror printed here
Number: 1.1.1.1 network unreachable
Address and route: nonefault lived here

Printed at the name. Lived in the missing address and route.

Two Programs, One Card

Here's where it got funny, in the way that only wastes an hour. Two programs were running, each convinced it owned the one wireless card: NetworkManager and iwd. Both were trying to manage the same interface, and neither knew the other existed. The kernel log told the story if you knew how to read it. The interface would authenticate, associate, and then immediately: deauthenticating by local choice. Over and over. Read those three words again. By local choice. The network wasn't rejecting the machine. The machine was severing its own connection, joining with one hand and cutting with the other, because its two managers disagreed about who held the card. The enemy was internal. It almost always is. I stopped NetworkManager and gave the card to iwd alone. One chair, one occupant.

Wi-Fi link: deauthenticating by local choiceerror printed here
Interface: one wireless card
NetworkManager and iwd both claim the cardfault lived here

Printed at the Wi-Fi link. Lived in two programs claiming one card.

Connect Is Not Configure

iwd connected. It stayed connected. And there was still no IP address. Still no route. Still the same silence I'd started with. This is the moment worth keeping. I had assumed that connecting to a network and asking it for an address were the same act. They're not. iwd had joined the network and then stood there politely, requesting nothing, because I had never told it to configure what it joined. One line of config (permit it to run DHCP on the connection it makes) and the address arrived, the route arrived, and the name at the very top of the stack finally resolved into a number. Connect and configure are two different verbs. I had spent an hour debugging the wrong one.

Name: could not resolve hosterror printed here
Address and route: none
iwd: not told to run DHCPfault lived here
Wi-Fi: connected

The first symptom, could not resolve host, had been true the entire time. It just lived four layers below where it was printed. I taught myself Linux years ago at Logpoint, documenting security products I didn't understand until I did, and the instinct that got me through that is the same one that got me here: when the abstraction leaks, go down, not sideways. I should have remembered that before day two.

The Update That Broke Everything

Online and pleased with myself, I did the most reasonable thing a person does with a fresh system. I updated it. One command. It pulled a new kernel, rebuilt a graphics driver, and I rebooted, the way one does. The machine came back with a kernel panic. Unable to mount root fs. The kernel had booted. And it couldn't find the disk it lived on. A newborn kernel knows almost nothing about your hardware. It carries a small bag of drivers packed in advance, the initramfs, and inside that bag has to be the tool for opening this specific disk. Pack it wrong, or fail to repack it when the kernel changes, and the kernel arrives at its own front door without a key. The update did exactly what I asked. I just hadn't understood the full contents of my own request. That's abstraction leaking again: the command was smooth and one word long, and it was standing on machinery I'd chosen to forget.

Kernel panic: unable to mount root fserror printed here
The disk the kernel lives on
initramfs: no tool for this diskfault lived here

Printed at the kernel. Lived in the initramfs.

bsdtar Was Innocent

So I booted a rescue USB, entered the installed system, and tried to repack the bag. It failed and named a culprit: bsdtar. I did what the message suggested. Reinstalled the tool. Reinstalled its libraries. Same failure, every time. Then I read the line above the verdict instead of the verdict. Because a program reports where it was standing when it fell, not what tripped it. No space left on device. bsdtar wasn't broken. bsdtar had nowhere to set the thing down. It was the porter, not the crime.

bsdtar failederror printed here
No space left on device
/boot: fullfault lived here

Printed at bsdtar. Lived in a full /boot.

Two Hundred Megabytes

The boot partition was 200 MB, and it was completely full. Next to it, my root partition was 900 GB and one percent used. An ocean of empty space beside a thimble that had overflowed, and the machine couldn't boot because of the thimble. The new kernel plus the initramfs plus a fat graphics driver did not fit through a 200 MB door, and no amount of fiercer compression would squeeze them through. This was not a broken network, or a broken kernel, or a broken tool. It was a decision made once, on the first day, and then forgotten. A room built too small at the founding of the house. It sat harmless as long as nothing needed to grow. Founding decisions always wait. That's the whole danger of them.

Day one

Can you see it? That is the 200 MB /boot, full. The root beside it: 900 GB, 1% used.

Rebuilt

1 GB /boot. Five times the room, and still a sliver.

Slivers enlarged so you can see them. At true scale, the 200 MB /boot would be under a quarter of a pixel wide.

Measure the Prisoner Before You Build the Cell

Day two. New question: how do you make the room bigger? The tempting answer is surgery. Shrink the 900 GB partition next door, slide it down the disk, hand the freed space to the boot partition. It's also the answer that can destroy a filesystem if the power blinks mid-move, and I'd be risking all of it to reclaim a few hundred megabytes. The risk didn't fit the prize. When a mistake is young and cheap to unmake, you don't renovate around it. You tear it down and lay the foundation again. Same afternoon of work, minus the knife at your data's throat. So I reformatted and reinstalled. And here I nearly made the opposite mistake. Burned by too little, I reached for far too much. My instinct was to make the boot partition 10 GB and never think about it again. But a boot partition holds a kernel, its initramfs, and a spare of each. Under a gigabyte, even with the bloated driver. Ten would just sit there, ninety percent empty, a monument to a fear rather than a need. The wound teaches not too small. Maturity adds the second half: and also not absurdly large. I built it at one gigabyte and moved on. One more discipline mattered more than any command. Two disks live in that machine: the fresh SSD I was installing onto, and a second one running Windows, entirely innocent. A wipe doesn't ask twice. So I read the target disk's name out loud before every destructive step, checked it until I was faintly embarrassed, and never let my hand drift to the wrong one. Ceremony around irreversible acts isn't superstition. It's the tax you pay to keep the bystander alive. The correct version of the install was, notably, silent. No panics, no overflowing partitions, no tools crying out. That's the signature of a fix made at the right layer: the floors above simply stop complaining, all at once, without fanfare. Good foundations are boring.

Existence Is Not Permission

Almost silent. The script that installs the desktop stopped immediately, and it had me wedged between two doors. It refused to run as root, which is correct behavior for that kind of installer. And my ordinary user, the one it wanted, wasn't in the sudoers file. He existed. He could log in. He could command nothing. Existence and permission are two separate grants, and I'd only issued the first. The same shape as the dormant network card from day one: present, but sworn to nothing. Then the story folded back on itself in a way I still find funny. Prompted for a password, the machine kept telling me it was incorrect. And I was certain I was typing it correctly, because I was. I'd given root and my own user the same password. Whichever one the prompt wanted, it was getting the right word. The password could not possibly have been the problem, because there was, in effect, only one password. I'd spent my worry on the password. The real fault sat one layer over, where it had waited the whole time: the user was not in the sudoers file. Incorrect password was one more finger pointing a layer too high. I added the user to sudoers, and the desktop installed.

"Incorrect password"error printed here
Password: same for root and user
User not in sudoersfault lived here

Printed at the password. Lived in the sudoers file.

The password was never wrong. The permission was missing. Same lesson, wearing new clothes.

The Resolver That Worked, For Everyone But Me

One more, because the pattern wasn't done with me. The desktop was up. I went to install a command-line tool, and the installer died with a DNS timeout: getaddrinfo ETIMEDOUT. Couldn't resolve the download host. I'd been here on day one, so I moved fast. Cleared a stray proxy that had been sitting in my environment as a literal placeholder. Pointed the resolver at reliable public DNS. Then I tested it: resolvectl query came back instantly with an IP. Resolution worked. I ran the installer again. Same timeout. The resolver worked. The installer, doing the exact same lookup, timed out. Two tools, one name, opposite results. Which meant the name wasn't the problem and the network wasn't the problem. Something between them was. The tell was in a file I'd glanced past twice: /etc/resolv.conf. Empty. Comments, no nameserver. And that is the seam. resolvectl talks to the resolver directly over its own socket, so of course it worked. But ordinary programs (curl, the installer, everything) call the old libc lookup, which reads /etc/resolv.conf, finds nothing, and gives up. My manual test had been passing precisely because it bypassed the broken part. I'd been checking the one path that couldn't fail.

resolvectl querysystemd-resolved socketanswer: an IP

curl, the installerlibc lookup/etc/resolv.conf: emptytimeout

fix: one symlink, /etc/resolv.conf → the resolver that already worked

Two paths for one name. The test took the path that works. The installer took the path that was empty.

One symlink pointed that file at the resolver that already worked, and the install went through on the first try. There was a smaller lesson folded inside it. The installer kept suggesting I re-run with --force. I never did, and it would never have helped: --force skips a version check, but the failure wasn't the check. It was the DNS underneath it. Forcing past the check just marches the same broken lookup one step further before it falls over.

When the flag that skips the check doesn't help, the check was never the thing that was broken.

Can't Find Ext4 Filesystem

The last one arrived a day later, and it was the most frightening message of the week: EXT4-fs: VFS: Can't find ext4 filesystem, over and over, dropping me into emergency mode on a locked root account. The machine was telling me the filesystem holding everything I'd built was gone. I booted the rescue USB and checked. The partition was there. ext4, valid UUID, correct label. A full filesystem check came back clean across all five passes. The filesystem the kernel swore it couldn't find was sitting right there, intact. Which meant the kernel wasn't wrong about failing. It was wrong about why. It hadn't lost the filesystem; it had been handed the wrong address for it. When I reformatted the disk, mkfs had minted a brand-new UUID, a new identity for the same room, and something downstream was still holding the old one. The system was knocking on a door that had been renumbered.

Can't find ext4 filesystemerror printed here
Filesystem: intact, fsck clean
Boot entry: holds the old UUIDfault lived here

Printed at the filesystem. Lived in the address the boot entry held.

Then the fix introduced its own small comedy. I updated the boot entry to reference the partition by its stable ID, rebooted, and watched it hang: a start job is running for /dev/disk/by-partuuid/..., waiting patiently for a device that would never appear. I'd transcribed one character wrong off a console screen. A lowercase l where the real value had a 1. They are the same handful of pixels in that font. A partition ID is hexadecimal; it cannot contain the letter l at all, which means the typo was detectable from first principles and I still made it. The lesson isn't "be more careful." I was being careful. The lesson is that any value you retype by eye is a value you can corrupt, so don't retype it. Have the machine read its own identifier and write it into the config directly. Remove the human from the copy and you remove the entire category of failure.

typed /dev/disk/by-partuuid/…l…real /dev/disk/by-partuuid/…1…

Same pixels in a console font. Only one is hexadecimal.

The Floor Below

Count the layers this held across. A resolver that blamed a name; the fault was a route. A card that blamed the network; the fault was two programs fighting over it. A manager that connected but wouldn't fetch; the fault was one missing line. A kernel that blamed the disk; the fault was a mispacked bag. A tool that blamed itself; the fault was a full partition. A partition built too small once and forgotten. A password that was innocent, because the wall I needed was permission, not a secret. A DNS timeout that survived a working resolver, because the program and I were reading from two different files, and I kept testing the one that wasn't broken. A filesystem reported missing while sitting intact, because the name it answered to had changed and nobody told the thing doing the calling. Every failure pointed up. Every fix was down.

Name: could not resolve hostprintedAddress and route: nonelived
Wi-Fi link: deauthenticating by local choiceprintedNetworkManager and iwd both claim the cardlived
Name: could not resolve hostprintediwd: not told to run DHCPlived
Kernel panic: unable to mount root fsprintedinitramfs: no tool for this disklived
bsdtar failedprinted/boot: fulllived
Boot failsprinted/boot made 200 MB on day onelived
"Incorrect password"printedUser not in sudoerslived
Installer: getaddrinfo ETIMEDOUTprinted/etc/resolv.conf: emptylived
Can't find ext4 filesystemprintedBoot entry: holds the old UUIDlived

Nine failures. Red is where each error printed. Green is where it lived.

This is not a story about Linux. It's the same argument I keep circling in everything I write. Abstraction is a deal: you forget a layer to gain leverage, and the deal is good exactly as long as nothing breaks. The moment it breaks, the only person who can fix it is the one willing to see through the floor to the basement underneath. Past the error message, past the obvious suspect, past their own certainty about what the problem must be. The tools are getting smoother. The commands are getting shorter. An LLM will now hand you a fix for a stack you've never read. All of that is chosen forgetting if you understand what's being forgotten, and blind trust if you don't. The difference is invisible in normal operation and total in failure. I don't debug systems because I enjoy suffering. I do it because every time I follow an error down to the floor it was actually standing on, I learn the shape of a layer I'd been trusting without seeing. That's not a Linux skill. It's the whole method.

The error is never where it says it is. Keep walking down until you reach the quiet layer that's actually broken and has been saying nothing at all.