How a Fusion Drive Ate Our Gopher

If you follow Somnolescent, you’ll know we’re big fans of the gopher protocol and formerly maintained a server at gopher.somnolescent.net for people to come and peek at.

That gopher’s offline and likely will be for a little while, and you can thank a Fusion Drive for taking it, along with my entire music library and all the project files and renders for my next album.

So how’d it happen?

To explain the setup that enabled this to happen, my home server is a 2017 5k iMac that I also use as a second computer and my media hub. It wasn’t meant to be a home server and I also didn’t pay for it, hence why I didn’t go with something less ill-fitting and ridiculous.

This machine had one of Apple’s Fusion Drives in it, a 32gb SSD and a 1TB spinning platter that get smushed together in software to make for a bigger SSD or a faster hard drive, however you wanna look at it. I don’t live with the computer at the moment, so I just SSH into it a lot. It works for what I need it to, it runs our Minecraft server, /shrug.

This all started around the 16th of April, when it stopped responding to any remote access. At first, I thought the machine was simply off, and I texted someone at home to switch it back on.

And got this response.

Turns out, my startup disk had gone limp. I made a Sierra repair disk from a flash drive and came home a few days later to poke at it. Booted into the drive and found it’d been wiped completely!

At least, according to Disk Utility. Either the disk was legitimately wiped or the partition table had gone bad and needed a rebuild. I wasn’t gonna trust it yet, so I made an Ubuntu live USB and got to work poking at it with TestDisk, which is some pretty solid partition recovery stuff for damaged disks.

Could I have been better served using Knoppix or a distro built for data recovery? Yeah, probably, but I wanted to try what I knew first, and that meant Ubuntu. Man, what a nostalgic experience. I’ve used various computers both at home and at camps that ran Ubuntu, but I hadn’t actually booted it up in a solid seven years or so. Memories of its weird, inky, bruised and orange UI came flooding back. Good OS.

So the first step with TestDisk is always to let it scan the entire drive to find available partitions. It takes a while, but I had a fresh copy of the Kids soundtrack to bop to while I waited, so that wasn’t a huge deal.

Something I found amusing was the way that it detected a batch of Mini vMac System 7 disk images as partitions. Indeed, those were my old disk images I use to boot up my writing VM. Reassuring, too, as it meant the drive was still intact.

Kind of.

By the end of the scan, it told me that it did find my main boot partition…but that the hard disk was “too small” and they were unrecoverable as a result. According to the internet, this usually comes about when the detected partitions add up to more than the size of the disk…problem being, I literally never reformatted that disk. Everything on it, down to the partition map, was stock aside from upgrading to Mojave.

I decided to go for broke and write whatever it could find to the hard drive anyway. Naturally, nothing booted. Even better is that it wrote all my disk images to the partition map as if I could boot my modern Mac off emulated System 7 disks (that’d be pretty cool if I could though).

Onto next plan, PhotoRec.

PhotoRec is the companion tool to TestDisk, the thought behind it being that, even if your shit is totally unbootable, toast, or has been wiped, PhotoRec can still pull whatever it can out of the burning wreckage. dcb’s successfully used this to pull photos off cameras, so I figured I’d give it a shot myself. And to its credit, it did find stuff!

Random junk. We’re talking LastPass assets and Vivaldi cache junk. A few pieces from my art hoard came up, but not much else, and worse yet, I didn’t have another disk handy to write all this shit to so I could sort through whatever it did manage to get. What nonsense.

By this point, I just wanted the damn computer to work again. I accepted that everything on it was gone, and that was fine if I could just get it to boot again. I went back to my Sierra boot disk and saw the absolute mess of unbootable, unmountable partitions that’d been left in TestDisk’s wake.

Burn it clean.

I got to work killing all the partitions (which most of the time would error out, making me think my shiny new drive was about to fail), including the SSD. Nothing survived the inferno.

Soon, the hard drive was whole again, and the SSD was clean as a whistle. Both came up clean on a First Aid check.

Reinstalled Sierra, the OS the machine came with, and came out of it with two split drives instead of the single logical volume it was originally mapped as. I’m only using the hard drive as a boot device now; I owned two iMacs before this point, a 2011 and a 2014 (what I’m writing this on), both with only hard drives, and they continue to chug along to this day.

So if there’s anything in this swirling, messed up world I can trust, I guess it’s Apple’s internal hard drives.

I’m blaming this entire shitshow on the Fusion Drive. I can’t see any other reason for two completely healthy drives to fail the way they did. A lot of this is my fault for not keeping backups. I’ve been lazy and unwilling to invest in external hard drives after I had two of them bite the big one (protip: don’t leave them plugged in all the time), and I’m not sure what other options I have. Disc backups pretty much require me to be there in person and take a lot of discs, even if I’m limiting what I back up, web storage is effective, but not a cure-all with my shitty internet, and a NAS box is a little overkill for what I’m doing.

I’m sure someone will come along and tell me some way I could’ve kept all my data if I just did this or did that, but I really didn’t feel like being without a computer as long as that would’ve taken. At some point, just losing everything is the lesser of two evils.

I’m still occasionally remembering things I lost in the crash, and it’s a shitty place to be for sure. This is why I don’t hoard stuff anymore; I inevitably lose it. I lost my 90gb bank of rhythm game stems to one of those aforementioned externals, and my big lossless music library, augmented with a ton of rare demos and torrents and things collected over the span of about five years, is all gone now.

About a week later, though, and the machine is running stable. It’s back to hosting our group Somnocraft sessions (and now I manually download the world after each game), and I’ve re-ripped what albums I have and I’m looking to replace the torrents with proper CD copies now. This is why you buy physical, kids.

Keep backups and split your Fusion Drives. It’s the American way. o7