Holiday memories

We used to have a lot of physical devices on our network*. Servers, firewalls, file-shares, staging servers, development machines… all sitting on the network with their hard drives endlessly spinning, spinning, spinning!

System administrators are fond of referring to platter-based hard drives as “spinning rust,” partly as a reference to the ferro-magnetic iron crystals that store the actual data, but also to remind us that it’s always decaying and corroding. Over time, drives start generating errors or becoming unreliable. When we had physical devices that exhibited issues, we’d yank the hard drive and replace it. Over the years, we’d accumulated a pile of a dozen or more drives that were unreliable or bad but still contained data.

The data is not especially sensitive, but there could be stuff that could be abused or belongs to other parties. There may well be meeting notes, source code, sample data files, or there could be cached passwords or other credentials. It’s not worth just hoping it’d be OK to release to the world. So it’s a chore to render this data unreadable.

Pulling apart spinning platter hard disks is humbling. These are incredible little devices, with incredibly precise machining and elegant engineering. Going through a pile that spans a decade, you can actually see the improvements in technology: new vibration damping systems, different head-parking strategies, traps for dust, and more. I see these parts, and am inspired by the craftsmanship that goes into them.

So in the spirit of admiration, I offer these (hopefully unreadable) holiday memories.

* Now, of course, we have few physical devices but all those same services are implemented on “the cloud.” This means that someone else has physical devices somewhere, with their hard drives (or SSDs) endlessly spinning, spinning, spinning (or trimming, trimming, trimming).

There’s a very fuzzy golden bumblebee bombing around the garden. I haven’t seen one like it before, and it’s too fast for me to photograph. Perhaps a Sonoran Bumblebee (Bombas sonorus)?

@reassuringurl my guess? A programmer who was super proud of implementing secure connections, and wanted people to be aware that something was different.

What in TARNation?

(technically, a vernal pool not a tarn, but I wanted to try to sound clever).

Anyone know why a few intersections on Beaubien have elevated stop signs? They’re not all near bus stops or parks or student crossings. And drivers seem ignore them anyway and blow through those intersections.
#montreal

Solving a VPN Mystery

The Department of Water and Power is doing work near the office, and over the weekend, there was a sustained power outage. I came in Monday to shrieking UPSes and had to power up the firewall and a few other machines. It was the normal stupid kind of stuff.

We have a few virtual servers out in “the cloud,” and we use point-to-point VPNs to make them seem local to our network. Those VPNs also needed restarting.

Through the course of the day, however, one VPN connection kept unceremoniously disconnecting. Looking at logs on the various servers was unenlightening. Everything was running normally, other than the surprise disconnects.

In the evenings, I’ve been watching the old Grenada TV/Jeremy Brett Sherlock Holmes series, so I had to apply Holmes’ deductive process. The virtual servers had experienced no changes except being disconnected, so I needed to focus on the firewall. The firewall had experienced no change, except being restarted. What could have happened?

I finally found a configuration that was incorrect (it was a netmask that was insufficiently restrictive, allowing devices not on the VPN to collide with VPN IP addresses). I fixed the netmask, and the VPN has been up and stable ever since.

But how could this be? It had been running properly literally for years. It had to be something to do with the power outage. But if that had corrupted the configuration, it wouldn’t have been a single IP netmask changing. “[W]hen you have eliminated the impossible, whatever remains, however improbable, must be the truth.” The bad configuration file could not have been in use.

The best theory is that the configuration file had been (accidentally?) modified at some point in the past, but never loaded. When the firewall was restarted, it loaded this modified configuration for the first time.

Mar Vista moonrise.

Flowering plum already bursting out. But you can see the drought if you compare to last year’s efflorescence.

@quephird CSG is awesome! I first saw it in DKB-Trace (the precursor to POV-Ray), and it was fascination at first sight.

Spending all day fighting metaphorical fires. Meanwhile, the countryside’s on real fire, and the political firestorms are growing.
#prayForRain #metaphoricalAndReal

We used to spend a lot of time coming up with film double-features as a thought experiment. It's interesting to think of theme nights, like "The Warriors" with "Rumblefish" or "Night of the Condor" with "Medium Cool".

Now I'm thinking the book equivalent. It's not as easy, because they're not a single sitting. My first suggestion: _Amatka_ by Karin Tidbeck along with _The Naming Song_ by Jeddidiah Berry.

Backups

So, computer folks always talk about the 3-2-1 strategy of backups: have three copies of your data, stored on two different types of media, with one geographically separated. They also like to repeat slogans like “if you have one backup you have no backups.”

For years, I’ve relied on Time Machine, the backup system Apple includes with their operating system. It not only provides a backup, but it keeps multiple versions of files, so if you, for example, accidentally clobbered your book manuscript by searching and replacing a badly-chosen term but didn’t notice for a week, you could go back to the version you had backed up last week. I felt like I was doing a pretty good job of securing my data: I back up onto an external drive at home, and I also back up on an external drive at the office, a little over 1km away. These external drives are encrypted, so if someone were to break into either place and swipe a drive, they’d have the hardware but not my data.

A few years ago, I also added another layer of redundancy: an encrypted cloud backup. I hadn’t liked the cloud backup services I’d seen before, because all of my files would be on someone’s machine where I had no control over them. A screw-up on the part of a system administrator somewhere could make my files available to the open internet! However, a bunch of new services started offering encrypted backups, where the encryption happens locally and the service doesn’t have view into your files other than it’s a big chunk o’ data (more on this later).

To make a long story short, I tried a few services, and went with Backblaze (disclaimer: that’s an affiliate link, I get credits if you follow it and subscribe. You can always avoid that by going directly to https://backblaze.com).

Fast forward a few years. A friend who’s not particularly computer savvy needed help with some IT stuff. They had an external hard drive connected to their machine and used Windows backup, but the process had silently failed a year before. In diagnosing and fixing this, I also convinced them to pay for and use cloud backups.

This friend lost their house and everything in it during the wildfires last week. Among the long list of things that they didn’t have time to grab before evacuating was that backup hard drive. Cloud backups to the rescue! I was able to download all their files for them.

The surprising scope of the fires also brought one thing into sharp focus: my original strategy of “one backup at home and one at the office” is really insufficient. One kilometer’s not far enough away! Having a remote backup somewhere is an important part of backup plans.

I mentioned above that encrypted cloud services like Backblaze have no visibility into your data. This is not completely true. If you use their encryption scheme, the data is encrypted on your local machine before the data is transmitted over the network. So it’s true in normal operations that there’s no way for them to see the contents. However, when you use their interface to restore files, you need to give them your encryption key so they can identify which file(s) you wish to restore. That means the data is (at least temporarily) decrypted on their servers. When I did a full restore of my friend’s files, I provided the key and they generated a zip file for me to download. That zip file was not encrypted. They say it’s on their server for only a seven days, and I don’t have any reason to distrust them.

I want my data encrypted when it’s backed up because I have financial information like account numbers, etc, that could be abused. That these could exist as clear-text on someone else’s server for short periods of time is not ideal, but it’s also a pretty minimal threat. That being said, if you are involved in journalism, political activism, or other activities where your information could impact people’s lives, this may not be the best solution.

@mcc @whimsy I want that extra byte of friends!!

I just read up on Jirayr Zorthian. Wow. I was probably 7 or 8 years old when I attended the art camp at the ranch. What I remember was livestock wandering around, sculptures reminiscent of Simon Rodia's Watts Towers, making sand-cast candles, and being given a jar of honey from the hives they kept (because I got stung by a bee while swimming). We also got root beer at snack time, which was a big treat for me, since my parents were health-food people and we very rarely got soda.

via Angeles National Forest 🐦

Mt. Wilson, known for its stellar views of the night sky and scenic backcountry trails, has landed a spot in many of the surrounding communities’ hearts. Fire personnel continue to keep their focus on protecting and defending this historic area.

On January 10,2025 the #EatonFire reached #MountWilson and the adjacent communications towers. Due to previous fuel treatment work done by the Angeles National Forest and the heroic efforts of firefighting personnel, no impacts were sustained to the values are risk in the area. Crews were able to extinguish the fire defending this historic site. Forces will remain in the area to monitor for any potential hotspots.

@jfmezei there’s some mention of it in local publications, but I fear people are under-informed of the risk. Similarly, folks are cleaning driveways and streets around town with leaf-blowers, kicking a lot of ash back into the air. I don’t think they understand the hazard.

I’m most posting from an account on a different instance these days. What’s the etiquette for that? Should I migrate completely?

Fuck fuck fuck

Looking bad for La Cañada, JPL, and upper Glendale.