A record of things done, undone, and redone differently.

"the prometheus incident"

• fixedPrometheus was scraping itself and storing those metrics. It was fine. Then it wasn't.
• fixedGrafana dashboard "is anything on fire" was showing a false positive for the NAS. NAS was fine. Dashboard was misconfigured. Classic.
• changedRetention policy lowered from 30d to 14d after disk fill. — turns out metrics take space
• noteUptime counter still running from before this incident. Technically accurate. Spiritually questionable.

"138 devices and counting"

• addedZigbee coordinator. One coordinator became two after the first one "just stopped." — it was the USB port
• addedZ-Wave stick for the locks. The locks did not want a Z-Wave stick. Convinced them eventually.
• addedHome Assistant automation: lights off when nobody is home. Works great except when it thinks nobody is home but someone is.
• changedIoT VLAN now properly isolated. Devices can reach the internet but not the lab. They seem angry about this.
• removedThat one Tuya bulb that kept rejoining the wrong network. Gone. Physically.

"backup roulette, resolved"

• securityActually tested the backups. One restore worked. One did not. Unclear which one mattered more. — both fixed now, probably
• fixedRestic forget policy was keeping 365 daily snapshots. Storage was 80% full. Policy is now sane.
• addedOffsite cold storage target. A rented VPS in a datacenter I've never visited. Good enough.
• noteThe CHANGELOG.md in the restic repo is more detailed. This is the highlight reel.

"four VLANs became eight"

• addedFour new VLANs. Named them at 23:00. Renamed them at 23:30 because the names were bad. Renamed again at 00:15.
• addedFortigate IPS profiles on WAN. It blocks things. Mostly scanners from Russia and one IP in Ohio that really wants to say hello.
• changedManagement VLAN is now actually management-only. Took two evenings and one accidental lockout to get right.
• removedThe flat network. It had a good run. It trusted everyone. That was the problem.

"one NUC is none"

• addedSecond NUC. For redundancy. Also because the first one was warm and that felt wrong.
• addedvSphere cluster. Two nodes, one vCenter. The vCenter runs on one of the nodes it manages. This is fine.
• addedECC RAM in both nodes. Explained what ECC means to several people who didn't ask.
• changedTotal power draw went from 54W to 102W. Monthly electricity cost went up. Worth it.
• noteStill no HA for vCenter itself. This is a known issue. It's on the list. The list is long.

"the meterkast grows up"

• added12U rack in the meterkast. Cut a hole for cable management. Regret nothing.
• added1500VA UPS. Finally. The homelab now outlives power blips. So does the ego.
• addedFortigate 60F. Replaced the ISP router. Turned off features I didn't understand. Turned some back on later.
• breakingMigrated everything from a single Pi to proper VMs. Some things did not survive the migration. Those things are in a better place now.
• removedThe Raspberry Pi 4 as primary server. Retired with honors. Now runs a weather station because it deserves better.

"the raspberry pi years"

• noteRecords from this period are incomplete. A lot happened. Some of it on purpose.
• addedFirst Pi-hole. The internet became quieter. Also some things broke. Mostly the internet became quieter.
• addedFirst NAS. A repurposed desktop, FreeNAS, four drives. Ran for years. Never lost data. Was surprised.
• changedStack rebuilt approximately four times as knowledge increased and past decisions became embarrassing.
• removedSeveral containers that seemed like a good idea at the time. They know what they did.
• noteThis is where the interest became a homelab and the homelab became a problem. Proudly.