Why do I run a homelab?

It used to be tidier, honest. NTP server, PDU, patch panel, UDM Pro, PoE switch, proxmox servers, fans, NAS, AI PC and shonky cooling. UPS not pictured. The small yellow boxes are wideband RF preamps.

So… I’m a geek. No hiding from it. Gone are the days when being a geek mostly meant having the shit beaten out of you in the playground. These days geeks rule the world. So, I’m a geek. I love playing with computers. I love playing with radio. I love playing with electronics. I’m a hacker, in the original sense of the word. That is to say I derive enjoyment from making tools, products, systems do things they weren’t designed to do.

Why? It’s educational, it’s fun, it’s frustrating enough to (sometimes, not always) give a great sense of satisfaction and achievement after solving a problem which has been bugging me for sometimes weeks. It’s inspiring, to learn new tools, to figure out what they’re good at, and what they’re bad at, and to use them to solve problems, more often than not, of my own making.

So I run a VPS (using Linode). I’ve run that VPS since 2008, running every Ubuntu release from 8.04 to 24.04. There I host my own MTA, my own web server for multiple domains, I host websites, databases, dev environments and access endpoints for other people – it’s very handy – and at home I run a homelab.

What’s a homelab? It’s intended for experimentation and learning systems, tools, applications, configurations and all that. All things you can and should learn on your day job, but without the knowledge from first-principles you’ll unlikely find that job int he first place.

Well perhaps my setup isn’t quite a homelab. A large proportion isn’t particularly experimental. A lot of the services I run at home keep me cosplaying as a sysadmin, that’s for sure. If you’re going to do it, you have to do it properly, so here goes:

  • I run a highly-available cluster using proxmox
  • I run highly-available DNS using unbound which provides some Ad-blocking controls
  • I run a Ubiquiti unifi controller and network built around a UDM Pro – easily one of my favourite pieces of networking infrastructure
  • I run failover internet lines (Gigaclear FTTH 1Gbps + Talktalk ADSL 70Mbps)
  • I run multiple VLANs for LAN, Guest, IoT, Surveillance
  • I run a Step-CA certificate authority and use it for SSL certs for all internal services.
  • I run a Squid caching-proxy server to (slightly) reduce internet traffic and add further Ad-blocking controls.
  • I run a Lancache for game library caching
  • I run CCTV/NVR using Frigate
  • I run remote access using Tailscale
  • I run Ollama, edge-tts
  • I run Gitlab source control
  • I run n8n for workflow automation, news aggregation, agentic AI, and I’m exploring what else I can do with this.
  • I run home assistant for home automation
  • I run Observium for monitoring
  • I run Wazuh for XDR/SIEM
  • I run a reprepro server for deb package distribution
  • I run a docker registry for docker image distribution (yes I suppose I could use gitlab for this, but I like keeping it separate)
  • I run another internal, satellite postfix MTA which hands off to the VPS
  • I run a CUPS airprint spooler – less useful these days than it used to be when half the clients didn’t have the correct printer drivers
  • I run nextcloud for DIY cloud storage & sharing
  • I run Calibre-web for book/magazine/academic paper library
  • I run Plex for home media library, sharing with iPads and Amazon Firesticks
  • I’ve just this week installed a local Penpot server to see if it’s useful enough to use instead of Figma
  • I run a weewx weather-station
  • I run a satellite-weather SDR receiver, driven by “predict
  • I run an ADSB receiver driven by flightaware-1090
  • Other devices on the network which aren’t directly in the homelab include things like a trusty DS1819+ Synology, HD Homerun, hamclock, Pi-Star, Echolink node, Allstarlink node, experimental Packet node, NTP server, enviro RPi Pico monitors and a raintrackr, not to mention the family laptops, PCs, TVs, iPads, smartphones, smart plugs, smart lights, NVR cameras, Echos, Google Homes and all that other junk.
  • Mostly kept online by an old Dell UPS cast-off from work – PoE Wireless APs to the rescue!

I’m sure I’ve forgotten a bunch of things, but that’s the gist of it. I do not like paying for subscription services if I can avoid it. I’m sure my response of installing everything and running it at home is pretty atypical, but I like being (mostly) in control of my own services.

The experimental side of my homelab actually comes in a different form – that’s the portable kit I take to field events (scouting and similar). That comes with a whole other set of hardware for servicing a mobile LAN but still having local fast file storage, server redundancy, backhaul links, VPN, caching, DNS and similar. That’s for another post.

There was a time when I would have STRONGLY preferred hiring developers and engineers into my teams who do this sort of stuff in their spare time. I’ve interviewed and hired a lot of developers and engineers into my teams over the last quarter century and I often look for the tinkerer in the candidate. I want them to be excited not only about the company or team but about technology itself.

To be honest I still think like that, apparently because I’m some sort of tech dinosaur. It seems learning and experimentation in one’s spare time has fallen out of fashion for a huge proportion of people. I don’t expect to see a stupid amount of github commit history – though that’s often encouraging to see. I know that if you’re working for most companies, that intellectual property is private, commercial, top secret, verboten, and certainly isn’t going on a public repository for a someone else’s AI training.

I get it – I was never paid for solving massive, difficult, long-standing system engineering and architecture problems for work in my sleep, which happened on many occasions, as might be expected when working on groundbreaking novel science. It certainly used to piss me off that what little recognition was received was never really proportional to the amount of effort put in either in work, or in overtime, or in own-time. I need to get over that – that’s a very much a “me” problem. In the meantime I need a bigger server cab.

What should I run on my homelab? What do you run on yours?

A photo/calendar frame with the Inky Impression 7.3″

I’ve been excited by e-ink displays for a long while. I leapt on the original, gorgeous reMarkable tablet as soon as it came out and have been a regular user and advocate ever since. I would dearly love to have one or two of these enormous 42″ e-ink art poster displays on the wall, but that’s for another day.

I’ve also been a long-time customer of Pimoroni and was aware of their range of nifty Inky displays. I recently came across this neat project by @mimireyburn and managed to pick up a 7.3″ Inky Impression after it being on back-order for only a week or two.

The Inky Impression 7.3″ with protective film and screen reflection
The Inky Impression 7.3″ rear with mounted Raspberry Pi Zero 2W

After flashing RPi OS on a clean card for the Pi Zero 2W, downloading the project, setting up Python, compilers, virtualenvs, prerequisites, etc. I was presented with a complete failure of the underlying driver and inky library to communicate with the display. This isn’t a fault of the inky-calendar project at all, may I reiterate, but unfortunately a very regular occurrence I’ve found when using many Pimoroni products.

Searching around I tried a few different things, including the usual modifications to boot parameters to enable the drivers/kernel modules and fiddling with permissions, users etc. but with no success. Now I’ve never deliberately pretended to be a Python programmer, nor do I particularly wish to be one, but I’m pretty good with debugging weird stuff and this was definitely presenting as a driver/library issue. Specifically some of the differences with the Inky Impression 7.3 seemed to be tripping things up, and it wasn’t a hole I fancied spelunking in today.

A little more digging highlighted a NodeJS package by @aeroniemi with working Impression 7.3″ display support. I definitely have masqueraded as a JavaScript programmer in the past so things were looking up. Some light Claude.AI vibing and I had two working scripts – one to fetch images from PicSum and another to replicate the calendar fetching+rendering, both from public iCal and authenticated Google Cal sources – awesome!

Some dremel butchery on the back panel of an old 7″ picture frame to fit around the sticky-out components on the back of the board and I was in business.

The rear of the photo frame with cut-outs for most of the components on the rear of the display
Extra clearance given to the left-most microUSB power socket on the Pi Zero 2W

Improvements

The only slight drawback with using this NodeJS library is that it only handles the image-display side of things – there’s no built-in support for the function buttons – something to revisit another day.

Another improvement would be to better-handle power – the main benefit of e-ink is that it doesn’t need power once the display has been set, and that’s not being utilised here at all – there’s a cronjob running on the Pi which displays the calendar before 10:00AM and photos after that, refreshing every half-hour.

*/30 6-10 * * * cd /home/frame/ ; node ical2png.js --calendar x --calendar y --google-calendar z --service-account KEY.json --view week ; node main.js --image calendar.png
*/30 10-23 * * * cd /home/frame/ ; node main.js --dither

Lastly, obviously, the display needs to load images from a folder rather than from the internet. That’s super-quick to do, and that’s this afternoon’s job. The calendar-rendering – fonts, sizes, colours etc. could do with a little more spit and polish too.

The code for this project can be found at https://github.com/rmp/inky-frame.

Signing MacOSX apps with Linux

Do you, like me, develop desktop applications for MacOSX? Do you, like me, do it on Linux because it makes for a much cheaper and easier to manage gitlab CI/CD build farm? Do you still sign your apps using a MacOSX machine, or worse (yes, like me), not sign them at all, leaving ugly popups like the one below?

With the impending trustpocalypse next month a lot of third-party (non-app-store) apps for MacOSX are going to start having deeper trust issues than they’ve had previously, no doubt meaning more, uglier popups than that one, or worse, not being able to run at all.

I suspect this trust-tightening issue, whilst arguably a relatively good thing to do to in the war against malware, will adversely affect a huge number of open-source Mac applications where the developer/s wish to provide Mac support for their users but may not wish to pay the annual Apple Developer tax even though it’s still relatively light, or may not even own any Apple hardware (though who knows how they do their integration testing?). In-particular this is likely to affect very many applications built with Electron or NWJS, into which group this post falls.

Well, this week I’ve been looking into this issue for one of the apps I look after, and I’m pleased to say it’s at a stage where I’m comfortable writing something about it. The limitation is that you don’t sidestep paying the Apple Developer tax, as you do still need valid certs with the Apple trust root. But you can sidestep paying for more Apple hardware than you need, i.e. nothing needed in the build farm.

First I should say all of the directions I used came from a 2016 article, here. Thanks very much to Allin Cottrell.

Below is the (slightly-edited) script now forming part of the build pipeline for my app. Hopefully the comments make it fairly self-explanatory. Before you say so, yes I’ve been lazy and haven’t parameterised directory and package names yet.

#!/bin/bash

#########
# This is a nwjs (node) project so fish the version out of package.json
#
VERSION=$(jq -r .version package.json)

#########
# set up the private key for signing, if present
#
rm -f key.pem
if [ "$APPLE_PRIVATE_KEY" != "" ]; then
    echo "$APPLE_PRIVATE_KEY" > key.pem
fi

#########
# temporary build folder/s for package construction
#
rm -rf build
mkdir build && cd build
mkdir -p flat/base.pkg flat/Resources/en.lproj
mkdir -p root/Applications;

#########
# stage the unsigned applicatio into the build folder
#
cp -pR "../dist/EPI2MEAgent/osx64/EPI2MEAgent.app" root/Applications/

#########
# fix a permissions issue which only manifests after following cpio stage
# nw.app seems to be built with owner-read only. no good when packaging as root
#
chmod go+r "root/Applications/EPI2MEAgent.app/Contents/Resources/app.nw"

#########
# pack the application payload
#
( cd root && find . | cpio -o --format odc --owner 0:80 | gzip -c ) > flat/base.pkg/Payload

#########
# calculate a few attributes
#
files=$(find root | wc -l)
bytes=$(du -b -s root | awk '{print $1}')
kbytes=$(( $bytes / 1000 ))

#########
# template the Installer PackageInfo
#
cat <<EOT > flat/base.pkg/PackageInfo
<pkg-info format-version="2" identifier="com.metrichor.agent.base.pkg" version="$VERSION" install-location="/" auth="root">
  <payload installKBytes="$kbytes" numberOfFiles="$files"/>
  <scripts>
    <postinstall file="./postinstall"/>
  </scripts>
  <bundle-version>
    <bundle id="com.metrichor.agent" CFBundleIdentifier="com.nw-builder.epimeagent" path="./Applications/EPI2MEAgent.app" CFBundleVersion="$VERSION"/>
  </bundle-version>
</pkg-info>
EOT

#########
# configure the optional post-install script with a popup dialog
#
mkdir -p scripts
cat <<EOT > scripts/postinstall
#!/bin/bash

osascript -e 'tell app "Finder" to activate'
osascript -e 'tell app "Finder" to display dialog "To get the most of EPI2ME please also explore the Nanopore Community https://community.nanoporetech.com/ ."'
EOT

chmod +x scripts/postinstall

#########
# pack the postinstall payload
#
( cd scripts && find . | cpio -o --format odc --owner 0:80 | gzip -c ) > flat/base.pkg/Scripts
mkbom -u 0 -g 80 root flat/base.pkg/Bom

#########
# Template the flat-package Distribution file together with a MacOS version check
#
cat <<EOT > flat/Distribution
<?xml version="1.0" encoding="utf-8"?>
<installer-script minSpecVersion="1.000000" authoringTool="com.apple.PackageMaker" authoringToolVersion="3.0.3" authoringToolBuild="174">
    <title>EPI2MEAgent $VERSION</title>
    <options customize="never" allow-external-scripts="no"/>
    <domains enable_anywhere="true"/>
    <installation-check script="pm_install_check();"/>
    <script>
function pm_install_check() {
  if(!(system.compareVersions(system.version.ProductVersion,'10.12') >= 0)) {
    my.result.title = 'Failure';
    my.result.message = 'You need at least Mac OS X 10.12 to install EPI2MEAgent.';
    my.result.type = 'Fatal';
    return false;
  }
  return true;
}
    </script>
    <choices-outline>
        <line choice="choice1"/>
    </choices-outline>
    <choice id="choice1" title="base">
        <pkg-ref id="com.metrichor.agent.base.pkg"/>
    </choice>
    <pkg-ref id="com.metrichor.agent.base.pkg" installKBytes="$kbytes" version="$VERSION" auth="Root">#base.pkg</pkg-ref>
</installer-script>
EOT

#########
# pack the Installer
#
( cd flat && xar --compression none -cf "../EPI2MEAgent $VERSION Installer.pkg" * )

#########
# check if we have a key for signing
#
if [ ! -f ../key.pem ]; then
    echo "not signing"
    exit
fi

#########
# calculate attribute
: | openssl dgst -sign ../key.pem -binary | wc -c > siglen.txt

#########
# xar the Installer package
#
xar --sign -f "EPI2MEAgent $VERSION Installer.pkg" \
    --digestinfo-to-sign digestinfo.dat --sig-size $(cat siglen.txt) \
    --cert-loc ../dist/tools/mac/certs/cert00 --cert-loc ../dist/tools/mac/certs/cert01 --cert-loc ../dist/tools/mac/certs/cert02

#########
# construct the signature
#
openssl rsautl -sign -inkey ../key.pem -in digestinfo.dat \
        -out signature.dat

#########
# add the signature to the installer
#
xar --inject-sig signature.dat -f "EPI2MEAgent $VERSION Installer.pkg"

#########
# clean up
#
rm -f signature.dat digestinfo.dat siglen.txt key.pem

With all that you still need a few assets. I built and published (internally) corresponding debs for xar v1.6.1 and bomutils 0.2. You might want to compile & install those from source – they’re pretty straightforward builds.

Next, you need a signing identity. I used XCode (Preferences => Accounts => Apple ID => Manage Certificates) to add a new Mac Installer Distribution certificate. Then used that to sign my .app once on MacOS in order to fish out the Apple cert chain (there are probably better ways to do this)

productsign --sign LJXXXXXX58 \
        build/EPI2MEAgent\ 2020.1.14\ Installer.pkg \
        EPI2MEAgent\ 2020.1.14\ Installer.pkg

Then fish out the certs

xar -f EPI2MEAgent\ 2020.1.14\ Installer.pkg \
        --extract-certs certs
mac:~/agent rmp$ ls -l certs/
total 24
-rw-r--r--  1 rmp  Users  1494 15 Jan 12:06 cert00
-rw-r--r--  1 rmp  Users  1062 15 Jan 12:06 cert01
-rw-r--r--  1 rmp  Users  1215 15 Jan 12:06 cert02

Next use Keychain to export the .p12 private key for the “3rd Party Mac Developer Installer” key. Then openssl it a bit to convert to a pem.

openssl pkcs12 -in certs.p12 -nodes | openssl rsa -out key.pem

I set this up the contents of key.pem as a gitlab CI/CD Environment Variable APPLE_PRIVATE_KEY so it’s never committed to the project source tree.

Once all that’s in place it should be possible to run the script (paths-permitting, obviously yours will be different) and end up with an installer looking something like this. Look for the closed padlock in the top-right, and the fully validated chain of certificate trust.

In conclusion, the cross-platform application nwjs builds (Mac, Windows, Linux) all run using nw-builder on ubuntu:18.04, and the Mac (and Windows, using osslsigncode, maybe more on that later) also all run on ubuntu:18.04. Meaning one docker image for the Linux-based Gitlab CI/CD build farm. Nice!

Proxy testing with IP Namespaces and GitLab CI/CD

CC-BY-NC https://www.flickr.com/photos/thomashawk/106559730

At work, I have a CLI tool I’ve been working on. It talks to the web and is used by customers all over the planet, some of them on networks with tighter restrictions than my own. Often those customers have an HTTP proxy of some sort and that means the CLI application needs to negotiate with it differently than it would directly with a web server.

So I need to test it somehow with a proxy environment. Installing a proxy service like Squid doesn’t sound like too big a deal but it needs to run in several configurations, at a very minimum these three:

  • no-proxy
  • authenticating HTTP proxy
  • non-authenticating HTTP proxy

I’m going to ignore HTTPS proxy for now as it’s not actually a common configuration for customers but I reckon it’s possible to do with mkcert or LetsEncrypt without too much work.

There are two other useful pieces of information to cover, firstly I use GitLab-CI to run the CI/CD test stages for the three proxy configurations in parallel. Secondly, and this is important, I must make sure that, once the test Squid proxy service is running, the web requests in the test only pass through the proxy and do not leak out of the GitLab runner. I can do this by using a really neat Linux feature called IP namespaces.

IP namespaces allow me to set up different network environments on the same machine, similar to IP subnets or AWS security groups. Then I can launch specific processes in those namespaces and network access from those processes will be limited by the configuration of the network namespace. That is to say, the Squid proxy can have full access but the test process can only talk to the proxy. Cool, right?

The GitLab CI/CD YAML looks like this (edited to protect the innocent)

stages:
- integration

.integration_common: &integration_common |
apt-get update
apt-get install -y iproute2

.network_ns: &network_ns |
ip netns add $namespace
ip link add v-eth1 type veth peer name v-peer1
ip link set v-peer1 netns $namespace
ip addr add 192.168.254.1/30 dev v-eth1
ip link set v-eth1 up
ip netns exec $namespace ip addr add 192.168.254.2/30 dev v-peer1
ip netns exec $namespace ip link set v-peer1 up
ip netns exec $namespace ip link set lo up
ip netns exec $namespace ip route add default via 192.168.254.1

noproxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- test/end2end/cli

proxyauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid apache2-utils
- mkdir -p /etc/squid3
- htpasswd -cb /etc/squid3/passwords testuser testpass
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.auth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://testuser:testpass@192.168.254.1:3128/ https_proxy=http://testuser:testpass@192.168.254.1:3128/ ip netns exec $namespace test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxyauth

proxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.noauth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://192.168.254.1:3128/ https_proxy=http://192.168.254.1:3128/ test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxynoauth

So there are five blocks here, with three stages and two common script blocks. The first common script block installs iproute2 which gives us the ip command.

The second script block is where the magic happens. It configures a virtual, routed subnet in the parameterised $namespace.

Following that we have the three test stages corresponding to the three proxy (or not) configurations I listed earlier. Two of them install Squid, one of those creates a test user for authenticating with the proxy. They all run the test script, which in this case is test/end2end/cli. When those three configs are modularised and out like this with the common net namespace script as well it provides a good deal of clarity to the test maintainer. I like it a lot.

So then the last remaining things are the respective squid configurations: proxyauth and proxynoauth. There’s a little bit more junk in these than there needs to be as they’re taken from the stock examples, but they look something like this:

 visible_hostname proxynoauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128

and for authentication:

 visible_hostname proxyauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid3/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED

http_access allow authenticated
http_access deny all
http_port 3128

And there you have it – network-restricted proxy testing with different proxy configurations. It’s the first time I’ve used ip net ns without being wrapped up in Docker, LXC, containerd or some other libvirt thing, but the feeling of power from my new-found network-god skills is quite something :)

Be aware that you might need to choose different subnet ranges if your regular LAN conflicts. Please let me know in the comments if you find this useful or if you had to modify things to work in your environment.

Bookmarks for February 2nd through March 11th

These are my links for February 2nd through March 11th:

Bookmarks for December 4th through January 10th

These are my links for December 4th through January 10th:

Bookmarks for May 6th through May 22nd

These are my links for May 6th through May 22nd:

Bookmarks for February 8th through April 23rd

These are my links for February 8th through April 23rd:

XBMC on Ubuntu 11.10 Oneric Ocelot

Yesterday I upgraded my XBMC media centre, an Acer (bleugh!) Revo 3610 from Ubuntu 10.10 to 11.10 (Oneric Ocelot).

The upgrade itself went fine but (re)installed a few things I’d previously removed, things I didn’t want and things which break a few XBMC features. This is what I had to do to reset things:

  1. Reset the xbmc user’s login session to ‘custom session’ using the gear icon on the top-right of the login window
  2. Add a .xsession file containing
    #!/bin/sh
    exec xbmc

    and chmod +x .xsession

  3. apt-get remove nautilus ubufox xul-ext-ubufox network-manager* pulseaudio
  4. Reset network settings (e.g. /etc/resolv.conf) if you made the mistake of logging in, resulting in NetworkManager resetting everything
  5. Check your xbmc user is still in the ‘audio’ group
  6. apt-add-repository ppa:ubuntu-x-swat/x-updates
  7. apt-get update
  8. apt-get install nvidia-current # if you hadn’t previously done this
  9. apt-get dist-upgrade
  10. apt-get autoremove

It’s probably worth saying I use plain stereo output from the headphone jack, and a Grand Hand III VGA adapter rather than HDMI because my TV is about 9 years old.

Bookmarks for October 4th through October 11th

These are my links for October 4th through October 11th: