Proxy testing with IP Namespaces and GitLab CI/CD

CC-BY-NC https://www.flickr.com/photos/thomashawk/106559730

At work, I have a CLI tool I’ve been working on. It talks to the web and is used by customers all over the planet, some of them on networks with tighter restrictions than my own. Often those customers have an HTTP proxy of some sort and that means the CLI application needs to negotiate with it differently than it would directly with a web server.

So I need to test it somehow with a proxy environment. Installing a proxy service like Squid doesn’t sound like too big a deal but it needs to run in several configurations, at a very minimum these three:

  • no-proxy
  • authenticating HTTP proxy
  • non-authenticating HTTP proxy

I’m going to ignore HTTPS proxy for now as it’s not actually a common configuration for customers but I reckon it’s possible to do with mkcert or LetsEncrypt without too much work.

There are two other useful pieces of information to cover, firstly I use GitLab-CI to run the CI/CD test stages for the three proxy configurations in parallel. Secondly, and this is important, I must make sure that, once the test Squid proxy service is running, the web requests in the test only pass through the proxy and do not leak out of the GitLab runner. I can do this by using a really neat Linux feature called IP namespaces.

IP namespaces allow me to set up different network environments on the same machine, similar to IP subnets or AWS security groups. Then I can launch specific processes in those namespaces and network access from those processes will be limited by the configuration of the network namespace. That is to say, the Squid proxy can have full access but the test process can only talk to the proxy. Cool, right?

The GitLab CI/CD YAML looks like this (edited to protect the innocent)

stages:
- integration

.integration_common: &integration_common |
apt-get update
apt-get install -y iproute2

.network_ns: &network_ns |
ip netns add $namespace
ip link add v-eth1 type veth peer name v-peer1
ip link set v-peer1 netns $namespace
ip addr add 192.168.254.1/30 dev v-eth1
ip link set v-eth1 up
ip netns exec $namespace ip addr add 192.168.254.2/30 dev v-peer1
ip netns exec $namespace ip link set v-peer1 up
ip netns exec $namespace ip link set lo up
ip netns exec $namespace ip route add default via 192.168.254.1

noproxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- test/end2end/cli

proxyauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid apache2-utils
- mkdir -p /etc/squid3
- htpasswd -cb /etc/squid3/passwords testuser testpass
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.auth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://testuser:testpass@192.168.254.1:3128/ https_proxy=http://testuser:testpass@192.168.254.1:3128/ ip netns exec $namespace test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxyauth

proxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.noauth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://192.168.254.1:3128/ https_proxy=http://192.168.254.1:3128/ test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxynoauth

So there are five blocks here, with three stages and two common script blocks. The first common script block installs iproute2 which gives us the ip command.

The second script block is where the magic happens. It configures a virtual, routed subnet in the parameterised $namespace.

Following that we have the three test stages corresponding to the three proxy (or not) configurations I listed earlier. Two of them install Squid, one of those creates a test user for authenticating with the proxy. They all run the test script, which in this case is test/end2end/cli. When those three configs are modularised and out like this with the common net namespace script as well it provides a good deal of clarity to the test maintainer. I like it a lot.

So then the last remaining things are the respective squid configurations: proxyauth and proxynoauth. There’s a little bit more junk in these than there needs to be as they’re taken from the stock examples, but they look something like this:

 visible_hostname proxynoauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128

and for authentication:

 visible_hostname proxyauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid3/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED

http_access allow authenticated
http_access deny all
http_port 3128

And there you have it – network-restricted proxy testing with different proxy configurations. It’s the first time I’ve used ip net ns without being wrapped up in Docker, LXC, containerd or some other libvirt thing, but the feeling of power from my new-found network-god skills is quite something :)

Be aware that you might need to choose different subnet ranges if your regular LAN conflicts. Please let me know in the comments if you find this useful or if you had to modify things to work in your environment.

Remote Power Management using Arduino

2016-03-04 21.20.07

2016-03-07 Update: Git Repo available

Recently I’ve been involved with building a hardware device consisting of a cluster of low-power PC servers. The boards chosen for this particular project aren’t enterprise or embedded -style boards with specialist features like out of band (power) management (like Dell’s iDRAC or Intel’s AMT) so I started thinking about how to approximate something similar.

It’s also a little reminiscent of STONITH (Shoot The Other Node In The Head), used for aspects of the Linux-HA (High Availability) services.

I dug around in a box of goodies and found a couple of handy parts:

  1. Arduino Duemilanove
  2. Seeedstudio Arduino Relay Shield v3

The relays are rated for switching up to 35V at 8A – easily handling the 19V @ 2A for the mini server boards I’m remote managing.

The other handy thing to notice is that the Arduino by its nature is serial-enabled, meaning you can control it very simply using a USB connection to the management system without needing any more shields or adapters.

Lastly it’s worth mentioning that the relays are effectively SPDT switches so have connections for circuit open and closed. In my case this is useful as most of the time I don’t want the relays to be energised, saving power and prolonging the life of the relay.

The example Arduino code below opens a serial port and collects characters in a string variable until a carriage-return (0x0D) before acting, accepting commands “on”, “off” and “reset”. When a command is completed, the code clears the command buffer and flips voltages on the digital pins controlling the relays. Works a treat – all I need to do now is splice the power cables for the cluster compute units and run them through the right connectors on the relay boards. With the draw the cluster nodes pull being well within the specs of the relays it might even be possible to happily run two nodes through each relay.

There’s no reason why this sort of thing couldn’t be used for many other purposes too – home automation or other types of remote management, and could obviously be activated over ethernet, wifi or bluetooth instead of serial – goes without saying for a relay board -duh!

int MotorControl1 = 4;
int MotorControl2 = 5;
int MotorControl3 = 6;
int MotorControl4 = 7;
int incomingByte = 0; // for incoming serial data
String input = ""; // for command message

void action (String cmd) {
  if(cmd == "off") {
    digitalWrite(MotorControl1, HIGH); // NO1 + COM1
    digitalWrite(MotorControl2, HIGH); // NO2 + COM2
    digitalWrite(MotorControl3, HIGH); // NO3 + COM3
    digitalWrite(MotorControl4, HIGH); // NO4 + COM4
    return;
  }

  if(cmd == "on") {
    digitalWrite(MotorControl1, LOW); // NC1 + COM1
    digitalWrite(MotorControl2, LOW); // NC2 + COM2
    digitalWrite(MotorControl3, LOW); // NC3 + COM3
    digitalWrite(MotorControl4, LOW); // NC4 + COM4
    return;
  }

  if(cmd == "reset") {
    action("off");
    delay(1000);
    action("on");
    return;
  }

  Serial.println("unknown action");
}

// the setup routine runs once when you press reset:
void setup() {
  pinMode(MotorControl1, OUTPUT);
  pinMode(MotorControl2, OUTPUT);
  pinMode(MotorControl3, OUTPUT);
  pinMode(MotorControl4, OUTPUT);
  Serial.begin(9600); // opens serial port, sets data rate to 9600 bps
  Serial.println("relay controller v0.1 rmp@psyphi.net actions are on|off|reset");
  input = "";
} 

// the loop routine runs over and over again forever:
void loop() {
  if (Serial.available() > 0) {
    incomingByte = Serial.read();

    if(incomingByte == 0x0D) {
      Serial.println("action:" + input);
      action(input);
      input = "";
    } else {
      input.concat(char(incomingByte));
    }
  } else {
    delay(1000); // no need to go crazy
  }
}



Amazon Prime on Kodi for Slice

slice-boxI’m lucky enough to have both a Raspberry Pi “Slice” media player and an Amazon Prime account but it’s not supported right out of the box. Here’s how I was able to set it up today.

Requirements:

  1. A Slice
  2. An Amazon Prime account

Firstl make sure your Slice is correctly networked. Configuration is under Setup => OpenElec Settings.

Next you need to download a third-party add-on repository for Kodi. Download XLordKX Repo zip into a folder onto the Slice. I did this from another computer and copied it into a network share served from the Slice.

Now we can install the add-on. Setup => Add-on manager => Install from zip file. Then navigate to the file you downloaded and install it. Now Setup => Get Add-ons => XLordKX Repo => Video Add-ons => Amazon Prime Instant Video => Install

Now to configure Amazon Prime. Setup => Add-ons => Video Add-ons => Amazon Prime Instant Video.

I set mine to Website Version: UK and left everything else as defaults. Feed it your Amazon username & password and off you go.

The navigation is a little flakey which is a common Kodi/XBMC problem but the streaming seems fully functional – no problems on anything I’ve tried so far. I also see no reason why this wouldn’t work on raspbmc or openelec on a plain old Raspberry Pi. Happy streaming!

 

HT https://seo-michael.co.uk/tutorial-how-to-install-amazon-prime-instant-video-xbmc-kodi/ where I found instructions for Kodi in general.

Apple Watch Adventures

2015-06-25 20.38.48Recently we’ve been exploring our customers’ user journeys, mapping out their touchpoints and reevaluating how we engage in the user experience of everything we do both digitally and in the physical world. Part of that requires the use of personas – model customers who in theory fulfil various different criteria in order to test the functionality and experiences of those digital touch points. I couldn’t help thinking about that and wondering which persona I might fit into in some nutjob’s head at Apple. Here are my first 12 hours’ experience with the much talked about Apple Watch.

As a bit of background: I’ve used work-owned Mac Laptops with OSX for 11 of the last 13 years but once made the mistake of spending my own money on an iPhone 3G, my first smartphone, which I hated more than I liked and swore never to buy another Apple device.

0900 Arrive at the office, coffee, email.

1000 Done responding to email for now. Time to look at what’s in the new box on my desk this morning. Ooh an Apple Watch. Great!

It’s heavy. Really heavy. The box is heavy, the plastic case is heavy, the magnetic charger is heavy. I think someone told Apple Heavy = Quality or something. I requested the smaller 38mm watch as my wrists are pretty thin and I didn’t want it to look ridiculous. The watch is small in width and height but it’s heavy. And fat – looks like a toy of some sort. Heavy, clunky, ugly. The leather strap feels it’s made of the same foam as my childrens’ play mats. The buckle is flabby and horrible. The face is already covered in fingerprints – I thought these things were oleophobic.

At least it came charged though, mostly because my IT Support team wanted a laugh and took it out of the box to play with earlier.

1030 meetings until 1200.

1200 Turn it on. It asks me to choose English (UK) as my preferred language fifteen times for some unknown reason.

1210 Meetings until 1400.

1400 I know it’s an Apple device and they’re very well known for being “open”. Not. Is there any way in the known universe to make it pair with my S6 Edge?
Read some webpages.

1420 Anticipate pairing compatibility answer. Scrounge an iPhone 5S from my IT Support team.

1430 meetings until 1800

1800 IT Support team managed to locate previous iPhone owner to deactivate account locks & device security so it can be reused.

1820 catch a lift home

2010 get home, cold dinner.

2030 No it doesn’t pair, but found a video of a guy who managed to make it run OS 7. Neat, I wonder if it could run Android. Read stupid Mashable articles for a bit.

2045 Try and set up the iPhone. Needs Wifi. Try to type in my long WPA code using the soft keyboard. Three attempts before typing it right – keyboard is noticeably less responsive than the S6 as well as being much smaller and non-Swype (yeah yeah, non-security-compromised, haha).

2050 Past the wifi setup screen. Yes!

Won’t proceed without a SIM. Full Fiscal Shambles! I can register a Galaxy without a SIM. Why must I have one for an iPhone? What happens if I use a SIM from something else? Is it locked somehow?

2055 No idea. Let’s try. Extract the SIM from my old S3. It’s a mini SIM. Too big. Don’t really want to cut it down as it’s already cut down from a fullsize one and I won’t be able to put it back in my S3.

2058 I wonder if I have something other than my S6 has a micro sim. Losing the will to live.

2100 Look for iPad everywhere in the house.

2110 Realise child has pinched iPad to play Clash of Clans and hidden it somewhere. Look for child.

2115 Location-aware child has left the house. Look for SIM extraction tool for S6. Find paperclip. Extract SIM.

2120 Insert SIM, complete iPhone setup.

2122 Complete watch setup. Manage to zoom in the app that tells the time. Can’t unzoom it. Didn’t read the instructions two seconds ago about how to unzoom. Can’t figure out what the magic pinch-press-zoom-standonhead combination is. I guess I need to hold Apple-Meta-Cmd or something.

2123 Press all the buttons at once and repeatedly in various combinations. Discover scrollwheel strafes across the display. Wow that’s a really horrible interaction.

2130 Have no content on iPhone to drive Watch applications.

2140 Get bored. Throw it in the bin. What a PoC.

2150 Realise we’re supposed to be writing Metrichor apps for it. Fish it out of the bin ready to give to the developers. Should set the project back a couple of months.

Update 2015-06-26
How could I forget? There is one thing I like – the UK plug with the retractable pins – finally! Sorry Samsung, only retracting one out of three pins doesn’t cut it.

Bookmarks for February 2nd through March 11th

These are my links for February 2nd through March 11th:

Bookmarks for December 2nd through January 12th

These are my links for December 2nd through January 12th:

Bookmarks for November 19th through December 2nd

These are my links for November 19th through December 2nd:

Bookmarks for August 29th through November 12th

These are my links for August 29th through November 12th:

Bookmarks for March 22nd through August 21st

These are my links for March 22nd through August 21st:

Pushing Jenkins Job Build Statuses to Geckoboard

geckoboard

I love using Geckoboard. I love using Jenkins. I do have a few issues connecting the two though.

My Jenkins build cluster sits inside my corporate network and while there is a Jenkins plugin for Geckoboard it will only connect to Jenkins instances it can see on the public internet. I haven’t yet found a Geckoboard plugin for Jenkins to push results out through either. One day soon I’ll be annoyed enough to learn some Java and write one but until then I have a hack.

The core configurations of most of my Jenkins jobs runs approximately on these lines:

make deb && scp *deb deb-repo.my.net:/var/www/apt/incoming/

i.e. build a .deb (for Ubuntu) and if successful, copy and queue it for indexing by reprepro on my .deb repository server.

Now in Geckoboard I can configure a 1×1 Custom Text widget for PUSH data and publish data to it like so:

curl https://push.geckoboard.com/v1/send/F639F1AE-2227-11E4-A773-8FE5A58BF7C4 \
-d "{"api_key":"AC738FE5A58BF7C4","data":{"item":[{"text":"packagename.deb","type":0}]}}"

Let’s make it a little more sustainable. In the main Jenkins configuration I set up a global environment variable called GECKO_APIKEY with a value of AC738FE5A58BF7C4. Now the line reads:

curl https://push.geckoboard.com/v1/send/F639F1AE-2227-11E4-A773-8FE5A58BF7C4 \
-d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"packagename.deb","type":0}]}}"

I know I’ll need to change the posted data on failure which most like means duplicating some or all of that line so I’ll extract the widget id too. The job is now configured like:

export WIDGET=F639F1AE-2227-11E4-A773-8FE5A58BF7C4
make deb && scp *deb deb-repo.my.net:/var/www/apt/incoming/
curl https://push.geckoboard.com/v1/send/$WIDGET \
 -d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"packagename.deb","type":0}]}}"

But it’s not yet triggered differently on success or failure, so…

export WIDGET=F639F1AE-2227-11E4-A773-8FE5A58BF7C4
make deb && scp *deb deb-repo.my.net:/var/www/apt/incoming/  && \
curl https://push.geckoboard.com/v1/send/$WIDGET \
 -d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"packagename.deb","type":0}]}}" || \
curl https://push.geckoboard.com/v1/send/$WIDGET \
 -d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"packagename.deb","type":1}]}}"

The duplicate URL and packagename.deb are annoying aren’t they? A quick look at the Jenkins docs reveals $JOB_NAME has what we want.

export WIDGET=F639F1AE-2227-11E4-A773-8FE5A58BF7C4
export GECKO_URL=https://push.geckoboard.com/v1/send/$WIDGET
make deb && scp *deb deb-repo.my.net:/var/www/apt/incoming/  && \
curl $GECKO_URL \
 -d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"$JOB_NAME PASS","type":0}]}}" || \
curl $GECKO_URL \
 -d "{"api_key":"$GECKO_APIKEY","data":{"item":[{"text":"$JOB_NAME FAIL","type":1}]}}"

Not too bad. It even works on Windows without too many modifications – “set” instead of “export”, %VAR% instead of $VAR and a Windows curl binary added to %PATH%.

 

Note: All API keys and Widget Ids have been changed to protect the innocent.