Proxy testing with IP Namespaces and GitLab CI/CD

CC-BY-NC https://www.flickr.com/photos/thomashawk/106559730

At work, I have a CLI tool I’ve been working on. It talks to the web and is used by customers all over the planet, some of them on networks with tighter restrictions than my own. Often those customers have an HTTP proxy of some sort and that means the CLI application needs to negotiate with it differently than it would directly with a web server.

So I need to test it somehow with a proxy environment. Installing a proxy service like Squid doesn’t sound like too big a deal but it needs to run in several configurations, at a very minimum these three:

  • no-proxy
  • authenticating HTTP proxy
  • non-authenticating HTTP proxy

I’m going to ignore HTTPS proxy for now as it’s not actually a common configuration for customers but I reckon it’s possible to do with mkcert or LetsEncrypt without too much work.

There are two other useful pieces of information to cover, firstly I use GitLab-CI to run the CI/CD test stages for the three proxy configurations in parallel. Secondly, and this is important, I must make sure that, once the test Squid proxy service is running, the web requests in the test only pass through the proxy and do not leak out of the GitLab runner. I can do this by using a really neat Linux feature called IP namespaces.

IP namespaces allow me to set up different network environments on the same machine, similar to IP subnets or AWS security groups. Then I can launch specific processes in those namespaces and network access from those processes will be limited by the configuration of the network namespace. That is to say, the Squid proxy can have full access but the test process can only talk to the proxy. Cool, right?

The GitLab CI/CD YAML looks like this (edited to protect the innocent)

stages:
- integration

.integration_common: &integration_common |
apt-get update
apt-get install -y iproute2

.network_ns: &network_ns |
ip netns add $namespace
ip link add v-eth1 type veth peer name v-peer1
ip link set v-peer1 netns $namespace
ip addr add 192.168.254.1/30 dev v-eth1
ip link set v-eth1 up
ip netns exec $namespace ip addr add 192.168.254.2/30 dev v-peer1
ip netns exec $namespace ip link set v-peer1 up
ip netns exec $namespace ip link set lo up
ip netns exec $namespace ip route add default via 192.168.254.1

noproxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- test/end2end/cli

proxyauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid apache2-utils
- mkdir -p /etc/squid3
- htpasswd -cb /etc/squid3/passwords testuser testpass
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.auth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://testuser:testpass@192.168.254.1:3128/ https_proxy=http://testuser:testpass@192.168.254.1:3128/ ip netns exec $namespace test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxyauth

proxynoauth-cli:
image: ubuntu:18.04
stage: integration
script:
- *integration_common
- apt-get install -y squid
- *network_ns
- squid3 -f test/end2end/conf/squid.conf.noauth && sleep 1 || tail -20 /var/log/syslog | grep squid
- http_proxy=http://192.168.254.1:3128/ https_proxy=http://192.168.254.1:3128/ test/end2end/cli
- ip netns del $namespace || true
variables:
namespace: proxynoauth

So there are five blocks here, with three stages and two common script blocks. The first common script block installs iproute2 which gives us the ip command.

The second script block is where the magic happens. It configures a virtual, routed subnet in the parameterised $namespace.

Following that we have the three test stages corresponding to the three proxy (or not) configurations I listed earlier. Two of them install Squid, one of those creates a test user for authenticating with the proxy. They all run the test script, which in this case is test/end2end/cli. When those three configs are modularised and out like this with the common net namespace script as well it provides a good deal of clarity to the test maintainer. I like it a lot.

So then the last remaining things are the respective squid configurations: proxyauth and proxynoauth. There’s a little bit more junk in these than there needs to be as they’re taken from the stock examples, but they look something like this:

 visible_hostname proxynoauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128

and for authentication:

 visible_hostname proxyauth
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid3/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED

http_access allow authenticated
http_access deny all
http_port 3128

And there you have it – network-restricted proxy testing with different proxy configurations. It’s the first time I’ve used ip net ns without being wrapped up in Docker, LXC, containerd or some other libvirt thing, but the feeling of power from my new-found network-god skills is quite something :)

Be aware that you might need to choose different subnet ranges if your regular LAN conflicts. Please let me know in the comments if you find this useful or if you had to modify things to work in your environment.

Bookmarks for August 29th through November 12th

These are my links for August 29th through November 12th:

Bookmarks for May 6th through May 22nd

These are my links for May 6th through May 22nd:

Bookmarks for November 6th through November 26th

These are my links for November 6th through November 26th:

Bookmarks for December 7th through January 21st

These are my links for December 7th through January 21st:

Bookmarks for November 15th through November 25th

These are my links for November 15th through November 25th:

Pseudo-VPN stuff with SSH

Firstly, there are *lots* of ways to do this. This is one way.

Secondly, poking holes in your corporate network is occasionally frowned upon and may contravene your workplace Acceptable Use Policy or equivalent. If you have a VPN solution (HTTPS, L2TP, or whatever) which works on everything you need, then I shouldn’t need to tell you to use that instead.

Anyway…

At home, on the end of my DSL line I have a PC running Linux.

At work I have lots of PCs running Linux.

Sometimes I’m using a random machine and/or a platform unsupported by my corporate VPN and I want to connect to work without using the (recommended) HTTPS VPN or (complicated) L2TP. So I turn to a trusty source of cool networky stuff: SSH.

Importantly, SSH understands how to be a SOCKS server. This allows applications which understand SOCKS (most Windows stuff for example) to redirect all their traffic over SSH without the addition of a proxy server like Squid on the corporate end.

So, how do you set it up? It’s fairly easy:

1. Set up the work-to-home connection:

user@work:~$ while [ 1 ]; do ssh -NR20000:localhost:22 user@home.net; done

2. Set up the laptop-to-home connection:

user@laptop:~$ ssh -L15000:localhost:20000 user@home.net

3. Set up the laptop-to-work connection:

user@laptop:~$ ssh -D15001 localhost -p 15000

If you’re at home and your “other” machine is on the same network as your home server you can be a bit more adventurous and do the following:

1. set GatewayPorts yes in your sshd_config

2. Set up the work-to-home connection, where home_ip is the IP of your home server on your internal network:

user@work:~$ while [ 1 ]; do ssh -NRhome_ip:15000:localhost:22 user@home.net; done

3. Set up the laptop-to-work connection:

user@laptop:~$ ssh -D15001 home_ip -p 15000

Passwordless authentication can be configured by setting up your ssh host keys correctly.

In both scenarios above, SOCKS-aware applications can be configured with server as “localhost” and port as “15001”. For non-SOCKS-aware applications, you can generally get away with using tsocks.

You’ll also notice that step (1) needs bootstrapping while you’re on the corporate network. Persuade someone to su to you, or do it while you’re in the office one day.

Generally you also want to reduce the possibility of your work-to-home connection failing, so run it in screen, or in a nohup script or use something like autossh or rstunnel to bring it back up for you.

Don’t forget you’ll also need to open appropriate holes in your home firewall, generally some sort of NAT, PAT, or DMZ settings to allow incoming SSH (TCP, port 22) to be forwarded to your home server.

Update 2010-06-30 17:57
It’s worth mentioning that if you don’t have a static IP on your home DSL line that you’ll need to use a dynamic DNS service (like DynDNS) to keep a static name for your dynamic IP. Personally I do other stuff with Linode so I’ve set something cool up using their web-service API.

Bookmarks for March 17th through April 12th

These are my links for March 17th through April 12th: