7 utilities for improving application quality in Perl

I’d like to share with you a list of what are probably my top utilities for improving code quality (style, documentation, testing) with a largely Perl flavour. In loosely important-but-dull to exciting-and-weird order…

Test::More. Billed as yet another framework for writing test scripts Test::More extends Test::Simple and provides a bunch of more useful methods beyond Simple’s ok(). The ones I use most being use_ok() for testing compilation, is() for testing equality and like() for testing similarity with regexes.

ExtUtils::MakeMaker. Another one of Mike Schwern’s babies, MakeMaker is used to set up a folder structure and associated ‘make’ paraphernalia when first embarking on writing a module or application. Although developers these days tend to favour Module::Build over MakeMaker I prefer it for some reason (probably fear of change) and still make regular mileage using it.

Test::Pod::Coverage – what a great module! Check how good your documentation coverage is with respect to the code. No just a subroutine header won’t do! I tend to use Test::Pod::Coverage as part of…

Test::Distribution . Automatically run a battery of standard tests including pod coverage, manifest integrity, straight compilation and a load of other important things.

perlcritic, Test::Perl::Critic . The Perl::Critic set of tools is amazing. It’s built on PPI and implements the Perl Best Practices book by Damien Conway. Now I realise that not everyone agrees with a lot of what Damien says but the point is that it represents a standard to work to (and it’s not that bad once you’re used to it). Since I discovered perlcritic I’ve been developing all my code as close to perlcritic -1 (the most severe) as I can. It’s almost instantly made my applications more readable through systematic appearance and made faults easier to spot even before Test::Perl::Critic comes in.

Devel::Cover. I’m almost ashamed to say I only discovered this last week after dipping into Ian Langworthy and chromatic’s book ‘Perl Testing’. Devel::Cover gives code exercise metrics, i.e. how much of your module or application was actually executed by that test. It collates stats from all modules matching a user-specified pattern and dumps them out in a natty coloured table, very suitable for tying into your CI system.

Selenium . Ok, not strictly speaking a tool I’m using right this minute but it’s next on my list of integration tools. Selenium is a non-interactive, automated, browser-testing framework written in Javascript. This tool definitely has legs and it seems to have come a long way since I first found it in the middle of 2006. I’m hoping to have automated interface testing up and running before the end of the year as part of the Perl CI system I’m planning on putting together for the new sequencing pipeline.

Hiring Perl Developers – how hard can it be?

All the roles I’ve had during my time at Sanger have more or less required the development of production quality Perl code, usually OO and increasingly using MVC patterns. Why is it then that very nearly every Perl developer I’ve interviewed in the past 8 years is woefully lacking, specifically in OO Perl but more generally in half-decent programming skills?

It’s been astonishing, not in a good way, how many have been unable to demonstrate use of hashes. Some have been too scared of them (their words, not mine) and some have never felt the need. For those of you who aren’t Perl programmers, hashes (aka associative arrays) are a pretty crucial feature of the language and fundamental to its OO implementation.

Now I program in Perl sometimes more than 7-8 hours a day. For many years this also involved reworking other people’s code. I can very easily say that if you claim to be a Perl programmer and have never used hashes then you’re not going to get a Perl-related job because of your technical skills. With a good, interactive and engaging personality and a desire for self-improvement you might get away with it, but certainly not on technical merit.

It’s also quite worrying how many of these interviewees are unable to describe the basics of object-oriented programming yet have, for example, developed and sold a commercial ERP system, presumably for big bucks. Man, these people must have awesome marketing!

Frankly a number of the bioinformaticians already working there have similar skills to the interviewees and often worse communication skills, so maybe I’m simply setting my standards too high.

I really hope this situation improves when Perl 6 goes public though I’m sure it’ll take longer to become common parlance. As long as it happens before those smug RoR types take over the world I’ll be happy ;)

DECIPHERing Large-scale Copy-Number Variations

It’s strange.. Since moving from the core Web Team at Sanger to Sequencing Informatics I’ve been able to reduce my working hours from ~70-80/week all the way down to the 48.5 hours which are actually in my contract.

In theory this means I’ve more spare time, but in reality I’ve been able to secure sensible contract work outside Rentacoder which I’ve relied on in the past.

The work in question is optimising and refactoring for the DECIPHER project which I used to manage the technical side of whilst in the web team.

DECIPHER is a database of large-scale copy number variations (CNVs) from patient arrayCGH data curated by clinicians and cytogeneticists around the world. DECIPHER represents one of the first clinical applications to come out of the HGP data from Sanger.

What’s exciting apart from the medical implications of DECIPHER’s joined-up thinking is that it also represents a valuable model for social, clinical applications in the Web 2.0 world. The application draws in data from various external sources as well as its own curated database. It primarily uses DAS via Bio::Das::Lite and Bio::Das::ProServer and I’m now working on improving interfaces, interactivity and speed by leveraging MVC and SOA techniques with ClearPress and Prototype.

It’s a great opportunity for me to keep contributing to one of my favourite projects and hopefully implement a load of really neat features I’ve wanted to add for a long time. Stay tuned…

VoIP peering & profits

So… shortly, I believe from February next year but am probably mistaken, prices in the UK go up for calling “Lo-Call” 0845 numbers. As I understand it they’ll be similar, or the same as 0870 rates at 20p/min or so.

Now I wonder if the regulator has missed a trick here. It so happens that the nation is converting to broadband, be it ADSL or cable-based, and that very many of those broadband packages now come with VoIP offerings as standard.

My point is that these bundled broadband VoIP packages invariably come with 0845 dial-in numbers and no other choice. Dialing out via your broadband ISP may well be cheap for you but spare a thought for those calling in at much higher rates.

Having been tinkering with VoIP for a good few years I realise that actually this should be ok because calling VoIP-to-VoIP should be free, right? Wrong. Most of these ISPs don’t peer to each others’ networks – for two main reasons as far as I can see –

  1. They’re competitors and have little business reason to peer, apart from keeping the small proportion of aware customers happy.
  2. These ISPs make profits from users dialing in – 0845 is a profit-sharing prefix with which both BT and the ISP in question have a stake. This old story is of course also true of many telephone help-desks and similar. Keeping the customer on the line longer means more profits for the company and its shareholders.

It seems to me that the world could be a better, more communicative place through more thorough VoIP network peering but I simply can’t see it becoming widespread whilst profits stand in the way.

The Simplest of Organisation

Ever since I started implementing SCRUM for my application development at work friends of mine have expressed an interest in the way it works.

Recently even people passing through my office – there talking to my colleagues and who I don’t know very well – have been remarking on the backlogs which are displayed in a prominent position above my desk. I think they’re impressed by the simplicity of the system and how effective it seems to be for me.

I must admit my backlogs are simpler than the full blown setup. As I’m still in the process of hiring, I currently only really develop alone so I’m not bothering with the intermediate item-in-progress stickies.

I also have tasks organised in a 2-dimensional area with axes for complexity and importance. Although sprint backlog tasks are prioritised by my customers, it’s been proving useful to have my take on these attributes displayed spatially rather than just writing ‘3 days’ on the ticket.

In fact I keep my product backlog organised this way as well, as soon as tickets come in. It allows me to relay my take on the tasks to the customers straight away, whether or not we’re building a sprint backlog at the time. When a sprint has finished the product backlog is reorganised to take account of any changes, e.g. to infrastructure, affecting the tasks.

Picking up momentum

It seems people are fairly taken with the BarCamb idea. It’s been lightly advertised internally at Sanger and has been picking up some interest via that and also on the upcoming page .

I wonder how many of the people already signed up actually have something to present. Having been at the WTSI for nearly eight years now I’ve a number of things I could talk about, it’s just a case of deciding which of them would be more interesting for people and that really depends on where attendees are coming from.

So… one or more of the following, of the things I’ve been working on recently – Bio::Das::Lite & Bio::Das::ProServer, ClearPress or the new sequencing technology. Now I’m not a biologist or a chemist either by trade or by hobby and I’m pretty certain that talking about NST is going to be asking for a whole bunch of biology and chemistry-question trouble. I guess DAS-related things are the most useful to present as they have the widest scientific application.

Though there’s nothing like a good bit of self-promotion so maybe something short on ClearPress would be a good thing too. Might need to improve the application builder and test-suite a bit more for that.

In related news, not wanting to be outdone by Matt’s BarCamb I coauthored and submitted a venue proposal for YAPC::Europe 2008 last week. Woohoo! Nail-biting stuff. The genome campus would be a great place to host it for all sorts of reasons – integrated and well supported conference centre; secured financial committment; great science to talk about and a tremendous perl resource to tap into just to list a few.

All I need to do now is submit my travel application for YAPC::Europe Vienna later this year and see how it’s done (again). It’s been a while since I’ve been to a YAPC::Europe!

Barcamp Cambridge

So… BarCamp Cambridge, or BarCamb as we’re affectionately calling it is definitely green for go.

To be hosted at the Wellcome Trust Sanger Institute near Cambridge it’s hopefully going to be a day of grass-roots science and technology talks on the 24th of August. That’s two months away last Sunday so plenty of time to unorganise it.

Should be interesting and I think I’m looking forward to it though I’m not sure what to expect. It could, of course, be an utter disaster, but what better area to have it than Cambridge, and what better site than the Genome Campus, however biased I might be?

I always dread saying this, but “more coming soon” I hope!

Sporting Developments

I recently started reading Agile Software Development with Scrum by Schwaber and Beedle. It’s a great introduction to this branch of the Agile movement. It’s easy to read and contains practical advice and straight-forward explainations of the terms and processes involved with Scrum.

Even more satisfying than the read itself was the realisation that I’ve been using a good number of the Scrum techniques in managing projects within my team for the last three years or so. I love the idea of a development team reaching a nirvana-like hyper-productive state though one of the examples of a four-person team at Quattro producing 1000 lines of C++ a week took me aback.

In the middle of last month I moved to a new position at WTSI, Team Leader for the New Sequencing Pipeline development team (currently consisting of me). Since then I’ve been working on what I’ll now call a code sprint and last week I had my first product increment. The product is a smallish system for tracking runs on the new technology sequencing machines but is around 10,000 lines of Perl (excluding templates, CSS & tests) built on a light MVC framework I produced in the same time. A one man-team producing 3,333 loc in a week seems ultra-productive and I can’t believe it’s *purely* down to the fact that Perl is easier to write than C++.

Anyway, I’m on a C++ course all next week, so I’ll soon be able to tell. Shame it’s not about Rails instead ;)

Hosting, advertising, content

Well more than a year has passed and it’s just plain embarrassing having not written anything here. So much has happened the choice is either to update or delete. I’ll try updating…

So… I’ve switched ISP again. This site is now free from shackles. I’ve delegated the DNS over to http://afraid.org/ which is updated by web-client from one of the psyphi.net servers. Everything you now see is running over my home ADSL line. It’s not too bad but could seriously do with better upload bandwidth. This new setup now allows me full control over everything server-wise and as much storage as I can eat. Hopefully it saves a small bucket of cash too.

I’ve switched advertising from AdBrite to AdSense which seems to be a little more effective. If and when it starts generating a few cents revenue it may even pay for the domain and the meagre subscription to afraid.org .

It mostly boils down to content in the end and yes, I know there is very little around here. Of course I’d like to change that little by little but I won’t promise anything because then it definitely won’t happen. Steve Pavlina reckons he generates $9000/month with his productivity site and whilst I know I won’t reach that for some time it’s still fairly inspiring. There are still a couple of months left to make my million before the deadline.

Going Green / The Iceman Cometh

Well it’s turning cold again. Doesn’t say too much for this country but we’ve definitely seen the last of summer. The fog’s setting in and it won’t be long before I’m scraping the ice off my car in the mornings.

So it’s cold, and I want myself, my family and my house to be comfortably warm as cheaply as possible. I’ve always tried to be fairly green but it turns out it’s not particularly cheap in terms of initial outlay although it tends to pay for itself in the longer term.

It turns out of course that it’s not just more heat but also more light you need come winter. So when we moved into the new house I set about replacing the lightbulbs with the disproportionately-priced energy-saving lightbulbs (the folded fluorescent-tube variety). Unfortunately the kitchen and bathroom are each fitted with a series of 50-watt mains-voltage spotlights, GU10 if memory serves. Energy-saving (7-watt) equivalents are nigh-on £10 each from Homebase , and not particularly common, either.

That’s really not a particularly enticing price when simply compared to the full-on regular bulbs for well under £2 each. But, like I say long-term they will pay for themselves pretty quickly, depending on how much you pay your power provider.

So switching out these bulbs got me thinking about how to cut out the electricity supplier all together. Where my place is located, it’s not directly near any sources of water or constant wind (of the renewable-energy-source variety). I figured solar panels were my only option. Not bad when you find out you can have some of the installation paid for by the government, but not so great when you find out that the total cost of an average installation is of the order of £10,000. Hmm – we’ll be saving up for a while!

Well, then I widened my criteria – a lot of my searching for energy efficiency is actually driven by the need to lower expenses and outgoings. Obviously finding the lowest-cost service provider is a good start – I tried out Uswitch to move from Powergen to EDF (but not before I’;d claimed my free energy-saving lightbulbs as a “new” Powergen customer!).

So I realised a large chunk of my outgoings are car-related. With the current, wallet-shattering prices of diesel it’s a good place to look for economies. Biodiesel definitely seems to be the way forward. From a little scouting around I see that there are two common types of useful fuel, 100% biodiesel and 95/5% petroleum diesel and biodiesel mix. A little more digging and you can find lists of cars and which approved mix their warranties allow. Sadly it was at this point I found my Peugeot 306 seems to only accept the 95/5 mix (I forget which standard code this is) so it’ll only save me 5 pence in the pound. Next time I change my car I’ll be sure to make this one of my criteria.

Of course there’s a thriving cottage industry in recycling frying oil from vans, chip shops and the like. Making your own biodiesel seems to be relatively easy given a simply chemistry kit and the right license. Only a little more surfing and it turns out one pretty neat thing to do is to couple the electricity supply problem with the biodiesel phenomenon. Buy a little generator and either run it straight on home-brew biodiesel or modify it to convert plain old vegetable oil on-the-fly. Fantastic idea but my wife doesn’t want the “pleasant aroma of french-fries or popcorn” in a cloud sat in the back-garden. We’re sufficiently close to the neighbours that I suspect it wouldn’t go down too well with them either.

Oh well, back to the drawing-board.

Back, at least to the energy saving… The right windows and doors are obviously important and that’s next on the big list of home-improvements. I think it’ll wait until next summer or so, when we’ve saved up quite a few pennies. I reckon we should be able to fit the place out (a moderate-sized 3-bedroom semi-detached) for around £4000-£4500 . Not insignificant, but again worth it in the long term.

Until then I’ll have to make do with simply replacing the loft-insulation. It’s gone from around 1cm thick ancient rubbish to shiny, new, pink, 17cm thick over the half I’ve done. The only problem now is I’d like to add loft floorboards without popping out the plasterboard ceilings below. Doh! Maybe I’ll go with 10cm thick stuff for the rest. It’s all currently 3 for the price of 2 rolls, again from Homebase.