Roger Pettett - psyphi.net blog

What can Bioinformatics learn from YouTube?

Caught Matt’s talk this morning at the weekly informatics group meetings

What can Bioinformaticians learn from YouTube?

There were general murmurings of agreement amongst the audience but nobody asking the probing questions I’d hope for as a measure of interestedness.

Matt touched upon microformats in all but name – I was really expecting a sell of http://bioformats.org/ , websites as APIs and RESTful web services in particular.

Whilst I’m inclined to agree that standardised, discoverable, reusable web services are largely the way forward (especially as it keeps me in work) I’m not wholly convinced they remove the problems associated with, for example, database connections, database-engine specific SQL, hostnames, ports, accounts etc.

My feeling is that all the problems associated with keeping track of your database credentials are replaced by a different set of problems, albeit more standardised in terms of network protocols in HTTP and REST/CRUD. We now run the risk that what’s fixed in terms of network protocols is pushed higher up the stack and manifests as myriad web services, all different. All these new websites and services use different XML structures and different URL schemes. The XML structures are analogous to database table schema and the URL schemes akin to table or object names.

At least these entities are now discoverable by the end user/developer simply by using the web application – and there’s the big win – transparency and discoverability. There’s also the whole microformat affair – once these really start to take off there’ll be all sorts of arguments about what goes into them, especially in domains like Bio and Chem, not covered by core formats like hCard. But that’s something for another day.

More over at Green Is Good

7 utilities for improving application quality in Perl

I’d like to share with you a list of what are probably my top utilities for improving code quality (style, documentation, testing) with a largely Perl flavour. In loosely important-but-dull to exciting-and-weird order…

Test::More. Billed as yet another framework for writing test scripts Test::More extends Test::Simple and provides a bunch of more useful methods beyond Simple’s ok(). The ones I use most being use_ok() for testing compilation, is() for testing equality and like() for testing similarity with regexes.

ExtUtils::MakeMaker. Another one of Mike Schwern’s babies, MakeMaker is used to set up a folder structure and associated ‘make’ paraphernalia when first embarking on writing a module or application. Although developers these days tend to favour Module::Build over MakeMaker I prefer it for some reason (probably fear of change) and still make regular mileage using it.

Test::Pod::Coverage – what a great module! Check how good your documentation coverage is with respect to the code. No just a subroutine header won’t do! I tend to use Test::Pod::Coverage as part of…

Test::Distribution . Automatically run a battery of standard tests including pod coverage, manifest integrity, straight compilation and a load of other important things.

perlcritic, Test::Perl::Critic . The Perl::Critic set of tools is amazing. It’s built on PPI and implements the Perl Best Practices book by Damien Conway. Now I realise that not everyone agrees with a lot of what Damien says but the point is that it represents a standard to work to (and it’s not that bad once you’re used to it). Since I discovered perlcritic I’ve been developing all my code as close to perlcritic -1 (the most severe) as I can. It’s almost instantly made my applications more readable through systematic appearance and made faults easier to spot even before Test::Perl::Critic comes in.

Devel::Cover. I’m almost ashamed to say I only discovered this last week after dipping into Ian Langworthy and chromatic’s book ‘Perl Testing’. Devel::Cover gives code exercise metrics, i.e. how much of your module or application was actually executed by that test. It collates stats from all modules matching a user-specified pattern and dumps them out in a natty coloured table, very suitable for tying into your CI system.

Selenium . Ok, not strictly speaking a tool I’m using right this minute but it’s next on my list of integration tools. Selenium is a non-interactive, automated, browser-testing framework written in Javascript. This tool definitely has legs and it seems to have come a long way since I first found it in the middle of 2006. I’m hoping to have automated interface testing up and running before the end of the year as part of the Perl CI system I’m planning on putting together for the new sequencing pipeline.

Hiring Perl Developers – how hard can it be?

All the roles I’ve had during my time at Sanger have more or less required the development of production quality Perl code, usually OO and increasingly using MVC patterns. Why is it then that very nearly every Perl developer I’ve interviewed in the past 8 years is woefully lacking, specifically in OO Perl but more generally in half-decent programming skills?

It’s been astonishing, not in a good way, how many have been unable to demonstrate use of hashes. Some have been too scared of them (their words, not mine) and some have never felt the need. For those of you who aren’t Perl programmers, hashes (aka associative arrays) are a pretty crucial feature of the language and fundamental to its OO implementation.

Now I program in Perl sometimes more than 7-8 hours a day. For many years this also involved reworking other people’s code. I can very easily say that if you claim to be a Perl programmer and have never used hashes then you’re not going to get a Perl-related job because of your technical skills. With a good, interactive and engaging personality and a desire for self-improvement you might get away with it, but certainly not on technical merit.

It’s also quite worrying how many of these interviewees are unable to describe the basics of object-oriented programming yet have, for example, developed and sold a commercial ERP system, presumably for big bucks. Man, these people must have awesome marketing!

Frankly a number of the bioinformaticians already working there have similar skills to the interviewees and often worse communication skills, so maybe I’m simply setting my standards too high.

I really hope this situation improves when Perl 6 goes public though I’m sure it’ll take longer to become common parlance. As long as it happens before those smug RoR types take over the world I’ll be happy ;)

DECIPHERing Large-scale Copy-Number Variations

It’s strange.. Since moving from the core Web Team at Sanger to Sequencing Informatics I’ve been able to reduce my working hours from ~70-80/week all the way down to the 48.5 hours which are actually in my contract.

In theory this means I’ve more spare time, but in reality I’ve been able to secure sensible contract work outside Rentacoder which I’ve relied on in the past.

The work in question is optimising and refactoring for the DECIPHER project which I used to manage the technical side of whilst in the web team.

DECIPHER is a database of large-scale copy number variations (CNVs) from patient arrayCGH data curated by clinicians and cytogeneticists around the world. DECIPHER represents one of the first clinical applications to come out of the HGP data from Sanger.

What’s exciting apart from the medical implications of DECIPHER’s joined-up thinking is that it also represents a valuable model for social, clinical applications in the Web 2.0 world. The application draws in data from various external sources as well as its own curated database. It primarily uses DAS via Bio::Das::Lite and Bio::Das::ProServer and I’m now working on improving interfaces, interactivity and speed by leveraging MVC and SOA techniques with ClearPress and Prototype.

It’s a great opportunity for me to keep contributing to one of my favourite projects and hopefully implement a load of really neat features I’ve wanted to add for a long time. Stay tuned…

VoIP peering & profits

So… shortly, I believe from February next year but am probably mistaken, prices in the UK go up for calling “Lo-Call” 0845 numbers. As I understand it they’ll be similar, or the same as 0870 rates at 20p/min or so.

Now I wonder if the regulator has missed a trick here. It so happens that the nation is converting to broadband, be it ADSL or cable-based, and that very many of those broadband packages now come with VoIP offerings as standard.

My point is that these bundled broadband VoIP packages invariably come with 0845 dial-in numbers and no other choice. Dialing out via your broadband ISP may well be cheap for you but spare a thought for those calling in at much higher rates.

Having been tinkering with VoIP for a good few years I realise that actually this should be ok because calling VoIP-to-VoIP should be free, right? Wrong. Most of these ISPs don’t peer to each others’ networks – for two main reasons as far as I can see –

They’re competitors and have little business reason to peer, apart from keeping the small proportion of aware customers happy.
These ISPs make profits from users dialing in – 0845 is a profit-sharing prefix with which both BT and the ISP in question have a stake. This old story is of course also true of many telephone help-desks and similar. Keeping the customer on the line longer means more profits for the company and its shareholders.

It seems to me that the world could be a better, more communicative place through more thorough VoIP network peering but I simply can’t see it becoming widespread whilst profits stand in the way.

The Simplest of Organisation

Ever since I started implementing SCRUM for my application development at work friends of mine have expressed an interest in the way it works.

Recently even people passing through my office – there talking to my colleagues and who I don’t know very well – have been remarking on the backlogs which are displayed in a prominent position above my desk. I think they’re impressed by the simplicity of the system and how effective it seems to be for me.

I must admit my backlogs are simpler than the full blown setup. As I’m still in the process of hiring, I currently only really develop alone so I’m not bothering with the intermediate item-in-progress stickies.

I also have tasks organised in a 2-dimensional area with axes for complexity and importance. Although sprint backlog tasks are prioritised by my customers, it’s been proving useful to have my take on these attributes displayed spatially rather than just writing ‘3 days’ on the ticket.

In fact I keep my product backlog organised this way as well, as soon as tickets come in. It allows me to relay my take on the tasks to the customers straight away, whether or not we’re building a sprint backlog at the time. When a sprint has finished the product backlog is reorganised to take account of any changes, e.g. to infrastructure, affecting the tasks.

Picking up momentum

It seems people are fairly taken with the BarCamb idea. It’s been lightly advertised internally at Sanger and has been picking up some interest via that and also on the upcoming page .

I wonder how many of the people already signed up actually have something to present. Having been at the WTSI for nearly eight years now I’ve a number of things I could talk about, it’s just a case of deciding which of them would be more interesting for people and that really depends on where attendees are coming from.

So… one or more of the following, of the things I’ve been working on recently – Bio::Das::Lite & Bio::Das::ProServer, ClearPress or the new sequencing technology. Now I’m not a biologist or a chemist either by trade or by hobby and I’m pretty certain that talking about NST is going to be asking for a whole bunch of biology and chemistry-question trouble. I guess DAS-related things are the most useful to present as they have the widest scientific application.

Though there’s nothing like a good bit of self-promotion so maybe something short on ClearPress would be a good thing too. Might need to improve the application builder and test-suite a bit more for that.

In related news, not wanting to be outdone by Matt’s BarCamb I coauthored and submitted a venue proposal for YAPC::Europe 2008 last week. Woohoo! Nail-biting stuff. The genome campus would be a great place to host it for all sorts of reasons – integrated and well supported conference centre; secured financial committment; great science to talk about and a tremendous perl resource to tap into just to list a few.

All I need to do now is submit my travel application for YAPC::Europe Vienna later this year and see how it’s done (again). It’s been a while since I’ve been to a YAPC::Europe!

Barcamp Cambridge

So… BarCamp Cambridge, or BarCamb as we’re affectionately calling it is definitely green for go.

To be hosted at the Wellcome Trust Sanger Institute near Cambridge it’s hopefully going to be a day of grass-roots science and technology talks on the 24th of August. That’s two months away last Sunday so plenty of time to unorganise it.

Should be interesting and I think I’m looking forward to it though I’m not sure what to expect. It could, of course, be an utter disaster, but what better area to have it than Cambridge, and what better site than the Genome Campus, however biased I might be?

I always dread saying this, but “more coming soon” I hope!

Sporting Developments

I recently started reading Agile Software Development with Scrum by Schwaber and Beedle. It’s a great introduction to this branch of the Agile movement. It’s easy to read and contains practical advice and straight-forward explainations of the terms and processes involved with Scrum.

Even more satisfying than the read itself was the realisation that I’ve been using a good number of the Scrum techniques in managing projects within my team for the last three years or so. I love the idea of a development team reaching a nirvana-like hyper-productive state though one of the examples of a four-person team at Quattro producing 1000 lines of C++ a week took me aback.

In the middle of last month I moved to a new position at WTSI, Team Leader for the New Sequencing Pipeline development team (currently consisting of me). Since then I’ve been working on what I’ll now call a code sprint and last week I had my first product increment. The product is a smallish system for tracking runs on the new technology sequencing machines but is around 10,000 lines of Perl (excluding templates, CSS & tests) built on a light MVC framework I produced in the same time. A one man-team producing 3,333 loc in a week seems ultra-productive and I can’t believe it’s *purely* down to the fact that Perl is easier to write than C++.

Anyway, I’m on a C++ course all next week, so I’ll soon be able to tell. Shame it’s not about Rails instead ;)

Hosting, advertising, content

Well more than a year has passed and it’s just plain embarrassing having not written anything here. So much has happened the choice is either to update or delete. I’ll try updating…

So… I’ve switched ISP again. This site is now free from shackles. I’ve delegated the DNS over to http://afraid.org/ which is updated by web-client from one of the psyphi.net servers. Everything you now see is running over my home ADSL line. It’s not too bad but could seriously do with better upload bandwidth. This new setup now allows me full control over everything server-wise and as much storage as I can eat. Hopefully it saves a small bucket of cash too.

I’ve switched advertising from AdBrite to AdSense which seems to be a little more effective. If and when it starts generating a few cents revenue it may even pay for the domain and the meagre subscription to afraid.org .

It mostly boils down to content in the end and yes, I know there is very little around here. Of course I’d like to change that little by little but I won’t promise anything because then it definitely won’t happen. Steve Pavlina reckons he generates $9000/month with his productivity site and whilst I know I won’t reach that for some time it’s still fairly inspiring. There are still a couple of months left to make my million before the deadline.