Back in February I wrote about an exciting project from the Cabinet Office: a complete overhaul of the UK government's "citizen-facing" Web sites. It was exciting in part because it was rather good, which made a nice change for a government computing project, but more particularly because it was open source through and through.
At the time, I gave a list of the main software it was using. Here's what the site is currently running/built with – there are quite a few additions:
HTML/CSS/JS – using HTML5 where appropriate, with a heavy focus on accessibility, and validating where we can
Video playback with the Nomensa Accessible Media Player
Backend admin systems make use of Twitter Bootstrap
We use SCSS, as seen in our frontend toolkit
We've worked with A2-Type for font production.
The core of the servers:
We're making use of Infrastructure As A Service from Skyscape
Our servers are running Ubuntu GNU/Linux 10.04, we're hoping to upgrade to 12.04 soon.
Servers are managed with Puppet, using PuppetDB
Web serving is handled by nginx, proxying to unicorn for our ruby applications. We're also using gunicorn to run some supporting services. One of the team wrote Unicorn Herder to make Unicorn play nicely with upstart.
We load balance internally with haproxy and cache requests using Varnish
nginx deserves an extra mention as it's letting us do all our redirection
we're using perl to manage and test our redirections
there's some php to add useful links to the "gone" pages where DirectGov and Businesslink content has been retired
node.js was used to build a side-by-side browser for reviewing the redirections
The majority of our applications are written in ruby, based on either Ruby on Rails or Sinatra.
A few components are written in Scala and built on top of Play 2.0
We're running Mapit from MySociety which is built on top of Django
Databases and other storage:
We use MongoDB for most systems, with a few apps also making use of MySQL. PostgreSQL is used by Mapit and Puppet.
Most search on the site is powered by Elasticsearch, though solr is currently the backend for the need-o-tron.
A few event-driven systems use RabbitMQ
Monitoring, managing and alerting:
We gather metrics from our apps with statsd
We collect logs with logstash
We monitor systems with ganglia
Graphite helps us make many, many graphs to understand what's going on
Nagios tells us if we need to act on any of that data
All our code is tested by Jenkins, which we also use to deploy it to servers
We track usage of the site with Google Analytics, using their API heavily to build dashboards
We occasionally use New Relic RPM for performance reviews
DNS is hosted by ja.net / Dyn
Email (internal alerts) sending via Amazon SES
Font handling and preparation with FontForge and FontTools
We keep on track and in touch using Google Apps, Pivotal Tracker and Campfire
Github helps us manage and discuss our code
Zendesk keeps the feedback flowing
We use jekyll & heroku for some of our prototyping
We've built all sorts of internal dashboards. They're very much our playground and you can find them written in a mixture of Ruby, Clojure, Node.JS, and PHP
That's a pretty impressive array. The big news is that GOV.UK has finally come out of beta, as Francis Maude explains in this blog post:
Today marks the start of a new way of delivering public services digitally. GOV.UK is a platform for future digital innovation. In stark contrast to the way IT has been delivered in government in the past, GOV.UK can rapidly accommodate new standards for development and security, catering to emerging technologies and user requirements quickly and effectively. It has been built the way Amazon built Amazon, and in the way that BA transformed their online business, by being agile, iterative and focused on users. GOV.UK has also been built using open source technology, which means we don't have to pay expensive software licensing costs.
It's great to see the open-source nature of the site highlighted – something that would have been unthinkable a few years ago.
There's an interesting reflection from one of the site's developers about some of open source's key advantages for this kind of project – reasons that apply not just to government projects, but commercial ones, too:
By choosing open source we can quickly try the tools that seem right, test them out, and make a decision about whether this is the right solution and start using the software straight away. A good example of that is the Elasticsearch search engine that Rob wrote about a few months back. From what we were reading it seemed like it'd be simpler for us to work with than solr, and we were able to dive in and try it out without a lengthy pre-sales process or costly and complex licensing arrangements. Instead, we test, we decide, we get on with tailoring our choice to best serve the user needs we're focussed on.
There are open source tools that cover most of what we're doing. We can analyse which are going to fit together in the ways we want, and where we should write our own code to make sure we're building the best product we can in a way we can maintain. One of my fellow GDS architects, Paul Downey, has often likened our approach to that advocated by JP Rangaswami in his "open source pyramid":
For common problems use Opensource.
For rare problems use Buy.
For unique problems use Build.
GOV.UK is already an extremely high-profile example of how open source can improve services and save money. And it looks like it will be built on immediately. Quoting Maude again:
The release today of GOV.UK, the new home of government services and information, proves that we can deliver digital public services at a fraction of the price of previous government IT projects. GOV.UK will cost taxpayers up to £70m less per year than the services it replaces, and substantial further savings will be achieved as more departments and agencies move on to the platform.
The details are still rather thin, but there's this:
Websites of all government departments and many other public bodies will be merged into the Inside Government section of www.gov.uk
This will happen in batches, starting soon and finishing in 2014.
Since the same platform will presumably be adopted for all those other departments, this effectively means that huge chunks of the UK government's computing infrastructure will be run on open source software. The more that happens, the more impressive the data point it represents, and the better the case for rolling it out even further.
Assuming that all goes smoothly (and there's no reason to suspect that it won't given the GOV.UK experience), we should congratulate Francis Maude and his team at Government Digital Service for achieving what a decade of government promises to use far more open source has failed to deliver.