A hobbyist hacker, spreadsheet warrior & an API advocate walked in to a pub…

..and they all tell the bar staff they can’t pour a pint.

That is basically the life of being a data publisher at the moment.

Leigh has written a great post about ‘Who is the intended audience for open data?’ where he articulates some of the problems and clashes of cultures better than I could but I just want to pick up on a couple of things.

For the purpose of this post I have magicked up three new ‘personas’ –

The hobbyist hacker — someone usually extremely technical, advocates (hard) for machine-readable open data, gives up a lot of their own free time to support hackdays and volunteer projects, has particular passions about how this stuff should be done and whose day job is in a parallel field (i.e. they work in software development or similar but not somewhere actually providing the data.) Just to be clear the hobbyist bit isn’t supposed to sound dismissive — I just like the way ‘hobbyist hacker’ scans 🙂

Like Leigh mentions this group can be pretty hard on any efforts to improve data release that is not focused on the needs of machines. Basically they want it to be (much easier) to programmatically query, aggregate and visualise the data (from multiple publishers). I absolutely understand where they are coming from, agree with the needs and have a lot of the same objectives in my role. The problem is that this isn’t the whole story.

The vast majority of ONS data users would fall in to the next entirely made up ‘persona’ — the spreadsheet warrior. These people work with ONS data day in and day out. Their weapon of choice though is not Python or even R — it is Excel. They are the audience the GSS (and FullFact) spreadsheet guidance sought to support. They need ‘human readable’ spreadsheets. The Venn diagram of needs between persona one and two would have more than its share of overlap but they are not the same. To be clear most data publishers (us included for sure) don’t do this well enough and if we are looking at a hierarchy of user needs (which I am not as I am just babbling away on a weekend) then providing the best possible experience here would be where we would get the biggest win (well assuming we made the data findable but that is another post).

The third, and final for the purposes of this, persona is the API advocate. This group — again for the purposes of this — tend to be somewhere between the two previous camps. They are often in-house technical staff whose role it is to support organisations who have to use the data day in and day out but they themselves are not always the end user. They share a lot of digital DNA with the hackers but the core difference is that they are trying to find a way to improve tools and processes at their day job. They see APIs (and open data more widely) as an opportunity to improve things but they need them to be documented, supported and stable as they want to integrate them in to, often vital, business systems. They are a growing, but still relatively small, group and one that really needs cultivating. My feeling is that finding a way to help these users will create an environment that improves the offerings to everyone.

Leigh ends his post with the advice

Publish for machines, but don’t forget the humans. All of the humans.

I’d just like to reorder that a little

Publish for humans, all of the humans. But don’t forget the machines.

Going viral: the digitalisation of our iconic institutions

Like most people of a certain vintage in a digital-ish profession I went through a stage where Techcrunch was my bible and I desperately wanted to be a part of a successful startup. In the end I spent a year at a [not] successful example and learned loads — not least that it wasn’t the world for me.

Over time I realised that what I really enjoyed was helping rather staid organisations embrace these new internet age opportunities. I liked the idea of being the digital patient zero — infecting organisations with virulent strains of transparency, open source, agility and ownership. None of it was rocket science and a lot of it was just attitude but amongst the frustrations there is a lot of fun to be had.

Clearly things have changed at GDS but I will forever be thankful for their work in infecting the whole blooming Civil Service. There are amazing digital teams now at DWP, Home Office, Ministry of Justice, HMRC, DVLA, Companies House, Land Registry and every other corner of Government. Hell even we punch above our weight in Newport.

Now it is fun to watch this contagion spread even further afield and how it effects these big, iconic organisations.

The new Parliament Digital Service are starting to do amazing work — hell they are running internal unconferences within the House of Lords! They have empowered some great advisors and are building a brilliant team. It is exciting to watch.

Like many, many people I love the NHS and everything it stands for and over the years it has served me well on a number of occasions (I am prone to breaking bones and my kidneys have been on a work to rule for 30 years) so the NHS Alpha project is something that is great to see. They are a small team working in an insanely complex space with an enormous amount to tackle but they have brilliant, passionate people and a sensible approach so it is worth keeping an eye on.

I mentioned on Twitter a couple of times recently it also seems like ‘retail is the new government’ as both the Co-operative Group — with their ‘lift and shift’ of the GDS leadership — and also Marks and Spencer with Meri Williams building #notjustanyteam (and picking up the odd ex GDS colleague on the way!) making serious moves towards becoming digital-first organisations. The Co-op and M&S? Is there anyone more iconic in the UK 🙂

Sure places like the BBC and the Guardian have been doing it longer — and their alumni have had more than their share of influence on all of this activity but there is something of a real goldrush going on now (which means it is going to get even harder to recruit I imagine!)

So what will be the next big institution to catch the digital transformation sniffles?

I have to say reading this brilliant piece about the Bank of England I did think what an amazing and satisfying challenge that would be to be part of a change there.

Who else?

I tried hard to think of someone in the Bristol area in this sort of company but I really couldn’t 😦

Little Rice. Little Read. Large Revelations.

41whesm6SwL._SX331_BO1,204,203,200_Just finished reading Little Rice the latest book by Clay Shirky after a recommendation from Russell Davies.

First up I have to say training my brain to read Xiaomi as ‘show (but like shower) – me’ has been less than easy and it is safe to safe I will not be conversing in Mandarin (or Cantonese) anytime soon.

It is a short read – more a long report than a book (not that it effected the price!) but was perfect commute fodder and like some of the Seth Godin books in the past I rather appreciated the brevity rather than stretching an idea in search of a word count (Chris Anderson I’m looking at you.)

Xiaomi is the hot mobile phone manufacturer (and software provider) in China at the moment and the book is basically a view of a number of social and economic trends through the lens of their rise. The Chinese economy and the tension with their political environment, perceptions of malfeasance via Chinese based software services, the Shenzhen manufacturing ethos and their cheap clones of successful electronics (especially phones) but also a company (Xiaomi) that exemplifies the spirit of the ‘Lean Startup‘ concept.

A laser focus on user needs, taking every opportunity to gather feedback and engage with users. The split of their users into essentially two personas – ‘fever fans’ who are early adopter, power users who provide detailed technical feedback and who are treated almost as part of the company and ‘flood fans’ who are more traditional consumers whose feedback is more ’emotional’ but provides insight in to trends and opportunities – is really interesting and clearly they spend a huge amount of time cultivating these groups (they avoid traditional marketing and prefer to focus on this networked marketing instead.) Constant (weekly) software upgrades and iterations – again always working to make sure that these are user driven. Focusing on online sales only and offering pre-sale signups (to judge demand rather than manufacturing on spec) and flash-sales (to create buzz and again stimulate demand). It is a hardware company operating like the best of Silicon Valley start-ups (from central Beijing) and in only five years it has risen to a dominant position in the local (huge) Chinese market – the challenge will be whether it makes the leap to global player or is held back by the requirements of its governments and the concerns of others.

Really enjoyed reading it and I learned quite a bit about both the mobile phone industry and modern China.

The challenge of ‘user driven development’


It has been 18 months now since we officially started the project to build a new ONS website – the most consistent thing throughout our initial discovery, the alpha and now the latter days of the beta has been a commitment to the principle of being ‘user focused’.

Over the life of the project I have taken to [mis]using the term ‘user driven development’ for the approach we have endeavoured to take on. At every stage of the project we have had at least one user researcher dedicated to our work and have regularly undertaken the full spectrum of research activities from lab based usability testing through to unmoderated online card sorts via critical friend interviews and guerrilla testing.

From the start we followed the advice from GDS (nicely summed up in this blogpost from Leisa) about how to integrate user researchers and their findings and priorities with an agile development team and despite a few hiccups it pretty much worked well for us throughout the alpha. The whole concept of ‘user research is a team sport’ was really something we embraced and it had a lot to do with the success of that phase I think.

As the project became considerably more complex during the beta – as we moved from producing a prototype to something production ready with all the additional requirements that brings – it became less straightforward to integrate the insights from the user research as quickly into the development cycle as we’d like.

Now we have never let these difficulties from stopping us doing what we believe is the right thing – and being user focused really is how we define ourselves as a team – but we have finite capacity and are a small team. The juggling act between different priorities and compromises to ensure a constantly improving product (including the back-end) is getting to a ‘Britain’s Got Talent’ level. Pressing issues identified via user research still always get prioritised but the backlog of nice to haves/fix has grown (though if issues re-occur then of course they get re-evaluated.)

I recognised a lot of our challenges in this article by Chris Gray particularly the challenge of participant recruitment and getting support systems in place. We are lucky to have a really engaged audience and especially our online research activities are extremely well supported but participant recruitment for particular slots on particular days is often hard work and time consuming (Alison in the team has been a trooper with this recently!) Especially as the personas we need to recruit for our slightly more specialist perhaps than some other projects.

After a year of managing our participant volunteer list via an increasingly complex spreadsheet (we are ONS – we do things in Excel!) we have just started using a proper contact management / CRM tool which looks like it will make a real difference. The ability to add custom fields, tags and segment entries is going to be a godsend longterm.

The ‘Fieldwork Fridays’ approach that is mentioned in the previous article as being used at Google is probably closest to our model – we schedule research opportunities and then poor Jonathan (our user research lead) works out what is the most pressing research need at the time. We have started to supplement this with more online testing – including some Optimal Workshop products but also Usability Hub and Loop11 as we try to broaden our reach in the run up to the end of the project.

I also found this post about ‘Incremental UX’ interesting recently – again I recognise a lot of the problems it is trying to address and I think without really formally realising it we have been working in a quite similar way for a while now. Starting by providing the ‘minimal positive experience’ to our users as and when we change the UI or add features but always aware of the longer term aims and using the research to keep checking that those aims remain valid.

Like I said about working agile in a previous blogpost ‘user driven development’ might be the right thing but it definitely isn’t the easy thing.

Don’t do Agile. Be agile. Or something.


I’ve read a couple of great posts about ‘agile’ in the last week or so – it is like 2008 all over again!

First Paul Downey strayed from the topic of lists & registers to contemplate ‘scope creep’ and then Nik Silver wrote about ‘getting partial value from partial delivery‘. Both posts eloquently cover the issue of bringing an agile approach and mindset into organisations or teams who remain uncomfortable without the security blanket(s) of a 50 page functional requirements document and a folder full of Gantt charts.

I’m a big exponent of agile working – like so many in the public sector I am a convert from the world of Prince2 and MoSCoW so in the past I have had that been pretty militant about it. There is no evangelist like the born again after all. Over the years I hope I take a more even handed approach and I have certainly become more inspired by the original Agile Manifesto than any particular methodology.

Organisationally we are making real efforts to move to a more ‘agile’ way of working but I think a lot of people are still uncomfortable with it and see it as a somewhat lazy approach – while in fact I often find I’m working harder than I ever did as a project manager back in the day (admittedly I was a HORRIBLE project manager). I find working in an agile way incredibly helpful and useful but it is not always straightforward and is certainly not an easy option.

I see the rituals and artefacts of the Agile methodologies as a ‘pick ‘n’ mix’ although over time our bag of tricks has become pretty Scrum-like/lite.

Things that really work for us:

– Daily stand-ups
– Retrospectives
– Show & tells
– Kanban walls
– Visualising everything, everywhere
– User stories*
– Sprint planning/prioritising
– Short sprints
– Regular deployments
– Fast feedback (from real users)
– Pair programming

Things we have struggled with or don’t do:

– Estimating user stories
– Burn down charts
– Release planning

Our greatest challenge from the earliest days of the Alpha has been pitching our user stories at the right level of granularity – this has been a constant battle and while we got much better around very UX focused stories elsewhere it is often still pretty painful. We have tried a number of approaches and techniques (taking an agile approach!) and at the moment (a year in) we seem to be in a good place – but we’ve been here before and then a complex set of stories in a sprint messed that up.

This problem getting the level of our stories correct has regularly made estimating in any kind of traditional Scrum like manner a nightmare so we gave up on that and have been taking a much more holistic approach with everyone having their say and the taking the temperature of the room as to what is doable. It is far from scientific but it is showing results.

I’ve always hated ‘Burndown charts’ – something about them always felt like piece-work to me – so I have always fought against their use in this project. As it happens because of the problems I mentioned above it would have been a disaster for us to try given our estimation issues. One of the original principles underpinning the Agile Manifesto was;

Working software is the primary measure of progress.


If it was good enough for those guys it is good enough for me.

I can see the appeal of ‘release planning/management/trains’ and maybe it is something that works better for software products but I never really felt it fitted our needs for what has been a pretty traditional website build really. I do wonder though – it is something that I return to on occasion.

So to reiterate – agile is not an easy way out. There are a lot of moving parts and in my experience there is a lot of thinking and contemplating how to find ways to make the processes better for the team – to make sure we meet the first principle of agile;

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

*User stories have been invaluable in keeping us focused on real user needs – we are all about ‘user driven development’ – we just haven’t cracked what level to pitch those stories at all the time 🙂