Ben Balter has the rather impressive job title of  ‘Government Evangelist’ at Github. He was one of the original Presidential Innovation Fellows in the U.S. and been a vocal advocate for a move towards a more digital by default federal government from within the White House and in his time at Github. He knows of what he speaks.

A few weeks ago he wrote a post about why it is so hard to attract quality technologists to work in government over there. Now from over here it doesn’t look like they are having that many problems with a slew of high profile hires in recent months but there is no doubt that hiring digital professionals is a challenge for everybody in the public sector on both sides of the Atlantic and it isn’t always just about the salary. I’ve written some thoughts of my own so read his post with great interest.

I’ve been through point by point and compared the issues he has identified to how we are organising my team at work. There is no doubt that internally we have been very privileged and have been lucky enough to work almost like a start-up within an organisation that can track its history to being founded by Churchill in the 40s. That said the whole organisation, particularly the digital/IT elements, seems on the verge of huge change (I almost wrote disruption there – BAD Jukesie!) and this list feels like a great way to judge just how far we still have to go.

[I couldn’t think of another way of doing this apart from copying the original post and responding point by point email like – so Ben if you follow a trackback to this I do apologise!]

 

You force developers to use tools designed for lawyers — At most agencies, technologists will be given the same hardware (computer, phone, etc.) as would be given to attorneys and other non-technical civil servants. Not all knowledge workers have the same needs. While a lawyer may live primarily in Outlook and Microsoft Word, developers spend the bulk of their time writing and compiling software, a task significantly easier on Unix-based systems like Macs or Linux that more closely mirror production environments or are better optimized for the creative arts. Before I can write a single function or draw a single line, it’s likely I’ll be forced to make the case for why I should be able to use hardware that’s standard in the private sector.

>> We recognised this really early. The developers/designers on the team work on Mac Pros with dual displays (albeit NOT Apple displays – we are public sector after all). We also have a couple of members of the team working on machines running Ubuntu.

You distrust your employees – You’d be hard-pressed to find an agency IT department that would give a developer full dominion over their workspace. When it comes to IT security, developers are distinct from the average employee in two ways: first, as engineers, they often have a critical understanding of the software we use on a daily basis, how it works, and how others might try to exploit it. As a result, although not entirely immune, software developers are less likely to fall prey to the types of traps system-level IT restrictions are designed to prevent. Second, while the average employee may primarily use one or two consumer applications that can be easily installed during on boarding (“set it and forget it” IT), a developer’s daily workflow often requires low-level system access — installing or upgrading development libraries, running local test servers, or compiling common frameworks — tasks impossible when an agency doesn’t trust developers enough to give them the necessary permissions on their own machines.

>> At the moment every member of the team has complete admin rights to their machines. This is a side effect of the decision made above and the next one regarding wifi. The amount of time this saves longterm as each dev sets their machine up to their own preferences as soon as they get started.

You tether developers to their desk – At most government agencies WiFi is the exception, not the norm. Many government IT departments hold on to outdated preconceptions that WiFi isn’t secure, or the agency simply failed to make the necessary investment a few years back, and is now behind the curve. WiFi will never be as secure as plugging directly into a hard-wired connection, but there’s a tradeoff to be made in terms of the ability to bring rich media in to anchor meetings in the practical, or have serendipitous interactions with coworkers someplace other than one’s desk. Most startups look physically nothing like government agencies, and for good reason. The ability to roam around a space cross-pollinates ideas and breeds creativity. While you don’t have to go out and buy neon green beanbag chairs, allowing internet-centric technologists to untether from their desk is a great first step.

>> Wifi is a slow burn. As a team we have our own entirely separate wifi network that is more than a little bit responsible for the freedom we are able to operate with. The organisation is on the brink of offering wifi across our offices though and importantly making the standard laptops wifi enabled – I know this doesn’t seem a big deal but it is HUGE for me.

You prefer government-specific service providers – If you go into a technical interview in San Francisco, they’re going to ask you if you’re familiar with services like Heroku, AWS, Travis, or GitHub. The thinking is that since there is a small set of industry standard tools that most companies use, when hiring, they’d prefer you know how to use their existing toolchain, rather than have to teach what the industry’s already settled on. In government, most of those tools are almost unheard of, being replaced by government-specific imitations that often pale in comparison in features, in quality, and in documentation and support. As a developer, I’d likely either have to unlearn what the rest of the world uses, or spend time making the case myself for why the agency should use the industry standard cloud services.

>> We still have a little element of this as nobody was buying in to my plan to just host the site on AWS or Heroku longterm 🙂 Generally though we use best of breed tools. Github, Travis, Puppet, Docker all feature significantly in the way we work. Also the team couldn’t function without Slack.

Temporary integration – In most companies (let alone open source projects) it’s sacrilege to commit a change without an accompanying test that verifies the intentioned functionality. As software projects get increasingly complex, standard QA in the form of “open up the browser and see if it works” isn’t sufficient to prevent regressions in the long run. Even if it is, developers need to be confident that when they make a change, they can do so knowing they didn’t break existing functionality. For many this function is accomplished by a process known as continuous integration. Every time a change is proposed, a server runs thousands of tests to give the developer instant feedback well before the change being merged. Yet in government, continuous integration is still a “we’d like to be there” development concept, with services like Jenkins or Travis being rarely used within government agencies. There’s simply not a test-centric culture to lean on, and without it, shaking things up becomes all the more precarious.

>> We are entirely committed to TDD. In fact I’d suggest it has been something of a preoccupation for us to date to ensure we got as firm a foundation in place as early as possible. David, our tech lead, is going to write up our approach as a blogpost soon (hint, hint!)

Sparse delivery — Continuous delivery is the idea that you should be constantly deploying small iterations of code, lines at a time, to test discrete changes individually. Unlike say, software that’s that’s distributed as a CD or installed by end users, websites and other service-based offerings aren’t versioned in the same way, and thus the cost of an update is negligible. Rather than wait for an all-or-nothing update from 1.0, 2.0, continuous delivery lets you incrementally test each change one-by-one, lessoning the risk of catastrophic failure (as an example, what version of Gmail are you using today?). A private sector firm might measure deploys per day or even deploys per hour, and still be unsatisfied. In government, there’s no need to measured deploys as they are scheduled well ahead of time, by day or by week, as each change must be reviewed by stringent change control boards which hold standing meetings for just such a purpose. Developers want to ship code — it’s how we innovate — and the more bureaucratic barriers you place between them and the ability to deploy, the harder that becomes.

>> This is totally the aim for the new website. Continuous delivery = continuous improvement and isn’t that the goal? Failure to deploy regularly damages morale and prevents momentum – things I am passionate about. 

You still see waterfall as a viable option — There’s no question that agile development is the hot new thing in government, but that doesn’t mean that it’s the default. Traditional waterfall development is still considered a viable option, with both procurement processes and agency policy encouraging large-batch development efforts. Developers want to move quickly, and want to be part of a team that moves as quickly as they do. It doesn’t have to be full-on scrum, Kanban, or whatever the hot new thing is, but the fact that waterfall development is still an option, let alone the default, speaks volumes about just how much development cycles differ between your agency and the private sector, and how much meta-work developers will have to do to reconcile their workflow with the agency’s.

>> We work in an agile manner but I doubt any ‘agile coach’ would recognise it as any particular approach. It works for us – providing enough structure for me to report upwards and enough freedom for the team to perform to the best of their abilities.

You don’t place process on a pedestal — Process matters. Take a look at the go-to project management flow in government: an agency might maintain an Excel spreadsheet with known bugs and desired improvements and hold a regular meeting with contractors to discuss priorities, with status updates being transacted via email. At the same time, the contractor likely maintains its own project management system in parallel to organize its own task orders. All this means that at the end of the engagement the final deliverable for a multi-million dollar contract may be as little as an email with a zipped attachment of the code. Agencies often don’t have day-to-day visibility into the project’s progress using professional project management tools (read: shared issue trackers), documentation is often an end-user manual in the form of a 100-page PDF, and rarely if ever, get more than a snapshot of the code (read: version control), two things essential to capture and expose process to stakeholders, not to mention, give developers the context they need when they join a new project.

>> We still need to get our heads around this. Currently we are small enough that we can muddle through based on the incredibly organised natures of a couple of the team. That won’t scale so we need to do better.

You erect a moat between developers and servers — There’s an odd anti-pattern whereby developers are distrusted to such a degree that they can never have access to production data. You write code, you zip it up, you email it to someone, and they deploy it. Need to run a migration? Send them a shell script. Why are system administrators trusted with production data while developers are not? Contrast that with many startup deployment flows. You write the code (in version control), you deploy the code, you monitor the effect your change has on production, and you adjust accordingly. The bigger the disconnect between technical talent and your users, the bigger the disconnect between the software and your users’ needs.

>> We are trying to encourage a set of DevOps type principles from day one to avoid this. Early days though.

New technologies are guilty until proven innocent — In government, modern development frameworks are assumed to be insecure unless the particular developer that wants to use them personally proves otherwise. It seems that in the late 90s, there was a big influx of spending in government IT, and the risk barometer was set, in terms of what technology can and can’t be used, and it has been set since. Technologies that could potentially be called past their peak in the private sector — things like Ruby on Rails or even PHP — are often seen as being too immature for government use, with agencies preferring to use legacy tools like Sharepoint and Cold Fusion due to technical debt and personal expertise. If you’d like to use a technology that was invented in the past ten years — a lifetime in web development years — be prepared to go to either bat for it, or learn a new government-specific framework.

>> I liked this post about ‘boring technologies‘ a lot. I recognise what Ben is saying here and there is no doubt we made choices based on them being ‘established’. The interesting thing I think is to take ‘boring technologies’ and use them in innovative ways.

You use open source as a verb — For government, open source is still something you do, not who you are or how you work. Few agency development teams embrace an open source ethos, and those that do, often face hostility from other parts of the organization for doing so. Private sector organizations see themselves as members of a larger open source community, with many developers serving as life-long open-source contributors, even as they change roles. For some in government, open source is merely a verb. It’s a switch you flip, to toggle the visibility of a line of code beyond the corporate firewall. Otherwise closed-source workflows remain the same, with stakeholders remaining guarded to the project’s outside influence and external contributions being vetted by the same process you might vet a press release or blog post.

Working in the open is a novelty, not a best practice – Why does your organization publish open source software? Is it because it’s the right thing to do? The best way to work? Do you simply publish open source because it’s in vogue? To get free labor? Good press? Who’s in charge of the effort? Is working openly an organization-wide initative? Does it go through your public affairs office? Are other teams skeptical of your efforts? Do your open source efforts have top-down support? Is there anyone to maintain the project once I leave?

>> As a team we operate in the open. We use and contribute to open source, we work in public Github repos, we blog about pretty much everything and speak to anybody interested. Open by default. We are a small team though with a specific set of aims and a team of experienced technologists. We certainly aren’t set-up to support open source tools or libraries in the long-term beyond our own work. I’d like to get there though.

Speaking at conferences is tightly controlled – What’s the process look like if I’m invited to speak at an industry conference? Do I ask my presumably technical supervisor who is familiar with the industry? A non-technical public affairs officer? How likely are they to say no? How far in advance do I have to seek approval? What information do they need? How long do they take to respond? Are they going to place restrictions on what I can say or do? Do I have to submit my slides in advance? Does the organization trust me to positively represent the agency to the broader technology community or does it try to hide me and my ideas in order to protect itself? Is my participation an asset or a liability in your eyes?

>> Well nobody seems to have been too worried about my conference speaking so far and given how late I usually finish my slides the idea of submitting slides is amusing. This would always be seen as an asset to the organisation I think and while there are always a few common sense guidelines (which I know I often push) this sort of thing would be encouraged.

Geeks are the bottom of your food chain – The single biggest different between a startup and an established organization, government or otherwise, is how they treat technical talent. In the early stages of a startup, when there’s only room for the bare minimum, developers reign supreme. Think The Social Network, Page and Brin, Coders in SF coffee shops. As the organization grows, that hierarchy is flipped. A professional class of managers enter the fray, to direct the developers efforts. In the long run, unless technical talent transitions to management, geeks will always be at bottom of your bureaucratic food chain, regardless of seniority or skill. This is especially the case for those without formal higher education. Career stagnation is all but guaranteed. There’s a reason they call it a bureaucracy, and not a technocracy.

>> Too early to tell though I am probably more manager than geek these days. We have a pretty senior layer of leadership now though who genuinely understand the challenges and opportunities created by technology (and could, at a push, still get their hands dirty).

Culture only happens outside of your working hours – Most small companies live and die by their culture. They pride themselves on the unique feel that the office has (architecture, foosball, open bar, and all), and when recruiting, look specifically for “cultural fit”. In many large organizations, culture happens, exclusively outside of business hours — at lunch and during employee-organized happy hours — if at all. Even during recruitment, culture is inoculated against, by necessity of the formulaic interview process designed to focus exclusively on documented hiring factors and paper-based skills. Developers want to work somewhere with a supportive culture, and unless you specifically seek to create it, it’s almost certain that any resemblance of culture will be stomped out by bureaucratic forces.

>> Again early days and most of our ‘culture’ seems to be sugar based at the moment. It is hard to identify a ‘fit’ when the culture is still evolving. One to watch though.

You measure your hiring process in months – It’s sometimes freaky fast how quickly startups can hire. You can go from screening interview to offer to on boarding in a matter of weeks. A big part of that is the intimacy of the hiring process. As a result, both sides are honest and forthright, and you really cut through the formalities. It’s easy to know where you are in the process, and you have a single, human point of contact, if at any point you have a question large or small. By contrast, there’s a good chance your hiring process is both opaque and arms-length. On the one hand, your formalized hiring process is a mystery to all but your hiring department, and chances are, you don’t do a good job of communicating that process to potential hires, and even if you do, I suspect the process is optimized for compliance and efficiency, not experience. Even within the process, interactions tend to be artificial and forced, by virtue of an uninterested HR department running the process (rather than those excited to onboard the hire), meaning I’m already burned out by day one.

>> Yes. Getting MUCH better but still a pretty long process which a number of opaque moments. I am getting much more directly involved in each stage of the process – one of the main lessons I took away from reading ‘How Google Works‘ was how important it was to make hiring a priority.

Onboarding is an afterthought – It’s my first day. What does the experience look like? Am I stuck in a room with lawyers learning ethics restrictions, or with visionaries talking about culture and the next big thing? How long does it take for me to get a laptop? An email login? When I show up? Hours? Days? Weeks? What does my badging process look like? Do I need to pee in a cup? How many of my high school friend’s shoe sizes do I need to remember to satisfy your background check? How do I get up to speed on the project I’m going to be working on? I’m there because I want to ship code. The longer it takes for me to write my first line of code, the harder you make it for me to do my job.

>> Again something that we need to get better at. We are certainly much better at this than in the past but there is loads that could be improved.

Recruitment is unheard of – In the private sector you can’t go a day without getting a tone-deaf recruitment email. It’s a running joke in the tech community. As much as we joke, that courting process, whether formal or informal, works. The uncertain is scary, and without someone pushing you, telling you just how much greener the grass is on the other side, you’re likely to stay put, public sector or otherwise. Whether it’s a hip startup or the White House calling, there’s something to be said for being wanted, for hearing the pitch, and for having someone to work through any questions or hesitation you may have.

>> Not sure what this is actually about…

You block half the internet – I honestly don’t know how I’d do my job without social networks like Twitter. Technology is about keeping abreast of emerging trends. Open source is about being part of a community. Without access to that community, without access to the dominant mechanisms of communication, it’s not unsurprising that many government employees are out of the loop. If reading /r/opensource is blocked, or @hackernews is inaccessible, you either sacrifice your personal time to advance your professional life, or simply fall behind. Even worse is when organizations lock down even more innocuous resources like Stack Overflow, Gist, or even GitHub, for fear of information getting out. Whenever you prevent something from getting out, you also prevent something from getting in.

>> Yes. Yes we do. Thanks to our own network we avoid most of these frustrations and the wider organisation is about to pull the trigger on some significant changes in this area that will make life easier for pretty much everybody.

 

So there you go. I reckon I’d give us a solid B on the Balter test 🙂 So much has changed in the last year or so and I have to be honest it hasn’t always been easy but it has been worth it.