Features

Why government should outsource technology

by Gavin Newsom '89 with Lisa Dickey |

Government, when it tries to handle tech issues itself, tends to get in way over its head. A perfect example of this was the Terry Childs incident, which happened toward the end of my second term as mayor. It was like a Keystone Kops episode, a cautionary tale if ever there was one.

Terry Childs was a 43-year-old network engineer for the city of San Francisco. He worked on our fiber-optic network, called FiberWAN, which handled crucial government data, such as email, legal documents, and payroll. By most accounts, he was a diligent and talented employee, and he seemed dedicated to making sure our systems ran safely.

Childs spent hours trying to perfect our FiberWAN system, upgrading equipment and installing firewalls. He had a high opinion of his own skills, even applying to get a copyright for the “technical artistry” he brought to the network. He unilaterally decided when to add hardware to the system, hooking up new computer equipment down at the city’s Department of Technology. Eventually it became apparent that he was the only person who knew all the codes and passwords to operate the system.

This was obviously not the ideal operating protocol, but it might not have been a catastrophe—except for the fact that in the summer of 2008, Childs was reassigned, and when he was instructed to hand over the password he had created for FiberWAN, he refused. Just like that, one man now held the city’s most valuable data hostage. No one could get into the FiberWAN without that password—emails were inaccessible, payroll couldn’t go out, and documents were under virtual lock and key. Childs had, under the noses of everyone, built a system that he could choke off at will.

Childs holed up at his home, refusing to tell anyone the password, as the local media ran wild with the story. Some commenters to local media website SFGate.com suggested that we waterboard him, while others saw him as some kind of folk hero. We had no choice but to try to compel him to give up the password, so the police arrested Childs on a charge of felony computer tampering and put him in jail. But he still wouldn’t give up the code.

The city was in crisis. We had to have that code! We spent at least $1 million, including hiring a team at Cisco Systems, to crack the code, but they couldn’t do it. Days went by, and the story went national. How could this be—how could one man hold an entire city hostage, and what did that portend for the rest of us in this new technological age?

In the midst of all this, I was scheduled to get married in Montana. I knew that leaving the city with this problem unresolved—even for my own wedding—would unleash a torrent of criticism. So when Childs’s attorney said Childs would be willing to talk to me, as long as I came by myself and there was no media fanfare, I jumped at the chance. Eight days after Childs was arrested, I went to the jail in secret, to ask him face-to-face for the code.

“Listen, Terry,” I said. “I’m going to lay it out here for you. I’m not going to moralize. I’m getting married in a few days, and I’m desperate. The city needs the code. Please.” I was as honest as I could be, asking him to give me the code as a favor, if nothing else. And he responded.

Childs said he didn’t trust his bosses or colleagues, but at that moment, he trusted me. He wrote down the code—all 28 numbers and letters of it—and handed it over. And although some  feared that this might be a booby trap set by Childs, a code that would self- destruct the entire system, I believed he had played it straight. And he had. The code unlocked the FiberWAN system, and order was finally restored.

Can you imagine anything similar happening at a company like Oracle, Microsoft, Twitter, or Yelp? This may be an extreme case, but the lesson is clear: Technology is not the government’s core competency. In government service, there tend to be a few people who are empowered and entrusted with an extraordinary amount of influence, usually based on seniority. Terry Childs was a brilliant technologist who cowed his colleagues into allowing him to amass total control over a critical city resource. No one in his department really knew what he was doing with all those new modems and servers, and we found out only when it was too late.

We just don’t have the resources in government to get the numbers and quality of tech employees we need. In fact, tech is usually the first place that funding gets cut. When you’re looking for programs to trim to satisfy a shrinking budget, what goes first? Police? Firefighters? Senior meals? Of course not. You cut costs that no one will protest. And every year, the first place that gets whacked is IT.

That’s how we end up with failed projects like the brand-new computer system for CalPERS, the California Public Employees’ Retirement System, which was intended to consolidate 49 old data systems into one streamlined system. Unfortunately, the project was completed two years late, cost a half-billion dollars—nearly twice the original estimate—and resulted in even worse services, with longer backlogs than before it was implemented.

In 2001, California governor Gray Davis signed a $95 million contract with Oracle for significant IT upgrades. A huge scandal ensued, mostly because it was a no-bid contract and Davis received a $25,000 check toward his re-election campaign from Oracle just days later. But the real scandal was that the contract was a boondoggle—Oracle’s upgrade was supposed to save the state about $111 million above the contract’s cost (if the state exercised an option for additional maintenance).

But instead, an audit committee found that the state would in fact spend almost $6 million more on Oracle database licenses and maintenance if it exercised that option and almost $41 million more if it didn’t. How could this happen? It happened because no one on the government side had enough knowledge to properly assess what Oracle was promising and whether it made sense. It happened because government will never be able to lure the kind of programming talent it needs.

“Finding great engineers is really difficult,” Yelp cofounder Jeremy Stoppelman told me. “The hot tech companies in the Valley—we’ll all complain until your ears bleed about how we can’t hire the best engineers. How can city government get a team of badass engineers to work on problems if Google or Facebook or Yelp can’t hire them?”

With no stock options, no hype, and only the promise of slow, steady advance within its civil-service ranks, government has very little to offer a talented young programmer.

We have to offer the next best thing, an open platform for people to experiment, to create and distribute government apps.

Excerpted from Citizenville: How to Take the Town Square Digital and Reinvent Government by Gavin Newsom with Lisa Dickey. Reprinted by arrangement of Penguin Press, a member of Penguin Group (USA), Inc. Copyright (c) 2013 by Gavin Newsom.

Back to article

Winter 2014

Table of contents

Features

Rise up, my love

There are the sanctuaries built for worship—and that carry beauty and grace for all to see. Then there are the improvised places of faith, perhaps more subtle in how they speak to the wonder worked there.

The chaplain is in the House

With the way things have gone recently in Congress, looking to the heavens for some help and guidance might seem like a very good idea. In fact, that’s what Pat Conroy, S.J., M.Div. ’83 is there to do.

Welcome to Citizenville

Who published the one book on government in 2013 that conservative firebrand Newt Gingrich told all true believers that they should read? Well, the author is now lieutenant governor of California. Before that, he was mayor of San Francisco. That’s right: It’s Gavin Newsom ’89.

Mission Matters

Goooaal!

Women’s soccer wins the West Coast Conference championship.

Patent trolls, beware

The White House has brought on SCU’s Colleen Chien, a leading expert in patent law, as senior advisor.

A sight of innocence

George Souliotes went to prison for three life sentences after he was convicted of arson and murder. Twenty years later, he’s out—after the Northern California Innocence Project proved he didn’t do it.