- PITS Computing
Non-Computing Fads
There are fads and trends in information technology, just like in the world of clothing or hairstyles. One of the latest fads is pie-in-the-sky computing (PITS), otherwise as “cloud computing” or software-as-a-service – SAAS (pronounced as in “sassy”).
But I’ll call it pie-in-the-sky (PITS) computing, just to be different and even a bit contrary.
PITS computing is only the latest in a long line of sea-changes in IT. Electronic data processing (EDP – now there’s an old term) was the very first of these trends, appearing on the scene in the 1950s and 1960s. EDP was a world of punch cards and paper tape. EDP was the era of “glass house” data centers and a computer “priesthood”. Computers were far too expensive and esoteric for normal human beings to comprehend or touch. So there was a “priesthood” of specially anointed and trained computer specialists whose job was the programming, care and feeding of the electronic monsters.
But the development of computing technology continued relentlessly. Along came mainframe computing (green-screen). personal computing, local-area-network computing, client-server computing and Internet or web computing.
Each one of these phases was driven by some significant technological advance. The development of microchips and the Intel 8088 processor, for example, drove the personal computing trend. (Thank you Intel and IBM!) The development of Ethernet standards drove networking which allowed individual computers to talk to each other.
And then computing, of course, became part of the mainstream culture. Any human being in a developed country knows “windows” doesn’t refer to that wonderful device for seeing through walls, the “glass window”, but rather the portal into the world of computers, an operating system developed and marketed by Microsoft. And almost no one thinks of the “web” as a home for spiders or the “net” as a tool for catching fish or butterflies.
In this context, PITS is the latest fad in computing and technology. PITS is driven by the appearance of more-or-less ubiquitous and reliable high speed networking. Networks today, thanks to fiber optic cable, the router/switch revolution (thank you Cisco) and advances in wireless (wi-fi and 3G telecomm networks), are virtually everywhere. Or at least everywhere where human beings live and companies and governments do significant business.
And these networks are reliable. The wired networks almost never go down, although the signal can get weak or strange with wireless. In my house for example, our Wi-Fi network connected to a wired DSL Internet connection has 105 megabits per second of throughput. Yet my commercial telecomm provided cell phone only works at a certain specific spot in the kitchen in front of the microwave!
Most enterprises now operate with giant central servers which store data and applications. At the City of Seattle, for example, we have computer aided dispatch systems which reside on central servers at a “highly secret” police department location. The police data resides there, but cops on the street can access criminal records and license plate information which reside not only in Seattle but also on the other side of the nation or even on another continent.
Our water utility manages pumps and valves and dams and reservoirs across the entire county and up into the Cascade mountains. City Light, our electrical utility, manages an electrical grid which spans the entire state of Washington.
We all routinely use the web to find information and read the news. But we also increasingly use it to store spreadsheets or photos or documents on our own websites or using servers such as Google apps. Microsoft is embracing the trend, with its Office 2010 now available “for free” in a PITS cloud.
So if Microsoft Office can be in a “cloud” somewhere on the Internet, why can’t our payroll system or e-mail system or financial management system be halfway across the State in a data center in Grant County, Washington (next to giant hydroelectric dams to supply the power) or even halfway across the United States, well outside the Seattle earthquake disaster zone?
Of course the applications and data can be almost anywhere. In the past, I’ve been skeptical of PITS / cloud computing because I didn’t trust the networks to stay up in a disaster, and I was concerned about the security of information stored in a non-descript data center in a distant location outside my personal control.
But with today’s reliable networks, the network is not the issue. And major companies like Microsoft or Amazon or Google handle the management and security better than most governments or small businesses. Furthermore, as demonstrated by the World Trade Center disaster, the data could actually reside in multiple different locations around the nation, increasing our ability to withstand a disaster like that 8.0 magnitude earthquake.
It will be a while before we in government embrace PITS, because the loss of control is a big cultural change for governments and many large companies to swallow. Just like people were concerned when their data moved off their desktop computers to a server, and servers moved out of the closet on the same floor to a centralized computer center in the government complex, so it will take us some time to embrace having those computers in an unnamed nondescript but super-secure location, possibly right next to the bunker where Vice-President Dick Cheney hung out after September 11th.
But embrace it we will.
- UASI, Bureaucracy and Terror
Homeland InSecurity
This week I was, in turn, amused, maddened and fasincated by disaster. I had the opportunity to do both leadership and followership as officials from the “Seattle Urban Area” considered projects to submit for grant funding from the fedgov Department of Homeland Insecurity.
This whole process is a look into the little-known culture of “homeland insecurity” which has blossomed over the last 7 years.
Let me say, right at the beginning, that September 11th, 2001, was a horrific event and a real wake-up call for the United States. We’ve been taught the same lesson we learned on December 7th, 1941 – two huge oceans do not make for a nation secure from the enemies of our way of life. My somewhat humorous tone in this blog is not meant to make light of the serious threats we face.
But what is our major response? Pretty typically, we create a giant bureaucracy, slosh a lot of political rhetoric and some money around, and get prepared to fight … the last war.
The major bureaucracy is, of course, the Department of Homeland Security, or Insecurity, as I’ve named it, because it heads an industry and set of programs based upon the fear and feelings of insecurity of the American People. DHS is really a myriad of individual bureaucracies and programs, sometimes working at cross-purposes and often duplicating effort.
But I stray from the subject – the Urban Area Security Initiative (UASI), usually pronounced You-Ah-Zee. In 2009 about $1.8 billion will be granted to urban areas through a series of programs as shown here. The UASI grants are made to “urban areas”, a carefully defined (by the fedgov) set of metropolitan areas which share vulnerabilities and threats. The “Seattle Urban Area” is the City of Seattle, its surrounding King County, and the counties to the north and south. The Seattle urban area is eligible for $11 million, more or less, of that $1.8 billion pie.
Now $11 million may sound like a lot of money, and it is. But think about the Seattle Urban Area – a population of 3 million people, two major ports (Seattle and Tacoma) into which ships with nuclear bombs or bioterrorism agents or chemical weapons could sail, three major military bases (Naval Station Everett, McChord Air Force Base, Fort Lewis), and vulnerability to a potential magnitude 8.0 earthquake. We probably have 12,000 cops and firefighters (Seattle alone has 2,200), each of which needs radios and hazmat gear and training. All of a sudden, $11 million doesn’t sound like a lot, and it isn’t.
Its also a little amazing that places like Vermont and Wyoming – not exactly your typical terrorist or disaster targets – get $6 million each. But that’s the reality of politics in our 2009 world with two senators from every state and a lot of dollars to be allocated.
Each year, a group of officials from the emergency management, law enforcement, firefighting, public health and technology disciplines – the officials responsible to prepare the region for disasters and terrorist events – get together to consider projects for UASI funding. We did that this week in the Seattle Urban Area. We have a strategy to guide us but it is still a give-and-take to determine which projects are most importnat to the region.
I lead the subgroup of technical and operational staff responsible to consider and prioritize interoperable communications projects. These are the networks and systems which allow first, second and third responders to communicate with each other and with their dispatch centers to prepare for and respond to disasters large and small. They include handheld and vehicle-mounted radios, computer and telephone systems, and fiber optic networks.
Small disasters are the ones which occur every day – such as when I fell off my bicycle, shattered my arm and called 911 for help. The large ones are the fires, landslides, earthquakes, and (God forbid) terrorist events which may afflict our region.
Finding projects which prepare us for those disasters and which fit into $11 million is daunting. In the past we’ve procured fireboats and helicopters and personal protective equipment. We’ve found money for training our responders, buying radios and for public outreach and disaster preparedness campaigns. It is hard work to winnow 100 proposed projects and $40 million in needs down to a dozen or so projects with $11 million or so, but we mostly did it this week. In future blog entries I’ll talk about some of those projects – the ones I can reveal without making us more vulnerable to terrorists (and that is most of them).
In the meantime, I’m just proud of the emergency managers, fire and police officials, and technical staff I was able to spend time with this week, prioritizing projects to keep the Seattle urban area safe.
And I’m tired!
- The 108 Degree Data Center
The 108 Degree Data Center
November in Seattle is always cool and rainy and sometimes stormy – windstorms, that is. Seattle’s all time high temperature – for any day of the year – is 100 degrees. That all time high is, of course, outside. But it reached 108 degrees here on Sunday November 16th. Inside a data center. The City of Seattle’s data center.
To make a short blog entry even shorter, I’ll skip to the root cause: a failed power breaker on a pump for the domestic water supply to the building housing the data center. The water supply flows to CRAC (“crack” or computer room air conditioning) units which, in turn, cool the data center. For HomeCity Security reasons, I won’t reveal the actual location of the data center, but let’s just say it is in a downtown 60 story skyscraper which also houses about 3500 office workers during the week. The problem started about noon and was fixed at about 8:00 PM.
The data center holds about 500 servers, storage systems and other equipment. We shut down a lot of servers and many services starting almost immediately. Nevertheless the temperature in the data center rose to that toasty 108 degrees, setting a new record high (sort of) for Seattle.
So why is this notable? For two reasons: the problem and the response.
In terms of the “problem”, let me assure you (especially if you live in Seattle) that cooling problems like this will be rare to non-existent in the future. Years ago we installed a one megawatt generator for backup power. This year we’ve been working a project to install “dry coolers”. These aren’t really “dry”, but the water cooling the data center will flow in a “closed loop” between the new coolers and the center, so we’ll no longer be dependent on external water or power supplies. Unfortunately, the dry coolers don’t come online until January, which is why we went to 108 degrees last Sunday.
But there’s a more general issue here – every city and county government has data centers and servers and vital information. Every area of the country is subject to some sort of a disaster and every government needs to have a backup and recovery plan.
But for what disaster should we prepare?
Here in Seattle, everyone is concerned about the “big one” – a magnitude 8.0 earthquake. While we need to be ready for an major earthquake, we have about one of those “big ones” every 300 years. Much more likely are disasters like last Sunday – a failure of water and cooling, a “meltdown” if you will (non-radioactive, however!). Or perhaps the disaster will be the opposite – too much water from a broken pipe, and a flood drowning those servers. Or – and this also happens in computer centers – a fire followed by (drum roll), a flood as the fire suppression system kicks in. Should we have a plan for “the big one”, that earthquake? Sure. But most of our disaster preparation effort should plan for the much more probable disaster of fire and water.
Finally, any disaster response plan has one element which is vastly more important than any other: people. And, on November 14th, the “people” (employees) of the City of Seattle and its Department of Information Technology performed splendidly. A dozen IT professionals showed up on site within two hours (despite interference from the traffic around a nearby Seahawks football game). The computer center manager – a 44 year employee and true hero Ken Skraban – was on site and immediately in charge. Two employees set up an IT operations center with an incident commander and support staff. Several responded to the data center and shut down servers in an orderly, pre-planned, color coded (red-green-orange-yellow) fashion, with the most critical servers (for example “Blackberry” support) staying up continuously. Server administrators from every major department in City government responded on site.
And when the crisis was past and cool water was again flowing to the “crack” units, those same folks brought all services up in an orderly fashion. And there was not a single call to the help desk on Monday morning as a result of our unanticipated “summer” high.
Disasters happen. Careful planning and skilled, trained staff will always mitigate their effects.
- Tech Nightmares Frighten a CIO
Tech Horror
Sometimes my job as a City CIO keeps me up at night. There are some pretty horrible things which can happen to the technology which keeps City and County governments running. Halloween seems like a perfect time to confront a few of our most frightful fears, and here are a few of mine.
Water. And Fire. Or Fire followed by Water. In my data center. The City of Seattle has multiple data centers, but our main one, constructed in 2001, has well over $15 million of stuff in it. Things like the e-mail servers used by the entire City government, or the disk array holding all our financial data. And about 500 mid-range servers. Our data center is in one of the most modern, earthquake-resistant buildings in town. But my real fear – and much more likely than the predicted 8.0 earthquake – is a fire or a gushing water leak. I guess it’s time to test that disaster recovery plan again!
E-mail Horror
E-mail. Gosh, e-mail is the most important application we have – more important than utility billing systems or computer-aided dispatch or financial management systems. We all get an avalanche of e-mail every day, and the City of Seattle’s great Postini spam filter from Google cleans out most of the viruses and junk mail. But it is really the content of the e-mail which scares me. Like that occasional email which says “hey, we’ve decided to cut your budget for xxx (fill in the blank) by $500,000 but you still get to do the project, on time, with reduced budget” or “oh, hey, Mr. CTO, your Wi-Fi network in the University District is down. Again. And the Mayor has a public meeting there at 3:00 PM”. The only thing more frightening than some of the e-mail messages is arriving in the morning to find that the e-mail system is … ah … “down”. And down HARD!
Tablet Computer
Tablet computers. Ah the great promise of laptops and tablets! You sit at your desk, and it is a desktop computer. You unhook it. You take it to every one of your meetings so you can view documents electronically, and don’t have to print paper to take along. You take notes using Microsoft One-Note on the tablet, rather than writing stuff on paper (and, like me, promptly losing the paper in one of the giant piles in my office). You demonstrate that you are “friendly to the environment” by personally reducing your paper use by storing everything electronically. Then you forget to back the tablet up, you trip on the stairs with the laptop in your hands, and it crashes. Into the wall. Literally.
BlackBerry
BlackBerries. An extraordinary combination of the two most nefarious technologies known to humankind, the cellular telephone and electronic mail. Now you get to be available to your customers 24 hours a day, 7 days a week, 365 days a year. The real killer is discovering, one September day, that one of my last havens where most cell phones and BlackBerries didn’t work – beautiful little Republic, Washington – had been jerked into the modern era (see blog entry Fossils and Technology). The Blackberry worked there! Arggh. The only thing worse than a fully functioning BlackBerry is one which doesn’t work, so you are out of touch! Arrgh!
Mayor’s briefing. You show up to brief the Mayor and his senior staff on your latest new hotshot tech project, hoping to convince them to make a relatively small (less than a million bucks) investment. But, as you walk into the Mayor’s Office, the Dow drops 500 points, Lehman Brothers fails, AIG needs a $85 billion bailout, sales tax revenues drop precipitously right along with consumer confidence, Boeing goes on strike and the room’s technology systems go on the fritz.
The Help Desk. So you call the Help Desk (206-386-1212 for the City) about any one of the problems above, and they fix the problem over the phone. Quickly. Efficiently. Surprisingly well. And you – the Chief Technology Guy – are really frightened, because the problem was “user error” and the user is you!
About
This website is the personal blog of Bill Schrier, Chief Technology Officer for the City of Seattle, used to express personal opinion. Comments on any blog entry are welcome, but will be moderated. The opinions expressed on this site and in the blog are solely those of Bill Schrier and do not necessarily reflect the official opinions of the elected officials or residents of the City of Seattle.
The focus of this blog is the intersection of technology and government, and specifically how technology influences Seattle’s government, but also how governments use technology on behalf of their residents, citizens, and visitors.
The official website of the City of Seattle is www.seattle.gov , and the official website for the City’s Department of Information Technology, which Bill directs, is www.seattle.gov/doit . Official City of Seattle blogs are on Citylink. Address general questions or comments to bill@schrier.org .
The Chief Geek
Bill Schrier
CTO and Director
Dept. of Info Tech
City of Seattle
