Asymmetrical Information

Large-scale software development projects have a terrible track record, whether done by government or by business. It's probably somewhat worse for government-sponsored projects, on the average, but there are plenty of skeletons in the business closet, too.

Here's a post on the failure of one very important software project, the FAA's Advanced Automation System.

To reinforce Dave Foster's point: half over half of all large IT projects are failures in the eyes of their sponsors.

Given that businesses can hide their IT failures easier than public agencies, I don't see the VA's and FAA's failures, lamentable as they are, as strong evidence that government is worse than buisness for these huge projects.

Nonsense :) There are many examples of large scale software projects done very well. To start with:

The Linux Kernel
The Apache Web Server
The Apache Java Ecosystem
Eclipse IDE

all of these involve millions of lines of code, design, maintenance, and planning across many years, etc.

On the commercial side I give you:

Mac OS X
Vmware ESX

OK, so maybe commercial examples are fewer and further between.

don't forget how much of a failure the FBI's ongoing effort to improve their computer systems have been.

There are far more massive and complex commercial projects than you listed. Companies invest millions into ERP and CRM applications every year - some are succesful and some are not. The government, for the most part is always a failure.

I actually met someone in the lauded VA and they told me stories of millions of dollars invested in SANs that sat in a datacenter for years waiting for something (don't remember what) by the time they were actually ready to implement the hardware was out of warrantee and wouldn't help them anymore.

When people tell me that the government can save money on healthcare through "efficiencies" I have to wonder which government they are talking about.

Missing from the observations about the VA, of course, is the simple fact that VA doctors typically make less than market wages. How many think that the government is willing to try to cap doctor salaries? Or how many are willing to discuss the harm that will come from capping perscription prices?

Most advocates of single payer health care do not point to better government implementation of IT systems as the source of administrative savings. Rather, the theory is that there will be less admin costs because of the nature of a single payer system. Reduced costs come from the removal of marketing costs, no "which insurance is primary" battles, no complexity on the physician's end of figuring out how to get reimbursed under 100 different sets of reimbursement rules. What the single payer advocates miss is that without competition, systems have no reason to become more (or even stay) efficient and therefore, don't.

Also, the advocates of saving money by reducing admin costs forget that there is a big need for oversight whenever anyone (government or insurer) is in the business of handing out money. Those admin costs are necessary to ensure that money goes to the right place, rather than to people placing fraudulent claims. When I worked for the Medicare program 15 years ago, it was common knowledge that the Inspector General made $10 back in savings from fraud for every $1 budget. Yet, their budget was cut every year because the Inspector General budget came from the general fund while the fraud savings accrued to the Social Security Trust Fund. It's virtually impossible to hold the federal government accountable for a program, which pretty much guarantees that waste, fraud and massive inefficiencies will creep in.

"The government, for the most part is always a failure"...this is a bit overstated. In aviation, for example: despite the failure of the AAS, which I described in the link above, computer systems for air traffic control are handling thousands of flights every day. In defense, very sophisticated systems do things like guide missiles to targets (as Zarqawi found out) and locate sources of enemy gunfire. In more mundane areas, tax returns do get processed and refund checks mailed out.

I suspect the main differentiator between success and failure in government systems is the same thing it is in business: leadership and culture in an individual agency. Most government systems are developed by contractors, of course, and they also play a nontrivial role in the success or lack thereof.

Regarding Jenny's point about the need for oversight to prevent fraud and insure that payments go to the right place, I wonder what Medicare and Medicaid's administrative costs as a percentage of their total spend would look like if they had enough personnel and sufficiently sophisticated systems and oversight to achieve as good a record as private insurers in controlling fraud. I suspect that most or all of the administrative savings promised by single payer advocates would disappear. Besides, Medicare and Medicaid are already effectively single payer systems for the large populations that they serve, and their costs are growing by leaps and bounds too despite the lowest payment rates in the system.

The Linux Kernel
The Apache Web Server
The Apache Java Ecosystem
Eclipse IDE

Large-scale open source projects like these are more the exception than the rule in software development, though, even in the open source world. They have millions of users, a large fraction of whom have advanced programming skills, solve relatively well-defined problems, and have a lot of prestige assosciated with making a contribution.

A national health service computer system would have a single customer whose employees are not selected for programming skills. It would also have very strict requirements for data security, which would not necessarily be compatible withallowing volunteer coders complete access to the system.

Reduced costs come from the removal of marketing costs, no "which insurance is primary" battles, no complexity on the physician's end of figuring out how to get reimbursed under 100 different sets of reimbursement rules.

I agree that this is the theory. But is this kind of squabbling honestly wasted expense, or is it just the way that our current system fights out the question of who pays what to get what?

It occurs to me that for society as a whole, it might not be a very good idea to try and save money if the cost of saving that money is getting a less satisfactory answer to that question.

Please note, in pointing out successful large scale programming projects, I was not suggesting that the government could manage IT savings. I don't think they can. I just wanted to point out that there are folks in the private and open source sector who can and do write excellent code.

OK. Looks like we agree, then.

quad...the examples you give are all opsys and middleware-level stuff, rather than functional applications...maybe there's a lesson there.

In general, IT spending pretty much never results in net cost savings. That's why the old gibe about computers seeming to be everywhere except in the productivity statistics rang so true. What IT does is allow you to do more things that you hadn't done before, rather than doing the same old stuff cheaper. When I saw Kerry's plan relied on large IT cost savings, I knew it was mostly pulled out of thin air.

As a simple example of the government process:

The quote is "Special price for government contracts. No, it isn't a discount..."

In reality, we are given permission to buy Dell computers that have been cleared as "cost effective" or whatever purchasing uses as a benchmark. It takes them a couple months to clear a computer. So, we can buy Dell computers, at 3 months ago's performance, at 3 months ago's price. Very frequently, this means we could get a cheaper, faster computer from the non-approved list (aka the Dell main page.)

Fortunately, in recent years, lab purchasing has noticed this, and will permit you to beg them to let you buy a better computer for cheaper. Proof that government functions best when it gets out of the way...

The point is, at my level, our sucky computers are not a massive IT failure, but rather the product of techno-illiterates telling the scientists what computers are cost effective.

I'm reminded of an old Dilbert: Dilbert tells the Pointy-Haired Boss (PHB): "Check me on this: You get a top of the line SparcStation while the engineers need to get along with 286s. All you know how to do is run the screensaver." PHB says nothing, Dilbert leaves. PHB thinks, "How does that ball keep on bouncing?"

There is no competetive market in our health care system. That's one of the great myths of the First Church of Free Market, whose primary tenent is that Free Market is all-knowing, all-wise, all-compassionate and capable of solving all problems. This, of course, is utter BS.

One poster muses that perhaps it is possible that the higher administrative overhead of the private insurance companies is worth it because they are so much better at weeding out fraud. But this too is a bogus argument because the private insurers aren't interested solely in rooting out fraud but also in denying legitimate claims and delaying them for months if they can't be completely rejected. Ask any doctor's administrative staff, you know, the one that is so big because of red tape that makes the government bureaucracy look piddling.

I have commented on my own blog about Paul Krugman's mistaken belief in the quality and efficiency of the VA health system. The Veteran's Health Administration informatics system is a gigantic mess, and the process for acquiring (or, shudder, internally building) a modern health informatics sytem is doomed to failure.

Here's an example of VHA informatics efficiency: The VHA Office of Information and Technology has been working on a relatively simple project (a Windows-based informatics module for blood bank labs) for over 8 years. The project was supposed to be completed in 2000. The program is still in 'beta' and has not been approved by the FDA. Regardless, the VA is spending millions of dollars this year installing hardware and training personnel on the yet to be finished, approved, and installed software. Oh, and the program will not come close to meeting our needs. (The VHA OIT says that version 2 will do so.)

The existing VHA electronic medical record system is written in MUMPS, an archaic programming language that was created in 1958! The code is so unwieldy that minor changes require thousands of hours of work. The VHA plans to migrate to a modern system over the next 10 years, but that won't happen unless they are forced to purchase a commercial electronic medical record system.

Is the source of Jane's question the whole "Insurance companies have adminstrative costs of 30% and Medicare's adminstrative costs are 1%" thing?

Does anyone have a link that explains how that calculation is performed?

OK... actual applications that don't suck:

Firefox
Google
Google Maps
GMail
Google Calendar

Please also roll in with Mac OS X: Mail.app, Keynote, Address Book, iCal, Quicksilver, Omnioutliner, Omnigraffle...

Notice the pattern: some development cultures produce really good stuff. I don't think you can foster such a functional culture inside government (to begin with, government workers are paid on seniority, not ability, and no able IT person will tolerate that kind of environment).

Don't mean to hijack the thread, but all this reading about governmental health care got me thinking about privacy. Apparently there are a bunch of people up in arms over the government looking for patterns various phone calls. This is all information that exists in the private sector and I imagine most phone companies would be able to run an analysis on your outgoing phone calls (it's called a phone bill you get it every month and it tells you exactly who you called).

Anyway, if the government is our health supplier, now big bad Bush or Kerry or Clinton or whoever can snoop around in our records right? Surely your private health information is as much if not more important than who you called last month? And the government would *have* to know if they're paying for it right? Your insurance company certainly wants to know what procedure you're getting and the justification for it before paying for it. I'd assume the government would require the same.

So...why all the fuss?

Oh, about about IT going out of style before the government can even procure it. Perhaps they should consider some form of leasing? Government leases the tech which gets updated every 2-4 years? The "old" machines get moved or donated to schools, 3rd world countries or who knows where else.

The Navy is doing leasing. Of course, the leased computer still arrived at the end user's desk six months after manufacturing.

quad..by a "functional application" I mean an application that does a specific business function, requiring it to embody knowledge about how that function is performed: accounts payable, material requirements planning, tax return verification, hospital order processing, etc.

My health plan just computerized. Now all the doctors and nurses walk around with notebook computers instead of paper files. It happened within a few months, training and all. I'm sure it was expensive, but I can imagine all sorts of extra capabilities and better accountability. Perhaps the greatest savings will be in avoiding mistakes, and thus hugely expensive litigation, because of obsolete or unavailable records.

Don't forget The Canadian Gun Registry. It was supposed to cost $119 million, and be almost self-supporting based on the income from registrations.
It's now well over $1 BILLION and still counting. It's a resource for thieves, who can now determine which citizens own firearms. They don't even have to case the homes before breaking in... IT has certainly helped them ;-)

Jane's original post was pretty sloppy. She reports on Paul Krugman's shoutout to the VA and then says that an article from the New Economist "offers another version of what our government could do with a big medical IT project, and it comports much better with what little I know about government IT."

But the New Economist article is about what's happening in the UK, which is not "our government." And since Jane says she doesn't know much about government IT, why does she think that she should be the one to say that the New Economist rather than Krugman is getting it right? This "analysis" has all the rigor of overcooked pasta.

Most of all, Jane doesn't even think to nail the glaring fault in Krugman's analysis. The VA's hospitals used to be horrible. They're better now because Congress was threatening to shut them down and turn veterans' health care over to the private hospitals. Hey, competition! Which wouldn't exist if the feds took over all health care. You'd think an economist would notice such things.

Why large government IT projects fail (along with most private business) and I am not talking about developing software but getting 1000's of people to be productive using the same software/hardware combinations:

1. Usually one group does not own enough of the project to have the final say. Example: In the VA or the DoD, the people who control healthcare enrollment (the demographic data) are separate from the care providers/clinical managers and the enrollment people do not care what the care providers want from the information systems. Thus, the enrollment database people design systems that do not work with clinical systems. On the other hand, in the corporate world, headquarters at say Wal-Mart has total say on how the information systems work.

2. The government has to deal with legacy systems. The VA does not get to start fresh with a new system because they have to maintain and use all of the old data. Thus all new systems have to be backward complatible. A good example is that all federal logistics system use the National Stock Number which is a legacy from punchcard days.

3. The government has failed many times by trying to get the new systems to act like their old systems instead of just using what the new system can actually Every time the feds have tried to do ERP, it has failed because the feds try to force the ERP products to function like their existing products. This is also why most private companies have failed using ERP.

Despite Dr.T's comments about the VA medical system (some of which is inaccurate and some points are simply misleading I will deal with them on a point by point basis below) it still manages to pull up the medical records for any vet at any hospital. This is QUITE an accomplishment.

"The VHA Office of Information and Technology has been working on a relatively simple project (a Windows-based informatics module for blood bank labs) for over 8 years."

This has nothing to do with the system written in MUMPS. It's merely an example of the failured project. The system Krugman is lauding is the Vista system. He's right to laud it. At least one VC backed company is taking open vista to hospital s for customization and deployment. (The VISTA system since it was written by the government is public domain). It has spawned an open source project openvista.org.

"The existing VHA electronic medical record system is written in MUMPS, an archaic programming language that was created in 1958!"

This particular point is meaningless. We have been taught with computers that newer is better! Well that's not always the case. Perfect example Linux which is just a variant of the Unix operating system which has been around since 1968! They are called legacy applications for a reason. Dr. T is misstating several facts there. From the Wikipedia,

"MUMPS (Massachusetts General Hospital Utility Multi-Programming System), or alternatively M, is a programming language created in the late 1960s for use in the healthcare industry. It was designed to make writing database-driven applications easy while simultaneously making as efficient use of computing resources as possible. Although it never gained widespread popularity, it was adopted as the language-of-choice for many healthcare and financial information systems/databases (especially ones developed in the 1970s and early 1980s) and continues to be used by many of the same clients today.

Because it predates C and most other popular languages in current usage, it has very different syntax and terminology. It offers a number of features unavailable in other languages and showcases some rarely used programming and database concepts."

What are those features you might ask? Also from Wikipedia (and some very cool features I might add)

"The core feature of MUMPS is that database interaction is transparently built into the language. Simply by using variables prefixed with a caret '^' character, you are referencing a database node. Assignment and retrieval uses the same commands as for interacting with standard RAM-based variables. Additionally, all variables (both RAM and database-based) can be treated as a multidimensional hash/array. Child nodes of a variable (called subscripts in M) can have numeric or string keys (the keys themselves are also called subscripts, such that with the variable name ^A("B",2,6), "B", 2 and 6 are the first, second and third subscripts of ^A). String keys are automatically stored in alphabetical order following all numeric keys. Numeric keys can have negative and/or floating-point values, all of which will be stored in order from lowest to highest. The MUMPS terminology for database-linked variables is a global, not to be confused with the C term for unscoped variables (see Variable scoping).

As a secondary language feature, you can abbreviate nearly all commands and native functions down to a single character to save space. Additionally, special operators exist to let you treat a delimited string (like Comma-separated values) as an array. To reduce the number of hard-disk reads, early MUMPS programmers would store a structure of related information as a delimited string, parsing it out after it was read.

MUMPS has no data types. Numbers can be treated as strings of digits, strings can be cast (coerced, in MUMPS terminology) into numbers by numeric operators. When a string is coerced, the parser turns as much of the string (starting from the left) into a number as it can, then discards the rest. Thus the statement 'IF 20Despite Dr.T's comments about the VA medical system (some of which is inaccurate and some points are simply misleading I will deal with them on a point by point basis below) it still manages to pull up the medical records for any vet at any hospital. This is QUITE an accomplishment.

"The VHA Office of Information and Technology has been working on a relatively simple project (a Windows-based informatics module for blood bank labs) for over 8 years."

This has nothing to do with the system written in MUMPS. It's merely an example of the failured project. The system Krugman is lauding is the Vista system. He's right to laud it. At least one VC backed company is taking open vista to hospital s for customization and deployment. (The VISTA system since it was written by the government is public domain). It has spawned an open source project openvista.org.

"The existing VHA electronic medical record system is written in MUMPS, an archaic programming language that was created in 1958!"

This particular point is meaningless. We have been taught with computers that newer is better! Well that's not always the case. Perfect example Linux which is just a variant of the Unix operating system which has been around since 1968! They are called legacy applications for a reason. Dr. T is misstating several facts there. From the Wikipedia,

"MUMPS (Massachusetts General Hospital Utility Multi-Programming System), or alternatively M, is a programming language created in the late 1960s for use in the healthcare industry. It was designed to make writing database-driven applications easy while simultaneously making as efficient use of computing resources as possible. Although it never gained widespread popularity, it was adopted as the language-of-choice for many healthcare and financial information systems/databases (especially ones developed in the 1970s and early 1980s) and continues to be used by many of the same clients today.

Because it predates C and most other popular languages in current usage, it has very different syntax and terminology. It offers a number of features unavailable in other languages and showcases some rarely used programming and database concepts."

What are those features you might ask? Also from Wikipedia (and some very cool features I might add)

"The core feature of MUMPS is that database interaction is transparently built into the language. Simply by using variables prefixed with a caret '^' character, you are referencing a database node. Assignment and retrieval uses the same commands as for interacting with standard RAM-based variables. Additionally, all variables (both RAM and database-based) can be treated as a multidimensional hash/array. Child nodes of a variable (called subscripts in M) can have numeric or string keys (the keys themselves are also called subscripts, such that with the variable name ^A("B",2,6), "B", 2 and 6 are the first, second and third subscripts of ^A). String keys are automatically stored in alphabetical order following all numeric keys. Numeric keys can have negative and/or floating-point values, all of which will be stored in order from lowest to highest. The MUMPS terminology for database-linked variables is a global, not to be confused with the C term for unscoped variables (see Variable scoping).

As a secondary language feature, you can abbreviate nearly all commands and native functions down to a single character to save space. Additionally, special operators exist to let you treat a delimited string (like Comma-separated values) as an array. To reduce the number of hard-disk reads, early MUMPS programmers would store a structure of related information as a delimited string, parsing it out after it was read.

MUMPS has no data types. Numbers can be treated as strings of digits, strings can be cast (coerced, in MUMPS terminology) into numbers by numeric operators. When a string is coerced, the parser turns as much of the string (starting from the left) into a number as it can, then discards the rest. Thus the statement 'IF 20

Other features of the language standard are designed to help applications interact with each other in a multi-user environment. Database locks, process identifiers and Atomicity of database update transactions are all required of MUMPS implementations that follow the standard."

All companies have old applications like this running. For example Instinet, moved their trading application (written in Cobol, running on VAXs) to the Linux hardware platform. How did they do this? They clustered linux servers together and ran an emulation of the VAX on the cluster. In short they didn't migrate any software at all but faked the old hardware on faster commodity hardware.

The VC founded startup that is taking VISTA and making it useful for hospitals is

http://www.medsphere.com/company

The current CEO is Larry Augustin, founder of VA Linux Systems who knows a thing or two about software (wrote Bison++, ran one of the largest linux companies on the planet, Linus Torvalds is godfather to at least two of his kids). Older isn't always worse in terms of software design. Older designs often had fewer resources and thus were more efficient. The abundance of hardware resources has lead sloppy coding practices which makes applications brittle.

Let's see the VCs funding them

Azore, Thomas Weisel Venture Partners, Wasatch Venture Fund. All top flight VCs whose due diligence was done. That should tell you something about VISTA. It's good enough for a software VC to spin off a start up on it.

In the end, most large IT projects fail because the requirements of a bureaucracy are incompatible with the requirements of software development. Governments and large companies, the kind of organizations that take on large projects in one big bite, are the organizational structures least suited to correctly planning, designing, implementing and testing large projects.

Agile development (and the agile architecture offshoots of it) are an attempt to fix the problem, but they are almost always (in my experience) snowed under by the bureaucratic requirements for control (sponsor must know every detail of every aspect of every project, at least theoretically, resulting in dead forests for unread documents and in countless hours wasted in meetings producing only an "informed" management) and panic response (as the manager panics about some perceived problem, all work stops to resolve the panic, frequently resulting in unnecessary rework as schedule and scope changes unpredictably.)

Most experienced IT people can tell you what makes a project fail. Most IT managers cannot.

Slightly off topic, but you did bring it up: Wang lives on, now as part of Getronics, a Dutch computer service company. Its systems not only continue to be used and supported, but customers now are moving from proprietary hardware to a Linux-based emulation.

Tylerh said,

"Given that businesses can hide their IT failures easier than public
agencies,..."

Actually it's my impression that the government is awash in IT failures.
Large and small, they are occuring all the time. Probably less
than 0.01% of the failures are ever reported by the media.

One may wonder why, but our modern media is not interested in reporting
government failure unless it's dramatic.

I doubt there's a week that goes by that some significant IT project
doesn't fail or is quitely dropped in Washington, DC. These are not hard
to find, all one has to do is pick a project at random and wait. The
odds are pretty good.

One poster muses that perhaps it is possible that the higher administrative overhead of the private insurance companies is worth it because they are so much better at weeding out fraud. But this too is a bogus argument because the private insurers aren't interested solely in rooting out fraud but also in denying legitimate claims and delaying them for months if they can't be completely rejected. Ask any doctor's administrative staff, you know, the one that is so big because of red tape that makes the government bureaucracy look piddling.

I think I'm the one you're talking about in this paragraph. The point isn't that private insurance is better against fraud, the point is that the whole complex of marketing and red tape is set up to decide who gets treated for what, and who pays for what.

It's annoying as heck, but insurance companies being utter jerks about paying for legitimate expenses is a major drag on escalating costs.

If you were setting up an entirely new health care plan, you would still have to answer the basic question of who pays what to get what. I'm not sure that it would be a good idea to scrimp on whatever system you have that makes that decision (which you'd pretty much have to do in order to squeeze $700 billion out of reduced administrative costs). Approving unnecessary procedures directly increases medical expenses. Disapproving necessary procedures directly hurts patient well-being. Considering that the whole point of your hypothetical system is to increase well-being while holding the line on total societal expenditures, do you really want to cut costs right at the point where those two major concerns meet? Any savings you make are directly competing with the hidden costs of making bad decisions in a crucial area.

It's like a baseball team trying to save money by firing scouts. The reduced expenses look like free money, but the cost of firing scouts is that you make bad decisions on signing players and pay too much for bad players or don't pay enough to land good players.

When talking about colossal government IT disasters, don't forget the IRS flushing $4 billion down the toilet in the '90s on a system that was just about as complete a failure as you can have.

As a general case, there is no such thing as a software project that will last five years. If it can't be done in two years, then it can't be done.

The exception is if you have both (a) extremely deep pockets, such as Microsoft, pre-Microsoft IBM, or the government, and (b) really smart people, like Microsoft, pre-Microsoft IBM, or, well, maybe just those two.

Dal,

Your mentioning of microsoft and IBM means that you do not really understand IT projects. Microsoft usually has very little do to with IT projects (which are very different from software projects). Examples of successful IT projects are the Airline reservation system, Wal-Marts POS system, the hotel reservation system, etc.

The question should be is why Hilton can have all of their hotels connected to the same reservation/inventory control/Frequent flyer/point of sales system but that the federal/state/local government keeps asking me for my phone number.

Comments are Closed.