NYT: Healthcare.gov Project Chaos Due Partly To Unorthodox Database Choice 334
First time accepted submitter conoviator writes "The NY Times has just published a piece providing more background on the healthcare.gov software project. One interesting aspect: 'Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.'" The story does not say that MarkLogic's software is bad in itself, only that the choice meant increased complexity on the project.
follow the money (Score:5, Insightful)
how much do they contribute to XXX???
There has got to be some reason that this DB that ive never even heard of (and i work with DBs, its not my main point of work but I know my way around DBs) got the gig over the more established players.
or, perhaps they went with it because it is less known and therefore reduce the risk of known attacks in other DB systems?
Noobs? (Score:5, Insightful)
FTA: "An initial assessment identified more than 600 hardware and software defects — 'the longest list anybody had ever seen,' one person involved with the project said. "
Strikes me as none of these people seemed to have ever worked on large projects before.
NIH syndrome (Score:4, Insightful)
You could take a handful of proven DB technologies such as Oracle/DB/MSSQL, throw a web (Apache/IIS) and app (.Net/WAS/Jboss) front end to it, and it would work. Why did these guys fuck up the whole thing? It's like the scene in The Fountainhead when the second-rate architects smash up the plans and add their own stuff, "to express their own individuality". This could have been a solved problem - hell, it WAS a solved problem.
Re:follow the money (Score:5, Insightful)
from wikipedia,
The company’s main product, MarkLogic Server, is an XML based database management server.
This sentence has basically summed up the entire problem.
Blow to NoSQL movement (Score:5, Insightful)
And here is a fantastic example of what happens when hype trumps common sense. NoSQL is the new hawtness, and apparently the dumbasses running the project wanted to be part of that. Now MarkLogic, and NoSQL in general, will have a massive blow to their reputations, and it's unknown how badly this will hurt them.
As someone who has done databases for a long time, I have very little respect for NoSQL, but that is mostly because everyone keeps trumpeting it as a killer of traditional databases. There are scenarios where NoSQL systems are an ideal fit. However, NONE of those scenarios require data to be very reliably stored in a guaranteed and predictable way.
If you don't get your tweets or your friends facebook posts as soon as they are posted, no one will really care. But for something as truly important as health insurance coverage? Are you f__king kidding me? And that's just from a reliability standpoint. Nevermind the fact that NoSQL is currently at the wild west stage where nobody is compatible with anybody else, there is nothing resembling a standard set of APIs between products, making it very difficult to develop expertise.
Sounds like the Gov was just begging for problems.
Re:Stop with the excuses. (Score:5, Insightful)
Bullshit. Since the law has been passed almost the entire implementation has been up to the administration. Which in case you had not noticed is lead by partisans in favor of the affordable care act.
If the GOP can be accused of anything material in terms of interfering with the implementation it would preventing the law from being amended, and if it needs amending there is your proof it was bad law to begin with. Sorry Obama owns this and really HE owns it not even the larger DNC as its HIS branch of government that has been responsible for implementation from the moment HE signed the thing.
*IF* it fails its either because it was unworkable in the first place as written and should not have been enacted or because the Obama administration failed to execute. Those are really the only honest high level conclusions.
The argument well because they had to pass the Senate bill as is it did not get the usual fixes and tweaking is why its got so many problems is also bogus. They had to pass the Senate bill by abusing the budget reconciliation process to deny the minority party in the House at the time its rights; they knew if usual procedures were followed it would never have become law. So again it either should not have been enacted or the administration has failed to execute.
How cait it be the problem (Score:2, Insightful)
Re:NIH syndrome (Score:5, Insightful)
Why did these guys fuck up the whole thing?
All actions by Federal government players seem rational when viewed from their point of view.
The problem with most analysis, here or on other sites across the web, is in the assumptions. Specifically, if you assume that the government actions are for some benefit to the people, or efficiently completes some task (which is assumed to benefit the people), then nothing they do makes sense. That assumption makes for good outrage which can attract readers, but it doesn't answer the question of "why".
When government actions are viewed in light of more obvious motivations, everything it does makes sense. You start to see them not as an pack of incompetent, bureaucratic screw-ups, but as a self-sustaining, self-absorbed, engine of corruption.
I'm not aware of *anything* the federal government's done in the past two decades that's been in the interests of the people. All the useful stuff, the stuff you would want to keep across a reboot, was several decades ago.
I don't know the specifics of why an obscure database was chosen over the obvious players, but the reason wasn't "best utility for the people". Look for another reason.
Re:Open Source (Score:5, Insightful)
Re:follow the money (Score:2, Insightful)
Never attribute to malice that which can be explained by incompetence.
To me, this sounds like a pretty open and shut case of "Hey, I've heard that these 'NoSQL' database thingies are trendy these days. Let's use one of those!"
There's a difference between using fun, exciting new technologies and learning something new while doing that... and doing a project which stays in schedule and budget, is based on technology you already know thoroughly, and on which people's lives can depend (well, indirectly).
Re:Blow to NoSQL movement (Score:2, Insightful)
NoSQL predates SQL because we were storing stuff back then to. Relational databases are nice for consistency, but you sacrifice a lot of control and speed improvements for that consistency. A good NoSQL database and a good NoSQL programmer will always outperform SQL based applications. However, it means you need to have programmers capable of operating at that level AND you need to have solid strongly enforced conventions.
I also think that you are largely misreading the problem. Being able to get a quote and select your insurance isn't a strongly time sensitive endeavor. Really, most of the information you need is static. And as soon as you put data into a properly configured NoSQL database it should be available everywhere. There really shouldn't be race conditions of import for this sort of system. The system essentially asks you a few questions. You give it those answers and it returns a set from which you pick. After agreeing to some conditions you submit and are done; at least, that seems like the best thing.
I honestly think either platform, configured correctly, would work. And I program in a NoSQL database that predates C. Not C++, Not C#. I mean C. This kind of system scales just fine when you build indices and if your NoSQL database presents the right tools. I don't know anything about MarkLogic; but would you judge C by how a programmer operates in whitespace?
If you think NoSQL can't or doesn't perform in high risk environments, think again:
http://robtweed.wordpress.com/2012/10/21/mumps-the-universal-nosql-database/
Re: follow the money (Score:3, Insightful)
XML soo sucks for db applications.
Re:Blow to NoSQL movement (Score:4, Insightful)
Re: follow the money (Score:5, Insightful)
I used marklogic when I worked at a previous job and after learning how it worked and understanding it better, it made our jobs incredibly easy. It just had a serious learning curve.
Marklogic is a nosql db, that uses XML for its object format and xquery for its query language. This thing is NOT mongodb. It actually works really well and allows for complex data modeling with the ability to do joins and have transactional isolation in making changes to the data as well as a really solid content processing framework with pipelining and all that jazz.
Now, I can't imagine a reason for using marklogic, or any non-relational db for a project like this. The only clue is that marklogic has a lot of government contracts; mostly for the military. So maybe that's why it was used. But the fact that they chose a database system that they weren't experts in for a project that had so much visibility speaks volumes on how mismanaged this whole project was.
Re:Taxes... (Score:5, Insightful)
Re:NIH syndrome (Score:5, Insightful)
I would actually put forward that the problem is not the government, our government is pathetically weak in many ways. The problem are all the private groups flexing their muscles and making the government dance. Most of the people I have worked with on these kinds of projects are good people who just want to get stuff done, but they are weighted down by a nearly impossible set of requirements, many of which are mutually exclusive, dropped on them by outside groups who mainly are interested in showing off their power to their own people and have no investment in the project itself being successful.... and often have an active interest in the project failing since they can always use that as fuel for more personal power.
Re: follow the money (Score:3, Insightful)
You're exactly right! I've done tons of "exploratory" coding over the years myself, either using some new techniques or new products, just for the fun of it or to learn something new. But that would always be on small, low visibility projects where the consequences of potential poor performance or other issues would be insignificant. To tread new ground on something so big with national visibility is foolish. You'd want the most well established and known reliable and performant tools and techniques possible. I played around a bit with these object based databases in the early days, and for some uses such as simple dictionary type access to a serialized object they're good. But the query syntax is almost always XPath oriented and less than optimal for complex joins such as you would find in massive systems like this. I guess they've learned their lesson now.
Re: follow the money (Score:5, Insightful)
To me, the fact that it was a major problem is an indicator of bigger problems. With systems like this, the database should be just a simple repository for persisting an retrieving domain models. If they want to do any non-trivial reporting on it the data should be replicated to a reporting repository. Treating the database a a simple persistence repository makes most database operations trivial, and better yet, decouples the database from the business, meaning that if you couldn't even manage the simple operations, replacing the layer is fairly trivial.
The problem is when people think the database is the starting point for a system, not just a tool to be used to support what your business logic is doing. It sounds like that's what happened here.
Re:Blow to NoSQL movement (Score:2, Insightful)
> MarkLogic's NoSQL managed to land a giant lucrative contract for the venture capitalists ...
+24 insightful and informative. This.
Here's another angle on it. Haven't you ever wondered why everything associated with the government takes longer and costs more?
I once worked with a guy who did government contracts all the time. He said, "you deliberately underbid* to make sure you get the deal. Just be sure to put a clause in there protecting yourself when the government (which is Rule By Committee, remember) makes changes. After you get the contract, you KNOW the government will make changes. You can charge whatever you want and take as long as you want."
* he also said, and I quote: "and you buy the people making the decision a bunch of hookers and get them drunk." Basically his exact words. You'd probably have to update that now to a government that includes many females in management positions, but you get the idea. :)
Re: follow the money (Score:5, Insightful)
In fact, all of those would suck donkey balls. That is because XML, JSON and XQuery are all *data-interchange formats*. Using them to run your database internals is like using a screwdriver as a hammer.
Re:Noobs? (Score:3, Insightful)
Re:follow the money (Score:5, Insightful)
For them, it is a wise choice. It brings job security using a non mainstream RDBMS, which means that the data would take man-years to be migrated to Oracle, DB/2, or even MS SQL server which may not have the horsepower of a SPARC or POWER7 box, but one can always partition and cluster the RDBMS. It also is expensive to keep maintained, so it provides additional job security, as the expertise for this architecture is rare.
Great win for the contracting company. Big loss for everyone else.
Re:follow the money (Score:5, Insightful)
It definitely explains the problems they've been having.
But this is only part of it. Check out this diagram. [nrcc.org] I'm no expert, but look at all the government systems in the upper left that Healthcare.gov is supposed to communicate with, in real time. And on the right, 50 state Medicaid systems. And at the bottom, all the insurance companies. Far, far simpler IT projects have failed. This site will not be ready by the end of the month as promised, and there is a good chance that it will never work as planned.
International incident in the making (Score:4, Insightful)
But how would a traditional relational database scale to the 1 billion, or 1,000 billion users, huh? Did you think about the need to future-proof the application?
Current population of the U.S. is a little over 300 million. That includes children, people who have company provided health insurance, etc. who don't need to access HealthCare.gov so that the number of users of HealthCare.gov is expected to be about 30 to 45 million people.
The only way HealthCare.gov would need to support 1 billion or more users would be if we inflicted it on the China or India. That could lead to war or poor technical support and customer service. Although the later does raise an intereting question of who does someone at a call center call when they need technical support? Likewise, the population of the planet is between 7 and 8 billion. For the 1,000 billion users are you planning on also providing health insurance to E.T?
Cheers,
Dave
Re: follow the money (Score:5, Insightful)
Re:MarkLogic is an XML repository, not a RDBMS (Score:5, Insightful)
Sounds like the vendor building the system had a favorite hammer and decided that a rather traditional database problem looked like a nail.
It was the Medicare folks that imposed MarkLogic over the objections of the prime contractor. Calling Medicare a "vendor" is a bit of a stretch. Like non-IT staff calling themselves "customers" of IT in a corporation, as if they have a choice of IT departments.
Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.
The CGI argument was that it would slow work. I have absolutely no doubt about that. Every NoSQL system is it's own distinct thing with unique APIs, protocols, conventions, quirks, etc. This is fine when you are dealing with a limited group of good developers that can be expected to rapidly climb a learning curve. This is not fine when you are the top contractor in a herd of contractors, populated by a vast number of mediocre programmers that last learned anything new in the late 90's.
In that case the correct choice is to select the most ubiquitous technologies. A boutique NoSQL XML-base is not appropriate, even if it is excellent, which MarkLogic may be. I don't know. I just know CGI was 100% correct in objecting to it. CGI's fault in this case was its failure to win that argument.
Not that it matters much. The project was doomed in any case — political agenda driving pie-in-the-sky requirements using crony contracting — Fucked at birth, as they say.
Re: follow the money (Score:5, Insightful)
Three days restoring is too long too. Unless your server is remote and upstream is super slow.
This is becoming more commonplace as companies are outsourcing or contracting out their admins and development.
But none of these things should give you permission to be a control freak. You are responsible for keeping the databases up, yes. But you are not responsible for everything that happens to them. Sometimes your dbs just get screwed up and you'll have to fix it. Just like sometimes business wants things done to the site/app that will totally jack up the fung shui i had going on in code. Nobody is purposefully trying to screw anyone over. We all want the business to grow and succeed. People make mistakes and shit happens. There's no need to also be an ass to each other on top of it all.
You'd think that, but when it's your job on the line because fucktards in management have 0 clue and assume that because it has something to do with the database and you are the admin, it's your fault. And having been subject to intentional sabotage, and seeing it done numerous times, it exists and happens.