Disastrous Oracle Implementation At Europe's Largest City Council. (theregister.com) 98
Longtime Slashdot reader whoever57 writes: Birmingham City Council, the largest such entity in Europe, has been declared effectively bankrupt. There are a couple of reasons for this, but one of them is a disastrous project to replace the city's income management system using Oracle. The cost of this has risen to $230 million, while the initial estimate was $24 million. There was a failed rollout of the new system earlier this year. "Original plans for the replacement of SAP with Oracle Fusion set aside a 19.965 million-euro budget for three years implementation until the end of the 2021 financial year," reports The Register. "Go-live date was later put back until April 2022 and the budget increased to 40 million euros. After the council realized it would need to reimplement all of Oracle, the budget for running the old system and introducing the new one increased to 131 million euros."
"In a hastily convened Audit Committee meeting this week, councilor heard how that date has now been put back until November, expressing their anger that the news hit the media before they were told." Testing failed with only a 73.3% pass rate and 10 severe deficits, "below the acceptance criteria of a 95 percent pass rate and zero severe deficits.
"In a hastily convened Audit Committee meeting this week, councilor heard how that date has now been put back until November, expressing their anger that the news hit the media before they were told." Testing failed with only a 73.3% pass rate and 10 severe deficits, "below the acceptance criteria of a 95 percent pass rate and zero severe deficits.
Perhaps Larry Ellison could help out (Score:2)
Toss a billion to Birmingham from the many that he's got, given it's an Oracle system that's a total mess...
Re:Perhaps Larry Ellison could help out (Score:5, Insightful)
Larry didn't make $400B by being generous.
Re: (Score:1)
Re: (Score:1)
Larry never gave a fuck about Oracle's customers. In fact he made his billions specifically at the expense of the customer base. Why would he suddenly grow a heart now?
Re: (Score:1)
Nog the only Oracle disaster. (Score:5, Interesting)
Look for the Millenium project fo VGR in Sweden where they were selected to deliver a new health care system that's also a disaster.
Re: Nog the only Oracle disaster. (Score:5, Interesting)
How come all of the so-called Oracle disasters end up requiring the client to pay Oracle 10x the initial estimate. I do not think Oracle views any of these as disasters.
How this company continually gets away with it is beyond me.
Re: (Score:3)
How come all of the so-called Oracle disasters end up requiring the client to pay Oracle 10x the initial estimate.
You are confusing cause and effect. These projects are designed to fail, because failure is where the profit is.
How this company continually gets away with it is beyond me.
They've been caught paying kickbacks for contracts.
That's how Gray Davis lost his job.
Re: Nog the only Oracle disaster. (Score:5, Interesting)
How this company continually gets away with it is beyond me.
Should ask: Why the hell are they using Oracle in the first place in spite of its extreme costs?
Considering $20 Million is money for enough work to develop a completely custom accounting software project and financial management solution from the ground up. You have enough money collectively there to found a new startup, develop a new product, and move to it... so why not plan to do that instead of being in a hurry to migrate to some unnecessary monstrosity that requires years of work to start implementing the product after buying software anyways? With custom-developed solutions.. the implementation effort is just part of software development process.
The answer to that question - Why are they buying an oracle product at that kind of price line.. probably has something to do with how Oracle is getting away with it.
Re: Nog the only Oracle disaster. (Score:2)
Until you need upgrade them to work on the latest os upgrades and find out that the company that wrote it no longer exists. Or no one left understands how it works.
Re: Nog the only Oracle disaster. (Score:4, Informative)
Except in the example above, the city would own the company that developed the software, mitigating all of those problems.
Re: (Score:2)
You have this backwards.
Oracle might no longer exist in the future, or they might no longer support the product, or they might significantly change their terms (see recent vmware news).
If they developed their own inhouse solution then they own it 100% and can always hire developers to work on it, which won't be difficult with a $20 million budget.
Also application software should be written in such a way as to not have hard dependencies on specific OS versions. It's not hard to do, and there are plenty of ap
Re: (Score:2)
or they might no longer support the product, or they might significantly change their terms (see recent vmware news).
Yeah.. If it cost $20 million for the team to Initially implement the project. Just wait to you see the quote of what it will cost 20 years from now to contract the software company for their team to help you implement the next major version of the software at a time after they Know all your data is locked in to their proprietary framework, and You are already paying a subscription
Re: (Score:2)
Until you need upgrade them to work on the latest os upgrades and find out that the company that wrote it no longer exists.
With the custom solution you can at least from a development management perspective require that everything is documented, and you can build your software upon a foundation that is not specific to an OS version - for example build your frontend using libQt, or a Javascript framework such as Electron, or as a website that runs in a standard browser with a Linux service running a standard
Re: (Score:2)
Likely skillful use of contracts and particularly the principles if ITIL and similar. Basically, you deliver what the customer asks for rather than what they need and keep repeating that until they give up.
ITIL, if you haven't heard of it, is a system designed to ensure that the customer is clearly and directly to blame for everything bad that happens to them whilst you have full documentation of that fact along with proof that you were locally reasonable every step along the process. It's got some integrat
Not a disaster for everyone (Score:1)
Re: (Score:2)
I mean: 131 million euros
You could hire 3,275 people at the going average salary in the UK ($40,000 EU) for a year with 131 million Euros. Or 328 people for ten years. Or 33 people for 100 years. Why do we seem to believe everything has to be done with software now, especially expensive software run by a madman billionaire?
Re: (Score:1)
Fixed price contracts (Score:4, Informative)
Make sure your contracts are a fixed price and the vendor has to suck up any overrun costs. Oddly these sorts of projects generally tend to come in on time and budget.
Re:Fixed price contracts (Score:5, Insightful)
I think you need penalty clauses and the option to take the contract away and give it to someone else if they fail to deliver on time. Otherwise when it all goes to shit they will quickly lose interest in fixing it since they can't bill for any more time.
Re: (Score:2)
I think you need penalty clauses and the option to take the contract away and give it to someone else if they fail to deliver on time. Otherwise when it all goes to shit they will quickly lose interest in fixing it since they can't bill for any more time.
This is actually exactly the trap here. If you do that, you end up having to specify what you want delivered, otherwise you can't apply penalties. Since, at the start of a project you don't actually know that fully, they end up delivering something useless on time and even under budget. If you realize what's happening before you get full delivery then you desperately try to negotiate changes, but the penalty clauses act as evidence that the changes are very very valuable. If you don't realize until it's del
Re: (Score:2)
Re: (Score:2)
Good luck getting Oracle to agree to a penalty clause they can't get out of. Sometimes I think Oracle is a law firm that happens to sell some software.
Re: (Score:2)
Re: Fixed price contracts (Score:4, Interesting)
Re: (Score:2)
A brilliant example of this, from outside software, is the UK high speed rail project to the North. According to the recent BBC podcast, because it was fixed price they put sound protection barriers all along the line into the original quote, even where they weren't needed. Then, when they realized these barriers weren't needed in many places because the line is already in a cutting they couldn't' get rid of them.
Re: (Score:2)
Re: (Score:2)
In this case the podcast. made it pretty clear that wasn't the point [bbc.co.uk]
Re: (Score:2)
Firm Fixed-Price contracts definitely do exist and are fairly common too
Re: (Score:2)
There's no way a company like Oracle enters into a firm fixed bid without a really robust change request process. You'll pay for the creep no matter what. You can argue the contract all you like but they have massive legal resources on hand that do nothing but this.
They write thousands of contracts. Most of the clients write a handful... maybe one. When the dust settles the contract structure will serve them better.
Re: (Score:2)
Firm Fixed-Price contracts definitely do exist and are fairly common too
There's a very specific understanding you have to have here. Small fixed price contracts exist and are actually a good idea sometimes. If you want to buy 50 high strength steel screws or a specific new report for your existing management console that's something that both you and the vendor understand. You agree what needs to be delivered, they do it, everything works.
Even in those little contracts something occasionally goes wrong. The vendor might have bought from a dodgy Chinese company that reduced the
Re:Fixed price contracts (Score:4, Insightful)
Re: (Score:2)
The UK government doing IT precurement hasn't exactly turned out grand in the past either.
Re: (Score:2)
I've seen a few software contracts in private industry and if they are negotiated by people who couldn't build the thing in the first place, then they are almost universally shit because the people paying for it don't fully understand the requirements. I can build software and am in a good place to negotiate small projects because I understand what's involved. If you've never been involved in building a system on that scale then you aren't going
Re: Fixed price contracts (Score:5, Interesting)
Phoning in from the 1980's (Score:2)
25% over would have been wildly successful. Some tiny projects would meet that threshold, but I became logically cynical early. 2-2.5 times budget would be about average, management never wanted to hear that testing was required. The big gov projects were multi year, multi administration cluster fucks. Each successive government would clean house, new buddies would be hired, and in
Re: (Score:2)
Cheop's Law: Nothing ever gets built on schedule or within budget (Robert Heinlein)
Re: (Score:2)
I went looking to see if there was a vendor running the implementation. Only went two articles down but I didn't find them. I found Oracle happily selling them the tech, but not necessarily the transformation work.
So are they rolling their own here? I mean, Oracle has been in the chair for lots of spectacular failures, but is this one of them?
Re: (Score:2)
Make sure your contracts are a fixed price
Contractee: "We insist on a fixed price contract."
Contactor: "No problem. We agree."
One month later:
Contractee: "We need to make some small changes to the spec."
Contractor: "That's gonna cost ya."
So what was SAP and hosting costing them? (Score:3)
Transition from self hosting to cloud should probably break even for them before heatdeath, maybe.
Considering that Oracle is in the run for TikTok (Score:3)
*grabs popcorn*
Re: (Score:2)
Their own money or their customer's money probably makes a big difference.
Their first mistake was using Oracle (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
at 100 million dollars the alternative is anything they want and it"s double that.
For 230 million dollars you could hire a team of mercs to kidnap larry, throw him in the basement and use a brain computer interface to turn him into the ERP system
For 230 million dollars you could just buy up all the old as/400s from scrap and staff the whole thing with boomers who never bothered learning Java or vb.net. (i know a place that actually did something like this because they prefer getting raped by IBM over payin
Re: (Score:2)
if they could fulfill the needs of the Birmingham council.
What special snowflake needs does Birmingham have that makes it any different from the hundreds of other cities running off-the-shelf solutions?
Re: (Score:2)
It's catching (Score:4, Interesting)
The people signing off these contracts have no idea they're about to get eaten by sharks and we all pick up the tab.
Re: (Score:3)
Oracle have been trying a new trick recently.
When they quote for a piece of work, the customer looks interested and asks about the payment schedule. "All up-front" the Oracle rep says.
"You're insane! Even if we trusted you like that, we don't have the money in this year's budget."
"Calm down, we can help with that. Let me introduce you to my friend over here at Oracle Financial Services. He'll lend you the money to pay us up-front and you pay it back in easy instalments."
"Eh? Okay, well, if that's the w
Re: (Score:2)
TIK TOK should go well. (Score:3)
What a shit show.
No excuse (Score:5, Informative)
With decades of experience on similar projects, there should be no reason for this to happen. Birmingham's income management system could not be that much more complicated than any other income management system already done. Yes, it will have its own quirks, but the underlying process is the same as other income management systems.
We keep hearing about "you get what you pay for" when it comes to programmers and database administrators and project managers, clearly that is not the case. Time and again we see projects like this go off the rails despite all this supposed experience. Perhaps it's time to reevaluate our conceptions. Perhaps we should stop making excuses that it's management's fault and just admit the overpaid people doing the work don't really know what they're doing since obviously they know nothing of KISS.
In this day and age of technology, with all the resources at our fingertips, to have another project like this fail so miserably is inexcusable.
Re:No excuse (Score:4, Insightful)
With decades of experience on similar projects, there should be no reason for this to happen.
Oracle has decades of experience fucking everything up.
Get off my dick (Score:2)
Which bitch with mod points and oracle stock shit here?
Re:No excuse (Score:5, Insightful)
Yeah, I don't get this either. There seem to be a few factors.
Firstly, it's a big consultancy doing government work. The consultancy rates start at £1k per person per day for a junior and head north from there. Let's say they average £2k per person per day, so your £200 million overrun works out to about 100,000 person-days or about 500-man-years.
Secondly, the organisations they are doing the work for are admittedly large, complex, bound up in endless regulation and unwilling to change. When a small business adopts an accountancy package, they basically change their processes to fit the package. At a certain scale, "the accounting software won't let me do that" is the default answer to why the accounts department insists on an employee jumping through a particularly ridiculous hoop. But these organisations are big enough that they expect the software to adapt to them, except every little group of people does things slightly differently and none of them are willing to change. In some cases, changing would mean all sorts of regulatory compliance work or is simply not possible because of regulatory constraints. This is the most valid excuse for the mess Birmingham is in.
Thirdly, planning this sort of work is non-trivial. People are not endlessly fungible, so if the process mapping necessitated by point 2 runs over, as it almost certainly will, you can easily end up with a large software engineering department sat twiddling its thumbs waiting for the output of the business consultancy department. At £2k per day. Do you redeploy them onto other projects, knowing that when they are needed you will have to get a whole new team up to speed and delay the project by even more? Or do you bill the customer - who probably caused they delay - for them to sit and do nothing for several months?
But it beggars belief that any piece of software can take 500 man-years to write and test. Especially when it's not being written from scratch, it's mostly just configuring an off-the-shelf piece of software.
Someone, somewhere should have learnt how to do these projects well by now and be eating Oracle's lunch for them.
Re: (Score:2)
Yeah but every globo corp ive ever worked at had a massive suite of expensive and barely functional software running things. I people are deluded thinking the software has adapted to their practices and not the other way around.
Re: (Score:2)
As much fun as it is to blame Oracle ( dog knows they deserve it ), this reeks of incompetent project management government-side. I can almost guarantee not a single person knew even a fraction of how their existing system worked or what it delivered, so during requirement gathering meetings would rattle off insane "wants" and "needs", while ignoring the real needs and requirements.
Oracle, happy to charge you as much money as you authorize, just went along with it I'm sure.
Re: (Score:3)
As with most IT boondoggles, there's plenty of blame to spread around from both the management and consultant side of the transaction. Even where seemingly water tight contracts are in place with KPIs, milestones and penalties, sooner or later the sunk cost fallacy will get triggered. The consultants know this, which is why quotes are largely fictitious.
I don't know what the solution is. Having been on both sides of that coin, I've seen how getting customers to come up with a well-defined spec and resisting
Re: (Score:2)
Sadly, that will probably also be the day pigs develop wings.
It's sad to see a municipality bankrupt itself because nobody had the balls to say, "enough is enough".
Re: (Score:1)
> Sadly, that will probably also be the day pigs develop wings.
I'm from Cincinnati and here pigs fly on a regular basis. :-)
Re: (Score:2)
Re: (Score:3)
Dude when projects this size fail it is always managements fault. They may have outsourced to an outsourcer that pays vibe coders on fiverr ... but it is managements fault that has happened.
My first question would be Did you choose Oracle because you thought it would be easy to defend as a prudent choice?
Re: (Score:2)
This is gross negligence coupled with gross incompetence: not a fixed-price project and apparently no penalties for non-performance. This is also not a very demanding type of project. I suspect there was a massive amount of corruption in the mix as well and a completely unsuitable implementer got the contract.
At the very least those responsible for this mess should be held personally accountable.
Re: (Score:2)
This kind of thing has been going on for at least 30 years with ERP systems. Usually the stories don't make it into the news. What's exceptional is that this one did.
Re: (Score:2)
Incompetence is only part of the story. In my meager experience, what tends to happen is that everyone who has a little fiefdom throws roadblocks in the way of anything they perceive as a threat to their gravy train. Trying to accommodate all such demands -- demands which are supported by higher-ups due to politics -- results in extreme complexity and often in logical inconsistencies.
Deployment went live despite numerous red flags (Score:2)
“The cloud-based enterprise system was expected to cost less than £19 million. It has already cost £171 million and played a decisive role in pushing the council into bankruptcy.”
Re: Deployment went live despite numerous red flag (Score:1)
All of the IT/tech press focuses solely on the Oracle overspend as being the reason for the council's bankruptcy. But the council also recently agreed a £760m settlement in a long-running pay discrimination case.
Yes the Oracle project has been a screw-up but it is a small part of the much wider problems in Birmingham.
Replacement of awful by terrible (Score:5, Funny)
replacement of SAP with Oracle Fusion
"We like pain and misery, but we'd like to get it from a different vendor".
Re: (Score:3)
I moderate you "funny but sadly true".
Re: (Score:2)
Genuine question from someone who has never worked in this space: What were the better options?
Re: (Score:2)
Red hot nails in the eyeballs is generally cheaper, less buggy, compatible with existing hardware, easier to maintain and deploy, and causes less pain to users.
Re: (Score:2)
Councils are the worst (Score:3)
When big organisations say, “we’ll modernize”, but then insist on keeping their old, messy business processes, they force the new system (SAP, Oracle, whatever) to bend around decades of legacy practice. That leads to:
Heavy customization fragile, expensive, hard to upgrade.
Data migration nightmares because the old crap gets pulled forward instead of cleaned.
User resistance staff cling to “how we’ve always done it.”
Project sprawl costs explode, deadlines slip, and the shiny new system just becomes a clunky version of the old one.
The councils and agencies that actually succeed are the ones that use the system to standardize, they adapt themselves to SAP/Oracle best practices instead of bending the software.
So yes, Birmingham didn’t really “modernize”, they basically tried to recreate their old SAP mess inside Oracle Cloud, and the cracks just widened.
Re: (Score:2)
At that price range we are talking custom software to for custom solutions. There is no forcing old systems into new. This is not an off the shelf accounting system.
On one end we have government who are used to passing laws without consideration to the consequences and on the other we have greed with lawyers who do not take responsibility for the products they are peddling. This is all done behind closed doors under a pretext that this is democratic but leaves the citizens holding the bag. Why would anyone
Re: Councils are the worst (Score:2)
Common knowledge, I say!
<MBA slams table>
That's what they told me anyways, and I had to look to different industries for contracts, because analysis and control was old fashioned, why reinvent the wheel?
Re: (Score:2)
"they basically tried to recreate their old SAP mess inside Oracle Cloud" That was inevitable and should have been foreseen. Their experience was with SAP, they didn't appear to have any experience with any other and SAP probably kept them in the torture chamber long enough that their institutional memory was stuck on SAP. So when they specced a new system, they expected they could do what they did with the old system and respecced their old system....or more likely simply specced what they knew, it was eas
Re: (Score:3)
Having been through several migrations of various ERP's and finance packages with my company, we always negotiated a fixed price contract, then we go by the "Configure, don't customize" motto and we will change our processes to match the software whenever possible rather than trying to customize the software.
When we migrated to Oracle, this led to an on-time and on-budget deployment of e-Business suite (Then version 11.something). There were a couple of areas where we needed custom code, mostly due to some
Looks like a project issue, not a platform one (Score:5, Insightful)
Re: Looks like a project issue, not a platform one (Score:2)
This. A serious vendor will convince the customer to changes their processes.
We used to sell a school admin system for adult schools (language, dance, etc ). One old-fashioned school owner, way past retirement age, was used to hashing out an individual price for each and every customer. It took a few meetings, but we finally convinced him that he *had* to have standard prices. Sure, he could offer discounts, but the system needed some sort of official price for his products. If he hadn't given in, we would
Who were the implementaion company? (Score:5, Interesting)
The linked article does not mention which IT company was involved.
This article https://www.cio.com/article/38... [cio.com] mentions EvoSys who I had never heard of before this
"Although the council had expected to implement the system out-of-the-box, it made customized modifications including the introduction of the BRS, which failed to function as planned."
Adding customizations defeats the purpose of implementing a package solution
Accountability (Score:1)
Re:Accountability (Score:5, Insightful)
I'm going to assume this was orchestrated by a smooth-talking consulting agency. They're known to promise you anything you ask for, bite off more than they can chew, and pass the costs on to the customer. Many of them will play the roll of friend and advocate, while charging outrageous mark-ups and billing for unnecessary services. They're very good at spending other peoples' money. And if it's a small-time group (or individual and handful of their firnds), they won't even have enough assets to reimburse you if you successfully sue them.
I spent some time working for a consulting company, and the president made it very clear that I worked for them and not the customer, and was to make decisions and tell the customers things that were in the consultant's best interest even if not in the client's best interests. Some may be okay with that, but it really ground my gears. People hire consulting groups because they're not experts in a field, trusting them to BE the experts, and make decisions in their best interest. Often they get neither. Some consultants do a great job of looking like they know what they're doing while scrambling in the background trying to make it happen when they lack the knowledge and experience to pull it off. Then by the time the client figures this out, a great deal of money's been burned without results. Worst case, they've committed you to a course of action and stalled out your company mid-stream in a transition when it all falls apart and it becomes obvious they can't handle it. This leaving you stuck in the middle where you can't function as-is, can't go back, and they can't figure out how to finish the transition. And that's what this is looking like.
Consulting groups like that really ought to have to be bonded and insured in case they leave you stranded in a bad spot like that. At least then you'd have a safety net that can drag you back to where you were before critical systems got ripped out, and you can get back to business.
Re: (Score:2)
Re: (Score:2)
don't worry (Score:1)
Peter Navarro our glorious commerce secretary announced Oracle will be America's "great firewall"
all the competence on display here will do is well
How hard can it be to count nowadays? (Score:4, Insightful)
When the cost of counting how much money you have coming in, the income management system, is greater than the amount of money you actually have coming in I can't help thinking somebody did not think things through very much. If at all.
Wut? (Score:2)
Oracle is selected because of ho's, not merit (Score:1)
Oracle wines and dines executives.
Oracle disaster? (Score:2)
Hmmm... must be a day that ends in "y".
Wow what a shocker! (Score:2)
Life is wonderful when your spending other peoples money!
The UK's New Digital ID (Score:2)
And? Did the city have penalties in the contract? (Score:2)
No? Sounds like criminal behavior to me. The responsible city officials should face a rather long prison time with damage this bad.
I worked for a state agency in the late '80s (Score:3)
Finally we got a new IT director who cancelled the project, we took it over and developed it in-house in two years and it worked great. It did exactly what we wanted and was very reliable. After I left the state, I was working for the police dept (through the '90s) who had an insanely complicated SQL Server payroll pre-process system, it crunched our OT and leave forms into standardized input for the city payroll mainframe process. Every rank it seemed had a different union contract and different rules for how they could be paid overtime or hold the pay for comp time off. The city wanted us to scrap that system and use their Peoplesoft system that they'd implemented city-wide.
We agreed to a meeting with their sales rep and one of their consulting programmers who knew the city system. We had 2-3" of documentation to present. My boss started describing how if you were a basic officer and didn't get your lunch, and elected to get paid for it, you got a code X, but if you wanted that to be held as comp time off, it was a code Y. Programmer smiles 'We can do that'. My boss continues 'But if you're a sergeant, that's a B or an F'. Programmer starts looking a little concerned. 'And if you're a lieutenant, that's a D or a Z.' Programmer starts looking a little twitchy. And then we started talking about getting called back out after the end of a shift, and all sorts of weird things that police do because it is a job with strange demands on your time. My boss did a brilliant job of designing his presentation to steeply ramp up the complexity of how our system worked.
Eventually the programmer's response became 'We can't do that' and for over a decade after I left, as long as I remained in contact with people there, that payroll system was still in use.
Sometimes it’s better to just keep legacy co (Score:2)
Failed, exploding-cost projects like this are a good reason to consider whether replacing your legacy stack is worth the risk and expense.
Sue (Score:2)
Seems to me that signing a contract for x, and being asked for ten times that, and it FAILS, is "fraudulent contract".
Being a government, they can even file criminal charges.
Far from the Only Failed Oracle Implementation (Score:2)
I know of a few personally that are YEARS delayed. Oracle skates on through all these failures knowing that their customers will be too ashamed to talk about these massive failures.
It would be wise for them to go public and maybe bring suit in a class action.