Dev vs. Ops: The State of Accountability (overops.com) 92
Here's an analysis by OverOps on how shared accountability affects the delivery of reliable software in a DevOps environment, and what are some of the top challenges teams face when it comes to building and maintaining quality applications. Conclusion from the report [PDF], which relies on a survey of over 2,000 IT professionals around the globe : At the center of this DevOps adoption chaos is the evolving relationship between development and operations. Many organizations are already taking a shared approach to accountability for application health, however they still lack the tools and application visibility needed to know who is ultimately responsible for addressing and fixing each issue. As the lines between these two teams continue to blur, organizations will need to focus on adopting tools that deepen visibility into their applications. Clarifying ownership of applications and services, and avoiding the "multiple owners = no owner" syndrome is a crucial for even the most bleeding edge organizations.
The "Dev vs. Ops: State of Accountability" survey revealed that as more organizations begin the transition to DevOps workflows, defining roles and processes becomes more difficult and more important. Furthermore, businesses of all sizes are building and releasing new code and application features faster than ever before, which adds additional pressure across the entire software delivery supply chain. Organizations going through the DevOps transformation are more likely to face visibility challenges that make it difficult to maintain or improve application quality and reliability.
The "Dev vs. Ops: State of Accountability" survey revealed that as more organizations begin the transition to DevOps workflows, defining roles and processes becomes more difficult and more important. Furthermore, businesses of all sizes are building and releasing new code and application features faster than ever before, which adds additional pressure across the entire software delivery supply chain. Organizations going through the DevOps transformation are more likely to face visibility challenges that make it difficult to maintain or improve application quality and reliability.
And thats why (Score:1)
... devops actually sucks just like every other âoeagileâ fad that came before and produced nothing of value other than the ability to avoid accountability
Re:And thats why (Score:5, Interesting)
Without devops, devs tend to product projects that are brittle and need to be micromanaged. Why would devs care if ops gets called at 1am on a Saturday? If the devs are getting called in the middle of the night, they'll start to care.
Ops should be limited to maintaining infrastructure services like VMs and creating tools to allows devs to deploy to prod on their own in a controlled way.
Re: (Score:3)
Ops should be limited to maintaining infrastructure services like VMs and creating tools to allows devs to deploy to prod on their own in a controlled way.
That's a valid approach for sure. What I've found in that case, however, is that taking Ops out of the loop and relying completely on developers for operational response to an incident in production yields poor results. Put simply, it's not their core competency, and most don't take to it well (although some do!). It's a very different type of pressure to which they're unaccustomed.
In my experience, a hybrid approach is best, where Ops personnel quarterback a production incident (or Support, assuming it's a
Re:And thats why (Score:5, Insightful)
Unless your project is very small or your management is stellar this just builds a road to disaster.
Developers who do not feel the pain of their actions are not incentivized to correct issues. In the beginning, there may be a few bright eyed engineers who take to heart the messages, but eventually that gets lost in the sea of priority. A number of things start to happen which can increase page count. It could be a hard to find bug that really only produces a few escalations, but combined with a number of those the issues can severely add up. Increasing escalations that don't get attention because individually they are not severe enough and well the operators can handle those.
Perhaps the largest issue only hinted at is the brittle nature that tends to evolve. It may take the form of circular dependencies or poorly considered dependency trees. Oh, such as taking a dependency on a tier 2 service for your tier 1 service or perhaps no one discussed the volume of traffic they would be placing on this other service. Those types of failures start to creep in when you disconnect yourself from your platform and multiple specialized teams start making changes to a cohesive whole. The reality is, despite everyone's hopes, is that living in the service keeps you connected to the details. Personally, I witnessed this type of degradation occur so severely I was likely one of maybe three people who understand the entire platform in an organization of hundreds.
The last service I helped build and design was built with the concept of "I don't want to be paged in the middle of the night." We didn't get it to that point by passing the micro issues to an oncall. No, the other teams did that and many of our pages are the result of those teams poor implementation or lifting work to our successful team.
I've seen whole operation teams nearly up and quit after development teams were completely disconnected from their projects and shouldered maintenance. It usually goes that way or you get some very battered and dedicated people who eventually trickle out until the lynch pin fails.
Pain is a great motivator to fix problems or at the very worst fix their poor alarms.
Re: (Score:3)
Re: And thats why (Score:2)
It's a balancing act, but if the devs can't work with the ops tools, the ops tools should be reworked until they can. Devs are generally smart. If you can't build tools they can use, then you probably aren't managing your ops issues well enough. You're probably reacting to problems instead of fixing them.
Re: (Score:3)
Bullshit. I've never known a dev not to care about maintainability and stability of their service. What devops causes is burnout for developers, as people who never signed up for 24/7 on call get forced into it, and people who know absolutely nothing about being a sysadmin get forced to do it, and do it half assed as a result.
Its one thing to have devops at a small startup, where you literally can't afford more people and need to wear multiple hats. At anything bigger, its a complete joke- you either ha
Re: (Score:2)
I'm perfectly prepared to be outsourced- I have all the manager's emails, and I'll let them know that my hourly rate is 4x what I charge as an employee to fix things after they fuck it up. As the inevitably will-- the percentage of oursourcings which work is way down in the single digit percents. The number that work but don't end up costing more in the end are even smaller.
Not to mention your argument is non-sensical- if they can outsource dev, they can outsource ops. So it would make someone no safe
Re: (Score:2)
Sure they do. And ops != architecture at any rate.
Re: (Score:2)
Without it, devs just throw the responsibly to operate over the wall and divorces them from the consequence of poor design.
I have no clue where you have been working (and hope never to work there), but in all the places I have worked, dev is *NEVER* divorced from the consequences. If there is a problem that isn't obviously hardware (failing disk, machine rebooting from kernel panics, router failing, ...), dev, almost always in conjunction with ops works on the problem. Sometimes the problem will still be hardware (a failing drive in one system manifested as a DB slowdown in a different system), sometimes it will be in the app.
Re: (Score:2)
Devops the concept is great, "devops" the buzzword is just adding more complexity with little or no benefit.
No, the whole thing is a dumb concept invented by people who don't know how to run a software shop properly.
The idea of devops is the devs design their projects such that they maintain the operations. ...
Without devops, devs tend to product projects that are brittle and need to be micromanaged. Why would devs care if ops gets called at 1am on a Saturday? If the devs are getting called in the middle of the night, they'll start to care.
Again, what the hell? I deny your above premise scenario as complete BS (or incompetence). This is NOT the way ANY software dev shop should be run!!! I've been doing professional software dev for 20+ years and have worked in half-a-dozen places over that time - and in every single shop I have worked in, the devs have ALWAYS been "third line support".
First line of support: customer service reps handlin
Re: (Score:3)
When I first heard about devops it was in the Ruby web dev community ~2007, and it referred to a role that was a liaison between the programmers and the sysadmins. Their job was to understand the dictates of the BOFH, and to help the programmers find a way to implement what they wanted in a way that was consistent with all the organizational rules. Mostly security or technology restrictions.
Then whoever decided to pay to hype it changed it to mean some sort of management service, a type of telemetry for the
Re: And thats why (Score:5, Insightful)
Sysadmins mostly work at Amazon and Google and Microsoft now. So, devops is largely now the process of automating cloud services.
Re: (Score:2)
That doesn't differentiate between anything, though.
We already had a word for the person that automates VM services; sysadmin.
Re: (Score:3)
DevOps is pretty cool when done correctly, where infrastructure is fully automated to the point where can you deploy new servers with the latest code and security fixes with just a mouse press.
Of course, most organizations don't have the resources or technical skill to pull that off or maintain it correctly. Worse yet, some of those same organizations also tend to be staffed with clueless managers who think that "DevOps" means that they can hire a junior developer out of college to replace their senior syst
Re: Reliable software? (Score:2)
Apple is based on Unix, not Linux. And as someone who spends a lot of time writing shell scripts, I assure you they are quite different.
More Crap!! (Score:5, Insightful)
IT and the "movement disease".
I am sort of tired of this constant "revolution" garbage that surrounds the IT industry in general. I work in this shit industry, I am well paid for what I do, but one thing is always certain... It will always suck because everyone in charge of IT came from college where stupid is the only thing being taught when it comes to computer science.
There is never anything innovative being done, by the time I am done listening to a sales pitch I realized I have heard all of this shit before, it's the same shit product emulating another shit product surrounded by proprietary technology that works like shit just in a different shitty way.
There is also the problem that every industry has... 20% of the folks do 80% of the work. Do you know what else tends to happen? 20% of the people are the only ones that knows what to do or what is going on. Do you know what else? IT is not a meritocracy either... it is still the same brown-nosing ass licking who you know path to success, like every other department. Those in the know are constantly assaulted by their lesser skilled and capable "co-workers". Those in the know are constantly waiting for some other knob in a different department to do their own damn job. And all of this while management keeps not getting a fucking clue and piling on more and more work to the point that more than 50% of projects fail by either never having the proper amount of time & expense dedicated to it.
These bullshit "cultural DevOps, ITIL, Agile, Waterfall, blah blah blah" are all stupid ideas people keep coming up with to address the problem of an industry that is riddled with incompetent management trying to rule over an incompetent group of pseudo intellectual nerds that know far less than they put on. And that is another problem as well... people hate IT personnel that do not sound "over confident" it is a practical requirement for IT pro's to act like they know every fucking thing there is to know and yet those of us at the top know different. We are all running around trying to figure out every little fucking thing on the fly because experience has taught us to just roll up our fucking sleeves and work it out... regardless of whichever newfangled fucking "operation ideology" that someone pushes.
Re: (Score:2, Offtopic)
Who asked you to whine?
I did. I was listening to Astral Plane by Jonathan Richman, and I thought, "gosh, are these devops guys just astral travelers, or what?"
Now, thanks to SirAstral, we know the answer. Or to paraphrase, "revolution in, revolution out."
Re: (Score:1)
old concept learn to operate a mainframe and nothing fundenmentally new will be under your sun
Re:What bollocks (Score:5, Informative)
Docker is getting more and more heavy weight, i.e. becoming full blown VMs (Though in the strict sense of the word they always were). Now part of the problem I see is that projects end up with containers scattered around like jack straws, like the "DLL hell" many experienced in the past, or plug-in hell. All docker does is allow the complexity to take a different form. Also stateless containers in my experience are pretty useless. Working on back end "heavy lifting" applications somewhere you need to maintain state and Docker and stateless containers, by definition, cannot do that.
Cloud is just putting the application somewhere else and paying by the cpu cycle. No different than before, it just makes it opaque as to what is going on and who is responsible. There's really not much new under the sun and good ideas are basically reinvented over time. Mainframe == cloud hosted apps on your mobi or browser. Nothing to see here, move along...
Re: (Score:2)
this is so spot on!!!
Re: (Score:2)
I thought you were supposed to maintain the state in whatever framework you're using to generate the docker containers?
I guess that is the problem; the more common practice is to receive containers rather than to build them. So whatever state you need is tacked onto the side.
IMO if you're using cloud computing to run your services, it should only be providing horizontal scaling. Nothing else should change. In that way of thinking, the statelessness is something you build into parts of your architecture by c
Re: (Score:2)
Re: (Score:2)
Application Virtualization is nothing new.
Just because you change it does not make it innovative.
Innovative means NEW... application virtualization is not.
I have been doing this for a while... sure I might be a problem, but likely only to the people recycling old hat like it is a new hat.
Re: (Score:3, Informative)
You sound like you're referring to corporate IT throughout your comment, or at least that's my impression. DevOps concepts don't really apply there...or at best it's a square peg in a round hole situation.
Building custom software, and more specifically SaaS, truly do have a lot to gain by adopting DevOps, especially when combined with Agile development.
Re:More Crap!! (Score:5, Insightful)
I have lived in both, once again... just because you change how things get managed it does not change the underlying idea I am putting forward.
No matter what you put forward the basic problem with incompetent people running the show exists. DevOps will fail just like every other doctrine before it.
I don't hate DevOps, Agile, ITIL, Waterfall or any of that stuff. They are all perfectly valid ways of doing things... and I do mean PERFECTLY VALID, the problem is that it is another smoke and mirrors attempt to move the goal post because no matter what you say or do... the blame has to go somewhere other than at the people in charge.
Re: (Score:1)
Re:More Crap!! (Score:4, Insightful)
Technology, aka tools, can never fix human problems, like incompetence. Incompetence in the work force is reinforced by incompetent people hiring others more incompetent than themselves.
Re: (Score:1)
Re: More Crap!! (Score:1)
Re: More Crap!! (Score:2)
Why do you think software should be innovative? Software is largely the translation of human-focused business processes into machine-readable code. If you are spending time "innovating" in this space, you aren't servicing your employer.
Some people get to work on innovative shit, but you shouldn't expect that to be the norm.
Re: (Score:2)
I've noticed that the technologies change quite slowly, but the buzzwords are redefined annually (;-))
For example, Communicating Sequential Processes were first described in a 1978 paper by Tony Hoare, but only widely adopted by Golang folks in the 200Xs.
You are the cause that it becomes shit. (Score:1)
don't fix what ain't broke too... (Score:1)
I'm in a situation now where a new VP of IT was hired late last year at the smallish (approx 500 employees total) company where I work but he came from a large multinational. Since coming in he started introducing various changes to our development and deployment processes, wringing his hands frequently about how bush league we supposedly are and frequently invoking "well in other companies they _typically_ do X".
The "funny" thing being that every time he introduces some new process that supposedly will im
Re: (Score:2)
Re: (Score:1)
Dev Ops is responsible for the Great Lab Hack where I work.
Short version: because some genius thought Dev Ops means that no one needs people with actual operations experience, we wound up with hundreds of old VMs that were spun up during some ancient sprint, then left online and never updated, never properly secured, and using weak passwords. (Some genius dev-op declared that all dev VMs for his team should use the same weak root password so every developer could use them.) Eventually someone found the lab
Dev Ops (Score:5, Interesting)
Dev Ops is an example of the willing, led by the unknowing, doing the impossible for the ungrateful.
Our Dev Ops team has adopted the slogan, "Delivering Yesterday's Technology Tomorrow!"
Re: (Score:1)
Monitoring (Score:5, Insightful)
I was our company's monitoring department and was checking systems and applications and it fits this question quite well, and you know what, IT SUCKS!
Between management that does not give two f**ks, developers who don't understand infrastructure and systems administrators who cannot manage applications, no one wants to be on the hook for anything. Just TRYING to get them fix issues without pointing a finger is a nightmare. It is like being the IRS, you never get a call from them saying you did a good job.
Everyone is afraid of looking bad because management, which does not understand IT or process, falls to politics to address issues and everyone else is afraid to make a move that make get them into trouble.
DevOps and Agile crap, will not fix broken management.
Jacks (Score:2)
DevOps typifies the phrase "jack-of-all-trades, master of none". This is a trend I've started to see where I work. A push to get everyone at least a little knowledgeable about everything, so anyone can fill in for anyone else. The idea that specialized skills allow a good team to be more efficient and productive is mostly lost on my coworkers.
Re: (Score:2)
There's nothing wrong with knowing a bit of someone else's job in addition to your own. If nothing else it aids understanding.
The problem is when you have a dev who sort of knows a bit about system administration because he kept the lights on that time the actual sysadmin was away on a course. Then some PHB decides you don't need the sysadmin any more, because the dev can do it. And then something crops up that isn't in the "For Dummies" book, and the safety rope - interrupting the guy on the course - is
Docker, Docker, Docker! (Score:2)
Here's the thread where we talk about developers bundling ancient versions of libraries with vulnerabilities in their Docker images, calling Ops people obstructionists, and then blaming them for security failures. Also, people who refused to use CI, introduce breakage, and then try to cast blame while someone else fixes it.
Don't work for a tech company where the CEO isn't an engineer.
Re: Docker, Docker, Docker! (Score:2)
Yeah, I'd say Docker might one day be a useful tool for production, but it's still a ways off. And some other shiny thing will probably appear before that happens.
Non-hierarchical delegated responsibility (Score:2)
The complicated interconnected nature of this sort of thing means issues land initially in one expert's lap, only to be delegated to another expert. The key is to be flexible and responsive to issues and not worry so much about who is responsible for what, but who is responsible now as an issue progresses. The hand-off of responsibility is more important than defining the responsibility.
Smart people tight teams make popular software tha (Score:2)
Non-coding managers will always be be prey to anyone or anything insulating them from the unacceptable truth that making software system sing is an art form more similar to making music than a production line. Find the right artists and keep the band together and sweet music will flow to your customers.