Life in the Trenches: a Sysadmin Speaks

Anonymous Coward writes "A senior systems administrator at a big ISP in Australia offers a no-nonsense view about his line of work, the pros and the cons, ths ups and the downs."

Re:Don't forget

[dipipanone]

a sysadmin has to be ethical.

I imagine that's why the System Administrators Guild [sage.org] has a SysAdmin Code of Ethics [usenix.org].

Hmm. I wonder if BOFH's also have their own code of ethics too?

"sysadmin": a very general term

[ilikehardhouse]

Personally, I think the role of sysadmin suffers from having so many different facets: supporting people/applications, installing software, adding/removing users, deploying applications/troubleshooting/dealing with security etc.

Because most people can do some of these things, they can end up doing sysadmin work. Does that make someone a sysadmin? I have interviewed for sysadmin roles before and always been amazed at the people who have used an application, or watched and install, and then applied for the sysadmin job. It's not enough.

The problem is, lots of people doing this kind of work without the training and experience (and often, no mentor either - nontechnical boss) give the profession a bad name - hence the whole BOFH subculture.

This link [infrastructures.org] describes some of the issues related to this job that isn't very mature at all ...

What about the processes?

[sbjornda]

I would have liked to see the article talk more about the processes SysAdmins should be following. If he's really working for a major service provider then where are his hooks into:

• Change control?
• Incident management?
• Problem management?
• Change window?
• Service level negotiations?
• Capacity management?
• Security management?

As long as all the SysAdmins seem to be making it up as they go along, we will continue to be marginalized and geek-ified by management. Try on for size:

• ITIL (IT Infrastructure Library) [itil.org]
• More ITIL [itil.co.uk]
• IT Service Management Forum [itsmf.com] (the U.K. headquarters)
• CMM (Capability Maturity Model) Technical Engineering Practices [cmu.edu]
• The Open Group's Technical Architecture Framework. [opengroup.org]

Heck, even Microsoft [microsoft.com] is trying to get into the picture with its Microsoft Operational Framework, a kind of embrace-and-extend on ITIL, though I don't know of many places that are actually using it.

It's not that the SysAdmin necessarily has to manage these processes - though in a small shop no one else will - but he/she/it needs at least to be able to talk the language and understand the processes that the IT Manager has set up. And if you are managing the shop, then this is your job. You must know this stuff as a matter of professional responsibility and "keeping up" in your field.

A 20 min. presentation to the other managers on Best Practices and Processes in IT Management will gain you a lot of credibility and help lift you out of the geek gutter. There are decades worth of lessons that have been learned the hard way and documented into these processes. When you can demonstrate to management that you are drawing on a substantial body of knowledge that is geared towards improving service and reducing total cost of ownership, you will gain their respect (assuming that you care about their respect).

Beyond this, I want to emphasize an excellent point that Sanders makes in the article. The SysAdmin job is one that is invisible if you're doing it right. A good day at work is a boring day. Excitement is a sign that something has gone wrong. You should structure your environment to be as boring and reliable as possible.

Too many SysAdmins live off the adrenaline rush of fixing a broken server while everyone else in the organization sits on their thumbs waiting. That's costly for the organization, but ironically is the easy way out for the SysAdmin - you don't need to be disciplined or structure your time or do any planning or thinking, just jump from crisis to crisis. It's much more challenging to turn it into a boring desk job where most of your work is pushing paper and the machines pretty much take care of themselves. But guess which option is better for the organization's mission?

Once you do get to that Nirvana state of boring life, you can strategize how to produce some measurables so you can blow your department's horn at the monthly managers meeting. Because if you do your job well, with the result that your work is invisible, they'll cut your funding unless you keep in their face on a regular basis.

.nosig

Re:Crap...

[ArmedGeek]

I wouldn't say that. He probably missed the machines...

I agree. I've got a couple of small pentiums at home that run a webserver, email, ftp, etc. They are never under any serious load and there aren't very many users, so the machines pretty well take care of themselves. That said, I still routinely ssh into them from work Just to make sure my babies are OK.

A recent flame war Craig got in to

[Kiwi]

If anyone has been paying attention to the Debian Linux ISP mailing list, people may recall that Craig recently got in to a nasty flame war with Dan Bernstein about the importance of a DNS server supporting the legacy BIND Zone file format.

To make a long story short (and the flame war got ugly), Craig feels that a DNS server needs to support the legacy BIND zone file format. Dan, obviously, does not; he feels that it only matters that one can transfer the zone file over to the new format (losing all comments in the zone file in the process).

Now, I will side with Dan here. Keep in mind that my viewpoint is rather biased, being the person responsible for the MaraDNS server [maradns.org], a server which Craig uses but feels is "poorly written code". Now, the only specific that Craig went in to when pointing out that he did not like my DNS server is that fact that, like Dan's TinyDNS, MaraDNS has no support for BIND's zone file format.

Now, with all due respect for Dan, I think he should not knock a gift horse in the mouth. The fact of the matter is that the code for MaraDNS is open; if support for BIND-style zone files is important to Craig, I suggest that he start coding it himself. I will gladly accept code which can read BIND-style zone files and make it part of MaraDNS.

I am not saying that BIND style zone file support is unimportant. However, I think Craig should be a little more courtious in requesting this feature than badmouthing MaraDNS on the Debian ISP mailing list.

I am sure he is an excellent system administrator; I really wish that he would start up a serious open-source project so that he understands how we OSS coders feel. I think it would make him interact with us in a more mature fashion; and save both him and the developers he flames some grief.

- Sam

P.S. I know Craig already knows this, but there is a non-BIND DNS server which supports BIND style zone files called NSI. It is on the list of DNS servers [maradns.org] on my web page.