Life in the Trenches: a Sysadmin Speaks 219
Anonymous Coward writes "A senior systems administrator at a big ISP in Australia offers
a no-nonsense view about his line of work, the pros and the cons, ths ups and the downs."
How many hardware guys does it take to change a light bulb? "Well the diagnostics say it's fine buddy, so it's a software problem."
Pros and Cons (Score:2, Funny)
Cons: Users
i would rather be a sysadmin... (Score:1)
Re:i would rather be a sysadmin... (Score:2)
Being a dedicated sys admin is like being a hobo, and thus the saying: "I maybe a hobo (sys admin), but you are a nobo (users)..."
problem solving skills? (Score:4, Insightful)
I think problem solving skill are a must for the sys admin job, especially if you don't want to be a Jr. Sys Admin and perform backups all your life.
I worked for a relatively large institution, in the capacity of a Sys Admin, and I know for a fact that you need some serious problem solving skills.
Re:problem solving skills? (Score:3, Funny)
Admins are the janitors of IT. If they're lucky they're allowed to write a few perl scripts and run them in a 'production' setting. If they're unlucky, the best they are allowed is to push around little users for power trips. Kinda like the janitor and his floor sweeper. You'd better get out of the way when he goes through with that floor sweeper at 7PM each night...
Re:problem solving skills? (Score:1)
Re:problem solving skills? (Score:2)
It's especially rewarding when the little user you are pushing around is a programmer.
Re:problem solving skills? (Score:5, Funny)
I think hands are a must for the sys admin job, especially if you don't want to be a Jr. Sys Admin and perform backups (with your teeth!) all your life.
I worked for a relatively large institution, in the capacity of a Sys Admin, and I know for a fact that you need some serious hands.
Re:job ad (Score:2, Funny)
Wanted: Web master. 10 years experience.
Re:problem solving skills? (Score:5, Insightful)
I know that I personally would much rather work with someone who was an ace problem solver and a quick learner, but who had little technology knowledge, than someone who had memorised every certification book, but was unable to apply that knowledge to real-world problems.
what article did you read? (Score:2)
I think that having a good understanding of how something works is far more valuable than having a specific rote procedure to follow. If you understand it, you can deal with situations that haven't been pre-scripted i.e. you can deal with unplanned emergencies. If all you know is a set of rote procedures then you're in serious trouble when something crops up for which you don't have a set procedure.
As another poster mentioned here, his number one quality for the job is aptitude. If that's not problem solving, I'm not sure what is. So it seems that you and the article agree, except that the author expects his juniors to get it and would not keep them around long if they did not.
Re:what article did you read? (Score:2)
Re:problem solving skills? (Score:3, Interesting)
These people tend to make good admins creating accounts, setting permissions, things that involve following the rules. The SA's good at problem sovling tend to get boring with rote work like this and make simple mistakes. But the paper-cert people are good at learning and following procedures.
The trouble is the good SysAdmin usually like their work and not interested in becoming managers. The paper-cert people its their only way to make more money so you get these idiots trying to manage creative people. Not good.
Aptitude... (Score:4, Insightful)
Important for choosing YOUR future IT job (Score:4, Insightful)
Of course, its important to try your dreamsjobs during during university, but you never know if your dream wont turn into a nightmare after a few years but just working a few weeks there...
Re:Important for choosing YOUR future IT job (Score:1)
I liked this article. In fact, I'm going to hand it to my managers as quickly as possible! I'll bet many of you /.'ers have to deal with management who doesn't understand [our] role in the company and/or the technical issues...
Sometimes you have to make your own dream job by forging ahead and molding the management team around you by helping them understand your team.
Crap... (Score:5, Insightful)
Any sysadmin that has to log into a system while on holiday in *India* is a bad one. If you don't have enough redundancy built into your system that your junior admins/engineers can't hold down the fort for a week or two, something is wrong.
Second, "strong experienced based opinions" is crap. Open your eyes to new concepts and ideas. Like me trying to explain to two 10+ year network engineers that having a flat, layer 2 network across an entire Air Force base with 8000 users is a Bad Idea, and that adding layer 3 switching capability at the distribution points wouldn't slow down the network, and it would, in fact, be faster. Sure, hold on to your opinions, but understand things change, and if you don't change with them, you're a gorram dinosaur.
Re:Crap... (Score:5, Interesting)
Having said that, logging in from a cyber cafe? Speaking as a former sysadmin of one of those self-same cafes, this made me shudder. Even if he's using something secure, I've often found keystroke loggers on machines (amongst other stuff), and he's risking some serious compromising.
"strong experienced based opinions" is crap
That's your strong, experience-based opinion, is it?
Re:Crap... (Score:4, Insightful)
Good point, and always one worth keeping in mind. It's always good to treat systems and networks like bags at the airport (have they been under your control since the time they were packed?). However, perhaps he was using:
1) his laptop, or
2) OPIE, S/Key, or some other one-time-password solution (and checking the SSH key of the remote end).
Re:Crap... (Score:5, Funny)
I wouldn't say that. He probably missed the machines...
Re:Crap... (Score:2, Informative)
I wouldn't say that. He probably missed the machines...
I agree. I've got a couple of small pentiums at home that run a webserver, email, ftp, etc. They are never under any serious load and there aren't very many users, so the machines pretty well take care of themselves. That said, I still routinely ssh into them from work Just to make sure my babies are OK.
Better than no experience (Score:4, Insightful)
It's better than just 'strong opinions'. Anyone logical enough to realize that you should normally have opinions based on experiences is normally logical enough to be reasoned with regarding how those experiences may differ from other experiences, and how 'new' approaches may in fact be better.
In your Air Force situation, it sounds like the people you were dealing with had had little or no experience with the type of topology you were recommending.
Re:Crap... (Score:1)
Re:Crap... (Score:5, Funny)
From: Sysadmin
To: Management
Subject: Everything's OK!
Hi,
I just logged in from sunny Goa here to check up on things. Everything's going ok! My well trained junior admins are keeping everything ship-shape.
Must go back to the beach now.
See you in two weeks!
regards,
BOFH
Re:Crap... - too true (Score:2, Insightful)
> hell" mentality, he's just a little more polite about it.
Too true and still an unfortunate stereotype of all too many self-annointed sysadmins, or at least those who can get away with this attitude. Unfortunately, many inexperienced management types still think that this is acceptable behavior - but that is changing.
He sounds like he works at a relatively small and fairly autonomous site without too much interaction with other groups/departments using the systems on a day-to-day basis. His management also doesn't appear to know what is going on - but it probably doesn't matter and they don't care given the circumstances of this particular site.
Any one involved in system admininstration or interested in this type of job should consider the recent book "The Practice of System and Network Administration" (by Thomas A. Limoncelli and Christine Hogan) a must read. This is a far more realistic description of contemporary practices in system administration than the comments made in this article.
Re:Crap... (Score:1)
Re:Crap... (Score:2)
Good part of the system had to be re-engineered to accommodate particular vendor's solution with no aparent benefit. Extensive and very well argumented analisys of why this wouldn't be a good move did not help. Those two guys got sacked, there is always something that needs to be fixed on the new system producing unnecesarry downtime, company has spent a fortune on crap, users are bitching (with a good reason) all the time and new management have collected their EOY bonuses, so they're spending well deserved holidays on exotic islands.
So you see, it's all very relative and depends on all sorts of other circumstances. Common sense aproach to real world problems is often not possible, due to deviations caused mostly by the behind the scenes happenings that always seem to be beyond the logic.
And yes, I sound bitter because I am affected with the change, trying to sysadmin this new system. Gotta feed the family though, so there isn't much choice for me here.
Re:Crap... (Score:5, Insightful)
I take exception to this. After all, that is supposed to be the basis of "experience" for which it is worth paying a premium. Maybe they don't pay for experience anymore, though (sure looks that way sometimes).
Yes, things change, and in this industry at a sometimes-painful rate. However, a good problem solver (and a SysAdmin better be one when "strange, impossible" things happen) should be able to look at a problem or requirement, weigh the available options, and choose the best one.
While the options available may change, when the problem or requirement falls into a catagory that is not materially affected by new technology, experience is gold. This does not meen that conventional wisdom shouldn't be challenged when a better idea seams appropriate (and, if it isn't, it should be possible to show why), but it shouldn't be totally ignored either. The good SysAdmin will choose wisely.
From a developer's perspective, I have encountered SysAdmin "control freaks" that got in the way of me doing my job (as in, "I don't care if the product your department is developing is Linux-based, you must run Windows," where the real issue was integration with LAN-resources). I have also encountered those who did things differently than I would, but with damn good reason, usually because any perceived extra "bang" I might get would not justify the complexity "buck" he or she would have to face, and add overhead overall.
The best SysAdmins provide a service, make sure it is available, support it, and will bend somewhat to accomodate slightly different or unusual needs, with commensurately less support (i.e. "yes, you can connect that Linux box to the 'net, just don't do ...., and don't expect support beyond IP assignment, NFS filesystem exports, server identifications (DNS, NTP, etc.)).
Re:The BOFH (Score:3, Insightful)
I kinda feel sorry for any company which lets unknowlegdable people make decisions for the experts to implement.
Re:The BOFH (Score:2)
Re:The BOFH (Score:2)
Re:The BOFH (Score:2)
So, you're an *expert* now? Been reading up on cryptography, TCP/IP, programming, and so on, have we? Because a few weeks ago, you were, as you put it "unknowlegdable" to an painful degree. It is to laugh.
Luckily your utter stupidity is now well-documented in my journal for all to see.
"WSH is the most powerful language ever" - William "woogieoogieboogie" Platt
Time to get that pesky degree, I think
Re:The BOFH (Score:2)
Re:The BOFH (Score:2)
You were the one who said "I can learn *anything* in minutes". And "I should be CEO of a Fortune 500 company". God, it's hilarious.
You are not even intelligent enough to worthy of my time discussing, debating or trolling
No, you lost. I have smashed every attempt you came up with to "hide the web page source". You keep inventing ever more bizarre new ones, but they are all worthless. Every time one is destroyed, you move onto a new one, and never try to defend the old ones again. I'm pleased that you realize the limits of your intellectual capacity, finally.
Anyway, get back your web site in Nowheresville Florida, your Windows pointing-and-clicking, and Javascript "programming".
Re:The BOFH (Score:2)
Aptitude!? (Score:2, Interesting)
In rough order of importance:
Aptitude.
...
Is it just me, or is that a somewhat circular choice for first on the list? What IS aptitude, but the qualities essential for the purpose?
Re:Aptitude!? (Score:3, Funny)
No, you missed the part where it says he's a Debian developer. He's actually referring to the software package, "aptitude". Damn useful little tool. Don't know if I would put it above communication skills et al, though...
Spot On (Score:4, Insightful)
It has often been my experience that the sysadmin(s) for an organization is/are the best informed resources from an IT perspective (at least if you're a good one). Who else do you talk to when needing to discuss any significant change to an organization's computing infrastructure?
To the person who commented that there was no mention of good troubleshooting skills as qualification for a good sysadmin....I believe that fell under the comment that a component of the sysadmin's job was to keep the systems running. To be able to troubleshoot and solve problems is a prerequisite to keeping systems running.
I'd only disagree to the extent that. . . (Score:5, Insightful)
The primary difference between a really good admin and a BOFH is the realization that "lusers" are *part of the system.* A really, *really* good admin has to be that apparently rarest of geeks, the person with outrageously good technical *and* people skills.
After all, the admin isn't just responsible for the machines, he is also the primary interface between the machines and the people.
How do you know if your company has a really talented admin? If he kills all of a user's processes and deletes all of his files, and the user is so greatful the treats the admin to lunch.
Now *that* is evidence of an admin who has figured out what his job is and how to do it. Which is, unfortunately, rare.
KFG
Re:I'd only disagree to the extent that. . . (Score:3, Insightful)
Re:I'd only disagree to the extent that. . . (Score:2, Interesting)
> between the machines and the people.
In larger installations users usually don't contact sysadmins directly. Instead, they call a "helpdesk" person who uses uses tools (mostly written by the sysadmins) to fix routine issues. Unresolvable issues are escalated upto the sysadmin.
It's usually like this with ISPs (where the sysadmin mentioned in this article works). Users (i.e. subscribers) calling about technical issues usually speak to a tech-support/helpdesk person; and not a system administator.
Situations where "userness" is. . . (Score:2, Insightful)
I'd only point out that help desk people are themselves users of the system, and generally rank only a smidgeon above subscribers on the "luse-O-meter."
My point stands.
KFG
Re:I'd only disagree to the extent that. . . (Score:2)
I got a chuckle of of that one. The sysadmin just succeeded in turning a major disaster in something rather humdrum.
The "system" is the users and the corporate infrastructure as much as the hardware and software. It doesn't matter how good the system is if it isn't being used that way.
Admin flamebait... (Score:4, Insightful)
Why did you buy the computer? To run programs. And so step forward the programmer...
Why did the programmer write the program? Because it performed the task needed. And so step forward the analyst...
Who needed the task performed? And so step forward the end-user...
I've always thought Syadmins to have an over-inflated importance in the world. As I show above, I put them third or fourth in the pecking order (depending on whether the end-user and the analyst are not the same people). Many admins forget that the point isn't to have lots of wonderfully run locked-down computers that don't do anything (damned users! get in the way of my policies...). A computer is a tool - a beautifully polished tool that doesn't do anything is worthless.
Cheers,
Ian
Re:Admin flamebait... (Score:5, Insightful)
Granted the job of a Sysadmin is to keep the machines running so that the user can do their job, but to say they are "unimportant" is absolutly stupid.
The job is more like a janitor, you "own" the house, you make sure that everything is clean, that the kids are not running in then hallways and that the bathrooms are clean.
Having said that, that also means that I am going to restrict of what a user can and cannot do, in order to make the system work for EVERYBODY.
The problem is mostly not the endusers, they are EASY to deal with, the problem in my own experience are all those wonderful programmers who think because they can write some code they should have all the rights, all the power and oh yeah, root because "Well, the program can only do what it is supposed to do when it is run as root." Right, permissions are for wimps.
I never had a real problem with an enduser that couldn't be solved after some facetime, on the other hand I had Programmers who activly tried to root production boxes because they NEEDED to testrun a program that had failed on the dev AND test box (he later claimed they were broken, yeah right), never heard of permissions, it sometimes amazes me how little of an understanding programmers have about System Architecture and security.
Sorry, but face it, if you ARE on my System *I* am the one who tells you what you can do and can't do. I AM the cop on that system and if you don't behave I make sure you can't do much damage.
Sounds "God like"? No, I never kill processes without first knowing what they are doing or why unless they jepardize the system.
Oh, and for the guy who tried to root the box: He got a warning from the manager and I am sure he thinks about me the same way you think about Sysadmins.
M.
Re:Admin flamebait... (Score:2, Insightful)
I didn't. I said I rated them fourth in importance, behind the user, the analyst and the developer.
I had Programmers who activly tried to root production boxes
The more technically accurate term for these people is 'cretins'. You have cretins in all jobs and all walks of life.
Sorry, but face it, if you ARE on my System...
And here we run into the over-inflated opinion problem again. I am not on your system. I am on the end-user's system. You are to help me do whatever the end-user requires.
Cheers,
Ian
Re:Admin flamebait... (Score:3, Insightful)
Programmers
Allways think the hardware or system is broken and they can fix it aka I'm a better sys admin than the sys admin syndrome.
Need superuser privlages on any machine they touch including there own aka I am god you can not be god because I am the one and only god because I can program.
Allways think the best way to increase application performance besides easy things is to make the system faster aka ROI be damned it's just easy to spend more to make it work.
Sys admins
The machine is my responcibility thus the machine is mine all mine it's my sandbox and nobody else can play with it unless they ask realy nicly aka king of the hill.
Nobody else knows all the little things that I have done to make the system work aka undocumented bailing wire and bubblegum.
Users are stupid why because they ask me questions that I allready know aka if your not an admin you are dirt.
Now either side has there issues but guess what all that realy matters is that the system stay up for most companies it dosent matter that you make 3 times the salerie of the average users when they are affected by your bad programming or inability to trend and premtivly fix issues the users suffer. Admins deserve there sandbox to a point as they are the ones who get canned if things go realy bad. Programmers need the rights that they request sometimes so then can get there work done more expidiciously.
BTW yes my spelling and grammer is horid so dont complain. If you dont agree with my oppinions thats fine to they are mine and not nessicarly anybody elses.
Re:Admin flamebait... (Score:2)
It is not owned by the company admins. It is owned by the company. Admins are just people doing a job which the company requires.
We agree on most of the rest of the post.
Cheers,
Ian
Re:Admin flamebait... (Score:2)
The company buys the machine and then what? The company is not a person it might own the physical box but it is not it's responsibility. Owning == Responsiblity and at the end of the day it is a SysAdmins responsibility. I have never heard a user complain about the "fucked up job the company did" with a server, have you?
M.
Re:Admin flamebait... (Score:2)
Okay, how does this usually go?
The System is bought because someone (reads enduser) has a requirement for the system. The programmers than write the application (or we just install something stock like say Oracle), then the SysAdmins take over the daily tasks, monitor the thing, patch it, make sure it runs. In my book that means it is MY system. I am on the line for it.
Not convinced? Fine, let's see how a project should work out:
Enduser defines requirement. Software Architect and System Architect sit down and design the system and everything else that is necessary.
Then the SysAdmins start building the system and the programmers start programming their piece, everything is tested, and then the user gets access to the application.
Sounds simple, right?
By your account the next step would be that the programmers hand over the program sourcecode to the enduser because the enduser owns the program and the sysadmins hand over the root password to the enduser because (again according to you) they own the system.
Now, if something goes wrong (let's say enduser made changes to the code and broke it) it's the endusers responsibility? Or who is?
Do you get my point? Own == responsibility and the ENDUSER is NOT responsible for the system. They "own" the box and they have a stake in what is done to it (it is called change managment) but the responsibility for the System is in the hand of the SysAdmin and the responsibility for the application is in the hand of the programmer.
About the rest I agree with the other post, there is definetly an attitude problem with some people in the business.
Re:Admin flamebait... (Score:2)
Kintanon
Re:Sysadmin Context: IT vs. Entire Organization (Score:2)
The individual end user may have little voice in matters, but if the end users are not doing what they are supposed to be doing, the system is pretty much worthless.
Re:Admin flamebait... (Score:2)
There are always differences between systems, and different requirements may mean you demands are a bit higher than some, but it's almost inconcievable to me that you have that many people paid that much money and still have complaints with so few end-users.
Thankless Job (Score:5, Insightful)
Systems Administration is the kind of job that nobody notices if you're doing it well. People only take notice of their systems when they're not working, And they tend to forget that a lot of work and expertise goes into making sure that they continue working.
You only ever talk about IT when things go wrong. In my mind, that's a thankless job. I am SO thankful that there are people that don't mind that... And this guy is a professional through and through:
But that's as it should be - computer networks are infrastructure that you should be able to rely on, to take for granted, just like telephones and electricity. If you can't do that, then there's something wrong, something that can and should be fixed.
I like how he takes responsibility. This is unbelievable. I want him as my IT guy now.
Re:Thankless Job (Score:2)
Maybe you shouldn't work in IT then. Leave it to those of us who enjoy tinkering, and playing with new technologies.
Re:Thankless Job (Score:2, Insightful)
Re:Thankless Job (Score:2, Insightful)
Doesn't sound much like a job in IT to me, or that he has much of a choice about it. What's he supposed to do? Request a transfer to a new family? Tell them to hire a professional IT guy?
I take it you're volunteering to go over to my mom's house and help her the next time she has a problem? Thanks. She can pay you in comments about how you're not sitting up straight enough or alternate forms of nagging currency.
Re:Thankless Job (Score:3, Insightful)
To me that's the biggest problem with being a sysadmin professionally. Old-style, less competent managers don't believe that you're worthwhile because you appear idle while nothing ever seems to actually happen.
Once, a combination of a bad spot on tape and a very unusual ice storm combined to result in three days' worth of data. (This was before the advent of cheap and readily-available RAID.) I was called in to a vice-president's office and read a list of backup strategies that the guy had torn out of a Novell magazine, about half of which applied to the SVR3 we were running.
Thankless job, exactly.
Not thankless when you are in control (Score:5, Insightful)
I think many sysadmins on this forum will find that the following rings a bell. You begin with total control in a startup IT team, decide on and bring into operation all aspects of a solution and keep it all running perfectly for years, with near-zero downtime and great job satisfaction. Then the corporate machine takes over, basically overturns everything you've done and creates an absolute disaster, and despite ignoring utterly all your input, you are to blame since you're the sysadmin. Needless to say, job satisfaction is, let's just say, less. This ring a bell?
Craig Sanders has managed to avoid stage 2 so far. He deserves only praise, in my book.
Re:Thankless Job (Score:2)
Don't forget (Score:5, Insightful)
Re:Don't forget (Score:2, Informative)
I imagine that's why the System Administrators Guild [sage.org] has a SysAdmin Code of Ethics [usenix.org].
Hmm. I wonder if BOFH's also have their own code of ethics too?
Sysadmins are different all over (Score:4, Interesting)
I also fully agree that when you're on vacation, if your underlings can't keep the ship together, you're not doing a very good job.
What he doesn't hit on very well in his preachy missive is the importance of diplomacy. I work in a big enough operation that I don't even deal with the end-customers, we have an application support team for that. This means that (a) the problems are reduced, since I only have to worry about a handful of real "users" who can damage the systems and (b) the problems are greater, because those guys are vastly better at really kicking the legs out from under my boxes! So it's mightily important to always touch base with the application support teams, and keep a continuous stream of communication up. It's easy to lose that, especially in a giant operation, especially when your specialty is copping an attitude.
And finally: Why do so many sysadmins dedicate their lives to looking like freaks? Find a shower, a razor, a comb, and use them, people!
Re:Sysadmins are different all over (Score:1)
Sysadmin personality types (Score:2, Insightful)
He's dead-on with his observation that personality type and aptitude are the most important qualities in a sysadmin. I am fighting a battle with a boss who actually thinks you can train someone (anyone) to be a sysadmin. Unfortunately when these people fail miserably I get accused of poor training. Oh well, I can always work for a service provider in my next life.
Re:Sysadmin personality types (Score:2, Insightful)
Well, he lapsed in many of his tasks and others he did not do correctly. I feel the training was adequate since I had done it before with someone who has a very different *personality*.
On a side note, I also liked how he disregarded certifications. Most people I have met with these always seem to have an answer looking for a problem instead of spending time actually fixing stuff and making it run better.
Dream Sysadmin Job? (Score:3, Interesting)
I don't have much to complain about
HUH!? I'm gonna go out on a limb here using my expereince and the people I know and say this is the exception and not the norm... Is this guy for real? Every sysadmin professional I know complains about the users, the hours, the pay and their job security. And what's this Telecommuniting BS? 70% of the time he was able to stay at home? Am I missing something here? This does NOT sound like the average Sysadmin Job I've come to know. Most employers are too damn anal for that to occure, even if you could effectively...
Jeez... I must be missing something here... Talk about a raw deal...
Re:Dream Sysadmin Job? (Score:1)
But I think you are right: as a network admin I complain about management, users, etc. And who doesn't complain about pay or job security?
Yes, you are. (Score:3, Insightful)
I was a good sysadmin and I have greater aspirations than this guy does so, I have moved up and beyond these older jobs but, they were very good jobs while I was there.
You're missing out. The question you must ask is, why? Are you really as good at your job as you think you are? Are you able to relate to management or are you constantly trying to win pissing contests with them? Do the users like you, or do the fear or view you with disdane? Honest answers to these questions are harder to get than you might think. You may want to ask a peer or higher-up engineer type for brutally honest answers to these questions. Engineer types will usually oblige, provided they aren't close friends or subordinates. Once you have these answers, accepting them and working to truely address potential shortcomings could completely turn things around for you. Good luck.
Re:Dream Sysadmin Job? (Score:2)
If you're a good admin, you know your OS(es), you know your apps. You get along with your users because you take the time to understand what it is that they want to do, so you are able to help them do it. For this reason I don't think telecommuting is a good way to be a sys-admin, much as I'd like it to be.
To be a good admin you have the people skills to pretty much disregard most of the half-baked ideas you get from management and users whilst making them think you're going out of your way to accomodate them.
The best admins I've worked with have always been smart, friendly people with a wide range of interests. The worst have always been small minded, power crazed Linux bores with chips on their shoulders.
"sysadmin": a very general term (Score:2, Informative)
Because most people can do some of these things, they can end up doing sysadmin work. Does that make someone a sysadmin? I have interviewed for sysadmin roles before and always been amazed at the people who have used an application, or watched and install, and then applied for the sysadmin job. It's not enough.
The problem is, lots of people doing this kind of work without the training and experience (and often, no mentor either - nontechnical boss) give the profession a bad name - hence the whole BOFH subculture.
This link [infrastructures.org] describes some of the issues related to this job that isn't very mature at all ...
A good article (Score:5, Insightful)
Of course people on slashdot are always looking for something to disagree with, so a few of you have already lashed out at the "strong experience-based opinions" quote. Experience is the number one most important part of being good at *any* job. If you don't agree, then you probably don't have enough experience.
I'll also say this: You don't have to agree with everything someone says to learn from them. (In fact, if you only listen to people who you are in complete agreement with, you will never learn much of anything.) There are a lot of good points in this article, and even if you are somehow offended by the experience-based opinions remark or something else, you can still gain something from it.
To counter the Slashdot trolls (Score:4, Insightful)
Here's a positive comment.
I thought that was an insightful article. System administration is the process of keeping together an organization's information infrastructure. People often find this job to be non-human oriented, but it is in fact completely human oriented. The good sys admin is constantly thinking of, and even torturing themselves over how the users will be affected by anything he/she ever does and how it can make their lives easier.
The really good sys admins will unfortunately be perceived as adversaries because they would rather disagree and cause a political stir than develop a system that they believe is going to harm the users more in a long run.
Most intelligent people can figure this out, and will respect their sys admin's position in the company. The sys admins who stay quiet during meetings when they see the company making a wrong move are the ones who don't care, and IMO better fit the profile of BOfH.
At the heart of the matter, our profession is to increase the quality of life through information technology. Anyone who doesn't see their IT profession this way is in the wrong career.
What about the processes? (Score:5, Informative)
As long as all the SysAdmins seem to be making it up as they go along, we will continue to be marginalized and geek-ified by management. Try on for size:
It's not that the SysAdmin necessarily has to manage these processes - though in a small shop no one else will - but he/she/it needs at least to be able to talk the language and understand the processes that the IT Manager has set up. And if you are managing the shop, then this is your job. You must know this stuff as a matter of professional responsibility and "keeping up" in your field.
A 20 min. presentation to the other managers on Best Practices and Processes in IT Management will gain you a lot of credibility and help lift you out of the geek gutter. There are decades worth of lessons that have been learned the hard way and documented into these processes. When you can demonstrate to management that you are drawing on a substantial body of knowledge that is geared towards improving service and reducing total cost of ownership, you will gain their respect (assuming that you care about their respect).
Beyond this, I want to emphasize an excellent point that Sanders makes in the article. The SysAdmin job is one that is invisible if you're doing it right. A good day at work is a boring day. Excitement is a sign that something has gone wrong. You should structure your environment to be as boring and reliable as possible.
Too many SysAdmins live off the adrenaline rush of fixing a broken server while everyone else in the organization sits on their thumbs waiting. That's costly for the organization, but ironically is the easy way out for the SysAdmin - you don't need to be disciplined or structure your time or do any planning or thinking, just jump from crisis to crisis. It's much more challenging to turn it into a boring desk job where most of your work is pushing paper and the machines pretty much take care of themselves. But guess which option is better for the organization's mission?
Once you do get to that Nirvana state of boring life, you can strategize how to produce some measurables so you can blow your department's horn at the monthly managers meeting. Because if you do your job well, with the result that your work is invisible, they'll cut your funding unless you keep in their face on a regular basis.
.nosig
Re:What about the processes? (Score:2, Insightful)
o Change control?
o Incident management?
o Problem management?
o Change window?
o Service level negotiations?
o Capacity management?
o Security management?
All of these points are needed for sysadmin, but you don't really need a process for them. use common sense and knowledge of the big picture to manage systems. Real good sysadmins doesn't need processes for how to insert a floppy or how to eject a tape from a DLT drive.
Processes are for McDonalds employees (remember section A.6.2 and say "Do you want fries with that?" if customer has only ordered a burger).
A good mentoring system with experienced sysadmins is what you need. Then the IT systems/infrastructure can be blackboxed from a management perspective.
Re:What about the processes? (Score:3, Insightful)
what about when you have dozens or more machines. everything needs to be done exactly the same. I can't get on each machine and spend time figuring out if the apache root is in
which version are we on? when you get to have some machines where uptime is really essential you can't necessarily upgrade all the machines at the same time. Did you upgrade this one yet? where is your configuration database? where is the process that keeps it updated?
If you are working w/ another company you have to have processes. Everything has to be documented so when I say build me a new web server, I know exactly which options are going to be set.
If you have more than one sysadmin, i need to know that each machine is built the same way, has the same naming conventions etc. all of these are processes.
as you move to more than a couple machines you need processes to keep things in line.
Re:What about the processes? (Score:2)
A process isn't there to create extra work. A process is built to standardize and to take advantage of lessons learned. The ITIL (where we started this thread) isn't so much make sure your apache is in
A sys admin i respect at a company w/ the one of the highest uptimes on the network told me this. (i am paraphrasing) "we have good, smart people, but other companies do too. The real key to uptime is the processes you use"
Re:What about the processes? (Score:3, Insightful)
Standard Processes let you interface with management / customers / the owners of the system, in terms they understand.
Note that Processes are NOT procedures. The process defines who is repsonsible for what - a Problem Resolution process describes who takes the call, how it gets escalated, how the fix gets tested (depending on the type of problem, the system, the impact of the change etc), and how it gets implemented. In our company, these a fairly high-level, and apply in variuous degrees to applications development, network management, O/S Admin, application administration, process control,....
The procedure is how your team implement your part of the process. The nett effect is demonstarted by the Macca's analogy. One of the few (only ?) virtues of a McD burger is you know what you're getting. Management like answers / solutions they can trust.
Likewise, me and my team look after aprox 19 SAP landscapes. When a customer says "I want xxx", we either say no (and tell them why) or say yes (and tell them how long / much). this has two advantages -
1) Because we have standard procedures, we are accurate (to +/- 10%) 99% of the time.
2) Becauee we've proven ourselves, people believe us.
Profound.... (Score:5, Funny)
Ye gods, how true!
--K.
Ignore this man at your peril (Score:5, Insightful)
For everyone whining about the fact that he says a good sysadmin should have strong opinions based on experience... If you think that every problem is going to be so clear cut and so clean that you can just bang out an optimal solution and provide a clean and mathmatical defense for it, all you have done are home or academic excercises.
The problem domain for solutions is so incredibly broad, and so incredibly rich, that if you are not depending on collection of good solid abstract rules of thumb and effective practices, you will never get to a good solution. You have to use intuition to narrow down the problem domain to a few concrete approaches, and then apply logic and experience to decide which of them to implement and how.
These are not opinions like "NT Sucks, Linux rules", these are opinions like "I don't want to hinge my business case on an operating system controlled by a single vendor". I don't want an enterprise IT infrastructure that depends on technology that only runs on non-scalable hardware". "I don't want an operating system that I cannot remotely administer". "I want an operating system that allows me to update and maintian, stop, and start some subsystems without effecting other subsystems". "I want an operating system where I can apply security patches without being forced to install operating system updates". You get the idea.
Having an open mind is important, but at some point you have to get off your ass and decide something, and act upon that decision. The older I get, the more important I have realized this becomes.
A group of people with "strong opinions based on experience" can get together and hammer out a list of pro's and cons, and come up with an excellent solution to a problem, fully aware of what the solution does well and where it will be weak. It will be a stressfull meeting, and tempers may occasionally flare, but when you finally grind through it you will end up on solid ground, and everyone will likely be on board.
A bunch of people with "open minds and no strong opinions" are going to dither about endlessly and end up with an unfocused, innefective, designed by committe monstrosity.
Acedemia is all about exploration and investigation. Work is about getting things done. Note though that even the academia people typically won't get much "exploration" done if their home made router is down because it is an old Linux box built around a $20 commodity power supply that just went up in smoke, and the only guy that knew how to set up the IPTables to get the routing right left to go to grad school 3 months ago.
I am with this guy... a lack of a strong opinion and the ability to defend it, suggests to me a lack of experience. How on earth can you do something day in and day out, sweat over it, bleed over it, live and die by it, day by day and year by year, and not form an opinion?
Very unrewarding occupation (Score:5, Insightful)
I've been a sysadmin for the past 5 years, two of them at a large department in a very big educational institution. I have to say that of all jobs I've had in the past, this is the most personally unrewarding.
Sure, the pay is good, and the benefits are nice, and you get to sit in your comfy chair most of the time punching buttons and not really doing anything in particular. However, this "bliss" comes with the following drawbacks:
I was an education major in college, and during one of the classes our professor told us: "when you start teaching, there will be rich schools and poor schools. If you work in one of the rich schools you will have a good salary, good budget, nice classrooms, and decent lunches. If you work in a poor school, you will have none of that, plus drugs, violence, and complete lack of parental involvement. Believe it or not, some people prefer to work in poor schools simply because if they are doing their job well, there will be people who will stop them every day in the hallway and tell them how much they admire their work. Not only that, but people working in poor schools are able to see with their own eyes how much difference they are making in the lives of the children they teach."
That seemed weird to me then, but now I think I understand. It all comes down to what one thinks to be a good reward for their work. If it is good pay, quick career path, and a Porshe by the time you're 30, then being a sysadmin is your dream job (granted, of course, that you're good at it). However, if you are looking for something that is personally rewarding, something you want to feel good about doing... You might want to pick a different carreer. Or at least do it only until you start feeling burnt-out.
Me? Oh, I'm quitting as soon as I can afford it. :)
Re:Very unrewarding occupation (Score:3, Insightful)
I'm on contract (system administration and programming) with a small internet service provider.. very organized group. Generally, clients of this ISP are very friendly and some-what knowledgable, dispite not being system admins. When I fix a problem, or even if we are unfortunate enough to not be able to fix it (corrupted backups, etc), they are very thankful and give us lots of praise.
I have been on contract and employed by other internet service providers, but none have ever had the customer satisfaction and praise that this one gets. The cause of this is quite simple, fast results. If someone emails you, reply within 10 minutes.. even if that reply is just to inform them that you're examining the problem.
Telephone support is a bad idea, especially if your user's native languages don't match your native language. Many users will refuse to stop talking about unrelated topics, others will take 10 minutes to try and tell you their password. Telephone contact for contracts, especially for programming projects isn't bad.. but it is very difficult for many kinds of technical support.
Some of my contracts are scheduled or 'on my own time', others are 'on call'. Most ISPs don't require 'on call' unless you have some special skill that the others don't have, or if they are under-staffed.
Users can't blame you for not notifying them if you're organized enough. Make a webpage with notifications and then send emails to the users notifying them of the downtime. If they complain, point them to the webpage..
I do understand the 'changes in routine' example you gave, but this is again part of organization. DO NOT make user-interface changes in 'stable' software. Prepare a new 'major' release and then make notifications of a 'major' upgrade.. and then have a meeting discussing the changes. Of course, this may seem silly if it is only a small change... but the users are less likely to complain about an 'upgrade' than 'jim playing with our software'. Users complain less about upgrades than they complain about random, unannounced changes in their configuraiton... especially if the upgrade has been properly 'hyped' (Users pay to 'upgrade' to WindowsXP from Windows98.. they are not only willing to see change, but also pay for it!)
It can be a very rewarding occupation. If you're not rewarded, then you're either in the wrong field or the wrong company.
I agree to a point (Score:4, Insightful)
That said, I don't see uptime as the holy grail of SysAdmining. Uptime and Availability have to be measured from a user's perspective, can the users use the system (the way that they want to)? You can have a system that is always up that no user uses because you've made it too hard to use. High uptime, low availability. A good SysAdmin is looking for ways to make the usage experience easier for their users.
This guy claims that a good sysadmin is the best informed in a company about IT. I'd add that a really good admin is the best informed about IT, and about the user's attitudes. A really good SysAdmin will take any problems with a system from a user's point of view to management. It's so easy for SysAdmins to miss the point. IT isn't about machines, IT is about enabeling other departments to do their jobs better. Anything you can do to maximize the bennefits to others affects their (and your) bottom line.
A dash of modesty never hurts either. When people ask me what I do I always pick a modest responce. I'm a virtual wrench turner just trying to keep things working. That gives people an easy mental image that isn't so far off from the truth really.
True tips for SysAdmin enlightenment (Score:2, Insightful)
Bingo!
If you truly understand this point and take it to heart then you will do well as a system administrator.
Toss in some enlightened modesty and knowledge that you will never be the IT authority on all topics therefore needing to work with users to jointly solve problems then you will never be without a job. Get to a position where you are acting pro-actively to prevent potential problems, stay in communication with your users and offer them solutions they hadn't yet thought of - rather than constantly running around reacting to problems (and cursing the "lusers" for creating them). Doing all of this, you will become visible, respected, and considered to be an integral part of IT (versus being mostly invisible, associated with problems, and generally considered to be an obstacle to getting things done). Consider yourself to be a professional and act accordingly (which means not dialing in from India to check up on your systems - how inane ...)
And this is posted here why? (Score:2)
Small-timer (Score:5, Insightful)
The big shops have to be organized. They need automated everything, not people running around fixing stuff. They have to have an organized strategy for growth and replacement. (Some of those strategies are unusual. At one large service provider, machines are installed in clusters of 100 and never serviced thereafter. Dead machines are switched off remotely. When 20% of the machines in a cluster have died, the entire cluster is replaced.) They need written procedures and manuals. They need physical layout standards. They need three-shift coverage. Most importantly, they need overall architectures that limit the consequences of failures and make it easy to find out what failed.
I agree with the article (Score:2, Interesting)
First, the cons.
The biggest con is working with management such as micromanaging your job such as dictating the directory name. An example is a software package is installed called Test Expert. Mgt decrees the path name is "/opt/testexpert" and I want to use the path name "/opt/texp" which makes it easier for the users to type
Having to deal with paperwork related to inventory, ordering software, budgets which that should be dealt with by managment.
Having to do Configuration Mgt which means a lot of reports which is not technical but in the language understood by accountants and marketing
Now, the pro's
Instant gratification. When you make a change, you know immediately if it works or not.
Working independently
As long as the machines work and there is no outstanding work, there is time to surf the 'Net :)
Possibility to telecommute but the company I work for, telecommuting has been taken away due to the downturn of the economy
Experimenting with different setups which eventually gets used by the company
Re:I agree with the article (Score:3, Insightful)
Not always true and even if it were, there's nothing gratifying about a change that immediately makes everything not work.
Ade_
/
Great Quote (Score:2)
He must be a sysadmin... When asked about the worst thing that ever happened, he didn't say: ``Chinese government super-ninja hackers, bent on info-terrorism, and ruining our credit ratings, doing so by defacing web sites with `I ownz j00'."
Hmm, perhaps the rack builder should be held in jail for several years without a trial as the government trys to build a multi-million dollar case against him....
Steady job (Score:2)
Of course, this makes the H1-B problem [geocities.com] even worse for us. "If H1-Bs don't come in, the job will be shipped elsewhere" is not really true of us, or at least the bar is much higher. So it's more in the admin's (network and system) intereste to be against high H1-B caps.
Re:Steady job (Score:3, Insightful)
If by sysadmin you mean desktop support, then yeah, it's kinda hard to replace a toner cartridge from a few thousand miles away.
If, like most sysadmins, you don't need to physically touch your servers every day, then it's trivial to move your job overseas. Programmers are more secure because at least they need to speak to the end users frequently to gather requirements and reproduce bugs.
Neat Side-Affect of the /. Effect (Score:3, Interesting)
At last, a beneficial use of the
Wow, he started with computers at age 11 (Score:2)
He says that as if that's a particularly early age to screw around with computers!
</sarcasm>
A recent flame war Craig got in to (Score:3, Informative)
To make a long story short (and the flame war got ugly), Craig feels that a DNS server needs to support the legacy BIND zone file format. Dan, obviously, does not; he feels that it only matters that one can transfer the zone file over to the new format (losing all comments in the zone file in the process).
Now, I will side with Dan here. Keep in mind that my viewpoint is rather biased, being the person responsible for the MaraDNS server [maradns.org], a server which Craig uses but feels is "poorly written code". Now, the only specific that Craig went in to when pointing out that he did not like my DNS server is that fact that, like Dan's TinyDNS, MaraDNS has no support for BIND's zone file format.
Now, with all due respect for Dan, I think he should not knock a gift horse in the mouth. The fact of the matter is that the code for MaraDNS is open; if support for BIND-style zone files is important to Craig, I suggest that he start coding it himself. I will gladly accept code which can read BIND-style zone files and make it part of MaraDNS.
I am not saying that BIND style zone file support is unimportant. However, I think Craig should be a little more courtious in requesting this feature than badmouthing MaraDNS on the Debian ISP mailing list.
I am sure he is an excellent system administrator; I really wish that he would start up a serious open-source project so that he understands how we OSS coders feel. I think it would make him interact with us in a more mature fashion; and save both him and the developers he flames some grief.
- Sam
P.S. I know Craig already knows this, but there is a non-BIND DNS server which supports BIND style zone files called NSI. It is on the list of DNS servers [maradns.org] on my web page.
Re:A recent flame war Craig got in to (Score:2)
I meant to say "With all due respect for Craig"
- Sam
Re:I've heard that Fosters... (Score:2)
coopers (Score:2)
Re:Hire any CNN journalists lately? (Score:2, Insightful)
While reading the article, I found I was agreeing with almost everything written except, maybe, the MBTI bit.
The part I liked most was one of the last comments about knowing you've done a good job when nobody knows you you did anything at all.
I spent a weekend replacing the HDs in two Banyan servers (upgrading five 1.2 gig drives in each, with 9.1s in a RAID 5 array) then restoring and testing all services and data.
I walked in Monday morning and asked the users if everything was OK.
They said "everything's fine, why?"
"No reason." I said and walked away with a smile.
Like the admin in the interview, I also had a piece of furniture give way from underneath a server but I was in the room at the time and was able to stop its rapid decent to oblivion and eased it to the floor. It stayed on the floor until we got a proper rack unit.
So, there is "drama" but, I wouldn't call what was written an "over dramatization".
Re:INTP vs ESFJ - Personality types (Score:2)
I don't know if the slashdot poll code can handle 16 options, but I suspect that the "fairly uncommon (less than five percent of the population)" people are probably more like 90 around here. Perhaps it is time to find out.
I know some imformal polls have been done, and indeed the normally rare INTJ and INTP types show up far more often among computer types than in the general population.
Re:What's the deal with these Penguinistas?? (Score:3, Funny)
If I ever got a job as a systems geek, I would go back to suits just to be different. I like different.