Some tech jobs are literally dirty; digging around the innards of a data centre or running network cables through crawlspaces, for example, can leave you deeply in need of a shower.
More often, though, dirty IT jobs put people in tough positions - like having to explain to a crew of arrogant geeks why the network can't be upgraded the same day payroll needs to run; or why you're not a spammer despite what it says on your business card; or how lying about your company's products is probably not a good strategy for long-term growth. You may be forced to take the blame for a failed project even when it's not your fault or to expose wrongdoing at your workplace even if it puts your career at risk.
Dirty jobs never rest, and neither do the people charged with doing them. Be thankful you aren't one of them. And if you are -- well, at least you have a job. Right?
What dirty jobs have we missed? Nominate yours in our dirty IT jobs discussion thread.
Dirty IT job No. 7: Email ninja
Wanted: Email wonk intimately familiar with intricacies of Authentication, SMTP, DNS, DNSBLs, and MTA, as well as antispam laws across the globe. Must work closely with marketing hags and help desk zombies, and log quality time in airport lounges. Thick skin a plus.
The hardest part of Andrew Bonar's job is convincing the world he's not a spammer. It's not easy. Just having "email deliverability consultant" on his business cards is enough to start the Viagra jokes.
Bonar works with companies whose email isn't getting through to customers, thanks to overzealous spam filters. CEO and founder of EmailExpert, Bonar has to convince ISPs to let his clients' legitimate emails past their filters, while persuading his clients not to bend the rules.
For the record, Bonar says, "I have never done anything related to pharmaceuticals, nutrients, health supplements, weight loss, gambling, or lipo/cosmetic surgery/penis enlargement," though he once completed a project for the UK Tax Office.
There aren't many freelance email deliverability specialists out there, says Bonar, which is one reason why he's logged almost as many frequent-flyer miles as George Clooney in "Up in the Air" - jetting from his base in London to Australia, New Zealand, Germany, and back again. Thirty-five hours in transit is not uncommon.
The worst part? "Having to deal with marketing departments - their cooties can be worse than user cooties, because they are viral." The biggest challenge: "Trying to convince ISP abuse departments that I'm not a bad guy, finding out what they want me to change in these emails, and proving that these emails really are wanted - all 5 million of them," he says.
Bonar urges his clients to adopt the gold standard for commercial email - closed loop, double opt-in - so people are getting information they actually requested, not ads for Russian mail-order brides. And transferring or selling email address lists to third parties is definitely not kosher.
But many commercial emailers don't want to play by the rules, so Bonar gets a lot of stupid questions: Can I buy a list from you with the names of 10,000 doctors on it? Can we remove the unsubscribe link? How do we get users to stop hitting the "report spam" button? Can you set up a blacklist to block my competitor's email?
"These people will present one face in public and quite another behind closed doors," he says. "They don't realise that at the end of the day, doing these kinds of things will come back and bite them in the ass."
Next Page: Clocking on the pay cheats
Dirty IT job No. 6: Payroll cop
Wanted: Sys admin to log long hours and plan vacations around other people's paydays. Must be willing to put up with self-absorbed IT professionals. Ability to sniff out fraud essential.
You'd think the one person responsible for making sure everyone in an organisation gets paid would be treated with the utmost respect. You'd be wrong.
For three years, Jennifer Hoffman worked as sys admin for a major US telecommunications carrier. She was the only person who knew how to operate the time-and-attendance software essential to running payroll for some 12,000 full-time employees and 3,000 contractors. From the janitors to the CEO, if you didn't fill out an electronic time card, you didn't get paid.
Sure, you could get the company to manually cut you a check. But then you'd have to deal with "Bonnie" (not her real name), possibly the most loathed individual in the company, says Hoffman, who is now a life and business coach in Los Angeles.
"People actually offered me money to talk to Bonnie for them," she says. "I wouldn't do it."
Yet during nearly every payroll period, some IT contractor would make a change to the network that brought her system to a standstill. Every time the network went down, the lone server that ran the time-and-attendance software - nestled deep inside a data centre 1,500 miles away - had to be manually restarted. Jennifer would get on the phone and tell the data center techs where the server was and what they needed to do.
"Dealing with high-profile, highly paid, arrogant contractors who felt they could do network upgrades whenever they felt like it was challenging," says Hoffman. "They were so fixated on what they were doing they never thought about the peripheral effects of their actions."
Still, the other part of her job - keeping people from cheating - was worse. Several times she was called upon to play payroll cop by the carrier's internal affairs division. For example, there was the time a techie who got laid off in May logged into the system and filed time cards for the remainder of the year, so he'd continue to get paid while sitting at home on the couch.
Or witness the HR employee who pretended to rehire dozens of former employees during the middle of each payroll period, then "fired" them a few days later. Their paychecks were sent via direct deposit to a bank in a neighbouring state, where her partner in crime would withdraw the funds and split the proceeds.
That woman eventually got fired, though never prosecuted. Later, she tried to use Hoffman as a job reference. You can imagine how that went.
"I didn't want to have to spy on people, but after that incident with HR, I got called on by internal affairs a lot," she says. "When you work in a system like payroll, the things people do are unbelievable. You have to have really trustworthy people working for you or they can rob you blind. It's the dirty jobs like this that are the backbone of every organisation."Next page: Hot and sweaty in the data centre
Dirty IT job No. 5: Coolant jockey
Wanted: Individuals seeking close interaction with grease, dust, and high voltage while wearing full body gear in an oven-like environment. Must enjoy sweating. Personal counselling skills a plus.
You want hot, sweaty, grimy IT work? Talk to the guys in the hazmat suits who have to service the IT infrastructure in your data centre's hot aisle.
Whether working inside a windowless office that's been converted into a rack room (and garbage receptacle) or a gleaming state-of-the-art data centre, when the cooling system needs maintenance, the going gets hot and dirty fast, says Greg Grace, whose official title is "precision cooling team leader" for Emerson Network Power's Liebert Services.
Temperatures can easily surpass 100 degrees, says Grace, and there's no stripping down to your skivvies for relief. Cotton coveralls, hoods, face shields, and thick gloves are standard gear for protection against arc flashes from 480-volt power supplies.
"It can look like a bolt of lightning when it arcs," he says. "The way we dress we look like Marty McFly in our safety gear dealing with plutonium rods. By the end of the day, we end up smelling pretty gnarly."
Grace says new hot aisle containment systems designed to trap the heat coming off server racks turn up the temperature even more. "It's like wearing a snowsuit in a sauna," he says.
But when a data centre overheats, servers start failing. That's when Grace's phone rings, 24/7, usually with a frantic IT manager or business owner on the other end. And you need to stay until the job is done, even if it takes most of the night.
"When I'm on the site, I'm the guy the customer deals with," says Grace. "If the customer is having a bad day -- and if I'm there at 3am, he's probably having a bad day -- I'm the guy who takes the brunt of it. Oh, it can be a blast."
Dirty IT job No. 4: Marketing hag
Wanted: Dynamic go-getter who speaks (or can, at least, fake) fluent geek with the ability to transform interesting concepts into saleable products. Expertise in personal grooming and a working B.S. detector are required.
Relations between IT and marketing departments have never been exactly toasty. Geeks think marketers are too stupid to understand technology - which is, of course, the only thing that matters. Mar-com types would like them to change their shirts more often and learn to speak in English, not Acronymish.
Beyond the stereotypes, though, most technology companies would not survive without somebody out there crafting the right message.
"Someone has to slap some lipstick on this pig to get it sold and that is the marketing hag," says Robin Bectel, vice president of New Venture Communications. "The tech team may build a product they think is really cool, but it might not be what the market wants. It could do a million things, but there may be only a few functions that make it something people actually want to buy. We help bring their dreams into reality."
The dirty part? Techies often play a little fast and loose with the truth. But it's the marketing hag who catches hell for it.
"You have no idea how many times a client or boss has looked me in the eye and sworn a product was ready for prime time when it was barely into alpha," she says. "The features and benefits you tout as ground-breaking you find out later haven't even been built yet. You usually get the full story only when you ask for a review version to send for a tech shoot-out. By then, your reputation is shot and you have to salvage whatever brand you have left."
It gets worse, says Brenda Christensen, principal at Stellar Public Relations.
"You're asked to write a fictional press releases about 'expansion' into 'new offices,' but when you dial the number it's some knitting group in Brazil," she says. "Or you're asked to throw a 'TM' after a product name, only to find out later it's not really trademarked. Or you have a unit put through testing, only to find out from the review editor that someone Krazy Glued the power button back on. I had to cry to get out of that one. I could go on and on."
Then there are the little things not included in the job description, says Christensen -- like having to persuade the programmer/CEO to change his sweat-stained shirt before a big photo shoot or teaching him how to tie a tie in the back of cab on the way to a meeting. And, of course, handling their disappointment when their whizzy new product fails to make the front page of the Wall Street Journal or TechCrunch.
"The geek personality is very different," says Bectel. "I've worked in a lot of different markets, and techies have much higher expectations for coverage than virtually any one else. It's because they're so passionate about what they do, and they expect everyone else to be equally passionate about it."
Next page: Baaa if you're a professional scapegoat
Dirty IT job No. 3: Professional scapegoat
Wanted: Manager willing to take charge of enormous projects doomed to almost certain failure; ability to work tirelessly and alone; must be physically fit enough to shoulder all the blame.
When big projects go bad, they can go really bad. And few big projects have as big a reputation for badness as enterprise resource planning. ERP systems are notorious as multimillion-dollar sinkholes that aimed too high and often missed their targets entirely. Yet, for the thousands of organizations that adopted ERP, it was somebody's job to make the project happen or die trying.
As Mr. T might say, we pity the fool, and in the late 1990s, one of those fools was Michael S. Meyers-Jouan. As CIO of a small, family-run apparel maker, Meyers-Jouan's dirty IT job was to ease the DOS-era company onto what was then the cutting edge.
"When they recruited me, I found myself supporting an ancient order processing system running on an AIX box, a MAS 90 accounting system, and a collection of PCs," says Meyers-Jouan, who later became an independent technology consultant. "I was supporting an HR department that never quite understood Windows, relational databases, or network security. I was supporting process managers who would enter columns of numbers into an Excel worksheet, then add the numbers up on a calculator and enter the sum below the column, and CAD system users whose idea of 'backup' was to make copies on floppies."
Nonetheless, Meyers-Jouan was handed the task of bringing his new employer into the late 20th century. He produced a detailed RFP, and over several months, he winnowed the list of potential vendors from 30 to 3, and finally to a single winner.
"That's when the fun began," he says. "Installing the ERP software was relatively simple. Interviewing the users to develop business process road maps and job descriptions was tedious, but no worse than expected. Configuring the software to match the business needs was challenging and time consuming, but within our capabilities. But getting the users to adopt the new system was simply not going to happen."
The factory workers refused to participate in the process, despite promises it would cut their workload in half. The back-office employees couldn't understand how a factory worker covered in oily dirt had anything useful to tell them; they continued to rely on email, spreadsheets, and 10-key machines to do their jobs. And the last thing top management wanted to hear about was the need for "culture change."
"For the only time in my life, I was fired," says Meyers-Jouan. "The ERP system never did get put into use, and, not too long thereafter, the company stopped answering their phones."
Though his dirty IT job involved ERP, the same factors apply to any IT project that doesn't have the active support of both end-users and management.
"I've gone back to working as a developer," he adds, "but with increased respect for people who can make IT operations work in the business environment, and wariness of large implementation projects that don't have the full commitment of the customers."
Dirty IT job No. 2: The whistle-blower
Wanted: Highly moral individual willing to expose employer's dirty laundry. Risk is high; financial rewards can be considerable, though not guaranteed. Disgruntled workers welcome.
Nearly every organisation has dirty little secrets. More often than not, IT knows where they're kept. Sometimes, it takes a geek to step forward and bring them into the light. But it's not easy.
Just ask Roger Smith (not his real name). As a computer science teacher at an East Coast high school, Smith became concerned when the district bought single-user licenses of Adobe Creative Suite and Microsoft Office, then installed them on network servers where 5,000 users could access them.
Smith says he approached his superiors and the district's IT department and explained why that was wrong, but to no avail. So one day he called the Business Software Alliance and reported them.
"With some software, we were on day 120 of a '30-day' free trial," he says. "Part of what we're trying to do is to teach kids ethics. That's hard to do when the software you're using isn't licensed and the kids know that."
The BSA relies on tips from people like Smith, says Jennifer Blank, the BSA's director of legal affairs. Often it's disgruntled employees spilling the beans on their boss. Sometimes, though, it's just IT pros who want to do the right thing.
Of course, the BSA offers other incentives - up to $1 million for tips that lead to successful settlements with software pirates, though the largest reward it's ever handed out is in the "tens of thousands of dollars," says Blank. (To qualify for the $1 million reward, a tipster would have to expose piracy that generated at least a $20 million settlement with the BSA.)
Smith says he never asked for nor received any money, and as far as he knows, the district did not have to pay a settlement. But a few months after he contacted the BSA, his employers purchased legal licences for its software.
The BSA has its share of critics, who claim the trade group exaggerates the amount of software being pirated, targets smaller businesses that lack the resources to defend themselves, or acts as unofficial enforcers for well-heeled software makers like Microsoft.
"The people who call us names forget that the people they are defending are stealing software," says Blank. "You wouldn't go into Best Buy, put a copy of Windows in your pocket and walk out the door. Only in this case they're taking that copy of Windows and installing it on 100 computers, so they're 100 times worse."
Whistle-blowing applies to more than exposing software licensing violations. Earlier this month, biometric smart card maker e-Smart Technologies was ordered to pay $600,000 in damages and rehire an employee who was fired after alerting authorities to inaccuracies in the company's SEC filings.
But this dirty IT job is not for the faint of heart. On her blog, The Whistler's Ear database administrator Nell Walton details her three-year legal battle with her former employer, credit card processor Nova Information Systems (now Elavon), after reporting rampant security breaches. She ultimately lost. The breaches were quite real, but the court decided that, as a database administrator, she could not have had a "reasonable belief" her employer was breaking the law, as required by Sarbanes-Oxley.
"This is not a path I would recommend to anyone unless you have a completely ethical reason for doing so, have a backbone of steel, and a very thick skin (don't think you will make a million dollars, in other words)," she writes. "Something all IT people in the USA need to be aware of [is that] we don't have a lot of protections when it comes to whistle blowing."
Next page: Long suffering cable guy
Dirty IT job No. 1: Network sherpa
Wanted: Local and wide-area networking wonk who likes to get his or her hands dirty; must be willing to camp out in your car and possess the ability to talk customers out of committing felonies.
It's a pretty simple equation these days: no network, no business. Somebody's got to crawl through the muck dragging Cat 5 cable behind them or get two incompatible wireless technologies on speaking terms. Meet the network sherpa, whose job is to haul his clients up LAN Mountain and deposit them kicking and screaming at the summit.
But the dirtiest part of the job isn't squeezing into tight, dusty, rodent-filled spaces, says Bill Horne, who's spent years as an independent networking consultant and is moderator of the Telecom Digest . It's dealing with penny-pinching customers unwilling to upgrade their crumbling infrastructures.
"Hell hath no fury like a customer who hears he must pay for a wireless bridge in order to retire several hundred feet of RG-58A/U coaxial cable that's been serving as the Ethernet backbone between two buildings for twenty years," says Horne. "Even though the cable will be buried under the parking lot, damaged by rodents, and hanging from the ground wire the electric company has ordered him to vacate immediately, he will insist his network is still capable of '10 gigs at least.'"
Horne says he's often had to patiently explain to his clients that (a) bribing the electric company is not a good idea, and (b) 802.11 standards have matured dramatically since 10Base2 was invented.
Worse, if you do a good job for your clients, they'll want you to come to their homes and do the same thing there -- like the time an exec at Horne's largest client asked him to fix the Internet feed at his remote New Hampshire vacation home.
"One assumes when there's a problem with an Internet connection, the customer in question actually has Internet access," says Horne. "No such luck."
After a four-hour drive from Boston to the lip of the Canadian border, Horne arrived to find a Linksys wireless access point wrapped in a plastic food bag, duct-taped to a three-meter TVRO satellite dish, which was pointed at a distant hilltop where at one time there had been an unsecured Wi-Fi hotspot. Horne calmly explained to his client it's not a good idea to poach Wi-Fi, and having dependable Net access would require paying a company to provide it, even if you live in the middle of nowhere.
"FedEx ultimately came to my rescue," he says. "After spending a very cold night in my car - the propane tank was empty and I hadn't thought to bring an ax, although I do always carry a sleeping bag - I was rewarded with a satellite dish and associated paraphernalia, fresh from the Hughes assembly line."
Horne removed the old dish by tying a rope over a tree branch and pulling it off the mounting pole with his car. Then he installed the new system on the old post, which "delivered green lights and great download speeds at the first click of the circuit breaker."
Fortunately for Horne, that was his last hurrah. He's given up the network sherpa lifestyle and taken a job with computer security firm, where he only has to deal with hackers, malware authors, disgruntled employees, and corporate espionage agents. Ah, the good life.