Pirates, cheats and IT certs

Some ne'er-do-wells steal test questions and answers, and cheaters buy that information, share answers in chat rooms, pay other people to take tests for them and bring a range of technologies and techniques into test centers to gain an edge.


Attacking pirates from the cloud

The traditional computer-based testing approach of having full copies of IT certification tests and answers stored in thousands of test centers worldwide has made test theft difficult to stop. To reduce the risk, IT certification providers are beginning to adopt Internet-based technology (IBT), cloud-based software as a service methodology that delivers questions, one at a time, in encrypted form, to a secured browser on each test taker's desktop.

This approach eliminates the need to download and store tests and answer keys at each testing site, which can have different levels of security depending on their size and where they're located. "The use of IBT is still relatively small but growing," Caveon's vice president Steve Addicott says, and big players such as Microsoft and CompTIA are already starting to adopt it.

At Microsoft, "We use the traditional delivery engine as well as just-in-time, Internet-based delivery," says Shelby Grieve, Microsoft's director of professional certifications.

Internet Testing Systems LLC sells software and online proctoring services that IT certification programs and test centers can use via a private-label portal to deliver content over the Internet to test takers anywhere. "We stream encrypted test items one at a time and only decrypt them when rendered on the screen," says Cabell Greenwood, vice president of business development.

Kryterion offers IBT and online proctoring for IT certification programs. With online proctoring, "There's no opportunity for any level of collusion between the proctor and the test taker," says Dave Meissner, chief operating officer at Kryterion Inc.

CompTIA is working with Pearson VUE to deploy IBT, possibly later this year, and Bryan Kainrath, vice president for certification operations at CompTIA, is bullish on the technology's prospects. "We don't have to send the answer keys. We pull the items back, take it offline, do the scoring and send the results to the candidate. We can secure items for a lot longer."

But IBT isn't always a good fit. It requires significant bandwidth, and some testing centers, particularly in overseas locations where the most intellectual property theft occurs, don't have enough to reliably deliver tests in that way, Addicott says.

That process can present an expensive challenge, however, because organized theft rings can compromise entire tests within three to five weeks of when they're first released, while most IT certification exams are refreshed every 12 to 15 months, Addicott says.

Kainraith admits that's a problem, but he thinks that questions take a bit longer to appear on brain-dump sites, and says CompTIA replaces tests at a rapid pace. "We're able to churn our items a lot faster than 12 to 15 months," he says, although he declined to say how fast.

While CompTIA has the scale and resources to turn over its test questions more quickly, smaller IT certification programs are more limited because the cost of building and maintaining tests ranges from hundreds of dollars per question to thousands of dollars per test item, according to Caveon.

Countermeasures: Tripping up the cheats

Catching cheaters has become its own science. "More candidates are sharing knowledge than we've seen in the past," says Kainrath. But both test centers and IT certification owners have ways of figuring out who's using stolen and shared test data, as well as who might be coming in to steal it.

In addition to using live proctors, Microsoft and others are moving toward online proctoring, which combines the use of a video camera with a live feed of the test taker's screen. While an online proctor is limited by what he can see on a video camera, it's easier to take immediate action against cheaters, Grieve says. Because they can look for suspicious activity at the question level, online proctors can identify cheating sooner and end the test before the candidate can see -- and possibly compromise -- the rest of the exam content.

Test centers also have ways to tell if candidates have been memorizing stolen test questions and answers or sharing knowledge in chat rooms. "We leverage several different publication strategies and question types designed specifically to address cheating," Grieve says.

While Grieve declined to provide details, Addicott says some of the more basic anomalies include people who perform at "superhuman speeds" on the exam or who perform well on items that have been on the test a long time while scoring poorly on newer items -- an indicator that the individual may have memorized stolen test content.

Some IT certification exams also catch people who have memorized stolen test data by including "Trojan Horse" questions that deliberately include the wrong answer in the official answer keys. These questions don't count toward the candidate's overall score, but if the test taker answers a predetermined number of such questions with the incorrect answers listed in the answer key it's assumed that they used stolen information and the test is automatically invalidated, says Addicott.

Certification programs may also use different test designs in an attempt to thwart cheaters who have memorized test questions and answers. These include scrambling the order of questions on any given exam, randomizing the order of answers to multiple-choice questions, having a pool of questions from which to choose from for each test item and giving different candidates in the same test center entirely different versions of the test.

CompTIA and other certification organizations have also started to supplement or replace some of the standard multiple-choice test questions with adaptive and performance-based methodologies that are harder to compromise. With adaptive testing each successive question the user sees depends on whether or not he answered the previous one correctly. As soon as the test determines that the taker knows -- or doesn't know -- the content, the test ends. "It's a more refined manner of judging, but it also provides security," says Greenwood.

CompTIA is adding progressively more performance-based testing, which uses scenario-based questions that ask the user to perform specific actions in a simulated environment. Such questions are harder to memorize. "At that point it becomes easier just to study," says Kainrath.

And that, in a nutshell, is a key part of CompTIA's strategy. "We can't stop cheating, but we can make sure it takes a lot of time versus just studying."

Next section: Getting caught: A great way to kill a career

Find your next job with computerworld UK jobs

"Recommended For You"

The real dirt on programming certifications How to prepare for the PMP exam - Part 7: Exam day logistics