Anybody who reads the Microsoft Born to Learn blog can only go so long before encountering a post from Microsoft’s Psychometrician Program Manager. I regularly mention her in my blogs here (most recently on December 19 in a post entitled “Interesting Changes Coming to MS Cert Exams“). Earlier this week I had the pleasure of interviewing Liberty Munson by phone about exam development issues and how candidates with specific exam issues can bring them to Microsoft Learning’s attention for possible resolution. It was an interesting interview, chock-full of useful and illuminating information about how Microsoft evaluates the quality of their exams and questions, and keeps an eagle eye on them to make sure they stay fresh, useful, and relevant.
According to Liberty, Microsoft’s over-riding concern is to make sure any exam “…is measuring real-world, relevant information in a valid and fair way.” This requires constant evaluation of how questions work and behave in actual exams. Other than the typical psychometric evaluation of questions, Microsoft also uses information that candidates provide through a variety of channels. For example, candidates can always comment on questions at the end of their exams, but with a portfolio of around 110 active exams right now means only a sample of comments can be read closely for any given exam. Beta exams are a different story: not only does Liberty herself read all beta survey results, that feedback is incorporated into the published (final) version of each exam.
In addition, Liberty and her team track customer satisfaction metrics quarterly, on the lookout for up or down changes in satisfaction levels (so if you’re asked to complete a survey on your exam — please do it — because Microsoft takes this information seriously and it drives changes to exams), as well as exam metrics, on the lookout for trends in results on a question-by-question basis. This involves examining each question to make sure it’s still “performing” — which means distinguishing candidates who know the material as evidenced by overall exam performance, from those who don’t by the same measure. “We want to make sure the right people get it right,” says Liberty, in explaining that an exam’s primary purpose is to separate high performers with strong skills and knowledge from low performers who are only superficially familiar with a test subject and its various ins and outs.
There’s also a fair amount of what Liberty calls “data forensics” work underway on exam results. Her team of crack statisticians looks for what she calls “patterns of anomalous behavior” where unexpected performance on some combination of test items (which could indicate an unusually accurate brain dump or practice test that reproduces items verbatim) or at a specific test center (which could indicate shared information among test-takers at such sites, as has been documented in various test centers outside the USA from time to time) triggers closer investigation (and occasional invalidation of test results).
Based on some recent interaction with an unhappy friend and colleague about a specific MCTS exam, I also discussed Microsoft Learning’s complaint handling process. Liberty explains that when complaints are fielded her team looks for performance of items on the exam from a psychometric perspective to see if anything unusual or out-of-the-ordinary is present. They also work with the product planners who are responsible for the technical content of the exam to make sure items are accurate and relevant to real-world, day-to-day tasks and activities. These product planners “use strong relationships with business groups and SMEs to work out potential item pool issues and get them solved,” says Liberty.
Next, Liberty asked me to remind readers how to take best advantage of the “item challenge process” for Microsoft exams. Test-takers unhappy with specific exam items should do their best to remember as many question details as possible, and to include them when they submit a complaint. Item challenges are discussed on the “Item Challenge” tab on the Microsoft Exam Policies web page, where you’ll find a link to a challenge form download and an email address to which it may be sent (MCPHelp@microsoft.com, with “Exam Item Evaluation” as the subject line, and only one item per each such email, please).
Finally, Liberty also informed me that “2012 is going to be a big year for MS certification” and that readers interested in new exams should stay tuned to the Born to Learn blog to keep an eye out for the numerous beta exams that will be released throughout this year. She also indicated that betas fill up “really fast” these days, and reminded me that the best way for people interested in taking beta exams to get an exam invite is to register themselves in Microsoft’s SME Database. “That’s where most of our invitations go these days,” she adds, and explains why she happily emailed me the registration link after we’d concluded our call.