As part of my daily due diligence in writing this blog, I keep up with the postings on Microsoft’s Born to Learn blog. A recent post (1/17/2012) from Andrew Bettany entitled “Have MSTS/MCITP exams got easier?” caught my eye this morning, as much or more because of the comments to this post as for the post itself. Be sure to check it out if you’ve got a Microsoft exam in your near-term future.
But what really got my wheels turning were the concepts of “fairness” and “challenge” raised in a comment by Microsoft’s lead psychometrician, Liberty Munson, especially in light of my recent brokering of contact between herself, an exam developer, and a friend and colleague who had taken the 70-662 Microsoft Exchange Server 2010, Configuring exam no less than four times without managing to pass it on any attempt. As somebody who’s actively engaged in consulting with major Fortune 1000 companies to help them transition hundreds of thousands of email users into Exchange 2010, I have trouble accepting the notion that he simply doesn’t understand the platform and its capabilities. In that light, consider what Ms. Munson has to say in this posting:
Thanks for great post! It shouldn’t surprise anyone that people who pass exams tend to think they are easy while those that fail believe that they are too hard. The key is to make sure that those who fail also believe that they were “fair assessments” of their skills while those who pass also found them to be “challenging” (or “easy but challenging”). That’s nuance of “easiness” that is often overlooked by candidates and hiring managers. Exams should be “easy” for qualified candidates because we’re assessing skills that they have, but they should also be “challenging” and fair assessments of skills. We rarely get past the “it’s too easy” part of the conversation to understand the deeper nuance…Was it challenging? Was it fair? Does it differentiate you as someone who knows the technology? It’s possible that our exams haven’t been challenging–that this is assumed in the “it’s too easy” statement–but there’s a deeper question here that needs to be asked.
Enough with the philosophical aspects of this conversation :)…over the last year, we’ve made changes to our item writing guidelines and use of item types to increase the percieved difficulty of our exams because we understand that the conversation generally stops at “it’s too easy”…certain item types and item writing strategies can be leveraged to make exams feel more challenging without requiring more skills or knowledge. However, as we introduce new certifications, you’ll find that we’ll be requiring deeper knowledge and higher level skills to address comments that “it’s too easy.” Psychometrically, I regularly monitor the difficulty of the exams and ensure that it’s at an appropriate level for the target audience and programmatic goals/definitions of what it means to hold the credential. Changes are made when and where necessary to meet these targets.
I’d love to start hearing people talk about our exams as “challenging and fair assessments” as we continue to improve the ability of our certifications to differentiate people who really know their stuff from those who don’t or just sorta do. Andrew–Thanks for starting to get the word out!
Clearly, some of the ongoing renovation of existing MS exams and development of new ones incorporates great concepts and intentions on the part of the people who work with SMEs to put those exams together, and to measure and monitor them for statistical relevance to the populations being tested. But “fairness” is a difficult concept to get right, especially when exam questions veer into the realms of what my aggrieved friend and colleague perceived as mostly “administrivia” that real-world admins would simply look up if they had to deal with such seldom-used (or rarely-needed) features and functions. In his case, he’d readily concede that the 70-662 exam is challenging, but also vehemently claims that it’s grossly unfair.
I symthpathize with both parties in this kind of encounter. On the one hand, having created hundreds of practice tests for the certification exams covered in my many Exam Cram and study guide certification books, I get how difficult it is to create sound, meaningful and relevant exam items for candidates to ponder and learn from. On the other hand, I also really feel for my friend’s situation where his consulting company requires him to pass the 70-662 exam so he can present the “right credentials” to his many clients, but where he feels that the exam is more of a random check on obscure or little-used Exchange features and functions, than a meaningful assessment of his skills and knowledge of the platform.
This is a tough situation, and one that requires not only that both sides recognize each other’s motivations and priorities, but also that the testing party provide enough information to testees to let them understand what they must learn to get past the exam, and that testees take this seriously enough to make a concerted effort to learn and master the necessary skills, knowledge, and even administrivia details to get over the hump. Usually, taking a Microsoft Official Curriculum class, reading the books, and acing numerous practice exams is enough to guarantee a good result. But when this tried-and-true method fails, it’s time to start asking why and trying to reason one’s way into the proper information set to figure out what’s wrong. My colleague also believes that MS should provide testees with more detailed feedback about questions answered incorrectly rather than simply flagging certain concept areas where the test-taker failed to meet minimum score requirements.
All this goes to show that certification exams really aren’t easy to create, and sometimes can be incredibly difficult to pass as well. It should be interesting to see how this situation plays out, and how the two parties will find some kind of rapprochement.