Post By: Chris Campbell, co-founder CampbellBarr Ltd.
When talking with Compliance leaders there is an almost universal understanding that end of course testing is, at best, a blunt tool when trying to demonstrate understanding. Despite acknowledging this problem at some point someone will say: “Yes, but what should the pass mark be?” The only thing that exceeds the horror of that question is the answer – “It depends”. Possibly the most infuriating word pairing in the English language; with a special mention to the emotionally traumatizing “group activity” or perhaps “organized fun”. However, when it comes to setting pass marks it really does depend on a whole heap of considerations. In my 25 years in this industry I have lost count of the number of hours spent debating the merits of 65% vs 75% as an acceptable pass mark, what is the retest protocol and how do we select the right number of pool questions to randomize from. None of this really matters if you haven’t considered the really important question:
What do we want to achieve?
No really, what does the business hope to achieve by testing this individual? If it is awareness of a policy or regulation, then a 10-part multi-choice set at the end of a piece of learning doesn’t help. At best this is checking short term recall but most likely is an exercise in recognition (I saw the words “facilitation payment” 2 minutes ago when I completed the module; I see the word Integrity, that sounds like a value.) Surely, we want awareness because we want the learner to know what to do, who to ask, when to stop etc. All in order to make the business better protected and enabled. An end of module test does not do this; smaller, regular quizzes may help, and other pieces of spaced repetition will help. So, if you really want to test to check awareness or understanding you might want to stop end of module testing and think again.
If you want or need to test competency (and specifically the ability to apply knowledge) then testing can be helpful, really helpful, but it should be separated from the formal learning and be in context. It should also be challenging and meaningful. This is when the pass mark question comes in. There are some cases when anything less than 100% would be worrying. I don’t want my heart surgeon to have achieved 80% on their test, if the 20% they got wrong was about how to perform the angiogram procedure I am about to have! Thankfully we don’t test surgeons this way, why? because we know it does not work.
For the test and the learning to be effective it must require effort. You should be striving for desirable difficulty. A learner should know that it will be a challenge, that effort will be required to pass and that passing will deliver a sense of achievement. The same thinking should apply to all content design. For many organisations the need to demonstrate competency in a key risk area is an important part of their E&C strategy. It gives leadership comfort to know that 98% of employees identified as bribery exposed have completed and passed their online learning. We can focus on the recalcitrant 2% and target the managers whose teams are in these groups. It is measurable and therefore we ascribe meaning to it.
In addition to greater educational benefit for the learner, well executed testing can be an enormous source of helpful data for your wider program. It does however mean that you have to get granular and really think about each individual question – why are you asking this particular question? A well-constructed test can help expose knowledge gaps either in a topic area across the business, by a specific business unit or cadre of employees. This is valuable information and should inform how our increasingly limited resources are directed. A question that everyone always gets right should probably be retired. If a specific high-risk group is indicating that they are confused about what to do about contact with competitors, then this is a place to target resources. If very few people can correctly identify components of our due diligence procedures, then look at the question and most likely look at the learning being provided. Too often a set of questions are written, uploaded and never considered again. A valuable source of both education and information is being ignored and we risk undermining the entire process.
Conversely, when little thought is given to how testing becomes meaningful and valued, not only does the efficacy of the test diminish, but the perceived worth of the whole training. Now, the KPMG US audit teams cheating scandal wasn’t caused by bad testing alone, but when KPMG auditors including senior partners in charge of public company audits, cheated on internal ethics, integrity and compliance tests, sharing answers with other partners and staff to help them pass, it does demonstrate some disdain for the testing process. If the testing process was really valued, then this kind of thing could not have happened for so long.
There are many E&C teams doing a fantastic job in this area, where testing is meaningful. Some organisations lock employees out of systems if they can’t pass the test, others use Artificial Intelligence to look at testing data to start directing interventions and predicting potential breaches, great work is happening. So, what should the pass mark be – it depends…
About the Author: Chris Campbell has worked in E&C education for over twenty years. Chris co-founded E&C education specialist CampbellBarr ten years ago, during this time Chris has designed and delivered E&C education in 35 countries for global clients. chris@campbellbarr.com
Why don‘t we go a different way? Why not presenting real work life challenges to be resolved based on what they‘ve learned? It is more work for those who need to read this, but would give a real picture if lesson learned or not! Ethical behaviour never will be tick the box. I had the honour and pleasure to work with Chris for more than 10 years and did. dozens of face to face ethical trainings. For each participant I would have been able to say passed or not, without test!
Comments are closed.