I once watched an entire engineering team grind to a halt over a certification exam. Not because the material was hard. Because one of the questions was wrong.

The question had a specific answer it wanted, but the answer was incorrect. Everyone knew it. So they did what humans do. They gathered in a hallway, compared notes, and figured out which wrong answer the system would accept. They passed. Nobody learned anything. An hour of productive work disappeared so a dozen people could reverse engineer a broken multiple choice question.

This is not a rare event. If you’ve administered technical assessments at any scale, you’ve seen it. Someone writes a question, picks an answer, and moves on. Maybe the technology changed. Maybe the question was ambiguous from the start. It doesn’t matter. The test becomes a puzzle about the test, not about the subject.

Here’s what bothers me more than the broken questions, though. Even when the questions are correct, they only measure a very narrow skill. The skill of performing on paper.

Your best engineer, the one who debugs production issues at 2am and mentors three junior developers, might score lower than a new hire who’s good at studying. That doesn’t mean the new hire is better. It means they’re better at tests.

Great technical people are not always great at marketing themselves. The same way a brilliant salesperson might bomb a written exam about sales methodology. They just do it. They read the room, adjust, close. Ask them to explain their framework on paper and you’ll get a blank stare.

I think the only honest way to know what someone is capable of is to watch them do it. Not a simulation of doing it on paper. Not a multiple choice proxy. The actual thing, or something close enough that it demands the same kind of thinking.

That’s the gap we keep coming back to at Bubot. The difference between what someone can select from a list and what they can actually do when the situation is in front of them.

If you’re rethinking how you evaluate your people, take a look at what we’re building at https://www.bubotlearning.com or reach out at info@bubotlearning.com. I’d like to hear what you’ve seen.