Aug 9, 2026
How I made a growing interview panel more consistent
When more engineers start interviewing, consistency becomes a different problem from hiring capacity.
A small interview panel can work well for a while. Then the team grows, hiring picks up, and the same few people end up doing most of the interviews.
That was the situation I ran into partway through growing the org 3x. We needed more interviewers, but simply adding people created another problem: candidates could get very different interviews depending on who they happened to speak with.
The problem wasn’t the number of interviewers
We had senior engineers who had never interviewed before, and I wanted to get them involved.
But I didn’t want every new interviewer to build their own question set or decide their own level of difficulty. A good candidate shouldn’t have a significantly different experience because of which interviewer they happened to get.
I started with a bucket of problems
I put together a defined bucket of problems around the areas we actually wanted to evaluate.
The new panel of five seniors worked through those problems together. The idea wasn’t to turn them into interview robots. It was to make sure everyone understood what a reasonable problem looked like, what signal we were looking for, and roughly what could be covered in thirty minutes.
Candidates would then get one problem from that bucket.
That small constraint made a surprisingly big difference.
Interviewers still had room to ask follow-up questions and understand how someone thought, but the baseline became much more consistent.
Scaling the panel changed the job
The interesting part was that we weren’t really scaling interviews.
We were scaling a system for interviewing.
The first few interviewers had built up intuition over time. That intuition doesn’t automatically transfer when you add five more people.
You have to make some of it explicit.
Not every part of an interview needs to be scripted. But the things you’re actually trying to measure should be clear enough that different interviewers are looking for roughly the same signal.
What I took away
Adding people to a process doesn’t necessarily scale the process.
Sometimes you first have to remove the variation that those extra people introduce.
That’s really the same problem the org ran into at every stage of the 3x growth, just in miniature — hiring didn’t get easier by adding people to it. It got easier once the people already involved were pointed at the same target.
← Back to writing
Comments
Loading comments…