We’re running an A/B test on human and AI involvement in hiring

We’re hiring for two roles, and trying two very different experiments

We’ve spent many hours debating when, where, and how AI should fit into our hiring process. As a small team with a lot of ambition, we are trying to level up, and hiring is ripe for this level of innovation. At the same time, we want to keep a high level of human touch and judgment in hiring. It’s such a key area, when each hire has a huge impact on our goals and culture.

For a long time, our process rested on the idea that a Buffer teammate would carefully read every application. That came from a good place, and we’re proud of it. At today’s volume, though, it has meant long waits for candidates and stretched-thin hiring managers. We know we need to modernize our process so we can do better, both for the people applying to Buffer and for our hiring managers.

On the flip side, if you’ve looked for a job lately, you know the market is tough right now. It can be incredibly competitive, and at times fairly inhuman. And, it’s been tough to be on the hiring side in its own way. More and more of the applications we see are largely written by AI. We understand why; applying to a job (or many) can be draining, and job seekers need to preserve their time and mental health. But that has made it harder for us to spot the thoughtful and personal applications, and get back to people in a reasonable amount of time.

So, we’re testing two ways to get there. We’re opening two roles, with very different application processes, as an experiment. Our goal is to learn as much as we can about the pros and cons of each approach, as we work to build a thoughtful hiring process that’s both efficient and deeply caring in a post-AI world.

Here’s how we’re running these experiments.

Experiment 1: Engineering Manager: a much lighter way to raise your hand.

For our EM role, we’re making it much easier to say “I’m interested.”

• A short intro instead of a long application. Our applications have always asked a lot: a series of detailed questions about the work and the person applying. We loved getting to know people that way, but it was a big investment for the candidate, and it stings when it doesn’t work out. This time, we’re just asking for your profile and a short introduction, so we can do more of the work to understand your experience before asking more of you.
• Joining a talent pool, not applying for a single role. We know that lots of folks could be a fit for a variety of roles. And in fact, many of our teammates were hired on their second or third try, because the right role came along at the right time. Careers are long, and often more like jungle gyms than ladders. So, instead of simply applying to one role and getting a yes or a no, we’re inviting folks to join our pool of interested folks. We’ll assess for this role first, and then keep other applicants in mind for future opportunities (with their permission).
• We’re leaning more heavily on AI to assess applications. We’ll use AI to help us review profiles and surface those with experience that matches what we need. Our hiring manager will reach out directly to the people we’d love to meet for an interview.
• A faster and more focused interview process. Once an interview process begins, we aim for candidates to progress through it and hear back from us much faster than in previous rounds. With less time spent sorting through applications, our hiring team should have more time for the candidates in discussion.

Experiment 2: Security Engineer, a human-first application and review process.

For our security role, we’re going the other direction. We’ll use a more structured application, with a human review of every one. This approach to application review usually takes anywhere from a few weeks to a few months. We’ve grown our hiring team recently, so we’re setting a more ambitious goal of getting back to folks within a couple of weeks.

• A few quick yes/no questions first. This role has a timezone component, so before applicants invest a lot of time, we’ll confirm a few details and general experience and qualifications.
• A detailed questionnaire, including a short video. Once we invite candidates to step two, we’d like the human to shine through. We’ll ask for concrete examples from past work, and a casual video from every applicant.
• A human reads every application. Everyone who applies hears back from us personally, whether or not we move forward. This most closely mirrors our previous and traditional approach.

What we’re watching

This isn’t a perfect A/B test. These are different roles, in different markets, with different hiring managers. Still, running them side by side should tell us a lot.

For example:

• What biases do each approach introduce? We know that both humans and AI bring certain biases to a search. However, we expect the human-led approach to capture more nuance; a lighter application with AI review could overlook someone whose path doesn’t look typical on paper.
• How quickly can we move? We intentionally don’t rush our process, but speed matters; great candidates don’t stay on the market for long. This is one area we expect the AI-led process to easily win!
• How will candidates feel about each approach? Most modern tech folks will expect a certain amount of agent-involvement in everything, but it can be somewhat comforting to know that a human has double-checked something as personal as a job application. I’ll be curious to see if this makes a difference in how candidates experience the process, and as an extension, how they feel about us as a company.
• How will our hiring managers and team feel about each approach? Our teammates appreciate the time-saving in the AI-first approach, but ultimately what matters most is results: whether we’re able to find and hire a great candidate who accepts our offer and succeeds in the role.

In other words, the application process needs to give us enough information about candidates to evaluate their experience, without asking for an unreasonable amount of time or investment in the process, from either side. This experiment is designed to help us find that balance!

If the lighter, AI-assisted approach works well, we can respond to people faster and develop a true talent-pool-based approach.

If the human-first approach works well, that tells us the personal touch is worth protecting, and we need to design the application process to allow us to do it at scale.

As always, we’ll share what we learn, including what doesn’t work as well as we hoped!

A quick note on AI, if you’re planning to apply to any role at Buffer in the future:

We’re using AI more in our day-to-day work, and we’re all-in on those efficiency gains when it’s done thoughtfully. But we don’t outsource our interpersonal conversation or most important writing. We’d like to ask the same of you. Using AI to check your grammar, organize your thoughts, or jog your memory for examples is completely fine. What we want to avoid is answers generated whole from a prompt, because we want to hear from you.

If you’re an engineering manager or a security engineer who’s been curious about Buffer, now’s a great time to say hello: https://buffer.com/journey#open-roles