When Hilke Schellmann spoke German into a tool that checks how you perform in job interviews, instead of an error message, it generated a “total gibberish” transcript and said she was 73% qualified for the job.
We all know that AI tools promise efficiency, speed, and fairness. But in reality? *cough cough*
The above (unexpected) result gets to the heart of the conversation that unfolded at the 2025 Toast Summit in Toronto, where Schellmann, author of The Algorithm and Assistant Professor of Journalism at New York University, and Bahar Sateli, a partner at PwC leading advanced analytics and AI, examined how AI is changing hiring.
As the conversation revealed, these systems are shaping decisions in ways that often remain poorly understood, even by those deploying them.
The invisible gatekeepers of modern hiring
Many people have no idea their resume is being evaluated by an algorithm. Yet AI is present at nearly every step of the hiring funnel: screening, personality assessments, video interviews, even background checks that evaluate social media activity.
As Schellmann points out, this widespread automation is driven by scale. She learned that Google gets about 3 million applications per year, and IBM gets over 5 million. It’s no wonder there’s been a pivot to algorithms: not necessarily to find the best candidates, but to filter the volume.
As the as-seen-on-tv saying goes, there’s gotta be a better way. But is this it?
“If you use a couple of algorithms…on all of these people, you might actually hurt a lot of people if there’s something wrong with it,” Schellmann says. “There’s no regulatory agency where you have to tell them what algorithms you use, how you build them.”
But this efficiency comes at a cost. Schellmann shared examples of resume parsers that downgraded applicants with the word “women” on their resume, or rewarded those who mentioned “baseball” over “softball.” Others favoured names like Thomas, or flagged applicants based on postal codes. These data points often reflect gender, race, or socioeconomic status, whether intentionally or not.
“There’s no evidence [these tools] find the most qualified candidates,” she says. In fact, she adds, in one survey of C suite leaders found that 90 percent of leaders using AI tools know their software rejected qualified applicants.”
Bias is mathematical, but so is fairness
Sateli, who leads PwC Canada’s AI practice, added a technical perspective. “Bias is what makes machine learning work,” she explained. But not all bias is harmful. The challenge lies in ensuring algorithms do not base predictions on protected attributes such as gender, age, ethnicity, or proxies like hobbies or postal codes.
“You probably come up with a line and then you say, left of this line, it’s a yes, right of this line is a no. It’s a math problem, right? But what is the line based on? That’s the important piece.”
Still, explainability remains a barrier. The concept of explainability in AI, Sateli explains, is how a model reaches its final decision. We want models to be fully explainable, and we want the highest accuracy possible.
“In AI, there’s a trade off. You can create simpler models that are explainable, or you can create very high performing models that are absolutely not,” she says. “So what you desire versus what’s feasible — sometimes two different things.”
Regulation is not keeping up
While the EU’s AI Act is the first attempt at global regulation, Canada has yet to implement similar laws. That leaves hiring decisions in a grey area, governed by general privacy and human rights legislation, but not tailored for AI.
“Right now, companies are relying on vendors’ claims. But vendors are not neutral,” says Schellmann. “There’s no incentive for them to flag flaws in their own tools.”
Sateli agreed, noting that many of PwC’s clients are still in the early stages of understanding what questions to ask vendors. “We now bring legal counsel into AI discussions, just like cybersecurity,” she said. “Procurement teams need frameworks, not just promises.”
Rebuilding from the ground up
Both speakers called for innovation, not just in algorithms, but in process design.
“When I looked at the first generation of these tools, I felt sometimes… wow, we’re digitizing AI processes that kind of don’t work in the first place, or are not very good, and now we’re just amplifying it.”
Sateli suggested a shift: “You can use AI to build better AI.”
With generative tools, she outlined,you can create synthetic training data to fill gaps in representation, though rigorous validation is still needed.
She also proposed more holistic assessments. Instead of parsing resumes, why not test real job tasks in simulated environments?
While technical capabilities now exist to generate more representative data sets, Sateli explains, they still require careful oversight, validation, and transparency to avoid introducing new risks.
A human problem, not just a technical one
Ultimately, both experts returned to a common theme: AI does not solve bias. It mirrors it. And it does not make decisions. People do, based on the data and tools they choose.
“I’ve never found an AI tool where somebody said, ‘I really don’t like women, we’re going to build a tool,’” Schellmann said. “Everyone actually wants to be non-biased. They’re just not aware how easily biased data can turn into biased results.”
Sateli echoed this, urging technologists to pair technical rigour with ethical responsibility, “giving the opportunity [for] things like redress and also allowing the end users, or the people are being impacted by your application to be able to raise a concern and say, ‘I have a feeling something is not right, right?’”
Schellmann and Sateli ended with a challenge, not a conclusion. If AI is here to stay in hiring, then it is up to those building and buying the tools to ensure they do more good than harm.
Digital Journal is the official media partner for the Toast Summit.
