Credit: Sébastien Thibault
Ideas Made to Matter
Workers with less experience gain the most from generative AI
By
Generative artificial intelligence may be able to pass the bar exam or help CEOs write speeches. But those results have been generated in a controlled setting, and with relatively little at stake.
In the real world, enterprises face the age-old question of technology innovation: How will generative AI affect workers, particularly those with limited experience?
According to a new paper by MIT Sloan associate professor MIT Sloan PhD candidate Lindsey Raymond, and Stanford University professor Erik Brynjolfsson, PhD ’91, inexperienced workers actually stand to benefit the most from generative AI.
The co-authors found that contact center agents with access to a conversational assistant saw a 14% boost in productivity, with the largest gains impacting new or low-skilled workers. In other words, the workers were upskilled, not replaced, thanks to the technology.
“Generative AI seems to be able to decrease inequality in productivity, helping lower-skilled workers significantly but with little effect on high-skilled workers,” Li said. “Without access to an AI tool, less-experienced workers would slowly get better at their jobs. Now they can get better faster.”
A different kind of disruption
Generative AI is the latest in a line of technologies to disrupt the workplace. Computers that can complete data entry, bookkeeping, and basic assembly-line tasks have been replacing or augmenting workers for decades. More recently, computerization has streamlined research and analysis, helping workers do more in less time.
Many of these technologies work when provided with explicit instructions, Li said: If you provide the right input, the computer will give you the right output. The subset of generative AI known as large language models is different, though, as it can infer the relationships between the inputs and outputs. (“Language” is a bit of a misnomer, as models can also analyze audio and visual files, computer code, protein sequences, and many other data sources.)
“If you give a large language model enough pictures of your mother and pictures of women who aren’t your mother, it will be able to figure out whether a single picture is your mother,” Li said. “What’s the impact of a technology that can do that?”
Room for improvement in the contact center
To study the impact of large language models, the researchers looked at a customer contact center at a Fortune 500 company that sells software to small businesses in the U.S. Eighty-three percent of the agents were based outside the U.S.
Contact center agents with access to an AI assistant were 14% more productive, with low-skilled workers improving the most.
Average contact centers have much room for improvement, the researchers write. Supervisors can spend up to 20 hours each week training low performers. Improvement comes from experience, which often means dealing with stressed-out customers. This can lead to employee burnout and extraordinarily high turnover; annually, up to 60% of contact center workers leave, and companies spend as much as $20,000 to replace each employee.
“Highly skilled workers are good at reading customers’ frustration — and that can take getting yelled at for six months,” Li said.
An abundance of data is another reason contact centers are a good testing ground for generative AI. As anyone who has called a helpline knows, conversations between agents and customers are routinely recorded for quality assurance purposes.
That gives generative AI models access to a large training data set, which can be used to provide agents with recommended responses to common customer questions or links to relevant product documentation.
Li and her co-authors emphasized that the generative AI model was meant to augment and not outright replace the contact center employees. The model offered recommendations only if it was “sufficiently confident” in its answers, which reduced the number of incorrect responses. In addition, workers weren’t required to use the recommendations. They followed them 38% of the time, which is consistent with the industry average for generative AI tools, Li said.
Gains in efficiency and customer sentiment
Workers using the generative AI model increased the number of customer chats resolved per hour by 13.8%, the researchers found. Within two months, they were resolving 2.5 chats per hour, compared with 1.7 for colleagues not using the model, who needed eight months to hit the higher threshold. In addition, workers using the AI model spent an average of 35 minutes on each chat, compared with 40 minutes for their colleagues who lacked a model. (These figures also account for the fact that workers often manage more than one chat at once.)
As noted, productivity gains were highest among workers with the least experience, who resolved 35% more chats per hour when they used the generative model. Productivity was essentially flat for workers with the most skills and experience.
Using the generative AI model also led to improved customer sentiment. Requests to speak to a manager declined by 25%, and transfers to other departments tended to happen earlier in the conversation, which suggests that the AI model was able to help workers better match a customer’s problem to the right business unit for a solution.
“The large effects on the lower-skilled workers tell us a lot about how we’re changing what workers are saying,” Li said. “It’s not just a recommendation that they always follow. It’s not just auto-complete. It seems that the model was changing [how they responded], leaving the customer thinking the situation has been resolved.”
Important findings, uncertain implications
The paper’s findings show a clear benefit to using generative AI in the context of the contact center. They also raise a lot of questions, Li said.
Related Articles
First, it isn’t clear who benefits from the productivity gains. Are contact center agents paid more when their performance improves, or do the developers of the AI model get a bonus? If workers can resolve more chats per hour, does a company hire fewer agents? Will contact center volumes increase as customers begin to realize the experience is better than it used to be?
It’s also worth exploring whether workers are learning from the AI model’s recommendations or simply following instructions. Are they becoming what Li described as a “small business owner whisperer,” learning how to diagnose customers’ specific problems? Or are they more like programmers copying code snippets from GitHub or drivers getting to their destination faster using Waze — using the provided recommendations to augment their work without thinking about how to solve the larger problem.
“If you’re just typing the recommendations, then you may not necessarily be learning. You’re more productive, but the source of your productivity resides in the technology,” Li said.
On the other hand, if companies take the time to teach workers how the technology can augment what they do, “workers start to learn faster, “ Li said. “And being faster at learning is going to make a big difference.”
Read next: why generative AI needs a creative human touch
