Why ESG ratings vary so widely (and what you can do about it)

Environmental. Social. Governance. In recent years, ESG, also known as sustainability or corporate social responsibility, has moved from the feel-good fringes to the center of business decision-making.

Consumers want to purchase products from companies that reflect their values — environmental protection, the empowerment of women, or the absence of child labor, for example. Investors, meanwhile, want to know if they are financing activities that might pose a reputational risk.

Their message is getting through — by one estimate, some 80% of CEOs believe demonstrating a commitment to society is important and look to sustainability ratings for guidance and benchmarking. An estimated $30 trillion of assets are invested worldwide that rely in some way on ESG information, a figure that has grown 34% since 2016.

The challenge becomes how to accurately measure a company’s environmental and social impact, particularly given that ESG remains an evolving concept and reporting standards are still in their infancy.

Currently, interested parties typically contract with one or more independent agencies that evaluate and assign ESG ratings to firms. The problem, a team of researchers at MIT Sloan have found, is that ESG ratings diverge substantially among those agencies. A new working paper, “Aggregate Confusion: The Divergence of ESG Ratings,” documents the disagreement among the ESG ratings of five prominent agencies around the globe — KLD, Sustainalytics, Video-Eiris, Asset4, and RobecoSAM.

The research team — Florian Berg, Julian Koelbel, and Roberto Rigobon, all associated with MIT Sloan’s Sustainability Initiative — found the correlation among those agencies’ ESG ratings was on average 0.61; by comparison, credit ratings from Moody’s and Standard & Poor’s are correlated at 0.99. That means “the information the decision-makers receive from [ESG] ratings agencies is relatively noisy,” the paper states — a condition researchers call “aggregate confusion.”

Some major real-world consequences can flow from the discrepancy in ratings, the researchers write:

• Corporate stock and bond prices are unlikely to properly reflect ESG performance as investors struggle to accurately identify out-performers and laggards.
• The divergence can dampen the ambition of companies seeking to improve their ESG performance, thanks to the mixed signals they receive from ratings agencies about which actions are expected and will be valued by the market.

Taken together, those consequences can be significant.

“The ambiguity around ESG ratings is an impediment to prudent decision-making that would contribute to an environmentally sustainable and socially just economy,” the paper states.

What drives ratings divergence

In investigating what drives the divergence of existing ESG ratings, the research team found that ratings agencies may adopt different definitions of ESG performance, or they may take different approaches to measuring that performance.

The paper identifies three distinct sources of divergence:

• Scope divergence can occur when one agency includes greenhouse gas emissions, employee turnover, human rights, and corporate lobbying in its ratings scope, while another doesn’t consider lobbying.
• Weight divergence can happen when agencies assign varying degrees of importance to attributes, valuing human rights more than lobbying, for example.
• Measurement divergence occurs when ratings agencies measure the same attribute using different indicators. One might evaluate a firm’s labor practices on the basis of workforce turnover, while another counts the number of labor cases against the firm. While both capture aspects of a firm’s labor practices, they are likely to lead to different assessments, the research cautions.

All told, the research team was able to determine that differences in measurement explained 50.1% of total differences among ESG ratings, with divergence in weight explaining 13.2% of differences, and divergence in scope accountable for an average of 36.7% of differences. Taken together, weight and scope divergence can be seen as how a given rating agency defines sustainability.

In addition, the team also identified a “rater effect” — when the ratings agencies’ assessment of a company in individual categories seemed to influence their views of the company as a whole. Specifically, when a rater judged a company as positive for a particular indicator — human rights, say, or labor practices — they were then more likely to judge other indicators as positive too.

Re-do the math

Armed with awareness of the substantial discrepancy between ESG ratings organizations, how should companies and investors proceed?

Companies should work with individual ratings agencies to establish open and transparent disclosure standards and ensure that data is publicly accessible — both moves that will discourage agencies from basing their ratings on sources prone to divergence, according to co-author Berg, a research fellow at the MIT Sloan School of Management.

“One of the reasons why you have this divergence is that ratings agencies will try to find proxies if they don’t have access to data,” Berg said. “If you don’t release data on how many women you have in management, they will use some other proxy, which may match up with what you're actually doing, but it may not.” 

Investors can use the researchers’ methodology as a framework to disaggregate ratings and impose their own weighting on indicators, hopefully leading to the development of a more coherent decision-making process.

In the short term, Berg said, companies should conduct a thorough due diligence before choosing one rating agency over another.

“Keep in mind that ESG rating is still a young field, and the definition of sustainability is by nature a fluid one,” he said. “What’s important today might not be important tomorrow.”