Making the most of AI: The latest lessons from MIT Sloan Management Review

Organizations are increasing their investments in artificial intelligence — and learning that they cannot expect to achieve ROI with a simple “set it and forget it” mentality. The most recent insights from MIT Sloan Management Review highlight the importance of building a data-driven culture, knowing how to evaluate AI tools, and effectively managing and sharing data internally and externally. There is also a new framework for improving career development.

The human barrier to better AI adoption

By many accounts, enterprises are making progress in AI adoption. According to the latest survey from NewVantage Partners:

• 82% of companies are investing in data modernization.
• 83% have appointed a chief data officer or chief data and analytics officer.
• 94% plan to increase their overall investments in data this year.
• 98% expect to achieve ROI on data and analytics investments this year.

But there’s a catch. As Thomas H. Davenport at the MIT Initiative on the Digital Economy points out, the obstacles to becoming a data-driven company are largely human. Only 24% of companies describe themselves as data-driven, and only 2% have deemed data literacy an investment priority.

Organizations see that culture change is necessary. One key reason is that computer scientists define the business goals and success metrics for AI differently than other stakeholders do. Another is the dry nature of data literacy education. A third is that, for all their executive hiring, few companies are investing in talent to help build that data-driven culture.

Put another way, leaders seem to have little interest in identifying the barriers that prevent their organizations from becoming more data-driven, never mind allocating technical resources or personnel to break down the barriers. Until that happens, Davenport and co-author Randy Bean of NewVantage Partners write, organizations are unlikely to produce the ROI they expect.

Read: Action and inaction on data, analytics, and AI

Find the “ground truth” in AI models

Managers who lack a background in computer science can be tempted to evaluate AI products based on high-level performance metrics. The most common metric rates the model’s accuracy by comparing its output to the ground truth, or the “correct” answer based on the training data set. A model rating 1 on a 0-1 scale, for example, would correctly predict that a red street sign with eight sides means “stop.”

The challenge, as spelled out in a paper co-authored by Natalia Levina, PhD ’01, is that there’s rarely an objective ground truth to feed an AI model. In the absence of objective data, models often rely on expert opinions, which are highly subjective and difficult to validate. That proves especially problematic when organizations intend to use AI to influence sensitive decisions. A system using a single mammography image to predict the likelihood of cancer is far from the ground truth, as the standard for validation requires numerous images — not to mention physical examinations, an assessment of risk factors, and many medical appointments.

Levina and her co-authors urge managers to “peel back the layers” of AI models to assess the ground truth and determine whether a less-than-perfect training data set is worth the risk given the types of decision being made. This process has an added benefit, they said, as unpacking how AI models make decisions provides insights into how to improve human decision-making.

Read: The No. 1 question to ask when evaluating AI tools

4 steps to more strategic data sharing

Effective data sharing is partly tactical, given that technical infrastructure and management processes are necessary for making data available. The bigger hurdle tends to be strategic, because organizations might not see how data sharing contributes to developing data-driven products or new business opportunities.

Barbara H. Wixom and Ina M. Sebastian at the MIT Center for Information Systems Research provide four best practices for strategic data sharing:

1. Increase data liquidity. When data assets are standardized, accurate, and secure (and anonymized, if necessary), they are more easily shared internally and externally. Storing data in a centralized data warehouse or data lake further improves liquidity.
2. Reduce friction. Manual approaches to data governance hinder access, which slows  value creation. Automated processes for granting access to data based on a user’s  permission level provide both oversight and flexibility.
3. Level the playing field. Making data available to customers offers greater visibility into how products are used. It also helps establish trust and gets both parties excited about what’s now possible.
4. Manage the rollout. To ensure that value is created (and captured), three things should happen: Teams should be accountable for reaching goals, solutions should be readily deployable, and everyone should accept that change is inevitable.

There are two approaches to adopting these best practices. One is to work with a trusted customer on a pilot project, learn from the process, and repeat as desired. The other is to focus on improved data sharing among internal business units.

Read: Level up to strategic data sharing

Where AI can — and cannot — improve data management

One important element sits at the heart of sharing data, building trusted AI models, and encouraging a data-driven culture within an organization: data management. Davenport points to five areas where AI can play an important role in data management.

1. Classification. Optical character recognition and deep learning can extract data from unstructured sources like videos and faxes, determine what it is, and prepare it for analysis. 
2. Cataloging. Automatically searching data repositories helps organizations determine where data originated, how it has been modified, and where it currently resides.
3. Quality. Business rules can define allowed data values and delete duplicated data. Advanced AI systems can make corrections or predict how data gaps ought to be filled. 
4. Security. Automated threat assessments can predict attack paths, identify patterns in attacks, assign severity levels, and — in some cases — mitigate a threat without human intervention.
5. Integration. Probabilistic matching can merge disparate data sets into a single, “golden” data record, with little need for manual review.

That said, there are still several areas where humans are better than machines. These include defining business terms, developing data governance policies, deciding where to store and process data, and determining whether the right data is being used to solve a problem.

Read: How AI is improving data management

3 ways to improve professional development

Large organizations tend to approach professional development in one of two ways. Both are flawed: Leaving it up to line managers artificially limits what’s available to employees, while encouraging employees to pursue their own professional growth makes it easy to cast blame when the process fails. Not surprisingly, nearly two-thirds of people who change jobs say lack of advancement is a key reason why.

MIT Sloan senior lecturerGeorge Westerman and MIT Sloan Management Review editor-in-chief Abbie Lundberg offer three recommendations to improve professional development so all employees can create their own career paths without looking outside the organization.

1. Make career pathways visible. Managers, and even human resources personnel, often fail to see opportunities outside an employee’s current business unit. Even if they do, they also often fail to provide clarity on how employees can bridge the skill gaps that would let them move into a different role.
2. Provide a place to practice. Employees need to be able to learn new skills. Digital learning platforms help, as do learning pathways customized to an employee’s needs. Stretch assignments and rotational programs offer hands-on learning and work best when managers can match workers to the tasks they want to learn.
3. Deliver feedback and coaching. Asking managers to simply document career discussions isn’t enough; nor is saving these conversations for annual reviews. Employees need frequent feedback, both from HR and their managers. HR, meanwhile, should recognize that managers may need coaching to provide more valuable feedback.

Read: Why companies should help every employee chart a career path