MIT Policy Hackathon, co-sponsored by HSI, Addressed Health Equity

Participants in the MIT Policy Hackathon, organized and run by students from MIT’s Institute for Data, Systems, and Society and MIT’s Technology and Policy Program, are asked to address relevant societal challenges using both data and policy analyses. Over a weekend in October 2024, students from multiple programs at MIT and other local universities took on the real-world problems posed by the hackathon’s government and industry partners. Participants could choose from among four challenges; there was one each in health, climate, housing, and criminal justice. They then formed teams, devised a solution, and wrote a presentation and a policy paper over a day and a half. The final day was reserved for presentations and Q&A in front of a panel of judges who are familiar with the industry. 

As a sponsor of the event, HSI was asked to recommend judges for the health challenge; HSI was specifically asked for people with health industry experience who are not academics. Director Anne Quaadgras asked the Director of Communications, Lisa Abrams  (Sloan MBA ’99, Hopkins MPH '13), and Joanna Brownstein (Sloan MBA ’08), who had spoke at an HSI lunchtime seminar earlier this year, to participate. The four-person judging panel for the health challenge was composed of these two, a post-doc who was familiar with the challenge topic, and the Duke University professor who posed the challenge.

The Health Challenge

Dr. A. Ian Wong, Professor of Medicine, Duke University, was the outside collaborator for the health team. Dr. Wong spoke about his research into pulse oximetry and provided some data to the four teams addressing the health equity topic. He first described the subject of the challenge, pulse oximeters.  Pulse oximetry may not have been well known before COVID, but most everybody who has been to a clinic or ED in the past few years has had the white clip attached to their finger to measure the level of oxygen in their blood. If the reading is within a specific range, crucial treatment protocols are put into effect. Generally, a pulse oximeter is a non-invasive way to measure a critical medical level. 

The equity issue is that these devices do not reflect accurate readings for everybody. Melanin in someone’s skin interferes with the device’s light absorption and reflection properties, resulting in readings that overestimate the amount of oxygen. The end result may well be that the oximeter reports a normal reading when, in fact, the patient’s level is low and treatment is badly needed. 

Dr. Wong challenged the hackathon teams to come up with a solution to this problem and provided materials, including data on the FDA’s current metric for measuring oximeter accuracy, Accuracy Root Mean Square (ARMS). Of the four teams, the judges elected two finalists. Having a judging panel composed of people with diverse backgrounds made the wide-ranging judging conversations lively. Academically-inclined judges understood the science and statistical robustness behind the proposals in detail, while the others raised several concerns regarding whether the proposed solutions would have the intended benefit in the clinical environment.

Team One

This group’s starting point was the FDA’s current metric for oximeter accuracy, ARMS. They proposed changes to its calculation to take into account varying skin tones; they would test that new calculation both in the clinical setting and in future clinical trials of oximeter performance. They concluded that this “approach builds upon existing processes, and enhances patient outcomes without making onerous changes to the current regulatory framework for pulse oximeter evaluation.” By changing how ARMS is calculated, they proposed, oximeters would give accurate readings for people of any skin tone.

What was lacking in this team’s submission was an understanding of the clinical environment. Specifically, the judges were troubled by their not mentioning the workflow interruptions that would be necessary to collect the data their solution required. One of the judges with experience in the clinical setting had particular difficulty with the team’s solution hinging on giving nurses more work. The team also did not appear to understand how change happens in the clinical environment. They believed that changes could be made “immediately” and the benefits would accrue as soon as the changes were implemented.

Team Five

This team presented an excellent summary of the issues from a variety of perspectives. Their policy paper was clear and professionally organized. They made a point of comparing their proposed new oximetry policy to previous government policies that were successfully implemented. They also used the data to create a tool on GitHub that enabled the user to understand the disparity in more detail. This team was also the only one that made realistic implementation suggestions to reduce disruptions in the clinical environment.  

However, they did not manipulate the data for their solution. Instead, they proposed some ways of checking and comparing the oximeter data to other available data to come up with, they hoped, more accurate readings and appropriate treatment. Unfortunately, Dr. Wong pointed out, some of their work relied on data and classification systems that were known to be problematic, and on processes that were unlikely to be effective. 

Judges’ Decision

The judging team members were torn between team one that came up with an innovation on the data, and team five that had a better understanding of the many stakeholders and issues involved in both the problem and in deploying a change.

Team one proposed an innovative metric based on the data, but their explanation and policy paper were aimed at an academic audience, and the assignment was to write a policy paper the was directed to a mixed audience.  Also, the judges faulted team one for their misunderstanding of how healthcare actually works “in the wild”. Team five’s presentation and work was more well-rounded and appropriate to a non-medical/academic audience. However, their solution was not innovative and much less likely to be effective.

There could not be two winners and, in the end, the judges awarded the win to team one. In announcing the results, the judges cautioned team one that they needed to consider the clinical environment to truly make a difference. Innovation that only works in the lab is not going to move the needle on healthcare. The judges praised team five for their work and pointed out the aspects they managed that the winning team did not, but ultimately, they did not come up with an effective change to the FDA metric.

Conclusion

Current oximeter designs fail to account for melanin’s effect on light transmission thus resulting in inaccurate readings that miss low oxygen levels in patients with darker skin tones. This built-in defect results in these patients not getting the life-saving treatment they need. 

The hackathon made clear some of the difficulties from design to implementation of a solution that will make a true clinical difference. 

Dr. Wong has written extensively on this issue. See articles in Plos One and JAMA, for example. He has also won an award from Duke University’s Center to Advance Healthcare Equity for his project: Bringing equity to inequitable pulse oximetry with EHR data. 

In February 2024, the FDA published a study Performance Evaluation of Pulse Oximeters Taking into Consideration Skin Pigmentation, Race and Ethnicity and the first FDA-approved oximeter was manufactured and marketed by Masimo. 

This work is welcome, although it has come too late to help patients during the Covid pandemic. However as one immunologist remarked, with climate change, we’re likely to see another respiratory pandemic and this advance can make a difference between life and death for a large number of patients.