Why most health apps are biased even before their first lines of code are written

Many of us use digital health tools every day, such as period trackers, fitness apps or online quizzes to check our mental health.

These mobile apps, wearable devices, and diagnostics often based on artificial intelligence (AI), are shaped by the data they are trained on, and the knowledge and assumptions of the people who build them.

But the datasets behind these digital tools often represent a narrow segment of the population. So the resulting health technologies can cause real harm to those who are under-represented.

For example, algorithms used to diagnose skin conditions are often largely trained on light skin. We’ve known for almost a decade this can result in misdiagnosis in people with darker skin.

These injustices do not remain static. A biased AI algorithm may shape how health care is delivered, which may then inform how subsequent algorithmic models are developed. This creates self-reinforcing loops that can worsen health inequities over time.

Our recent article, which involves collaborators from ten countries, shows a better way of developing these digital tools.

If these tools are to be inclusive, equitable and truly valuable, we need to start having certain conversations even before a single line of code is written.

What’s the issue?

We draw together evidence showing digital health tools are mainly shaped by Western, Eurocentric approaches that may overlook diverse understandings of health and wellbeing.

They’re often built on invisible assumptions about what good health and wellbeing means, whose bodies are considered “normal”, and which forms of knowledge matter.

When it comes to shaping these health tools, technology companies, funders and research institutions hold the power. That is, they often influence which problems are prioritised and whose knowledge is recognised as legitimate.

Often these institutions do not consider the broader social and historical contexts that shape people’s health and wellbeing, which can inadvertently cause harm.

For example, an algorithm was used in US hospitals to identify people with complex medical needs so they could be referred to programs to improve their care.

However, researchers found under 18% of the patients the algorithm assigned extra care to were Black. This number should have been over 46%. The algorithm assigned risk scores based on annual health expenditure. However, Black patients tend to spend less on their health care due to a variety of reasons, including lower socioeconomic status and mistrust in the health-care system, partly driven by systemic racism.

So Black patients had to be significantly sicker than white patients to be flagged for extra care.

We outline how marginalised communities are often not consulted at all about new health technologies, or if they are, only after most key decisions have already been made. This includes consultations about whether digital technology is the best solution for the problem at hand.

We show that assumptions about some communities often prevent these communities from meaningfully shaping the technologies that directly affect their care. Assumptions might include “they are not digitally literate” or “they will not use this technology”.

We also highlight the tension between what funders and technology developers want or value (for instance, technological advancement and profit) and what communities value or even need.

Then there’s the issue of who owns the data collected as part of developing these health technologies.

For Indigenous and many other marginalised communities globally, their data may represent a cultural and economic asset connected to identity, sovereignty and collective rights.

Read more: AI has potential to revolutionise health care – but we must first confront the risk of algorithmic bias

What would be better?

Even before a single line of code is written, we can examine Western, Eurocentric assumptions about health and health care, by asking:

• what counts as health?

• whose knowledge is valued?

• which forms of wellbeing are prioritised?

• who gets to participate in shaping these technologies?

This way, community knowledge, often shared across generations orally, and people’s lived experience can meaningfully complement dominant Western-produced information to shape health technologies.

For example, for many Indigenous communities, health may encompass connections between body, mind, spirit, community and environment.

So AI models that include Indigenous communities’ understandings of the natural environment, climate and human health can help discern patterns of risk or resilience that may otherwise go undetected.

When done well, culturally appropriate digital technologies can have impacts beyond a Western understanding of health. For instance, such digital mental health programs for young Indigenous people help deepen cultural identity and honour traditional Indigenous knowledge. They help improve wellbeing and promote resilience.

Innovators also need to actively explore the values of the communities these technologies aim to serve, and make sure the technology’s values align.

An example comes from developing assistive technologies – those that help people with mobility, communication or cognition, for example.

One study of Indigenous people across Canada, Australia and the US showed people were more likely to adopt technologies when they enhanced family and community involvement in their care and nurtured stronger connections with health-care providers. Users preferred tools that promoted inter-dependence, rather than independence.

Communities should also be able to govern how their data are collected, analysed, linked, interpreted, shared and managed, a concept known as data sovereignty.

Why is this important now?

Building digital health differently requires more than creating technologies that are accessible or technically accurate. It requires changing how we think about health, knowledge, power and value.

This shift is more important than ever as digital health technologies, particularly AI, become increasingly embedded in health-care systems and everyday health decisions.

We thank the following co-authors of the paper mentioned in this article: Laura Vokey, Carrie Van Rensburg, Divya Kewalramani, Sipho Dlamini, Esmita Charani, Hasan Ferdous, Leo Anthony Celi, Chikondi Milanzi, and Anna Schneider-Kamp.