AI could turn China’s social credit system into surveillance tool: government-funded study

China’s social credit system, stereotypically imagined as an all-powerful numerical scoring project for authoritarian control, has gained notoriety in the West – although people inside the country actually feel little of its presence or practical influence.

In a government-funded study, however, a Chinese legal scholar warned that deploying artificial intelligence (AI) risked turning the rating system into a “large-scale surveillance tool”. The paper was published in the Journal of Sichuan Normal University last month.

Author Lin Jianxing noted that social rating, which uses big data and algorithms to translate citizens’ activities into quantifiable metrics, could “strengthen the risk assessment and automated regulatory capabilities of government agencies”.

Lin, a doctoral candidate at Sichuan University’s Law School, expressed concern that such algorithmic administration, typified by risk assessment, “could trigger issues such as opacity, large-scale surveillance and discrimination”.

The research was funded by a project on empirical studies of social governance models and institutional innovation in Sichuan province.

According to Lin’s definition, social rating refers to administrative bodies using big data mining and predictive algorithms to convert people’s online and offline activities into quantifiable scores.

This aims to predict an individual’s potential actions and risks, thereby making important decisions regarding that person.

The researcher pointed out that while a high or low score might not appear to directly affect an individual’s rights and obligations on the surface, in reality, it influences the treatment they receive in areas such as healthcare, financial lending and government services.

02:30

China’s ‘social credit system’ explained

The “social rating system” examined in the study falls under the broader framework of China’s better-known – and controversial – social credit system.

These can be categorised into four types, including blacklists for serious legal violations and a financial credit reporting system, according to the article.

Another category involves municipal government-run systems that collect behavioural data and combine rewards with punishments – echoing the mechanisms discussed by Lin.

Beijing has been advancing the application of AI in social governance.

An action plan released by the State Council last year called for the safe, steady and orderly advancement of AI in government sectors, aiming to create a new model of public service that “accurately identifies needs, proactively plans services and offers fully intelligent processing”.

For instance, the central province of Anhui has developed a dynamic social assistance monitoring platform.

By building an information database of low-income populations and promoting data sharing among various government departments, it aims to boost social services.

However, Lin warned that adopting AI within the credit system could entail certain risks. One risk is the reliability of the algorithms, which could be affected by factors such as the quality of collected individual data and model design.

In the absence of strict data compliance regulations and oversight mechanisms, using the diverse data footprints left by citizens to create precise profiles could cause the mechanism to “evolve from a risk management tool into a means of behavioural surveillance”, he said.

The researcher pointed out that the rating system lays the foundation for the government to accurately identify citizens’ needs, predict risks and allocate administrative resources.

Yet Lin noted that effective operation relied heavily on two factors: the quality of the data and scoring algorithms, and the standardisation of administrative organs’ behaviour.

China’s social credit system has often been portrayed as a uniform, frightening tool, where ordinary people’s behaviour is watched and rated in ways that affect all aspects of their lives.

But the Stanford article challenged these narratives.

It pointed out that only a small fraction of cities in China had such local systems, and the framework rewarded and surveilled government employees more intensely than ordinary citizens, reflecting the state’s effort to “discipline both the public and its own agents”.