Skip to content
Search

Latest Stories

Experts concerned as India rolls out AI tools without safeguards

Tech experts fear the lack of an official policy for the ethical use of AI will hurt people at the bottom, entrenching age-old bias, criminalising minorities and channeling most benefits to the rich

Experts concerned as India rolls out AI tools without safeguards

AFTER communal clashes in Delhi’s Jahangirpuri area last year, police said they used facial recognition technology to identify and arrest dozens of men, the second such instance after a more violent riot in the Indian capital in 2020.

In both cases, most of those charged were Muslims, leading human rights groups and tech experts to criticise India’s use of the AI-based technology to target poor, minority and marginalised groups in Delhi and elsewhere in the country.


As India rolls out AI tools that authorities say will increase efficiency and improve access, tech experts fear the lack of an official policy for the ethical use of AI will hurt people at the bottom, entrenching age-old bias, criminalising minorities and channeling most benefits to the rich.

“It is going to directly affect the people living on the fringes - the Dalits, the Muslims, the trans people. It will exacerbate bias and discrimination against them,” said Shivangi Narayan, a researcher who has studied predictive policing in Delhi.

With a population of 1.4 billion powering the world’s fifth-biggest economy, India is undergoing breakneck technological change, rolling out AI-based systems – in spheres from health to education, agriculture to criminal justice – but with scant debate on their ethical implications, experts say.

In a nation beset by old and deep divisions, be it of class, religion, gender or wealth, researchers like Narayan – a member of the Algorithmic Governance Research Network – fear that AI risks exacerbating all these schisms.

“We think technology works objectively. But the databases being used to train AI systems are biased against caste, gender, religion, even location of residence, so they will exacerbate bias and discrimination against them,” she said.

Facial recognition technology - which uses AI to match live images against a database of cached faces - is one of many AI applications that critics say risks more surveillance of Muslims, lower-caste Dalits, indigenous Adivasis, transgender and other marginalised groups, all while ignoring their needs.

Linking databases to a national ID system and a growing use of AI for loan approvals, hiring and background checks can slam doors firmly shut on the marginalised, said Siva Mathiyazhagan, an assistant professor at the University of Pennsylvania. The growing popularity of generative AI applications such as chatbots further exacerbates these biases, he said.

“If you ask a chatbot the names of 20 Indian doctors and professors, the suggestions are generally Hindu dominantcaste surnames - just one example of how unequal representations in data lead to caste-biased outcomes of generative AI systems,” Mathiyazhagan told the Thomson Reuters Foundation.

Caste discrimination was outlawed in India 75 years ago, yet Dalits still face widespread abuse, many of their attempts at upward mobility met with violent oppression. Under-represented in higher education and good jobs despite affirmative action programmes, Dalits, Muslims and indigenous people lag higher-caste Indians in smartphone ownership and social media use, studies show.

About half of India’s population – primarily women, rural communities and Adivasis – lacks access to the internet, so “entire communities may be missing or misrepresented in datasets... leading to wrong conclusions and residual unfairness,” an analysis by Google Research showed in 2021.

The ramificiations are widespread; not least, in healthcare.

“Rich people problems like cardiac disease and cancer, not poor people’s tuberculosis, is prioritised, exacerbating inequities among those who benefit from AI and those who do not,” researchers said in the Google analysis.

LEAD INSET AI GettyImages 1240101389 Police used AI tools to arrest dozens of men following communal clashes in New Delhi and most of them were Muslims

Similarly, mobile safety apps that use data mapping to flag unsafe areas are skewed by middle-class users who tend to mark Dalit, Muslim and slum areas as dodgy, potentially leading to over-policing and unwarranted mass surveillance.

“The irony is that people who are not counted in these datasets are still subject to these data-driven systems which reproduce bias and discrimination,” said Urvashi Aneja, founding director of Digital Futures Lab, a research collective.

India’s criminal databases are particularly problematic, as Muslims, Dalits and indigenous people are arrested, charged and incarcerated at higher rates than others, official data show.

The police registers are used for potential AI-assisted predictive policing to identify who is likely to commit a crime. Generative AI may come to court, with the Punjab and Haryana high court earlier using ChatGPT to decide whether to award bail for a suspect in a murder case - a first in the country.

“Any new AI-based predictive policing system will likely only perpetuate the legacies of caste discrimination and the unjust criminalisation and surveillance of marginalised communities,” said Nikita Sonavane, co-founder of the Criminal Justice and Police Accountability Project, a non-profit.

“Policing has always been casteist in India, and data has been used to entrench caste-based hierarchies. What we’re seeing now is the creation and rise of a digital caste panopticon.”

The ministry of information technology did not respond to a request for comment.

Governments worldwide have been slow to regulate AI. China’s draft rules for generative AI took effect last month, while the EU’s AI Act is in the final stage of negotiations, and the U.S. AI Bill of Rights offers guidelines for responsible design and use.

India does not have an AI law, only a strategy from government thinktank NITI Aayog that states that AI systems must not discriminate on the basis of religion, race, caste, sex, descent, place of birth or residence, and that they must be audited to ensure they are impartial and free from bias. But there is little discussion in India about bias in AI, even as there is growing awareness of caste in the tech industry in the United States, with California poised to become the first state to ban caste discrimination, after Seattle became the first U.S. city to do so.

South Asian immigrant communities make up large numbers of tech workers in the United States, where Dalit engineers - including women - have complained of discrimination and abuse from high-caste men.

Having mostly high-caste men design AI tools can unduly benefit the privileged and altogether bypass women, lower caste and other marginalised groups, said Aneja. “How much agency do women or lower-caste groups have to check or contradict what’s coming out of a system? Especially generative AI, which is designed to seem human-like,” she said.

“A technical fix cannot take existing bias out of the system; what’s needed is a better understanding of the biases and their impacts in different social contexts.

“We should shed the assumption that bias is going to go away - instead, we should accept that bias is always going to be there, and design and build systems accordingly,” she said.

More For You

Starmer

Starmer, who has faced negative coverage since taking office in July 2024, defended the appointment process.

Reuters

Starmer: I would not have appointed Mandelson if aware of Epstein ties

PRIME MINISTER Keir Starmer said on Monday he would not have appointed Peter Mandelson as UK ambassador to Washington had he known the extent of his links with convicted sex offender Jeffrey Epstein.

This was Starmer’s first public statement since dismissing Mandelson last week. The prime minister is facing questions over his judgement, including from Labour MPs, after initially standing by Mandelson before removing him from the post.

Keep ReadingShow less
Trump

Trump said the suspect had been arrested earlier for 'terrible crimes,' including child sex abuse, grand theft auto and false imprisonment, but was released under the Biden administration because Cuba refused to take him back.

Getty Images

Trump says accused in Dallas motel beheading will face first-degree murder charge

US PRESIDENT Donald Trump has described Chandra Mouli “Bob” Nagamallaiah, the Indian-origin motel manager killed in Dallas, as a “well-respected person” and said the accused will face a first-degree murder charge.

Nagamallaiah, 50, was killed last week at the Downtown Suites motel by co-worker Yordanis Cobos-Martinez, a 37-year-old undocumented Cuban immigrant with a criminal history.

Keep ReadingShow less
Starmer Mandelson

Starmer talks with Mandelson during a welcome reception at the ambassador's residence on February 26, 2025 in Washington, DC.

Getty

Starmer under pressure from party MPs after Mandelson dismissal

PRIME MINISTER Keir Starmer is facing questions within the Labour party after the sacking of US ambassador Peter Mandelson.

Mandelson was removed last week after Bloomberg published emails showing messages of support he sent following Jeffrey Epstein’s conviction for sex offences. The dismissal comes just ahead of US president Donald Trump’s state visit.

Keep ReadingShow less
Nepal’s new leader pledges to act on Gen Z calls to end corruption

Officials greet newly-elected Prime Minister of Nepal's interim government Sushila Karki (R) as she arrives at the prime minister's office in Kathmandu on September 14, 2025. (Photo by PRABIN RANABHAT/AFP via Getty Images)

Nepal’s new leader pledges to act on Gen Z calls to end corruption

NEPAL’s new interim prime minister Sushila Karki on Sunday (14) pledged to act on protesters’ calls to end corruption and restore trust in government, as the country struggles with the aftermath of its worst political unrest in decades.

“We have to work according to the thinking of the Gen Z generation,” Karki said in her first address to the nation since taking office on Friday (12). “What this group is demanding is the end of corruption, good governance and economic equality. We will not stay here more than six months in any situation. We will complete our responsibilities and hand over to the next parliament and ministers.”

Keep ReadingShow less
UK secures £1.25bn US investment ahead of Trump’s visit

US president Donald Trump and UK prime minister Sir Keir Starmer arrive at Trump International Golf Links on July 28, 2025 in Balmedie, Scotland. (Photo by Jane Barlow-WPA Pool/Getty Images)

UK secures £1.25bn US investment ahead of Trump’s visit

THE British government has announced over £1.25 billion ($1.69bn) in fresh investment from major US financial firms, including PayPal, Bank of America, Citigroup and S&P Global, ahead of a state visit by president Donald Trump.

The investment is expected to create 1,800 jobs across London, Edinburgh, Belfast and Manchester, and deepen transatlantic financial ties, the Department for Business and Trade said.

Keep ReadingShow less