Column | When machine enters the courtroom: A case for regulating AI?

Artificial intelligence (AI) has evolved over seven decades through the work of visionary mathematicians, computer scientists, researchers, and engineers. From the 1950s theoretical frameworks positioning computing machines as large-scale calculators to generative AI reshaping industries today, AI is now a daily phenomenon. Moving beyond the experimental stages, AI use in legal and judicial works is concentrated across research, drafting, case filing and management, transcribing hearings, summarising submissions, and flagging risks in sentencing decisions. The debate is not whether AI should be used, but where its role should end, and who bears responsibility for its failures and its going rogue.

There is a compelling case for AI in legal and judicial work. Judges are human. Limited by their mortal capabilities, susceptible to fatigue. Often influenced by hindsight, availability, and sometimes plain unconscious bias. These have a systematic impact on case backlog and an individual’s right to seek timely justice.

AI could unlock solutions to these problems and even surpass human physical limitations in reading and computing more legal norms than legal and judicial members, and contribute to faster, more efficient processes.

AI-assistance in foreign judiciary

Chinese Hangzhou internet courts report AI-assisted case management cutting hearing times by more than 50 per cent, while Estonia automates transcription and redaction through “Salme”, freeing up judicial time for decision-making. Closer to home, the Sikkim High Court became the first fully paperless court, with AI use being tested at district courts for assistive functions.

Against this, a body of evidence cautions counsel. Bias does not disappear when a machine takes over; rather, it hides inside the training data. An offence risk assessment tool used in the USA was found to assign harsher scores to black defendants, while a Dutch welfare-fraud detection system penalised applicants by nationality. Such flawed predictions may have compounding effects including higher policing, denial of rights, and breach of privacy expectations.

The problem is augmented by hallucinations. A language model does not retrieve case law as a legal database does; it generates the statistically probable word in a sequence, and because legal citations follow a predictable format, a fabricated one can look entirely authentic! A 2024 study by Stanford and Yale shows that even legal-specific AI tools have 17 per cent to 34 per cent hallucination margins, underscoring the systemic nature of the risk.

Story continues below this ad

Alongside this, the lack of reasoning transparency due to AI black boxes and dearth of disclosures around design exacerbates the risks. Blind reliance on erroneous AI outputs can cascade into a judicial due process concern, undermining the entire adversarial process. In civil cases, this affects credibility and liability outcomes for litigants. In criminal cases, the impact of manipulated synthetic content could be devastating, where a piece of evidence seeming authentic may be premised on plausibility.

Globally, courts, including in the United States, the United Kingdom, Canada and the European Union, have had to respond to the fallout. A tracker of such incidents has recorded more than 1,725 penalties worldwide as of mid-2026, with fines reaching 110,000 US dollars in a single case. India has voiced its aspirations, with the Supreme Court demanding a “zero-tolerance” approach to bad outputs.

Closer to home, on July 2, 2026, while setting aside NCLT’s judgment in Pooja Ramesh Singh vs Jammu and Kashmir Bank Ltd. and Anr., the Indian Supreme Court equated the problem to a noxious injection into the justice delivery system that remains “invisible”, “insidious”, and “catastrophic” for a very long time.

A month earlier, the Supreme Court had released the draft regulations for the use of AI in courts for consultation, premised on principles of human primacy, transparency, accountability, data protection, and judicial autonomy, proposing prohibition on the use of AI tools in the judicial decision-making process and limiting it to assistive functions only.

Story continues below this ad

AI models going rogue

Recent instances of foundational AI models going rogue and bypassing containment constraints in a less-than-robust sandbox are a wake-up call. While AI is very attractive with India’s bludgeoning docket of pendency of over 60 million cases, it must not be unfettered but subjected to safeguards and checks around credibility, reliability, and fit-for-use evaluations.

Faults in AI use in legal work are no longer speculative. Courts and regulators are moving beyond reprimands to sanctions, disbarment, and revised conduct codes. Every citation warrants a human check against the primary source, and each AI-assisted output warrants scrutiny proportionate to what is at stake. Every institution introducing these tools into a courtroom owes the public a clear account of what the machine decided, what a person decided, and why the two were kept separate.

The need is for a nimble mechanism to make and review regularly “informed” choices about the use of AI, accounting for the stakes at hand, the nature of disputes (administrative, commercial, civil, criminal, or impacting individual liberties), and whether the use is likely to move beyond assistive to actual decision-making domain assumes critical importance.

With an emergent need to fast-track deployment of nimble frameworks, legal and judicial members should envisage higher thresholds of diligence and duty of care that align with preserving judicial independence and due process rules, building and enforcing technical, organisational, and human guardrails against rogue outcomes, and constant vigilance and review mechanisms to course-correct in a timely.

Story continues below this ad

Practitioners must understand the use cases enabled by AI tools and remain abreast of their limitations. On the other hand, law societies, associations, and law schools stand to benefit from dedicated upskilling aimed at equipping AI-enabled lawyers. It would be in order and timely for legal professional codes to be suitably upgraded with permissible AI use cases, accountability and transparency matrices, and associated consequences to deter over-reliance and delegation of a lawyer’s immutable duties to a machine.

As we move forward to assimilate the new technology, we must draw inspiration from the wise words of Krishna Iyer J, written 50 years ago, “…sociology-cultural changes are the sources of the new values, and sloughing off old legal thought is part of the process of the new equity-loaded legality… the rule of law enshrined in our Constitution must and does reckon with the roaring current of change which shifts our social values and shrivels our deferral roots, invaded our lives and fashions our destiny.”

Amit Kapur is Senior Partner (Head – Northern Region and Chair – Disputes) and Arya Tripathi is Partner at Cyril Amarchand Mangaldas

Written by eminent law professionals, the Legal Minds Column aims to break down complex statutory frameworks into insights, both for members of the fraternity and the general public.

Story continues below this ad

For feedback and pitches, write to us at expresslegal@indianexpress.com