Navigating the Supreme Court’s Crackdown on AI-Generated "Hallucinations"

The integration of Artificial Intelligence (AI) into legal research has moved from a futuristic novelty to an everyday necessity. However, the convenience of Large Language Models (LLMs) has recently collided with the rigid sanctity of Indian jurisprudence. In a move that signals a paradigm shift in legal ethics, the Supreme Court of India has urged the Bar Council of India (BCI) to constitute an expert panel to address the rising "menace" of AI-generated fake judgments being cited in courtrooms.

This directive arises not from a fear of technology, but from a necessity to protect the stare decisis doctrine from "phantom precedents"—citations that look authoritative but are entirely synthetic fabrications of AI.

1. Professional Misconduct vs. Bona Fide Error: Does the citation of an AI-generated fake case qualify as a simple research error or professional misconduct?
2. Duty of Verification: To what extent can an advocate delegate the "duty to the court" to an automated tool?
3. Judicial Integrity: The impact of "synthetic jurisprudence" on the reliability of the adjudicatory process, especially when trial courts rely on unverified AI outputs.

The Legal Analysis: From "Co-Pilot" to "Misconduct"

The Ratio Decidendi of Judicial Concern

The Supreme Court, in the recent matter of Gummadi Usha Rani v. Sure Mallikarjuna Rao (2026), took a stern view of a trial court in Andhra Pradesh that relied on four non-existent judgments to dismiss objections in a property dispute. The Bench, comprising Justices P.S. Narasimha and Alok Aradhe, laid down a critical distinction:

"A decision based on such non-existent and fake alleged judgments is not an error in the decision-making. It would be a misconduct and legal consequence shall follow."

This observation elevates the failure to verify AI output from a negligent mistake to a breach of professional ethics. The court emphasizes that while AI can be a "co-pilot," the advocate remains the "pilot" responsible for every word filed.

The Hallucination Trap

The phenomenon of "AI Hallucination"—where LLMs like ChatGPT provide plausible-sounding but factually incorrect citations—has reached the highest echelons of the Bar. In a separate instance, a petition before the Supreme Court cited a fictional case titled "Mercy vs. Mankind", which existed in no database other than the probabilistic output of an AI tool. Similarly, the Bombay High Court in Jyoti w/o Dinesh Tulsiani v. Elegant Associates (2025) imposed a fine of ₹50,000 on a litigant for "dumping" unverified AI-generated material that wasted judicial time.

The BCI’s New Mandate

By urging the BCI to form an expert panel, the Apex Court is paving the way for a formal regulatory framework. This panel is expected to examine:

• Standard Operating Procedures (SOPs) for using AI in drafting.
• Mandatory disclosure requirements when AI is used for substantive research.
• Development of a "Sovereign Legal Database" to reduce reliance on open-source LLMs that lack access to verified Indian case laws.

Practical Takeaways for Practitioners

• Verification is Non-Negotiable: Every citation must be cross-referenced with authorized law reporters (e.g., SCC, AIR, or the High Court’s official website). "AI said so" is no longer a valid defense against a show-cause notice for misconduct.
• The "Human-in-the-Loop" Requirement: AI should be used for structuring and summarizing, but the Ratio Decidendi and specific paragraph numbers must be manually verified.
• Beware of LLM Limitations: Most generic LLMs are trained on broad internet data and do not have real-time access to the most recent SCC or Manupatra updates, leading to the "invention" of cases to satisfy user prompts.
• Ethical Disclosure: Moving forward, lawyers may be required to certify that all citations have been manually verified, similar to practices emerging in the US and UK.

The Supreme Court’s intervention is a timely "Caveat Advocatus" (Lawyer Beware). As we move toward a digital judiciary, the burden of ensuring that technology aids—rather than subverts—justice falls squarely on the shoulders of the Bar. The forthcoming BCI expert panel report will likely be the first step toward a code of conduct for "Legal AI" in India.