AI May Well Serve As Training Wheels But Entrusting It With Pilot's Seat Would Be Both Imprudent And Dangerous: Bombay High Court

MUMBAI: Emphasizing that reliance on fake or hallucinated Artificial Intelligence (AI) precedents vitiates the decision-making process, the Bombay High Court on October 30, 2026, set aside a customs adjudication order passed without independent application of mind. A Division Bench comprising Justice M. S. Karnik and Justice Sandesh D. Patil held that while AI can serve as an assistive tool to expedite decision-making, it can never substitute human adjudication in quasi-judicial or judicial proceedings under the Customs Act, 1962.

The petitioner, Quanta Logistics Pvt. Ltd., approached the High Court under Article 226 of the Constitution of India challenging an Order-in-Original dated August 11, 2026, passed by the Commissioner of Customs (Import-I). The petitioner asserted that the adjudicating authority had resorted to generative AI tools without applying its own mind, citing two entirely non-existent judgments attributed to the Supreme Court of India. When the petitioner sought copies of the purported decisions through written communication, the department failed to provide them because no such judgments existed in any legal database or law report.

Whether an adjudicatory order passed by placing reliance on hallucinated and fictitious precedents generated by Artificial Intelligence is sustainable in law.

Whether the entry of unverified, AI-generated material into the adjudicatory process violates the integrity of decision-making and renders the resulting determination void.

A Division Bench of Justice M. S. Karnik and Justice Sandesh D. Patil observed that judicial and quasi-judicial determination requires genuine, independent application of mind. The Court took note of the petitioner's submission that the two Supreme Court judgments cited in the impugned order were pure hallucinations of an AI system and did not exist in any legal compendium or database.

Zero tolerance must be maintained against unverified AI precedents

The Bench referred to the authoritative principles laid down by the Supreme Court in Vijay Ghanshyam Gadiya v. Union of India & Anr., which had extensively reiterated the landmark ruling in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd. dealing with the hazards of deploying unverified AI in judgments.

The Court highlighted that both the Bar and the Bench are under a strict obligation to verify legal authorities, noting the Supreme Court's warning that citing fabricated material strikes at the root of judicial integrity:

"It is necessary for courts to adopt a zero-tolerance mode for producing, citing or using artificial intelligence-generated precedents without verification. It is a misconduct on the part of an advocate to cite such judgments without verification. Equally, it is a serious lapse if a judge relies on such a fake or hallucinated artificial intelligence-generated material as precedents in support of the determination."

Hallucinated legal citations render adjudicatory determinations void

Reiterating the Supreme Court's mandate, the High Court observed that any decision contaminated by hallucinated citations loses its legal standing entirely. The Bench underscored that the integrity of the adjudicatory process cannot tolerate even minimal reliance on fabricated jurisprudence:

"We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication."

AI may serve as training wheels but cannot occupy the adjudicator's pilot seat

Addressing the broader implications of technological adoption, the Bench clarified that although AI holds utility as an assistive mechanism to streamline administrative and judicial workflows, it cannot usurp the adjudicator's role. Human judgment and scrutiny must remain non-negotiable throughout the adjudicatory exercise:

"Notwithstanding that the regulations are yet to be finalised, use of AI as an assistive tool to speed up the decision-making process cannot be denied. However, a note of caution must be sounded: assistance can never be substituted for adjudication. AI may well serve as training wheels but entrusting it with the pilot's seat would be both imprudent and dangerous."

Reliance on dubious AI-generated citations fatal to administrative orders

The Court observed that the use of AI resulting in reliance on dubious material is fatal to the sustainability of an order. Conceding to the patent flaw in the adjudication process, the counsel for the Revenue fairly agreed that the impugned order could not be sustained and submitted that the matter be remanded for de novo consideration.

In view of the consensus and established legal principles, the High Court quashed and set aside the Order-in-Original dated August 11, 2026. The matter was remanded to the Commissioner of Customs (Import-I) for fresh adjudication on merits, with directions to conclude the proceedings within eight weeks.

Date of Decision: October 30, 2026