NYC’s AI Chatbot Draws Ire for Dispensing Misleading Advice, City Refuses to Pull It Down

New York City’s endeavor to aid small business owners through an AI-powered chatbot has backfired as the tool dispenses erroneous guidance, including advice that contradicts local policies and encourages legal violations. Despite mounting criticism and acknowledgments of inaccuracies by Mayor Eric Adams, the city has opted to retain the chatbot on its official website.
Launched as a comprehensive resource for business owners in October, the chatbot generates algorithmic responses to queries regarding navigating the city’s bureaucratic landscape. While featuring a disclaimer about potential inaccuracies and a strengthened disclaimer that it doesn’t offer legal advice, the tool continues to furnish false information, prompting concerns about the lack of oversight in government adoption of AI technology.
Experts highlight the dangers of deploying unproven AI systems without adequate safeguards, emphasizing the need for accountability and responsible implementation. The chatbot’s responses, including misinformation on employment rights, waste management, and absurd scenarios, underscore the risks associated with relying on flawed AI models.
Microsoft, the provider of the bot’s Azure AI services, asserts collaboration with city officials to enhance accuracy based on official documentation. Mayor Adams defends the tool, attributing its shortcomings to the iterative nature of technological advancement, while critics deem such an approach reckless.
The incident reflects broader concerns about the reliability of large language models and the consequences of their deployment by both public and private sectors. Recent instances of misleading advice from chatbots, including Air Canada’s refund policy misrepresentation, underscore the urgency of addressing AI-related risks.
Experts advocate for prudent use of chatbots in public services, emphasizing the importance of content curation and accountability. Los Angeles’ approach, which avoids reliance on large language models, contrasts with New York’s experience, serving as a cautionary example for cities contemplating similar initiatives.
As cities weigh the adoption of chatbots, the imperative lies in understanding their purpose and ensuring accountability to mitigate potential harm and maximize their utility as public service tools.

Leave a Reply

Your email address will not be published. Required fields are marked *