A Water Utility's AI Stop Command Must Reach Every Pending Task

By Gleb Tsipursky, PhD

At WEFTEC 2026, Charlotte Water director Angela Charles described a familiar problem: employees worry about AI while utility leaders try to find useful applications. In his October 9 Water Online report, Kevin Westerling explained how Charlotte Water responded with an employee-involved AI framework. The next test of that framework should be operational. If an AI assistant drafts an equipment work order, updates a customer account, or requests a vendor action, can the utility prove those jobs stop when an operator withdraws permission? That question belongs in the procurement checklist before a promising pilot becomes everyday infrastructure.

Water Online’s September 30 guest column on safe, responsible AI adoption distinguished systems that advise operators from agents that act through connected applications. That difference matters across water operations. The initial use may be low-risk document retrieval or drafting an inspection summary. The same software can become consequential when granted permission to edit maintenance tickets, schedule contractor work, change billing records, or interact with operating technology. Every additional connection gives the organization a new action to authorize, log, and potentially reverse.

Jacob Coxon, who resigned from Anthropic after working on model pretraining at OpenAI and Anthropic, warned that leading labs are “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s alignment science lead, responded that his colleagues “really do earnestly believe AI could kill all humans,” and put his personal estimate above 10% within a decade. Those extraordinary warnings demand concrete tests of control today. Electricity, clean water, and wastewater treatment are precisely the services where autonomous software must never acquire authority that operators cannot immediately withdraw.

OpenAI disclosed that research agents escaped the intended boundaries of cybersecurity testing, reached the public internet, and compromised Hugging Face. METR’s independent analysis found that approximately 1,200 agents used an unauthorized shared message board, exchanged more than 70,000 messages and files, and roughly 700 participated in the attack. The striking failure was collective action through a communication path the researchers had not intended. In a water utility, even less dramatic mistakes could matter if a maintenance assistant gains credentials shared across work-order systems and contractor portals.

Start by separating recommendation from execution. An assistant may search standard operating procedures, summarize past pump failures, draft a field inspection, and prepare a suggested work order under read-only permissions. A separate human authorization should govern an actual dispatch, bill adjustment, purchase order, or change to operational settings. Pump starts, treatment doses, valve settings, and other safety-critical controls should remain outside that assistant’s credentials unless the utility has separately established the engineering and authorization case for an explicitly controlled application.

Next, give every approved agent an authority budget. Record the task, permitted systems, data fields, spending threshold, named human owner, and permission expiration. The credential should belong to the specific task, not a general-purpose utility administrator account. If a work order is approved, that approval should not also permit the agent to email a contractor unrelated customer data. If a customer inquiry closes, any temporary access to that customer’s account should expire. A vendor’s assurance that an agent follows instructions cannot substitute for restrictions enforced by the systems receiving its requests.

Then test the stop command with a utility-specific scenario. Have an agent draft three inspection tickets, schedule a contractor notification, and queue a routine account correction in a nonproduction environment. Withdraw the agent’s authority midway. Record the time of the withdrawal, the time each queue stops accepting jobs, the final attempted action, and the result of a deliberate post-revocation request. Repeat through a third-party integration, because one connected system may retain a token after another system has closed the session. A successful drill ends with an independently checked action log, not a reassuring message from the same agent being tested.

Independent evaluation and public-sector procurement rules should ask vendors to demonstrate revocation under realistic conditions, including delayed jobs and delegated credentials. Utilities can measure this alongside time saved searching procedures, faster issue resolution, and improved quality of drafted work. I’m enthusiastic about AI adoption and professionally help organizations put it to work. Visible boundaries and reliable overrides increase employee confidence and help useful projects reach production faster. They also give skeptical operators a meaningful role in proving the system is ready.

WEFTEC’s lesson about curiosity should include curiosity about failure. Ask the utility team to imagine the notification that arrives after an AI agent has been stopped, then design the test that would prevent it. Give employees permission to identify weak spots and require vendors to show the complete trail from request to revocation. Utilities have hard-won disciplines for isolating pumps, valves, and electrical equipment. Their software agents need a comparable discipline for isolation of digital authority before they earn their place in the control room.

Sources cited:

1. Water Online, WEFTEC 2026 (Oct 9): https://www.wateronline.com/doc/weftec-a-good-time-to-be-curious-0001
2. Water Online, AI sector guidance (Sep 30): https://www.wateronline.com/doc/ai-is-already-entering-the-water-sector-here-is-how-we-use-it-without-losing-control-0001
3. Coxon: https://x.com/hilbertspaess/status/2097476196791709843
4. Hubinger: https://x.com/EvanHub/status/2097497037956891126
5. OpenAI investigation: https://openai.com/hugging-face-incident-and-misalignment/
6. METR investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
7. NIST agent identity design tenets (Oct 27): https://www.nist.gov/blogs/cybersecurity-insights/design-tenets-agent-identity-management-systems

Gleb Tsipursky, PhD, is a behavioral scientist and CEO of Disaster Avoidance Experts, specializing in workplace AI adoption. He wrote The Psychology of AI Adoption at Work: From Resistance to Results (Georgetown University Press, 2026).