Warning: my company's chatbot started hallucinating contract clauses last month

We launched an AI customer service bot back in February and it seemed fine at first. Last week during a review I found it was making up refund policies that didn't exist, like offering free replacements after 90 days. Our legal team had to shut it down completely while we retrained it on the actual terms. Has anyone else caught their bot inventing rules that could cost them real money?

3 comments

3 Comments

anthonynelson2d ago

Used to think this stuff was overblown until it happened to us. We had a simple FAQ bot that started promising overnight shipping on items we dont even stock. The worst part was it sounded so confident. Legal basically said we cant turn it back on until we prove it won't happen again. Retraining takes forever because you have to feed it every single edge case. Honestly shook me how fast a useful tool turns into a liability.

felixm292d ago

Yeah, been there @anthonelynson, and it's a nightmare. After we had a similar fiasco, the trick that finally worked was having legal sign off on a hardcoded "can't promise" list that blocks the bot from making any claims about shipping, pricing, or stock. You basically treat the bot like a new hire who talks too much - you gotta lock down the risky topics before you even think about the edge cases.

casey8182d ago

Man, @anthonynelson calling it a "nightmare" is right. My buddy runs a small auto shop and his booking bot started promising free oil changes for life on every car, even the beaters that barely passed inspection.

-1