“Let AI summarise your support tickets” sounds fine right up until the tickets carry real stakes, the kind of queue where a wrong detail isn’t just an inconvenience (think healthcare, financial services, legal, anything regulated or safety-adjacent), and where a team will act on whatever the summary says. At that point “sounds fine” isn’t good enough. You need to know exactly what the model will and won’t do, every time, with no exceptions for a bad day.
I built this for a client whose support team fields both general enquiries and a smaller queue of higher-stakes requests through their helpdesk platform (Intercom and Zendesk are the two I see most often). When a ticket gets routed to that higher-stakes queue, a workflow automatically pulls the full conversation, summarises it, and posts the summary as an internal note before a human even opens the thread. The point isn’t to replace anyone’s judgement. It’s to stop them opening a wall of back-and-forth cold and having to piece together what actually happened.
Here’s what makes that safe to run unattended, because none of it is the interesting bit people usually skip.
Only fire on the events that actually mean something
The trigger isn’t “any event from the helpdesk.” It’s specifically a routing or assignment event, on a specific ticket type, with a linked conversation to summarise. Anything else, including the event firing without a linked conversation, gets dropped before it goes anywhere near the model. That sounds obvious written down. It’s the difference between “this runs when it should” and “this runs constantly and nobody trusts the output,” which is how most AI automations quietly get switched off six weeks in.
Clean the input before you trust the output
Helpdesk conversations usually come through as raw HTML, with authors, timestamps and formatting mixed in. Before any of that reaches the model, it gets stripped down to a plain “who said what” transcript. If there’s genuinely nothing to summarise (a ticket created without an opening message, or a conversation with no readable body), the workflow doesn’t send an empty string to the model and hope for something coherent back. It substitutes an explicit placeholder that says exactly that: no content was found. Garbage in, garbage out is a real problem with LLMs, and the fix isn’t a better prompt, it’s not feeding it garbage in the first place.
The prompt has to do more than “summarise this”
The instruction that actually matters isn’t “write a summary,” it’s the constraints wrapped around it:
- A fixed structure the note has to follow every time (who the customer is, what they needed, how the conversation went, current status, anything the receiving team specifically needs to know), so every note is scannable the same way regardless of who’s reading it.
- A hard word limit, because a summary nobody has time to read isn’t a summary, it’s homework.
- An explicit instruction to flag anything mentioning the specific handful of things that matter for that queue (adverse effects, complaints, refund requests, whatever the equivalent is for your business) by name, every time, rather than leaving it to the model’s judgement about what counts as important.
- An explicit instruction not to invent detail when the transcript says nothing was found, which matters more than it sounds like it should. Models are good at sounding confident about things that aren’t true, and the fix is telling it plainly what to do when it has nothing to say.
Don’t hide that it’s AI-generated
The note posted to the ticket ends with a plain line stating it’s an automated summary. It’s posted under a shared account, so without that line, anyone reading the ticket later has no way to tell a human wrote it from a machine did. That’s not a legal disclaimer bolted on for cover. It’s the difference between a colleague trusting a note at face value and a colleague quietly wondering if they need to double-check it, which defeats the point of writing it in the first place.
Idempotency isn’t optional
Webhooks get redelivered. Tickets get reopened and reassigned. Without a guard, that means the same conversation gets summarised twice, and now there are two slightly different AI notes on the same ticket, which is worse than one, because now someone has to work out which one to trust. The fix is a simple check before the expensive work runs: has this ticket already been notified? If yes, stop, no duplicate note.
Measure it, don’t just assume it’s working
Every run logs a fixed time-saved estimate to a central log, tagged by workflow, week and month. A few minutes doesn’t sound like much per ticket. Across a queue running continuously, it adds up into a number you can actually defend in a review, rather than “the AI thing, it’s probably helping.”
The actual lesson here
None of this is about the model being clever. A capable model summarising a conversation well is table stakes at this point. The trust comes from everything around it: firing only on the right event, cleaning the input, constraining the output format, being explicit about what must never be invented, disclosing that it’s automated, and guarding against duplicates. Strip any one of those out and you’ve still got “AI writes a summary.” You’ve just lost the part that makes it safe to leave running.
If you’re weighing up where AI can safely sit in your own support process, and where it genuinely shouldn’t, that’s exactly the kind of decision I walk through in The Automation Roadmap, my Maven workshop on building your team’s first 90-day automation plan. Details and dates are here.
This was the second build in a longer rollout for this client. See how it fit alongside the rest of the 90 days.