7% calls got abnormally high latency

Incident Report for Retell AI

Resolved

Incident range: 6:30am - 11am PST
impact: around 7% of the calls are having abnormally high latency

post mortem:
Some AWS automatic patches to our transcription clusters caused the container to lost GPU access, and used CPU for transcription, causing extra long latency there. Most calls are routed to the backup endpoint which was working fine, but around 7% did not trigger the fallback there. We are updating the containers to ensure it does not get impacted with the automatic patches.

Posted Oct 02, 2025 - 06:30 PDT