When your AI tool goes down
If you do real work in an AI chat, the service behind it will go down at some point. The natural move is to open a new chat. On the one night we measured it, that did not save us: both new chats died on their first request. This piece is for anyone who leans on an AI tool for a day's work. The lesson is to stay in the chat that still answers and give it a small job every few minutes. It is our reading of one night, not proof, and I will say why.
On 29 July our AI service declared a major outage, with errors across all its models. We had eight chats in use that night. At least four of them failed, and two of those were new chats opened during the outage. When we wrote the night up, one chat was still answering.
The likely reason is something called a prompt cache. Each time you send a message, your tool sends the whole conversation to the service, not only your new line. Reading all of that again is heavy work. So the service holds a recent conversation ready for a while, and only has to read the new part. Here is what that means, with made-up numbers kept easy.
Chat A has been working all morning. Its history is 100 pages, and you add 1. The service already holds the 100, so it reads 1 page.
Chat B has the same history, but the service has let it go. You add 1 page, and it has to read 101.
Chat C is brand new. It has, say, 20 pages of instructions and files the service has to read, so it reads 20.
An overloaded service seems to drop the heavy requests first. Chat A has the lightest job, so it gets through. Chat C is lighter than Chat B, but it is still 20 times the work of Chat A.
Our real numbers had that shape. Every chat that kept working was reading 84,000 to 224,000 tokens from what the service held, and sending a few hundred new ones. A token is about three quarters of a word. The two older chats that failed had to send 268,000 and 375,000 fresh.
Those two older chats prove less than they seem to. A chat stuck on errors goes cold in the end whatever broke it, within the hour on ours, so the cold could be a result of the failure. Our note from that night says they had sat idle first, but it gives no times, so we cannot check that. The two new chats are the cleaner evidence, because they had no history of their own for the service to hold. That is two chats on one night, and we cannot see how the service chooses what to drop, so this is an inference.
So when the errors start, find the chat that still answers and stay in it. Give it a small, useful job every few minutes so the service keeps holding it. How long it holds a conversation varies by tool and plan. Five minutes is the service's default and our main chats get an hour, so every few minutes is the safe rule. Leave the new chat until the service says the trouble is over.
If nothing answers, wait. Our tool keeps each chat on the laptop for about 30 days, so a dead chat's work can still be read the next day and picked up from there. Check whether yours does the same before you redo anything.
Lena