I wrote a command-line tool that turns a YouTube video into a written article. It takes the transcript, has a language model rewrite it into structured Markdown, and for very long videos splits the result into a folder of chapters, a book mode. Running it on a 7.4 hour video found failures that shorter runs had not, and most of them were quiet.
An answer that stops mid-sentence
In a real run, a blockquote in the output trailed off: "...His face and eyes resemble a". There was no marker that anything was missing and no unbalanced bracket for the checks to catch. The text just ended.
My first defence had been a heuristic: look for unbalanced brackets and trim. It cannot catch this, and no guess at a large-enough max_tokens can either. Reasoning models spend a variable number of hidden tokens thinking before they write, so the visible answer can be cut by a budget that was fine yesterday.
The reliable signal is the one the API already gives. A response that stopped because it hit the limit says so: stop_reason == "max_tokens" on Anthropic, finish_reason == "length" on OpenAI-compatible APIs. The tool now reads that field and retries with a doubled budget, up to a ceiling of 65,536 tokens:
# sketch of the idea, not the exact code
if response.stop_reason == "max_tokens" and budget < 65_536:
return call(prompt, max_tokens=min(budget * 2, 65_536))
It spends extra only when a call actually needed it. A call that completes normally costs exactly what it did before.
The same bug in a second place
The book-level summary, the "about this book" paragraph that goes on the index, used max_tokens=1024. It hit the same truncation, the failure was swallowed, and the index came out with no summary at all.
Three changes: the budget was raised, the failure is now reported through the progress callback so the command line can warn, and there is a deterministic fallback that needs no model, a plain "This book covers N chapters: ..." line. The principle is that an index should never be left blank because one optional call failed.
The rewrite prompt got a stricter rule too. When the source transcript is itself garbled or cut off, the model must write [unclear], not trail off silently.
A timeout that was never retried
One book run failed every chapter against the flagship model. Reading the code showed why. A read timeout, where the server hangs instead of answering, has no HTTP response object, so the check for retryable errors silently classed it as not retryable. One chapter failed after a single 300 second hang and zero retries.
Timeouts now get the same treatment as a 503. And because a sustained "high demand" outage on the main model can outlast retrying, the two calls that use the expensive model now carry a cheaper sibling as a fallback. After the main model exhausts its own retries, the tool tries the sibling once more before giving up on the chapter. An overloaded flagship and its differently provisioned smaller sibling do not necessarily share the same outage.
A default that no longer exists
Two days later a provider's default model returned "model not found". Its whole previous family had been retired with no same-name replacement. I checked against the provider's live model list and a real completion call, then moved the default and the cheap model to what was actually being served. While there, I checked the models configured for the other three providers against their live APIs. They all still resolved, and only one was dead.
The fix that matters is the habit: default model names are a dependency like any other, and they expire. A tool with provider defaults should have a way to find out when one has.
Money is a reliability issue too
Retrying costs real money, so the cost tracking records every attempt. A call that was truncated and then retried is two billed requests, and the report counts both. Lower-stakes calls (front matter, research notes, the speaker bio, the book summary) are routed to a cheaper model in the same family, and only the rewrite itself, the one call where quality really depends on the model, always uses the main one. A --choose flag analyses a video first, states a suggested path, and asks before it goes ahead.
What I would keep
- Read the API's own signal. Truncation, rate limits and overload are reported. Use the field and not a guess.
- Check that every failure path is retried the same way. A timeout is a failure like any other and should not be an exception by accident.
- Never leave an optional output blank. A fallback that is less clever and always present beats a clever call that can fail silently.
- Run the long case. A 7.4 hour video found what the short ones never could.