WatchTrail runs on Vercel, and twice in two months I found a bill I hadn't seen coming. Once my fault, once a crawler's. Both had the same signature: no page ever failed, every dashboard was green, and the meter kept climbing. Here are both, with the real numbers, because this is the kind of thing you don't find written down until you've lived it.
August: 58 GB of cache writes for pages nobody read
The site has one page per TV episode. Roughly 250 shows, about fifty episodes each, four languages: some 50,000 URLs. Those pages are thin, around a hundred words, and they are `noindex`: they exist for navigation comfort, nothing else.
In July I turned on page pre-generation with `generateStaticParams`. The intent was sound: less compute at visit time. What I had missed is that on Vercel every pre-generated URL is also STORED, and a cache write is billed. Not the render: the write.
The cycle report settled it: 7.21 million 8 KB writes, about 58 GB, and $34.60 — against one cent of reads. The ratio said everything: we were writing a cache nobody ever read back. Meanwhile those episode pages accounted for 251,772 renders in 24 hours, 87% of all server activity, for pages invisible to search engines.
The fix is one line: drop `generateStaticParams` from that route. No per-URL entry in the manifest, therefore no writes. Verified under load the next day: zero writes during a crawl at 350 requests per minute, against roughly 5,000 writes per minute the day before.
What I took from it: a page's cost is not readable from its traffic. It is readable from what the host writes for it. And a comment in the code claimed that route was dynamic — it was lying, and nobody had checked in weeks.
September: a crawler read 40,000 title pages in nine hours
On September 13, between 6 a.m. and 3 p.m., the compute meter went from two minutes a day to thirty-nine. Transfer, from 35 MB to 1.81 GB. The logs showed nothing wrong: no errors, only normal responses.
The per-route breakdown named the culprit: 28,383 requests on movie pages, 11,266 on show pages, 24,739 distinct URLs, nearly all in English. The visitor's name appears nowhere in the runtime logs — I had to open the host's firewall view to read it: `GPTBot/1.4`, OpenAI's training crawler, 39,800 requests from a single address.
The mechanism is as dumb as it gets. Every title page shows a "more like this" section linking to other title pages. The catalogue behind it is a movie database with hundreds of thousands of entries. Seen from a title page, that catalogue is infinite: a crawler following links never stops, and every unknown URL renders a full page and returns 200.
It wasn't an attack, and the crawler broke no rule: it was explicitly allowed by our `robots.txt`, a deliberate choice so that assistants could cite us. The trap was ours.
The remedy has two parts. `robots.txt` takes title pages away from training crawlers while keeping everything worth learning: the home page, the guides, the comparisons, the blog. And because a crawler in a hurry doesn't always read that file, the server now answers 404, with no render, to any crawler that is neither a search engine, nor a link preview, nor a human-triggered fetch, as soon as it asks for a title page that isn't in our local cache — that is, a title nobody here tracks.
Cost for the day: about $0.35. On a paid plan, absorbed. On the free tier, thirty-nine minutes of compute a day eats the monthly quota in a week and the site goes dark. Which is exactly what happened to us in August.
What the two have in common
- No errors. In both cases, zero failed pages, zero alerts. Monitoring that watches for errors sees nothing at all.
- The cost doesn't come from human traffic. It comes from what the host writes, and from what crawlers read. Neither shows up in any analytics tool.
- The cause was a reasonable technical choice, made for a good reason, whose effect on the bill was written down nowhere.
- Diagnosis required leaving the runtime logs. The first was read in the billing export, the second in the firewall. Neither was visible from the application side.
If you host a site that generates pages from an external catalogue, two questions are worth asking today rather than at the next invoice. Is every pre-generated page read back at least once? And does a crawler following my internal links ever stop?