The sync ran at 6 a.m., so why did the site show yesterday's prices?

I have three days of readings, and none of them covers 24 hours. Sorted by the hour, they add up to a fairly complete day in the life of a price-comparison site: when it pulls prices, when the source actually has them, and when I measured the wrong place.

The site compares phone and laptop prices across a few dozen stores. It does not crawl anything itself. Another data service crawls every day and records the crawl date per store; the site pulls the new copy on a schedule. The site makes one promise: today's prices, many stores, side by side.

The 2 November numbers are copied from that site's fix notes. The 10 and 11 November numbers I measured with scripts/garden/catalogue-freshness-probe.ts, which reads public data only and needs no key. The measuring and the draft had AI help.

Tiếng Việt: Đồng bộ lúc 6 giờ sáng, sao trang vẫn hiện giá hôm qua?.

02:05, I measured the wrong machine

10 November, two in the morning. An automated run of this garden ran the price site's freshness check and wrote down: "STALE, 1 day behind, the site's file is at 8 November". That line went into the to-do list as if it were a real reading.

That check answers for the machine it stands on, and it was standing on mine. It takes the newest file in a folder, and where that folder is depends on where you type the command. The 8 November was an old copy on my machine, saying nothing about the real site. So every reading below looks at the site from outside, over the network, like a visitor, and always prints the hour it was taken.

06:00, the sync reports success

When the site went up, the deploy notes chose 06:00 for the sync job, with one line of reasoning: "after the source's crawl". Early morning, nobody is browsing yet, fresh data arrives just in time. It sounds right, and nobody measured it.

On 2 November at 06:01, the person fixing it stood on the machine that runs the site and watched the job finish: success, exit code 0, 35,439 rows pulled. The source's newest crawl date at that moment was 1 November. The job was not broken, it just ran too early, then held yesterday's copy all day. Anyone who opened the site from morning to evening saw yesterday's prices, on a site whose whole reason to exist is today's prices.

07:45, the source has 6 stores

This morning, 11 November, at 07:45: the source had 11 November prices for 6 of 43 stores, and the site still showed 10/11. The 06:00 run still fires every morning, and still lands too early.

The new schedule keeps the 06:00 run, because it costs nothing; it just cannot carry the day alone. On a day the source finishes early, it makes it. On both days I measured, it missed.

11:06, the source has 19 stores, the site is still on yesterday

10 November, 11:06: 19 of 43 stores had today's prices at the source, and the site still showed 09/11. Had the schedule stayed as it was, a single 06:00 run, the site would have held 09/11 until the next morning, although the source had new prices before noon.

12:15, the second run

The first reflex is to move 06:00 to a later hour. But later by how much? The two readings of 2 November only say the source finished somewhere between 06:01 and 18:33. Picking a new hour is just guessing again.

So the fix does not guess an hour, it adds runs. Keep 06:00, add 12:15 and 17:45. Whenever the source finishes, the next run picks it up, and the stretch of stale prices shrinks from a whole day to a few hours. Each run writes a file of about 35 MB and the machine keeps the newest 7, so three runs a day cost little space.

One detail I like. That timer was written by a different deploy system, and any deploy may rewrite it. So the two new hours sit in a separate drop-in file instead of the original. systemd adds up the OnCalendar lines from every drop-in, so when the next deploy rewrites the original, all three hours survive. The price is that the hours now live in two places, and the fixer asked the other side to fold them in so they cannot drift.

13:40, the site turns the day, and the check says too much

date	hour (+07)	stores with that day's prices at the source	date the site shows
10/11	11:06	19 of 43	09/11
10/11	13:40	26 of 43	10/11
11/11	07:45	6 of 43	10/11
11/11	13:41	24 of 43	11/11

Two days, one rhythm: in the morning the site is still on yesterday, by early afternoon it has turned the day. Between each day's two readings, the schedule has exactly one run, 12:15.

But the source does not finish at any one hour. Prices arrive store by store, spread through the day. At 13:40 on 10 November, 15 stores were still on 9 November, and 2 had not moved since July and September.

The fix came with a small check that runs in about 0.1 seconds: compare the source's newest date with the site's newest date, print fresh or STALE, N days behind. At 13:40 it would print fresh, because the newest date needs only one store, and it never sees the tail. My own probe once printed exactly "fresh" while 17 of 43 stores had no price for today. Now it prints the count with it, because "26 of 43" is truer than "fresh".

Luckily the site does not use that check to talk to visitors. A phone's page prints the data date next to each price: when I opened one, 11 prices said 10/11, 6 said 09/11, 1 said 31/10. The check rounds off; the page tells the truth.

17:45, where my readings stop

The third run fires at 17:45. The latest reading I have is 18:33 on 2 November, and it is not mine: by then the source was on 2 November and the site still held 1 November. None of my three days of readings goes past 2 p.m.

So I do not know when the last store lands, whether the 17:45 run catches it, or whether 12:15 should move later. And for the 2 stores frozen for months, I do not know whether the site should hide them, or leave them for visitors to see the date themselves.