Discover the snapshot
/data/current/manifest.json is the canonical, mutable pointer to the latest generated snapshot. It is short-cached and may change; use its snapshot_date and generated_at fields to judge freshness.
The manifest’s summary_url, every groups[].pages[].url, and every all_pages[].url are immutable snapshot URLs with long-lived cache headers. Each group’s legacy url points to its first page. The summary contains exact totals, title-group discovery, current-active first-seen history, and projection data.
Counts and paging
The manifest’s total_count and summary counts.active_total equal the number of permitted database rows marked active in the published snapshot. Every page entry’s count is the number of jobs in that page; group count values equal the sum of their page counts.
Use the summary or manifest groups for reliable discovery. Cache immutable responses locally, refresh the mutable manifest only when needed, and compare its freshness fields before downloading again. Scrape politely and sequentially—fetch one page at a time rather than opening concurrent requests.