Ingestion

From auction pages to rows in Postgres.

Every price record passes through validation, bottle normalization, in-run deduplication, and a database write — with Celery beat keeping sources on staggered schedules so one slow spider never blocks the rest.

Four-stage pipeline

1 · ValidateRequired fields, type checks, currency bounds

2 · NormalizeMatch or create bottle, detect category, aliases

3 · DedupeIn-memory keys per scrape run

4 · PersistPostgreSQL prices + bottle stats refresh

Implemented in src/scrapers/pipelines/ — separate item types for auction lots (AuctionLotItem) and retail listings (RetailPriceItem).

Active sources

Schedules defined in src/tasks/celery_app.py. Auction spiders run every 6–8 hours; retail every 12 hours.

Spider	Type	Source	Schedule
`whisky_auctioneer`	Auction	whiskyauctioneer.com	Every 6h
`scotch_whisky_auctions`	Auction	scotchwhiskyauctions.com	Every 6h
`whisky_auction_uk`	Auction	whisky.auction	Every 8h
`whiskyauction_com`	Auction	whiskyauction.com	Every 8h
`whisky_hunter`	API	whiskyhunter.net	Daily
`whisky_hammer`	Auction	whiskyhammer.com	Every 12h
`dekanta`	Retail	dekanta.com (Shopify JSON)	Every 12h
`whisky_barrel`	Retail	thewhiskybarrel.com	Every 12h
`fine_drams`	Retail	finedrams.com	Every 12h
`cask_cartel`	Retail	caskcartel.com	Every 12h
`bottle_blue_book`	Valuation	bottlebluebook.com	Daily

Several sources (Whiskybase, Wine-Searcher, Whiskystats, RW101) were built but disabled when site structures changed — the spider registry makes toggling them straightforward.

Reliability work

Stale-run guards

Celery tasks mark scrape runs in Postgres; zombie runs are cleaned up before new jobs start so workers never double-write.

Source ID index

Database-level dedup on source_id plus an in-memory cache during high-volume runs reduced duplicate inserts.

Memory-safe exports

Pandas OOM'd on 1.3M rows — switched to psql COPY via scripts/export_dataset.sh for publishing.

Crawl scope limits

Retail and valuation spiders cap page depth so long-tail catalog crawls don't starve the worker queue.

Seed data

Three Kaggle datasets bootstrapped the catalog before live scrapers took over:

Brand catalog (~4,800 brands) → bottles
Cask auction CSV (~562 records) → bottles + prices
Market aggregates (~1,719 monthly rows) → market_stats