Every product price, traced to its source.
An operations console that turns scattered web scraping into a governed pipeline for product and pricing data, across four markets, for the client and five competing brands.
- Run and Resume: long catalogs continue from the next page instead of restarting at page one.
- Traceable records: every row carries its source URL, page number and extraction time.
- Operated, not engineered: business users find, run, schedule and export without an engineer.
Meet our client
- Client
- Fortune 1000 industrial manufacturer
- Industry
- Filtration products & aftermarket parts
- Markets
- APAC · EMENA · LATAM · US / Canada
In a nutshell
We designed, built and documented a market intelligence platform for a Fortune 1000 industrial manufacturer: a console where business users collect product and pricing data on the client's brand and competing brands such as Fleetguard, Mann-Filter, Hengst, Parker Racor and Baldwin. A Next.js console fronts a Flask API that orchestrates ParseHub and six custom scrapers, stores everything in Snowflake, and runs in production on Azure Kubernetes Service.
Long catalogs, broken runs, untraceable data
One early extraction captured 1,269 of 3,413 of the client's products before it stopped. Replay that run, then replay it with the platform.
- Engineers for routine work: starting a job, checking it had finished and gathering output all needed a technical operator.
- Shifting formats: results varied by site and by run, so analysts reworked them before comparing a price or part number.
- Demanding data: several brands, four regions, multiple currencies, European comma decimals and cross-reference part numbers in titles.
- No way back in: an interrupted run left a partial dataset and no reliable way to continue where it stopped.
Start from the operator. Work back to the architecture.
Our product lead defined the operating flow first. The constraints that came out of it produced five design decisions that run through the whole system.
Six stages from web page to export
Follow one product record through the platform. Pick a stage to see exactly what the record looks like there.
- Normalized once: product name, part number, brand, price, currency, country, source URL, page number and extraction time.
- Part numbers untangled: the first primary part number is kept, and cross-reference numbers go in their own field.
- One price format, source currency: European decimals such as 1.234,56 become 1234.56, and currency symbols map to ISO codes. Each price stays in the currency of its source, with the ISO code in its own field.
- Never twice: ingestion is idempotent per run token, so replaying a run never duplicates rows.
Pick up at page 39, not page 1
A cumulative session groups every run that belongs to one paginated scrape. Drive it yourself: run, interrupt, resume, and try to start a second run while one is active.
- Run opens a new session: a fresh lifecycle, starting at page one, with its own run token.
- Resume continues: only when a session is active, pages remain, the next page resolves and nothing else is running.
- No overlaps: a short database lock and a conflict check stop two people, or a person and a schedule, starting the same project.
- No silent loops: checkpoint logic blocks a project whose resume makes no forward progress, until someone reviews it.
Where ParseHub stops, six custom scrapers take over
ParseHub handles stable listing pages. Six Python scrapers cover dynamic sites, supplier portals and local-language catalogs, and all of them deliver the same record shape.
- Patterns over selectors: extraction reads product names, URL slugs and image alt text, so layout changes break less.
- Sessions that persist: saved cookies let logged-in supplier-portal work continue across runs.
- Built for the site: Selenium, Playwright or DrissionPage, chosen per target, with local locales where markets need them.
- One path out: scraper output goes through the same ingestion, storage and export path as ParseHub runs.
- Configured projects: business users find, run, resume, schedule and export them in the console, and import project metadata in bulk. The user guide covers each step.
- New or changed ParseHub sources: a project built in ParseHub reaches the console through project sync. The client guide covers templates, fields, regex and adapting a project to a new brand.
- Custom scrapers: these are Python code. Adding one, or repairing one after a site changes, is engineering work, documented in the scraping-suite guide.
- The platform itself: changes to the API, database, automation or deployment follow the technical handover's change procedures and the deployment guide.
Try the console
A working replica of the operations console with sample data. Filter by market, run and resume projects, inspect a record's provenance, export a real CSV, validate a metadata import, and switch automation on and off.
Working replica built for this page · sample projects and masked values · the project, run and record totals repeat the platform snapshot from the result · no live systems are called
- Check coverage: each project shows pages collected against the total pages in its metadata, and a project whose resume stops advancing shows as Blocked until someone reviews it. Coverage is checked against the catalog, not assumed from a finished run.
- Check the records: ingestion tests every record for a part number, a numeric price, a recognized currency and a valid source URL. A row that looks wrong traces back to its source URL, page number, run and session.
- Export the right scope: the latest run or a whole cumulative session, as CSV for business review or JSON for system-to-system handoff.
- Compare the market: the client's brand and competing brands arrive in one record shape, so analysts can set the client's prices and part numbers beside competitors' in the same market without reworking formats first.
Secrets stay on the server. Releases ship on every push.
Every browser action follows one path, and every release follows one pipeline. Pick one to trace it.
- A server-side gateway: Next.js routes inject the backend key, normalize errors and handle uploads and downloads; the browser never calls the API.
- Separate keys: standard routes need a bearer key, internal automation a separate key, and the ParseHub webhook its own secret.
- Containers on AKS: console and API in separate Docker images, behind NGINX ingress with managed TLS, inside the client's environment.
- Safe by default: schema creation at startup is off unless enabled on purpose, so a routine deployment never alters the database.
Inside the platform
Product screens with sample values and masked prices. Tap any screen to see it full size.
From engineer-dependent to operator-run
The team now spends its time on pricing analysis, competitor research and market insight instead of running scrapes.
Snapshot figures are platform-level counts at one point in time, alongside 9 completed runs. They are not live figures and do not establish full catalog coverage.
- Repeatable market analysis: structured product and pricing records in one shape, across sites and markets.
- AI-ready data: provenance on every row makes the records traceable inputs for pricing analysis and future AI workflows.
- Room to grow: new sources become projects; the client guide shows how to adapt a site project to a new brand.
- Owned by the client: technical, user, client, deployment and scraping-suite guides let the team run and extend it.
External data your analysts can trust.
A collection platform that operations staff run, with every record traced to its source.
- 01Your business objective
- 02The technical context
- 03The right next step