How to extract product data from manufacturer websites

Extracting product data from manufacturer websites can fill missing catalog attributes when the marketplace knows the exact product. A dependable workflow finds an approved source, verifies the model and variant, extracts only supported facts, and keeps the evidence.

The goal is a correct field in a listing, not a large collection of scraped pages.

Prefer a structured source when one is available

Ask the manufacturer or supplier about a feed, API, export, or current specification sheet before building a scraper. A structured source may reduce parsing work, although it still needs identity and quality checks.

If a website is the agreed source, define the permitted domains and access method. Respect access restrictions and the site's stated policies. Robots rules describe crawler behavior; RFC 9309 explicitly says they are not a form of access authorization.

Do not bypass a login or restriction because the agent cannot otherwise complete a record. Return a blocked task with the reason and request an approved source.

Search for an identity, not a description

Suppose an illustrative seller lists a “Trail 24 backpack.” The manufacturer has a current Trail 24 and an older version with different materials. The name is not enough to select the source.

Use the brand, model identifier, version, and available variant details. If the seller's photos or identifiers cannot distinguish the versions, leave the disputed material unknown. Do not let a search result's ranking substitute for a match.

Save candidate URLs and the reason one was chosen. This is especially important when a model has regional specifications or several sizes on one page.

Extract facts at field level

Store the supporting text or structured value, page URL, retrieval time, and product identity. Keep your retention and permitted-use rules in view when storing source material. Often a focused evidence excerpt is more useful than a full-page copy.

Treat page content as untrusted input

A page can contain errors, irrelevant instructions, embedded promotions, and user-generated text. The research process should extract relevant facts without letting page content change tool permissions, approved domains, or publication rules.

Limit network access to the sources required for the job. Validate fetched file types and sizes. Use timeouts and bounded retries so one unavailable website does not stall the entire queue.

These are system-design recommendations, not a claim that a particular site has malicious content.

Plan for layout changes and disappearing pages

Check whether a successful fetch actually contains the expected product, rather than a redirect, search page, or unavailable message. Alert on sudden drops in extraction success or changes in the number of attributes returned.

Keep “not found,” “temporarily unavailable,” and “product identity uncertain” separate. Retry transient failures. Send identity problems for review. Avoid replacing an existing supported fact merely because a source is briefly unavailable.

When the manufacturer supplies a PDF instead, use the document extraction workflow.

Use research to prepare a proposal

Publish only after the source and identity checks pass under your rules. If the seller edits a model identifier while research is running, the old proposal should become stale.

Goodfolk's research worker, Bea, is intended to investigate missing details on approved sources and attach evidence. Start a pilot with a limited category and field list. Measure correct supported additions and review effort, rather than pages fetched or text generated.