How to find duplicate listings without merging different items

Removing duplicate product listings requires deciding what is duplicated. Two sellers offering the same model are usually two valid offers. Two database records describing the same physical used item may be an accidental duplicate. Merging both situations in the same way can lose inventory and seller history.

Define the entity before choosing a matching algorithm.

Separate products, variants, and offers

Consider an illustrative camera marketplace. One camera model has black and silver variants. Several sellers offer used black units, each with a different shutter count and condition report. A seller accidentally imports the same unit twice.

The shared model can have one canonical product record. The black variant can have its own identifier. Each physical camera remains a separate item and offer. Only the repeated import is a candidate for collapsing as a duplicate listing.

Schema.org's ProductGroup vocabulary also distinguishes a product group from its variants. It is a useful reference for the distinction, not a requirement to adopt that exact internal schema.

Build candidates before deciding matches

Compare likely neighbors rather than every record with every other record. Candidate groups can use brand, manufacturer identifier, category, normalized model, or a combination of those fields.

Keep raw identifiers alongside normalized ones. Removing punctuation may help candidate retrieval, but punctuation can distinguish real model numbers. A broad candidate rule should not become an automatic merge rule.

Images can help identify repeated submissions, but a shared manufacturer photo does not prove two offers are the same item. Likewise, similar descriptions may have been copied from a template.

Use stronger evidence for destructive actions

Treat conflicting identifiers as a reason to investigate. A language model's similarity judgment should not override a clear storage, size, pack, or version difference.

Review a proposed merge as a relationship change

Show the reviewer both records, their seller IDs, external IDs, variants, inventory references, and transaction relationships. Explain why the system thinks they match and what would happen after approval.

Often the right action is to link records to a shared product rather than delete either record. For an accidental repeated import, retain an alias from the old listing ID to the surviving one. Decide how old links, saved items, and search documents will behave.

Do not move orders, refunds, or historical condition reports merely because two catalog descriptions match. Those relationships need an explicit migration plan if they change at all.

Test for false merges

Create a review set with near misses: adjacent model years, different pack sizes, similar colors, replacement parts, and separate used units photographed in the same style. Include real duplicates too.

Measure incorrect merges separately from missed duplicates. Their costs differ. A missed duplicate may create clutter; a false merge can show the wrong stock or condition to a buyer.

Start with suggestions only. After enough checked examples, automate narrowly defined cases such as an idempotent reimport with the same seller and external record ID.

Keep rollback at the identity layer

Record the original relationships and the approved decision. Reversing a grouping should restore the prior product links and refresh downstream search data. Deleting a source record first makes that much harder.

Goodfolk's catalog worker, Pip, is intended to identify inconsistent and duplicate records. A pilot should agree on the meaning of “duplicate” in your schema before running. Pair the workflow with variant normalization and an approval process so similar products remain distinct where it matters.