Lead deduplication: why duplicates come back

In March, RevOps runs a bulk merge and the duplicate report drops close to zero. By June it looks the way it did in February. The merge was correct. It ran once, and the CRM creates new duplicates every day.

Lead deduplication is stage four of lead routing. It sits between enrichment and account matching, and it decides whether one buyer becomes one record or several.

What is lead deduplication?

Lead deduplication finds records that describe the same person and merges them into one. A complete process does four things:

1. It detects likely duplicates on several rules, each with a confidence score.
2. It decides each field of the merged record separately.
3. It moves all activity and history to the record that survives.
4. It leaves ownership changes to a person.

It runs when a record is created, and again on a schedule.

Why duplicates come back

1. Intake is continuous and cleanup is not. Forms, list uploads, event syncs and integrations create records every day. A merge that runs once a quarter falls behind the day after it finishes.
2. Detection uses exact email only. That finds the easy cases. The expensive ones differ by a plus-address, a personal address on one form and a work address on another, or a job change.
3. Every source creates instead of checking. The webinar tool, the chat tool and the list import each write new leads. None of them looks for the person first.
4. The check ignores contacts. The person already exists as a contact on an account. A new form fill creates a lead, because the check only searched leads.
5. Nobody trusts the merge, so it stops running. One merge loses an activity history or moves a lead to a different rep. After that, every merge needs approval, and the review queue grows faster than anyone clears it.

What it costs

1. Two reps call the same buyer. The buyer notices, and so do the reps.
2. The score splits across records. The demo request is on one record and the webinar attendance is on the other. Neither record crosses the threshold, so neither is routed.
3. Routing uses the wrong record. The new lead has no account match and goes to a territory rep. The existing contact belongs to an account executive with an open deal.
4. Reports count people twice. Lead volume and campaign response look higher than they are, and conversion rates look lower.

What to fix first

1. Check before you create. Send every source through one entry point that searches leads and contacts before it writes. A duplicate that is never created never needs a merge.
2. Detect on several rules, each with a score. Use exact email, email with plus-addressing removed, name and company, and name and phone. Merge automatically only above a high threshold. Send the middle band to a review queue.
3. Decide survivorship field by field. Keep the newest phone, the title a person typed, the most complete firmographics, and the earliest created date and lead source, so first-touch attribution survives.
4. Move all history to the survivor. Tasks, emails, campaign memberships, opportunities and attachments. Check that nothing points at the losing record before you retire it.
5. Never change an owner in a merge. If the two records have different owners, flag it for a person to decide. If both have open opportunities, stop the merge.
6. Keep merges reversible for 30 days. Retain both original records, so a bad merge is an undo.
7. Run a nightly sweep. Some duplicates arrive in bulk or through a source you missed. The sweep catches them before the next morning’s routing.

The guide to building lead deduplication and merge has the prompts for detection, survivorship and moving history.

Where dedupe sits in the lifecycle

Order matters. Dedupe works better after enrichment, because a normalized company name and domain make name-and-company matches reliable. It has to run before lead-to-account matching, which resolves leads against accounts rather than against each other. It also has to run before scoring and routing, which should see one record per person.

List uploads and event syncs write records in bulk, so they are where a missing check creates the most duplicates. A duplicate that reaches routing also slows the first call, which speed to lead covers. Lead lifecycle management covers how each step feeds the next.

Which of your automations breaks first? Take the assessment. It ranks what your team already runs by who can change it, how you would find out it stopped, how many systems it touches, and what a break costs.