Scraping a market twice: the overlap that eats a third of a data budget

The expensive mistake in list building is not the price per profile. It is extracting the same person three times across three searches that felt different when you wrote them. On one run of ours, four searches returned 23,089 rows that resolved to 15,363 distinct people. A third of the spend bought nothing. Nobody noticed until the deduplication step, by which point every duplicate had already been paid for.

Why overlap is invisible

Search one: heads of sales in software companies in France. Search two: sales directors in SaaS companies in France. To a human those are two angles on a market. To a database they are two overlapping sets, and the people in the middle are returned, and billed, twice. The wider the second search, the more of the first it swallows.

It gets worse over months. A market you work repeatedly produces a new extraction every quarter, and each one re-collects the people you already have. Without duplicate detection against your own history, you are renting the same list over and over.

The order that fixes it

Count first, wherever counting is free. Then sort your searches from narrowest to widest and check whether any of them is a subset of another you already intend to run. If it is, delete it: the wider search will return those people anyway. Only then extract, narrowest first, and keep a record of what came back so that the next search can exclude it.

This is also why automatic duplicate detection against previously scraped profiles is not a convenience feature. It is the difference between a data budget that grows with your market and one that grows with the number of times you look at it.

The 2,500 ceiling

LinkedIn searches stop being exportable in one piece above a couple of thousand profiles, which is where people start slicing a query by hand: by region, by headcount band, by first letter of the company name. Every one of those manual slices is a chance to create overlap, and most people create plenty. A tool that splits the query automatically above 2,500 profiles removes the ceiling and the temptation at the same time.

Clean fields are not cosmetic

Emojis in a first name, a job title glued onto a surname, random capitals: these survive the extraction, travel through the enrichment, and come out the other end in the first line of an email. A merge field is the one place where sloppy data is read by the person you are trying to impress. Cleaning at extraction time is the cheapest place to do it, because after that the bad value has been copied into three systems.

What extraction does not give you

Profiles, not contact details. The email and the mobile come from an enrichment provider afterwards, billed per result, and the hit rate there is the number that decides your real cost per usable contact. Across one of our runs on French decision-makers, the whole waterfall returned a mobile for twenty percent of them. You size a campaign on that number, not on the size of the extraction.

Schedule it, then stop thinking about it

A market you care about is not a one-off extraction, it is a standing query. Scheduled runs and detection of new profiles on a saved search turn a list into a feed: the people who joined the segment since last month arrive on their own, already deduplicated against everyone you have. That is the difference between a file that ages from the day it is built and a pipeline that stays current without anyone remembering to refresh it.

The test to run before paying

Take a free plan, extract two hundred profiles from a search you know well, and check three things: are the fields clean enough to merge into a first line, does the tool recognise people you have already extracted, and does the split of a large query return the number the search preview promised. Those three answers tell you more than any feature list, and they take an afternoon.