How to Build a Lead Scraper with n8n and Apify

Most "lead scraper" tutorials stop at pulling a list of names. The hard part is everything after that: keeping the data clean, not scraping the same company four times, and knowing which of those 30,000 rows is actually worth a sales call. Here is the architecture I use.

I have built this pipeline more than once, and the version that survives contact with real data always has the same five stages. Apify does the scraping. n8n orchestrates everything around it. The value is not in any single node. It is in the boring middle, where messy web data becomes clean, deduplicated, scored records a human can act on.

This is a build-first walkthrough. You will not find a "sign up for my course" wall halfway down. If you want to see the finished version, the BYO Lead Scraper case study documents a production build of exactly this.

The architecture in one picture

Five stages, each with a clear job:

1. Target definition. What are we looking for, and where does it live? Search results, a directory, a map listing, a set of company domains.
2. Scrape (Apify). Run an Apify Actor to extract raw records. Apify handles the browser, proxies, and anti-bot problems you do not want to reimplement.
3. Orchestrate (n8n). n8n triggers the run, waits for it, pulls the dataset, and moves each record through the rest of the flow.
4. Clean and deduplicate. Normalize fields, drop junk, and make sure one company does not appear five times under five spellings.
5. Enrich and score. Add missing data (email, size, tech stack), then rank each lead so sales works the top of the list first.

Stage 1: Define the target before you scrape anything

The most common mistake is scraping first and thinking later. Decide what a good lead looks like before you write a single node. Industry, company size, location, and a signal of intent if you can get one. This definition becomes your scoring rubric in stage 5, so it is worth writing down now.

Keep your target list in something n8n can read: a Google Sheet, an Airtable base, or a static list in a Set node. The point is that your targets are configuration, not code. When the client wants a new vertical next week, you change a row, not a workflow.

Stage 2: Scrape with Apify

Apify's Actors are pre-built scrapers for common sources, and they are the reason this pipeline is maintainable. Instead of babysitting Playwright against a site that changes its markup every month, you call an Actor and get structured JSON back.

In practice you will:

• Pick an Actor that matches your source (a maps scraper, a directory scraper, a generic web scraper).
• Pass it your search terms and limits as input.
• Start the run through the Apify API and collect the dataset when it finishes.

Set sane limits. It is tempting to ask for everything. Do not. Pull a few hundred records first, run them all the way through your pipeline, and confirm the output is good before you scale to tens of thousands. A cheap dry run saves you from paying to scrape garbage.

Stage 3: Orchestrate with n8n

n8n is the spine. A scheduled trigger or a manual webhook kicks off the run. An HTTP Request node starts the Apify Actor, then you either poll for completion or use Apify's webhook to tell n8n the dataset is ready. When it is, pull the items and split them into a stream of individual records.

Two things matter here that beginners skip:

• Idempotency. If the workflow retries, it must not create duplicate records. Key every lead on something stable, like a normalized domain, and upsert instead of insert.
• Batching. Do not fire one enrichment API call per node execution across 30,000 items with no throttle. Use the Split In Batches node and respect rate limits, or you will get blocked and your data will be full of holes.

Stage 4: Clean and deduplicate

This is where a lead list becomes a lead database. Normalize everything before you compare anything:

• Lowercase and strip domains to their root (www.acme.co.uk/contact becomes acme.co.uk).
• Standardize phone numbers to a single format.
• Trim whitespace, fix casing on names, and drop obvious junk rows (empty companies, placeholder emails).

Then deduplicate on the normalized key. The same business shows up as "Acme Ltd", "Acme Limited", and "ACME" across three sources, and if you do not collapse those, your sales team calls the same company three times and looks careless. A simple approach is to build a key from the root domain, and where there is no domain, fall back to a normalized name plus city.

Stage 5: Enrich and score

Enrichment fills the gaps: find a contact email, estimate company size, detect the tech they run. You can use dedicated enrichment APIs, or use an LLM to extract and infer structured fields from the scraped page text. I lean on an LLM step for the fuzzy parts, because it is good at turning a messy "about us" page into a clean set of fields.

Scoring is where the whole pipeline pays off. Take the target definition from stage 1 and turn it into points. Right industry, add points. Right size, add points. Has a hiring signal or recent funding, add more. Sort descending. Now your client is not staring at 30,000 rows. They are working the top 500, which is the only part that was ever going to convert.

A lead list nobody trusts gets ignored. The scoring step is what turns a scrape into something a sales team actually opens on Monday.

The failure modes nobody warns you about

• Silent rate limiting. An API starts returning empty results instead of errors, and your pipeline happily writes blanks. Log response codes and alert on a drop in fill rate.
• Dedup that is too aggressive. Collapsing on company name alone merges two real, different businesses. Prefer domain-based keys and only fall back to names carefully.
• Enrichment drift. A free enrichment source quietly degrades, and three weeks later half your emails bounce. Sample the output regularly.
• No observability. If you cannot answer "how many leads came in, how many survived cleaning, how many scored above the threshold", you are flying blind. Track counts at every stage.

When not to build this

If you need 200 leads once, do not build a pipeline. Run an Apify Actor by hand, clean it in a spreadsheet, and move on. This architecture earns its keep when the scrape is recurring, the volume is high, and the same messy data problems would otherwise bite you every single week. That is the honest test. Automate the repeated pain, not the one-off task.

← Back to all articles