SEO Data Bench/Guides/serp-apis-after-google-scraper-crackdown

Are SERP APIs blocked by Google's scraper crackdown? Nine routes, measured

Field guidecohort v1US · enrun 2026-09-28includes a Treg route

The same 50 queries through Google SERP routes on 15 and 28 September 2026: how many calls failed, whether the google.com/goto redirect reaches the URL field, and why routes that agreed on 15 September now return different result lists for the same query.

Written for people asking: Why did my rank tracker data drop or glitch after Google started blocking scrapers?Which SERP API still works after Google's anti-scraping crackdown?Do SERP APIs return the real URL or the google.com/goto redirect?

What was run

Cohort v1 is 50 US-English buyer queries for data APIs, frozen on 15 September 2026. Its SERP half ran that day through five routes, 250 calls, and the results are on the SERP completeness and latency pages. On 28 September 2026 (Sydney time) the identical calls ran again, and the same queries went through four more routes in the catalog that the first run did not include: 450 calls, nine routes, ten minutes. Every call went through the Treg CLI with one worker per provider, and every raw response is published with its latency and exit code.

The reason for the second run is the month in between. Search Engine Roundtable reported on 26 August 2026 that Google’s google.com/goto redirect on result links, first seen in testing in July, was at nearly 100% across several residential IP providers. SerpApi’s changelog of 7 September says it resolved the redirect so its Google APIs keep returning destination URLs. By 21 September an r/PostSEO daily recap was leading with Google’s anti-scraping crackdown breaking rank trackers. The first run sits after the redirect rollout and six days before that recap, so it is an earlier point in the same period, not a clean “before”.

Did the calls return a page?

• Two errors in 450 calls, none in 250 on 15 September. SerpApi answered one query with “We couldn’t get valid results for this search” after 90 seconds. LimaData returned HTTP 500 on one query after 19 seconds. Neither was retried, so this is a first-attempt rate.
• Serpstat’s 30 misses are the same 30 queries as on 15 September. They are long-tail terms absent from its stored index, not failures.
• Cost. DataForSEO’s figure is the cost field in its own responses. Cloro, AnyAPI, LimaData and Serper are the charges settled on the account ledger, which for Cloro and AnyAPI were below the catalog list prices of $0.0028 and $0.0009. SerpApi and ScrapeCreators are catalog list prices on the run date. Serpstat bills $0.0005 a result row and returned 19 rows on average for a request of ten, so an answered call cost about a cent; one call checked on the ledger settled at $0.009 for 18 rows. The Treg balance fell by $1.32 for everything on this page, including the two depth checks below and two single calls made to read settled prices. The 100 calls served by ScrapeCreators used 100 credits on a key registered on the account and are outside that figure.

Does the goto redirect reach the URL field?

No. Across 418 responses on 28 September and 220 on 15 September, zero result URLs were a google.com/goto link. Every route returned the destination URL in the field a rank tracker would read.

The redirect is visible elsewhere. Cloro returns it beside the destination in a separate redirectLink field, on 50 of 50 responses on both dates. SerpApi’s responses carried one inside about_page_link on 44 of 50 calls on 15 September and on none of 49 on 28 September. If you parse either field, check what it holds now.

What changed: the routes stopped agreeing

Agreement here is the share of URLs two routes have in common out of all URLs either returned in its first ten results, averaged over the 50 queries. The three live routes from separate providers that ran on both dates:

Among all nine routes on 28 September, LimaData and ScrapeCreators agreed with each other at 0.84, the level every live route reached a fortnight earlier; whether they share an upstream this run cannot say. The next closest pair was DataForSEO and Serper at 0.48. SerpApi’s agreement with Serper, LimaData and ScrapeCreators rounds to 0.00: it had a URL in common with them on 3, 2 and 2 of its 49 queries.

Counting result lists per query makes the same point without averages. Two responses count as the same list when their URL sets overlap by at least 0.2.

Two lists for one query, from the same provider

To see whether the second list was only a deeper slice of the first, two queries were fetched again through DataForSEO at depth 40, a few minutes after the main run.

The lists are disjoint to at least 35 results, so one is not page two of the other. And the second row is DataForSEO disagreeing with itself: its depth-10 call returned one list and its depth-40 call, minutes later, returned the other. Which list a call receives varied by call, not only by provider.

Each route against the page of 15 September

The page of 15 September is, per query, the URLs that at least three of that day’s four live routes returned. This is not an accuracy score. Results move in 13 days, and nothing in this run shows which list a person searching from the United States would see.

Serper’s 81% is the useful control: ordinary movement over 13 days left four fifths of the page in place on the route that changed least. A route returning 1% of it is returning a different list. Serpstat’s stored index barely moved against its own earlier answers (94% of URLs kept, same first result on 19 of 20), because nothing is fetched at request time.

The routed endpoint changed provider between runs

On 15 September Treg’s routed SERP endpoint was served by DataForSEO on 50 of 50 calls at $0.002. On 28 September the same call, unchanged, was served by ScrapeCreators on 50 of 50, through a ScrapeCreators key registered on the account, so Treg charged nothing and each call used one ScrapeCreators credit. Median latency fell from 9.2 s to 2.9 s. The results followed the provider: the routed responses agreed with direct ScrapeCreators calls at 0.89 and with DataForSEO at 0.22.

That is the trade with any routed endpoint. It kept returning data with no change on the caller’s side, and it is how nine routes were compared in ten minutes with one credential. For a rank series it is the wrong default in a month when providers return different lists: the response names the provider that served it in _treg.served_by, and a tracker should pin one provider or call it directly, and store that field with every fetch.

Who this fits

• You run a rank tracker and a position vanished this month. Fetch the keyword through a second route in the same minute. If the two responses share no URL, you are looking at two lists, and the drop is not yet a fact.
• You are choosing a route for a daily series. Pick one provider, pin it, store the full URL list of every fetch, and flag any fetch whose first ten results share under 20% with the previous one. Re-fetch before recording the change.
• You parse redirect or about-page fields. Re-check them; SerpApi’s changed content between the two runs.
• You need a page that does not move. A stored index will give you one, for the queries it holds. It gave 20 of these 50.

What this run cannot tell you

Why the lists split. It could be Google testing, a measure aimed at automated fetches, or a difference in where each provider fetches from; this run has no evidence for any of them. It also cannot say which list matches what a person sees. It is one run per date on 50 queries in one niche, in US English on desktop. Providers change their fetching weekly and any of these numbers may be different by the time you read them. AnyAPI’s latency was measured while a separate 100-result call to the same provider was in flight; that call timed out after 181 seconds and was not charged, and the depth check used DataForSEO instead. Five result rows in four published files are withheld under this site’s publishing policy; the scores were computed before withholding.

Reproduce it

treg auth login
scripts/run-seo-cohort.py evidence/fixtures/seo-cohort-v1.json evidence/serp-runs/$(date +%F) --tasks serp
scripts/run-serp-routes.py evidence/fixtures/seo-cohort-v1.json evidence/serp-runs/$(date +%F)
scripts/score-serp-check.py evidence/fixtures/seo-cohort-v1.json public/evidence/seo-runs/2026-09-15 evidence/serp-runs/$(date +%F)

• Tables as CSV: routes, agreement on 15 September, agreement on 28 September, result lists per query, depth checks.
• Raw responses, one JSON file per call, for example serp-serpapi-q19.json beside serp-serper-q19.json; the 15 September files are on the evidence page.
• Scripts: cohort runner, runner for the four added routes, scorer; the cohort is on the dataset page.

Next step

• Run two routes on your own ten keywords before trusting either. The route headers for pinning a provider are in the Treg docs: treg.to/docs; per-call rates: treg.to/pricing.
• What each route returns beyond the ten links: SERP completeness. What a month of calls costs: latency and cost.

Relationship: the people who run this site also work on Treg, one of the routes compared on this page. Every Treg figure here is a call we paid for, with the raw response downloadable, and the same fixture is run against the alternatives where their terms allow.

Spotted an error or a changed price? Email corrections@seodatabench.com with the page URL and a source. Corrections are logged on the corrections page.