What we hit when we tried to scrape Google News

To scrape Google News with plain HTTP, use the RSS feeds. In our test they answered 270 polite requests with 270 200s, while the news.google.com search page blocked us on the first request. The Google News RSS feed guide said "I did not stress-test rate limits". This post is the follow-up. It covers the three things people mean by a Google News scraper: polling the feeds, fetching the web pages, and resolving the redirect links.

We ran everything on 2026-10-15 from the Netherlands, on a home connection, with Python's standard library and curl. We have not tested a datacentre IP, a long run or another country. Google's responses may differ for you.

What you can fetch without an API

Surface	What it returns	Needs JavaScript?	What we saw
RSS feeds (/rss/search, topic feeds, /rss)	XML, up to 92 items for our query	No	200 every time we sent a valid URL
Web pages (/search, /topics/..., /read/...)	HTML	Search and article pages: yes. Topic pages: no	Search blocked, topic page readable, article page empty
Redirect resolve call	Publisher URL as JSON	No	Covered in our earlier test

None of this is a documented API, so Google can change it without notice. Is there a Google News API covers the alternatives, and Newscatcher vs NewsData.io goes deeper on two paid ones.

The fetch loop we ran

The loop is one GET to a fixed search feed, a fixed User-Agent and a fixed delay. It logs a timestamp, the status, the body size, the item count and the time taken. Here is the core of it:

import time, urllib.request, urllib.error

UA = "Mozilla/5.0 (compatible; Feedmarkdown/1.0; +https://feedmarkdown.com)"
URL = "https://news.google.com/rss/search?q=offshore+wind&hl=en-US&gl=US&ceid=US:en"

def poll(delay, seconds):
    end = time.time() + seconds
    while time.time() < end:
        t0 = time.time()
        try:
            with urllib.request.urlopen(
                urllib.request.Request(URL, headers={"User-Agent": UA}), timeout=20
            ) as r:
                code, body = r.status, r.read()
        except urllib.error.HTTPError as e:
            code, body = e.code, e.read()
        print(code, len(body), body.count(b"<item>"), f"{time.time() - t0:.3f}s")
        if code in (429, 503):
            break  # a block is a result, not a goal
        time.sleep(max(0, delay - (time.time() - t0)))

poll(2, 300)    # 1 request every 2 s for 5 minutes
time.sleep(600) # 10 minute pause
poll(1, 120)    # 1 request every second for 2 minutes

Rate limits we hit, and the ones we did not

Phase	Rate	Requests	Status codes	Latency p50 / p95	Non-RSS bodies
A	1 every 2 s, 5 min	150	150 x 200	0.133 s / 0.197 s	0
B	1 every 1 s, 2 min	120	120 x 200	0.147 s / 0.188 s	0

We saw no 429, no 503 and no HTML "unusual traffic" page on the feed. Two more checks:

• User-Agent. The default curl UA, our UA, a Chrome UA and an empty UA all returned the same 122,611-byte feed with 92 items.
• Query diversity. 60 different company names at one request every 2 s: 60 200s. The same query 60 times: 60 200s. We found no limit keyed on query diversity.

Do not read this as a safe rate. We tried 1 request a second for two minutes and stopped. The feed also barely moves. Over the 150 requests we counted 94 distinct GUIDs, so two new items showed up in five minutes. Polling every 2 seconds buys you almost nothing.

The redirect call has the same story. Our earlier post resolved 300 distinct ids at concurrency 1, 4 and 16 plus 50 more, and saw no 429. That is the only limit test we had before this one.

Then a realistic pass. We fetched a feed (the newest 20 items of a when:1d search), resolved every unseen item 4 at a time, cached results by GUID, and repeated every 60 s for 10 cycles. Cycle 1 made 20 resolves. Cycle 6 made 2, for items that had just appeared. The other eight made none. That is 22 resolves in 10 minutes with no failures. A poller that caches by GUID is cheap to run.

Our own adapter works the same way. It sends one feed GET per run and resolves new items at a cap of 4 at a time, skipping ids it already has. The cap comes from a code comment saying Google's resolve call "rate-limits aggressively". It is caution, not a number we measured. Scheduling polls a feed at most once per interval, and the adapter has no retry or backoff of its own. The next attempt is the feed's next run.

Scraping the Google News web page with plain HTTP

This is where we got blocked. We fetched three kinds of page with curl -sL and a Chrome User-Agent:

• Search page (/search?q=offshore+wind): the first request redirected to www.google.com/sorry/index and ended on a 429. The page says "Our systems have detected unusual traffic from your computer network". The consent cookie made no difference. It was still 429 for two different queries when we checked again at the end of the session, so we cannot say how long it lasts.
• Topic page (Technology): 200, 3.5 MB, with 196 anchors carrying headline text in the raw HTML. No JavaScript was needed. Without a cookie it took one extra consent hop. With Cookie: CONSENT=YES+cb; SOCS=CAI it took none.
• Article page (/read/<id>): 200, 1 MB, with zero <a> tags. It holds no article text, only the data the resolve call needs, as our redirect post showed.

We do not know why the search page blocked us on the first try while the feed and topic pages did not. We did not try a headless browser or another IP. If you need the search UI, many IPs or rendered pages, a proxy or SERP API from a vendor is the usual route. We did not compare any, so we make no claims about them.

Redirects, consent pages and what breaks

What we sent	What came back	Status
Topic feed URL (/rss/headlines/section/topic/TECHNOLOGY)	302 to /rss/topics/<id>, then 200 XML. curl without -L stops at the 302	Observed
Unsupported topic (.../topic/NOPE)	301, then after 4 hops a 200 HTML page of 1.8 MB	Observed
Nonexistent feed path (/rss/nothing)	302, then a 200 RSS feed with one item titled "This feed is not available."	Observed
Missing hl, gl, ceid	302 to the same URL with en-US defaults added, then 200	Observed
Invalid language and country (xx-ZZ)	302, then 200 with en-US results	Observed
Empty q	404 HTML page	Observed
650-character q	200, 104 items	Observed
Consent page in front of the feed	None. Feed was identical with and without the cookie	Not seen from the Netherlands

Three of these matter if you write a consumer:

• A 200 is not always a feed. Check the body starts with <?xml. A bad topic gives HTML. A bad path gives a valid feed with one fake item.
• Follow redirects. Topic feeds always redirect. Go's HTTP client follows them by default, and our adapter's client does not turn that off. A bare curl does not.
• Bad locale parameters fail silently. You get English results with no error.

Link handling is a separate failure. curl -L on an item link never reaches the publisher. The redirect post has the two-request method. Google Alerts feeds use a different link type, covered in the Alerts RSS post.

The item data was tidy in our 92-item feed. All 92 had a <source> element, all 92 titles ended with " - Publisher", and all 92 links were news.google.com/rss/articles/.... One description held more than one link.

Failures in our own pipeline can hide. The adapter drops any item that fails to resolve and logs it at debug level only. A resolve outage therefore looks like an empty run, not an error. Our adapter's 500-character query limit is our own rule. Google accepted 650 characters.

A polite setup

• Send a fixed, honest User-Agent. We saw no difference between UAs, so a disguise buys nothing.
• Poll the feed no faster than the data changes. We saw 2 changes in 5 minutes.
• Cache resolves by GUID and only resolve unseen ids. It took a 10-cycle run from 20 resolves to 0 or 2.
• Cap resolves at a small number. Four worked for us. We have not found where it stops working.
• Back off on any non-200. On a 429, stop.
• Check the body, not only the status.

Skip the plumbing

Feedmarkdown's Google News source does the polite part for you. It fetches the feed on your feed's interval, resolves links four at a time, skips ids it already has and dedups by headline before any paid scrape. The result arrives as full-text Markdown in an HMAC-signed webhook (X-Feedmarkdown-Signature), or you pull it as JSON Feed 1.1 or Atom. Google News and Google Alerts are the live sources today. Start on the home page or see the pricing.