How often a Google News RSS feed repeats the same story
A Google News RSS feed repeats stories because every publisher gets its own item, and identical or rewritten headlines arrive as separate entries. In 7 searches we pulled on 2026-10-23, 53 of 469 items repeated an earlier headline. The biggest cluster of real news that headline matching found was 5 copies of one story. So "five times" is the worst case we saw, not the usual one.
This post is the diagnosis. For URL formats and the other two problems with these feeds, see the Google News RSS feed guide. For how to match and remove repeats, see why matching headlines isn't enough.
What the feed returned on 2026-10-23
We pulled 7 searches, each with when:1d, from the US English edition (hl=en-US&gl=US&ceid=US:en). One request per query, 5 seconds apart, one attempt each, no cookies, no API keys, and an honest User-Agent. All 7 returned 200. We took the first query, a breaking story, from that morning's top-stories feed.
curl -s -A "feedmarkdown-research/1.0 (+https://feedmarkdown.com)" \
"https://news.google.com/rss/search?q=lithium+prices+when:1d&hl=en-US&gl=US&ceid=US:en"
No feed repeated a GUID: every item in every feed had its own. The site:reuters.com feed returned 100 items from one publisher, which is the item cap, so it works as a control.
This is a one-day, US/English sample with one pull per query. It is a snapshot, not a rate.
The kinds of repeat, and how many of each
We sorted every item oldest first and ran the headline clustering code the Feedmarkdown server uses. We copied server/dedup.go and diffed it the day of the run: only the package line differs. We strip the - Publisher suffix first, as the Google News adapter does. A repeat is an item whose headline matches an earlier item in the same feed.
| Kind of repeat | Count (of 469 items) | Example | Caught by a headline check? |
|---|---|---|---|
| Same GUID twice in one feed | 0 | none | Yes, by ID |
| Same story, identical headline | 15 | "Is 2027 the Turnaround Year for These 4 EV and Air Taxi Stocks?" on AOL, The Globe and Mail and The Motley Fool | Yes |
| Same story, rewritten headline | 38 | "West Bank: Extensive military operation 'on the way', Netanyahu says" on France 24, against "Netanyahu says extensive West Bank military operation 'on the way' after attack" on 4 other outlets | Yes |
| Same story, headlines only a human would link | not counted across all feeds | "Kodal targets Ghana lithium growth beyond Mali mine" vs "Kodal plans West Africa lithium expansion, CEO says" | No |
The first three rows add up to 53 repeats. A second count overlaps them: 18 of the 53 came from a publisher that already had an item in the cluster. In the lithium feed, FXEmpire listed one forecast twice and AD HOC NEWS ran two headlines on one court date.
Repeats were not spread evenly:
| Feed | Items | Repeats | Largest cluster |
|---|---|---|---|
| Netanyahu West Bank operation | 42 | 11 | 5 |
| semiconductor export controls | 34 | 1 | 2 |
| lithium prices | 46 | 5 | 2 |
| FDA approval | 97 | 30 | 11 |
| central bank interest rates | 50 | 2 | 2 |
| electric vehicles | 100 | 4 | 3 |
| site:reuters.com | 100 | 0 | 1 |
The four ordinary queries (chips, lithium, rates, EVs) repeated 12 of 230 items, about 5%. The breaking story repeated 11 of 42, about 26%. These are headline-match counts, so they are a lower bound, as the hand check below shows.
The FDA feed is mostly not news
The 11-copy cluster in the FDA feed is not a news story. Headlines like "fda approved weight loss drug 2019" and "fda approved weight loss drug 2022" come from sites that publish one keyword-style page per search phrase. 32 of the 97 items came from two sites, BusinessWorld Online and polskieradio.pl. We did not open those pages, but the URLs we resolved had paths like /health-library/en-LV/ and /wellness-resources/en-BE/. A headline check merges them into one cluster. That is right in the sense that they repeat each other, but they do not repeat a story.
A real story in the same feed also had 5 copies: the FDA's full approval of pembrolizumab for TMB-H solid tumors, run by The ASCO Post, Oncology Nursing News, Oncology News Central and others.
The biggest cluster
The largest cluster of real news was five copies of one headline in the Netanyahu feed.
Five looks small until you read the rest of the feed: 35 of its 42 headlines name Netanyahu and the West Bank or Hebron. Headline matching splits them into 31 "stories" because outlets wrote one event in different words: "vows", "orders", "threatens", "lockdown", "demolish".
The feed held 42 items, well under the cap of around 100 described in the guide, so nothing was pushed out here. We did not measure a hotter story crowding out other stories.
Counting it yourself
Here is a version you can run on a saved feed. It is a simplified approximation, word-overlap (Jaccard) at 0.6, not the product's code. Save each pull as JSON Lines, one record per query with an items list, and pass the file as the argument.
import json, re, sys
from collections import Counter
STOP = {"a", "an", "the", "and", "or", "of", "to", "in", "on", "for", "at", "by", "with", "from", "as", "is", "are", "was", "after", "over", "says"}
def tokens(item):
title = item["title"]
if item["source"] and title.endswith(" - " + item["source"]):
title = title[: -len(item["source"]) - 3]
words = re.sub(r"[^a-z0-9]+", " ", title.lower()).split()
return {w for w in words if w not in STOP}
def jaccard(a, b):
return len(a & b) / len(a | b) if a and b else 0.0
for line in open(sys.argv[1]):
feed = json.loads(line)
seen, firsts, sizes = set(), [], Counter()
for item in feed["items"]:
if item["guid"] in seen:
continue
seen.add(item["guid"])
toks = tokens(item)
for i, first in enumerate(firsts):
if len(toks) >= 4 and len(first) >= 4 and jaccard(toks, first) >= 0.6:
sizes[i] += 1
break
else:
firsts.append(toks)
sizes[len(firsts) - 1] = 1
hist = Counter(sizes.values())
extra = sum(n * (k - 1) for k, n in hist.items())
print(f"{feed['id']:8} items={len(seen):3} stories={len(firsts):3} repeats={extra:2} sizes={dict(sorted(hist.items()))}")
Output on our saved feeds:
hot items= 42 stories= 34 repeats= 8 sizes={1: 30, 2: 2, 3: 1, 5: 1}
chips items= 34 stories= 33 repeats= 1 sizes={1: 32, 2: 1}
lithium items= 46 stories= 41 repeats= 5 sizes={1: 36, 2: 5}
fda items= 97 stories= 69 repeats=28 sizes={1: 61, 2: 3, 3: 1, 4: 1, 7: 2, 9: 1}
rates items= 50 stories= 48 repeats= 2 sizes={1: 46, 2: 2}
reuters items=100 stories=100 repeats= 0 sizes={1: 100}
ev items=100 stories= 96 repeats= 4 sizes={1: 93, 2: 2, 3: 1}
It lands near the real clustering: 8 repeats in the Netanyahu feed against 11, and 28 in the FDA feed against 30. The product also uses a containment check and a simhash, which catch a few more. The dedup post covers the thresholds and the risk of merging two different stories.
Why Google News RSS duplicates happen
What the data shows:
- Each publisher is its own item. The five Netanyahu copies came from five publishers, each with its own GUID. The feed did not collapse them.
- Identical and rewritten headlines both arrive. 15 repeats had an identical headline and 38 had a rewritten one.
- Repeated polls return the same GUIDs. We pulled the FDA feed again 10 minutes later: 95 of 97 GUIDs were the same, 2 were new, 2 had dropped out, and no GUID changed its headline. After 60 minutes, 88 of the original 97 GUIDs were still there, 7 were new and 9 had dropped out. One GUID kept its headline but changed the publisher suffix, from
streamlinefeed.co.ketohttps://streamlinefeed.co.ke/. So a GUID is a stable ID for dedup, and a feed that shifts by a few items per hour re-serves most of what you already have. - Overlapping queries share few items. Across the 7 feeds, 1 GUID appeared in two of them (lithium and EVs), and 1 story matched exactly by headline in two feeds.
- One article can have two GUIDs. The two FXEmpire items resolved to the same page, one with
?_c=963224added. The product's URL normaliser removes only known tracking parameters and_cis not one, so the two URLs hash differently. The headline check still caught the pair.
We did not test why Google lists items this way, and we do not know how its ranking works.
URL equality only works after you resolve the link, because every link is a news.google.com/rss/articles/<id> address. See how to get the real article URL. We resolved 20 links with the method in that post, 2 requests each, 5 seconds apart. That resolver sends the consent cookie from that post; the feed pulls sent no cookies. We picked the 20 from same-publisher pairs, a few same-headline pairs and the Netanyahu cluster. Only the FXEmpire pair was the same page. The two JFeed items with one headline and one publish time pointed to different pages, and so did every pair of keyword pages. URL repeats are a lower bound too: we resolved 20 links, not 469.
What a headline check misses
We hand-labelled the first 16 items of the Netanyahu feed and the first 24 of the lithium feed. This was a quick headline-only read, with no article bodies, so treat it as a rough estimate, not ground truth. "Same story" means the same news event.
| Feed (items labelled) | Stories by hand | Stories by headline matching |
|---|---|---|
| Netanyahu West Bank (16) | 3 | 13 |
| lithium prices (24) | 19 | 23 |
In the Netanyahu sample, 12 of 16 headlines were one event, "operation announced", and matching merged only 3 of them into earlier items. In the lithium sample, matching caught the FXEmpire pair and missed the Kodal story (3 items), the Mineral Resources sales story (2) and the European Lithium court date (2). The hand count is coarse for the breaking story, and a stricter reader would split it further. The direction holds: on a breaking story, headline matching under-counts repeats.
What removes them
Cheapest first:
- Skip IDs you have seen. This removes poll-to-poll repeats.
- Cluster headlines before you scrape anything. Here that removed 53 of 469 items with no page fetch.
- Compare article text after scraping, to catch a rewritten headline over the same body. Only this step can close the gap in the table above.
Narrowing the query with operators changes what comes back, see Google News RSS search operators. Do the first two before you pay for a fetch. The dedup post has the order and the cost argument. For a digest, see how to build a daily news newsletter from feeds. For duplicates around a regulator announcement, see regulatory news monitoring. One earlier data point: in media monitoring on a small budget, 20 of 100 headlines were one story from 16 publishers.
What we did not test
- Other editions (any
hl,glorceidother than US English). - Other days or times of day. This is one run per query on one day, with no second-day repeat.
- Whether Google's ranking or collapsing changes cluster sizes over time.
- Google News web pages, as opposed to RSS.
- Any live Feedmarkdown run. Counts come from an offline copy of the headline code on saved public feeds.
- Whether the keyword-style pages are spam. We did not open them.
Repeats across daily issues, rather than inside one fetch, are covered in how to stop sending subscribers the same story twice.
Skip the plumbing
Feedmarkdown handles this. It reads Google News and Google Alerts sources, clusters headlines before it scrapes so duplicate copies are not paid for, then compares content after scraping to catch rewritten headlines over the same text. A duplicate points back at the original article. You get each new story as Markdown in a signed webhook, or pull it as JSON Feed or Atom. See pricing or the home page.