How to stop sending subscribers the same story twice

November 2, 202613 min readFeedmarkdown#deduplication#newsletter#news-feeds

In our replay of real Google News items, going from a 24-hour to a 168-hour memory dropped only 2 to 3 of 271 items, while 9 repeats (3.3%) sat inside a single issue, where no window helps. So to keep duplicate articles out of a newsletter, deduplicate inside each issue first, then keep a record of what you sent and compare new items against it. That second step is how you deduplicate an RSS feed across days.

The pillar post on building a daily news newsletter from feeds covers dedup inside one fetch. This post is about the next problem: a subscriber opens Tuesday's issue and reads Monday's story again. How two headlines are judged "the same" is in why matching headlines isn't enough. Here we assume that part works.

Why a story shows up twice

Most repeats happen inside one issue: several outlets run the same story under near-identical headlines, and one fetch returns them all. Counts for that are in how often a Google News RSS feed repeats the same story. Repeats across issues are rarer, and come from two things: another outlet runs the story a day later, or the same item is still in tomorrow's feed. A follow-up or correction is a third case, but it is a product decision, not a duplicate. Repeats also cost money if the text goes to an LLM; see what 1,000 summaries cost.

What we measured: repeats under each window

We fetched six Google News RSS searches with when:30d on 2026-11-01, one request each, five seconds apart, US/English, no retries. "neobank regulation" returned only 7 items over 17 days, so we dropped it. Five were left: floating offshore wind, solid-state battery, direct air capture, quantum error correction and medical device recall. That is 374 items from 2026-10-03 to 2026-11-01. Google caps a search at 100 items, as we measured in the search operators post.

We cut the items into daily issues by UTC publish date. For each window we walked the issues in order, kept a ledger of delivered items, and dropped an item if its headline matched a ledger entry inside the window. The matcher is a copy of the server's headline code (TitleIndex: token Jaccard, containment, simhash). We counted 22 issues, 2026-10-10 to 2026-10-31, so even the 168-hour window has full history behind it: 271 items per window. As a cross-check we reran the code on our 2026-10-23 capture, 17 searches with when:7d: 984 items over the three full days 10-20 to 10-22, windows up to 72 hours.

Bar chart of items dropped as a headline repeat for 24, 48, 72 and 168 hour windows on a 30-day capture, plus a 7-day cross-check

Window	Dropped, 30-day capture (of 271)	Dropped, 7-day cross-check (of 984)
24h	2	10
48h	2	13
72h	2	14
168h	3	not testable (capture spans 7 days)

The window barely changes the result: 24 to 168 hours dropped one more item out of 271. The same-issue count did not change at all: 9 of 271 items matched another item in the same issue at every window, and 40 of 984 in the cross-check. That is why the first step is dedup inside the issue.

Limits: only headlines were replayed. Exact repeats were not measured in the replay: one capture puts each item in one issue, so "still in tomorrow's feed" is zero by construction. Items that left the feed before capture are missing, and five niche searches over one month may not match a busy topic.

How long a story keeps getting new headlines

We counted the days between the first and last item of every group of two or more matching headlines.

Histogram of days between first and last item for headline clusters, from a 30-day and a 7-day capture

In the 7-day capture, 30 of 52 groups ended the day they started and 9 lasted two days or more. In the 30-day capture, 8 of 13 were same-day and 3 lasted five days or more. Two of those three were not one story: two Department of Energy pages ("Top Trends in Offshore Wind", "Top 10 Things You Didn't Know About Offshore Wind Energy") and several Quantum Zeitgeist headlines that all start "Researchers ..." cluster on shared wording. Templated headlines are the cost of a long window.

Follow-ups: new development or the same story again?

We labeled from headlines only, one labeler, no second pass. Rehash: same event, no new fact. Development: a new fact, such as a vote, a number or a new party. False merge: a different story.

We labeled every pair the matcher dropped as a repeat of an earlier issue when the ledger remembered everything back to the start of each capture, no window cap: 6 in the 30-day capture and 17 in the 7-day one, 23 in total. We had planned a 40-item sample, but the windows rarely differ. Result: 18 rehash, 5 false merge, 0 development.

Earlier headline	Later headline	Label
Saluda Medical's shares hit record low on Evoke device recall (SRN News, 10-20)	same headline (TradingView, 10-21)	Rehash
Paymentology Partners with Asia United Bank to Accelerate Digital Payments in the Philippines (FF News, 10-19)	Paymentology partners with Asia United Bank on digital payments (The Paypers, 10-22)	Rehash
Axena Health Named Finalist for the Digital Health Hub Foundation (10-20)	Vespexx Recognized as Finalist for the Digital Health Hub Foundation (10-22)	False merge
Top Trends in Offshore Wind (DOE, 10-23)	Top 10 Things You Didn't Know About Offshore Wind Energy (DOE, 10-29)	False merge

None of the dropped items was a development. We did not label what was delivered, so we can't say how many follow-ups got through. A development usually has a new headline, so we would expect the matcher to deliver most of them, but that is an assumption. Of the 5 extra items a longer window dropped (24 to 72 hours in the 7-day data, 48 to 168 in the 30-day data), 3 were rehashes and 2 were false merges.

Feedmarkdown does not detect updates. A similar headline inside the window is a duplicate. A follow-up with a different headline is new, and nothing links it to the earlier article.

Stories that come back after the window

In the 30-day capture, one story returned after 7 days or more. Rutgers University's "Scientists Move a Step Toward Quantum Computers That Can Correct Their Own Mistakes" appeared on 2026-10-09, and Technology Org ran the same headline on 2026-10-21, 12 days later. A 168-hour window would let it through. The other long gap, Quantum Zeitgeist headlines on 10-04 and 10-25 (21 days), was a false merge, not a return.

One case is an anecdote, not a rate. Your options: let it through, tag it "revisited" in your template, or lengthen the window. A window cannot exceed your plan's article history: 7 days on Starter, 30 days on Pro, per the pricing page.

Which key identifies an item

Pick the key before you build the ledger. A Google News item's guid is a long opaque id, and its link points at news.google.com, not the publisher. Feedmarkdown stores the feed guid as the item's external id (falling back to the link) and resolves the redirect to the publisher URL, as described in how to get the real article URL. It then hashes a normalized URL: https, no www., no fragment or trailing slash, utm_*, fbclid and gclid removed, query sorted. Key your own ledger on the publisher URL the same way. A common source of duplicate items in RSS-to-email setups is the same link returning with a different utm_ tag: an unnormalized key sees a new item. Our snippet below just cuts the query string, which is cruder than the server.

How to deduplicate an RSS feed across issues

First, deduplicate inside the issue: before you send, compare each item with the others in the same batch. Then keep a ledger of what went out, not of what you fetched. If you record at fetch time, an item you cut from today's issue can never come back. Record after the send.

The script is a small SQLite ledger. It checks an exact URL key first, with no window, then headlines inside the window. It uses the server's stopwords and thresholds but not simhash, and it strips the trailing " - Outlet" from Google News titles, which the server does not. We ran it on real headlines from our 2026-10-23 capture with placeholder URLs; the 10-21 item reuses the 10-19 link to exercise the exact check. The exact-URL check is not what the replay measured.

import hashlib, re, sqlite3, sys
from datetime import datetime, timedelta

WINDOW = int(sys.argv[1]) if len(sys.argv) > 1 else 48  # hours
STOP = set("a an the and or of to in on for at by with from as is are was were be it its this that "
           "after over into says said report reports new how why what update updates live".split())

def tokens(title):
    title = re.sub(r"\s[-|]\s[^-|]{1,40}$", "", title)  # drop trailing " - Outlet" or " | Outlet"
    return {w for w in re.findall(r"[a-z0-9]+", title.lower()) if w not in STOP}

def same_story(a, b):
    a, b = tokens(a), tokens(b)
    if len(a) < 4 or len(b) < 4:
        return False
    small = min(len(a), len(b))
    return len(a & b) / len(a | b) >= 0.6 or (small >= 5 and len(a & b) / small >= 0.85)

db = sqlite3.connect(":memory:")  # use a file in real life
db.execute("create table sent (url_hash text, title text, sent_at text)")

def send_issue(date, items):
    now, picked = datetime.fromisoformat(date + "T23:59:00"), []
    since = (now - timedelta(hours=WINDOW)).isoformat()
    for url, title in items:
        h = hashlib.sha256(url.split("?")[0].encode()).hexdigest()
        recent = [t for (t,) in db.execute("select title from sent where sent_at >= ?", (since,))]
        if db.execute("select 1 from sent where url_hash = ?", (h,)).fetchone():
            print(f"  skip (exact url)  {title[:60]}")
        elif any(same_story(title, t) for t in recent + [t for _, t in picked]):
            print(f"  skip (headline)   {title[:60]}")
        else:
            print(f"  send              {title[:60]}")
            picked.append((h, title))
    # record after the send, not after the fetch
    db.executemany("insert into sent values (?, ?, ?)", [(h, t, now.isoformat()) for h, t in picked])

A = "Paymentology Partners with Asia United Bank to Accelerate Digital Payments in the Philippines - FF News"
B = "Paymentology partners with Asia United Bank on digital payments - The Paypers"
for date, items in [
    ("2026-10-19", [("https://x.test/ff", A)]),
    ("2026-10-21", [("https://x.test/ff?utm_source=feed", A), ("https://x.test/ftf", "Kort to acquire fellow Canadian paytech AptPay - FinTech Futures")]),
    ("2026-10-22", [("https://x.test/paypers", B)]),
]:
    print(" issue", date)
    send_issue(date, items)

Console output of the ledger script at 48 and 72 hour windows

The Paymentology story sent on 10-19 comes back under another outlet's headline on 10-22, 72 hours later. At 48 hours it ships again. At 72 it is skipped. The repeated FF News URL on 10-21 is skipped at both, because the exact check ignores the window.

Weekly issues and several newsletters

A weekly issue needs a window at least as long as the issue, and it still needs dedup inside the issue. We did not test weekly issues. With several newsletters in one account, dedup_scope: feed keeps each newsletter's memory separate. org shares it, so a story sent in newsletter A is skipped in B. Running news digests for several clients covers the agency case.

What we did not test

Body (TLSH) matching and database lookups. Only headlines were replayed.
Exact URL repeats across real daily fetches.
What was delivered, and whether follow-ups got through.
Real subscribers: opens, replies, unsubscribes. We have none.
Weekly issues, Google Alerts sources, other languages or regions.
A live Feedmarkdown run beyond two runs minutes apart: no multi-day run, no headline-match test, one query.
Other tools' dedup features.

What Feedmarkdown does with the window

Feedmarkdown has the same two layers. The window runs from when we stored the item, not its publish date.

Field	What it does
`dedup_window_hours`	How far back headline and content matching look. Default 48 hours. Cannot exceed your history: 168 hours on Starter, 720 on Pro.
`dedup_scope`	`feed` compares against that feed's items. `org` compares against every feed in the account.

Exact id and URL repeats are checked against everything stored in the scope (this feed's items, or the whole account with dedup_scope: org), with no time window. Items older than your history never arrive as new. A feed update is PUT /v1/feed/{id} with a Bearer API key and a JSON body:

{ "dedup_window_hours": 72, "dedup_scope": "feed" }

Without a subscription, writes return 402 subscription_required; new accounts get a 7-day trial.

Your ledger and what Feedmarkdown keeps

Kept by	What
Feedmarkdown	An account-wide article store, kept for your plan's history (7 days Starter, 30 Pro), with URL hash, normalized headline and content hash
Feedmarkdown	Per feed, what each run produced: `new`, `duplicate` (pointing at the article it matched) or `error`
Feedmarkdown	The window and scope that decide what counts as seen
Your code	What each subscriber or list received, and when
Your code	The policy for follow-ups, "revisited" tags and resurfaced stories

So Feedmarkdown answers "is this story new to this feed or account?" before you pay for the scrape, and your sending code answers "did this reader already get it?"

What a live feed did on two runs

On 2026-11-01 (UTC) we created a pull feed on a Pro account over one Google News source, "quantum computing when:1d", with items_limit 10, dedup_window_hours 48 and dedup_scope: feed. We triggered a run with PUT /v1/feed/{id}/run, then triggered it again about four minutes later. The run records read:

Run	Found	New	Deduped	Scraped	Errors	Time
1, 23:43:45 UTC	10	10	0	10	0	about 20 s
2, 23:47:45 UTC	10	0	0	0	0	under 1 s

Run 2 found the same 10 items, treated none as new and scraped nothing. The items_deduped counter stayed at 0 on run 2 even though nothing was new; we observed that and did not look into why. The JSON Feed returned the 10 items from run 1. This shows exact repeats only. We then deleted the feed and the source.

We also tried a window above the plan's history:

curl -X PUT https://api.feedmarkdown.com/v1/feed/FEED_ID \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dedup_window_hours": 721}'

The API answered 402. That account was on Pro, whose history is 30 days:

{"error_code":"plan_limit_exceeded","error_message":"dedup_window_hours cannot exceed your 30-day article history (720 hours)"}

Skip the plumbing

Feedmarkdown reads Google News and Google Alerts sources, drops repeat headlines before it scrapes anything, compares scraped text afterward, and delivers each new story as Markdown by signed webhook or as JSON Feed 1.1 or Atom. The sent ledger stays in your code. You can start a feed in the dashboard and run it twice, as we did above, to see the "new to this feed" half working.