Deduplicating news: why matching headlines isn't enough

News deduplication with headlines alone catches syndicated copies and merges some stories that are not the same. Matching article text catches reprints under rewritten headlines, but it needs the article body, and the body costs a scrape. So do the cheap checks first, scrape what is left, then compare text.

We ran one story through both methods on 2026-10-12: Pope Leo's announcement that he will have lung surgery. A Google News search feed returned 100 items about it. Below are the real headlines, the scores, and the cases where each method failed. If you are building a feed that goes into a daily newsletter or an LLM, this is the part that decides what you pay for.

The code we ran is the dedup code from Feedmarkdown's server. The thresholds are our choices, not universal ones. For how common repeats are across ordinary queries, see how often a Google News RSS feed repeats the same story.

One story, six headlines

We fetched the feed with the URL form from our Google News RSS guide, resolved every link to the publisher URL (see how to get the real article URL), and scraped 74 of the 100 with webcr -m. The rest were video pages. Six of them:

The facts and quotes are the same across them. The headline, the byline and the page furniture are not. That split is what dedup has to work with.

What exact matching catches

The first check is exact. Feedmarkdown normalizes the URL, then hashes it. The normalizer forces https, drops www., removes the fragment and the trailing slash, strips utm_* and a list of tracking parameters such as fbclid and gclid, and sorts the query. We ran it on four inputs:

http://www.Example.com/news/story/?utm_source=newsletter&id=7&fbclid=abc#comments  -> https://example.com/news/story?id=7
https://example.com/news/story?id=7&utm_medium=email                              -> https://example.com/news/story?id=7
https://example.com/news/story/?id=7                                               -> https://example.com/news/story?id=7
https://m.example.com/news/story?id=7                                              -> https://m.example.com/news/story?id=7

The first three collapse to one hash. The m. subdomain does not, because we don't guess which subdomains are mirrors. This only helps when the same URL arrives twice, for example from two feeds. Across outlets every URL is different, as the table shows. This is also why resolving Google's redirect links comes first: hashing news.google.com URLs tells you nothing.

Headline matching: how it works and where it misses

Feedmarkdown compares headlines after normalizing them: lowercase, punctuation removed, a trailing | Publisher of four words or fewer removed, and common words dropped (the, to, says, new, live and similar). Two headlines are the same story if any of these holds:

• Token Jaccard is at least 0.6. That is shared tokens divided by all distinct tokens.
• Containment is at least 0.85, when the smaller headline has at least 5 tokens. This catches "AP: headline" against "headline".
• A 64-bit simhash of the tokens differs by 3 bits or fewer.

A headline with fewer than 4 tokens never clusters. "Live updates" is too generic to merge with anything.

Of 15 pairs among those six, 5 cluster and 10 do not. Reuters, "Pope Leo says he will undergo surgery for small growth on lung", normalizes to pope leo he will undergo surgery small growth lung. It scores 0.31 against AP and 0.54 against NBC4. It joins no cluster with any of the other five. The Guardian and CNN miss each other at 0.50. Same facts, different words.

Feedmarkdown compares each new headline against every earlier one in the window, not just one pair. We replayed all 100 items, oldest first, through the real TitleIndex: 67 matched an earlier item and 33 did not. Reuters was one of the 33.

False merges

We looked for headlines that cluster but are different stories. Across the Pope Leo set, the closest case was Newsweek's "What Is a Lung Nodule? Pope Leo To Undergo Surgery To Remove Lump", which merged into the announcement cluster. Its body opens with the same announcement, so we would call that a fair merge, but the headline promises an explainer.

A real false merge needed a different search. In 302 headlines from five other searches that day (earnings, stocks to watch, storms), two MarketBeat items cluster:

• QuantumScape Q3 2026 Earnings Date and Estimates
• B2Gold Q3 2026 Earnings Date and Estimates

Jaccard is 0.71, so they cluster. Two different companies. The same happens with marketscreener's Swiss stocks - Factors to watch on October 12 and UK stocks-factors to watch on October 12. Templated headlines are the weak spot. Because stage 1 runs before the scrape, the second article is dropped and never read.

The fix on your side is a narrower window, or an org-wide scope only where you want it. Raising the Jaccard threshold also helps, but then you miss more real rewrites. We did not tune the thresholds against a labelled set, so we can't give you a false-merge rate.

Fingerprinting the article text

Wire copy is reprinted. Headlines, bylines and the box of "related stories" around it are not. So the text of the story is more stable than the headline.

Raw page text does not work, because a big publisher's navigation can be longer than the story and would make every article from that site look alike. Feedmarkdown first reduces the scraped Markdown to prose: lines of at least 12 words that are not mostly link text, and only if there are at least 400 characters in total. Headings, tables and menus are dropped. Then it computes a TLSH digest of that prose.

TLSH is a locality-sensitive hash. Unlike SHA-256, similar inputs give similar digests, and you compare two digests to get a distance. 0 means the prose is identical. Larger means more different. Oliver, Cheng and Chen published it in 2013 for security and digital forensics, not for news. That is why few people write about it for text dedup. Simhash, the other hash in this post, goes back to Charikar (2002), and Google reported using it to find near-duplicate web pages when crawling (Manku, Jain and Das Sarma, 2007). We use simhash on short headlines and TLSH on long bodies.

The default match threshold is 40. Distances between nine of the Pope Leo bodies:

Three things stand out:

• NBC4 Washington and NBC 6 South Florida are distance 0: the same text on two station sites. The LA Times and ABC4 Utah copies are 29 and 34 from NBC4. Those are reprints, and they match.
• NPR against WOSU is 39, a match by one point. NPR against OPB is 92 even though both carry the identical headline and the same story. OPB's page prose differs enough to push it past 40. Wire copy that is lightly edited can fall outside the threshold.
• Reuters is 96 or more from everything, and AP is 52 from the LA Times and 66 from NBC4. Those are independent write-ups, which TLSH is right not to merge.

What content fingerprinting misses

It finds reprints. It does not find the same event reported separately. Reuters, CNN and the Guardian each wrote their own piece, and their bodies are far apart. If you want one item per event in a digest, that is event clustering. It is a different problem, and Feedmarkdown does not claim to solve it.

It also returns nothing when there is too little prose. In our run, 1 of the 74 scraped pages produced no digest: The Washington Post's, a 338-character page that held no story text. An article with no digest can't be compared in stage 2 and can only be dropped by its headline.

Then there is the threshold. A lower value means fewer false merges and more missed reprints. When we replayed the Decart story below at 60 instead of 40, one more pair merged. At 40 it did not.

We ran a second story the same day to check the rewritten-headline case: Amazon's reported talks to buy Decart, from 20 hand-picked items. Reuters, SiliconANGLE, PYMNTS, Techzine and others each rewrote the Wall Street Journal scoop. Of those pairs, the lowest TLSH distance was 55, between two auto-generated posts from the same publisher, GuruFocus, with different headlines. The closest pair from different outlets was 80. At threshold 40 stage 2 caught nothing in that set.

On both stories, stage 1 did the work and stage 2 added nothing at threshold 40. We could not find a pair on the day that headline matching missed and text matching caught at the default. Stage 2 exists for that case, and we did not see it in this sample.

Why dedup has to run in two stages

Text comparison needs Markdown, and Markdown needs a paid scrape. If you only compare text, you scrape every item to find out that most were copies. So the order is:

1. Fetch every source. Drop anything older than the history window.
2. Stage 1, on feed metadata only: repeats inside the run, exact id or URL-hash repeats against history, then headline clusters against the dedup window.
3. Scrape what is left.
4. Stage 2: TLSH against recent bodies and the current batch.
5. Persist, then deliver.

In our replay of the 100 Pope Leo items, 67 matched an earlier headline, so 33 would have been scraped. That is a count from real functions on real headlines, not a feed run on the product, and it ignores URL repeats and the window. Treat it as the shape of the saving. We did not measure cost, so we don't quote one.

Two details matter if you build this:

• The earliest publisher is the canonical copy. Candidates are sorted oldest first before dedup. In our replay the first item in one of the biggest headline clusters was Vatican News, and later items matched against it. Earliest is not the best copy, only the first one seen.
• A late original does not replace a copy. If a copy was stored in an earlier run and the original arrives afterwards, inside the dedup window, the original is the one marked as duplicate. Stage 1 only adds stored articles to the headline index and points new matches at them. It never swaps a stored article out.

An article already scraped by another feed in the same org is reused rather than scraped again.

Windows and scope

You can't compare against all of history, and an old story that resurfaces is arguably new. Feedmarkdown compares inside a window.

Setting	What it does	Default
dedup_window_hours	How far back to look for the same story	48, up to the plan's history
dedup_scope	feed dedups against that feed's history, org against every feed in the organization	feed

For how to pick the window for a newsletter, with replayed numbers, see how to stop sending subscribers the same story twice.

If you build this yourself

• Normalize and hash URLs first. Resolve redirect links before you do.
• Normalize headlines, and set a minimum token count so generic ones never merge.
• Hash prose, not the raw page. Menus and related-story rails will otherwise make a whole site look the same.
• Scrape after the cheap checks, not before. What dedupe saves in LLM spend is measured in our cost post.
• Pick a window. Pick a scope on purpose.
• Log why each item was dropped, and what it matched. Without that you can't debug a false merge like QuantumScape and B2Gold.
• Test on your own feed. Our thresholds fit our data, not yours.

Skip the plumbing

Feedmarkdown runs this pipeline for Google News and Google Alerts sources. Stage 1 drops repeats on feed metadata before any scrape, headline clusters stay inside dedup_window_hours, and stage 2 compares the TLSH of scraped text. Duplicates carry the id of the article they matched, and each run records how many items it found, kept, deduplicated and scraped. We do not claim an accuracy figure, and we do not merge independently reported stories. You get the result as full-text Markdown by signed webhook, or pull it as JSON Feed 1.1 or Atom. Start on the home page or see how it works.