How to feed news to an LLM without paying for HTML

Getting news into an LLM takes four steps: fetch the stories, turn each page into clean Markdown, drop duplicates, and hand the result to your model. Dedupe is listed third, but half of it has to run before you pay for a scrape. More on that in step 3. An RSS feed alone covers only the first step, and only halfway. We ran each step on real data and counted the tokens.

We ran everything on 2026-10-13 from the Netherlands. The headline result: on three real articles, the raw HTML was 74,201 to 632,626 tokens, and the Markdown was 486 to 2,223. That is n=3, one tokenizer and one day, so read it as an example, not a benchmark. The method is below so you can rerun it on your own articles.

The source is a Google News search feed, built as in our Google News RSS feed guide.

The four steps at a glance

Dedupe shows up twice in the diagram because it runs on both sides of the scrape.

Step	What goes wrong with raw RSS	What to do
Fetch	Every link is a Google redirect, not the publisher page	Resolve each link to the real URL
Clean	The feed has a headline and a link, no article text	Fetch the page and convert it to Markdown
Dedupe	One story arrives from many outlets and you pay for each copy	Match headlines before scraping, compare text after
Deliver	Your model node needs one clean record per story	Push by webhook or let the model pull a feed

The order matters for cost. Anything you can drop from the feed metadata alone should be dropped before you scrape. Scraping is the paid step, and every LLM call after it is a second bill.

What a news RSS item gives an LLM

This is one real <item> from https://news.google.com/rss/search?q=openai+when:1d&hl=en-US&gl=US&ceid=US:en, fetched on 2026-10-13. The feed had 100 items. It is a rolling one-day window, so if you fetch it later you will get different items and counts. We shortened the ids with ….

What is in it:

• A title, with the publisher name appended (- Mashable).
• A link and guid that point to news.google.com/rss/articles/<id>.
• A pubDate.
• A description that is HTML-escaped: an <a> tag around the headline, then the publisher name.
• A source element with the publisher's home page.

What is missing:

• The article. The description repeats the headline. There is no summary and no body. For this item that is 185 tokens, and almost all of them are the long redirect id inside the link.
• The publisher URL. You get a Google link. We traced where it goes, and curl -L does not reach the article.
• An author. None of the six element types above carries one.
• Any cross-item signal. Ten of the 100 items mentioned Meta's "Muse Wonder" model. The feed does not tell you that eight of them are the same launch story.

If you feed this item to a model, it can only rewrite the headline. Everything else needs a fetch.

Step 1: fetch and resolve the real URL

A Google News link has to be turned into the publisher's URL before you can fetch the page. The redirect chain ends on a Google page, so a plain redirect-following request never gets there. Our redirect post shows the two-request method and a Python script. We used that script, unchanged, for every URL below. It uses an undocumented Google endpoint, so re-test it before you depend on it.

If you wonder whether there is an official API for any of this, there is not one for Google News. We covered that in Is there a Google News API?.

Google Alerts is the other feed source. It is better for narrow, named topics and has different quirks. Our Google Alerts RSS guide explains when to pick which.

We picked three articles from three publishers in the feed and resolved them:

Publisher	Resolved URL (shortened)
Tom's Hardware	tomshardware.com/tech-industry/artificial-intelligence/ai-causes-uproar-in-mathematics-…
Mashable	mashable.com/tech/openai-chatgpt-intelligent-ui-interactive-visual-graphics
CIO	cio.com/article/4233864/stop-sending-me-engineers-what-openai-anthropic-and-microsoft-misread-about-enterprise-ai.html

Step 2: turn the page into Markdown

An LLM reads text. A news page is mostly not text: navigation, scripts, ad slots, related links, cookie banners and styles. All of that costs tokens and sits in the context next to the three paragraphs you care about.

First the raw fetch, with a plain browser-style user agent:

curl -sL -A "Mozilla/5.0" --max-time 40 -o article.html \
  -w "%{http_code} %{size_download}\n" "$ARTICLE_URL"

Then the Markdown call. We used WebCrawlerAPI's /markdown endpoint, the same one Feedmarkdown uses. It scrapes the page and has an LLM clean it down to the main content:

curl -s -X POST https://api.webcrawlerapi.com/markdown \
  -H "Authorization: Bearer $WEBCRAWLERAPI_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$ARTICLE_URL\"}"

The response is JSON with a markdown field. Here is what we measured on 2026-10-13:

Article	HTML bytes	HTML tokens	Stripped text tokens	Markdown tokens	HTML to Markdown	webcr -m tokens
Tom's Hardware	1,891,189	632,626	3,402	1,611	393x	3,342
Mashable	223,033	74,201	1,706	486	153x	1,823
CIO	277,735	95,088	2,562	2,223	43x	5,380

Tokens are counted with tiktoken 0.7.0, encoding o200k_base. The median Markdown size is 1,611 tokens and the median ratio is 153x.

A few things in that table:

• The HTML ratio is a strawman. Nobody should send 632,626 tokens of HTML to a model. Most of it is inline scripts and styles. That is why we added the stripped-text column: remove <script>, <style>, <noscript> and <svg>, then take the text. That is a fairer baseline.
• Against stripped text, Markdown is smaller by 1.2x to 3.5x (3,402 vs 1,611, 1,706 vs 486, 2,562 vs 2,223). The stripped text still contains menus, related-story titles and footers. The CIO page is the closest call.
• webcr -m is not the same as /markdown. It is the main-content mode of the plain scrape endpoint (/v2/scrape), with no LLM cleaning step. It returned more text on all three: 3,342, 1,823 and 5,380 tokens. We have not read all of it to say what the extra is.
• Markdown keeps links. The Mashable output has inline links like [OpenAI](https://mashable.com/category/openai). Those cost tokens too. Strip them if your model does not need them.

We read the Mashable output in full and the first lines of the CIO output. It matched the articles. We did not check each Markdown file against its page for dropped paragraphs. If you summarize with a model, spot-check yours.

How we measured the token counts

The steps, so you can repeat them:

1. Fetched the feed, took 100 items, resolved links with the redirect post's script.
2. Fetched each page with curl -sL -A "Mozilla/5.0" and kept the file.
3. Called /markdown twice per URL to see run-to-run variance.
4. Ran webcr "$URL" -m once per URL for the extra column.
5. Counted tokens with this script:

import sys, tiktoken
from bs4 import BeautifulSoup

enc = tiktoken.get_encoding("o200k_base")

def tok(s):
    return len(enc.encode(s, disallowed_special=()))

def stripped(html):
    soup = BeautifulSoup(html, "html.parser")
    for t in soup(["script", "style", "noscript", "svg"]):
        t.decompose()
    return " ".join(soup.get_text(" ").split())

for name in sys.argv[1:]:
    html = open(f"{name}.html", encoding="utf-8", errors="replace").read()
    md = open(f"{name}.md").read()
    print(name, tok(html), tok(stripped(html)), tok(md))

What went wrong or limits the result:

• Some publishers blocked or timed out plain curl. We tried six articles. VentureBeat answered 429 with a Vercel "Security Checkpoint" page, and XDA Developers timed out. We dropped both from the table. The /markdown call returned text for both.
• curl sees the HTML as served. Pages that fill in content with JavaScript will look smaller to curl than they do in a browser. We did not test that.
• LLM-cleaned Markdown is not deterministic. On the two runs, Tom's Hardware and CIO gave identical token counts. Mashable gave 486 and 487. VentureBeat's output was 10,471 bytes on one run and 9,264 on the other. IT Pro's was 2,767 and 2,751. Expect small differences, and occasionally larger ones.
• Only one tokenizer. o200k_base is an OpenAI encoding. Claude, Gemini and Llama tokenizers count differently, so absolute numbers will differ. We did not cross-check.
• Three articles, one day, one query. Your pages will differ.
• We did not test answer quality. We did not check whether a model summarizes Markdown better than stripped HTML. The numbers above show size only.

Step 3: dedupe before you spend tokens

News repeats. On the same query, ten of 100 items mentioned Meta's Muse Wonder launch. Eight looked like launch reports by their titles. We resolved all eight links and sent them through /markdown:

Publisher	Markdown tokens
Bloomberg (video page)	96
Reuters	336
TIKR	597
Chosun	685
Neowin	756
The Tech Portal	1,069
Mashable	1,212
bloomingbit	2,268

That is 7,019 Markdown tokens and eight scrapes for one story. If you summarize each copy, add eight model calls. Dedupe first and you scrape one copy. The size of that copy depends on which one you keep: from 96 tokens (a video page that is nearly empty) to 2,268.

Treat that as an illustration. We scraped all eight to measure them, we did not summarize them, and the Bloomberg row shows that "one story" does not mean "eight equal articles". We measured the cost per 1,000 articles in What it costs to summarize 1,000 news articles with an LLM.

Two stages do the work, and they run on either side of the scrape:

1. Before the scrape, compare the cheap fields: the normalized URL and the headline. This catches syndicated copies and near-identical headlines without paying for any page.
2. After the scrape, compare the article text. This catches a reprint under a rewritten headline.

Neither stage is enough alone. Headline matching merges some stories that differ and misses rewrites. Text comparison needs the body, which is the thing you are trying not to buy twice. We went through real headlines and scores in Deduplicating news: why matching headlines isn't enough. Start there if you build your own.

A minimal first version, for a script of your own, is a seen-set on the resolved URL:

import hashlib
from urllib.parse import urlparse, parse_qsl, urlencode

TRACKING = {"fbclid", "gclid"}

def url_key(url):
    p = urlparse(url)
    host = p.netloc.lower().removeprefix("www.")
    query = sorted(
        (k, v) for k, v in parse_qsl(p.query)
        if not k.startswith("utm_") and k not in TRACKING
    )
    clean = f"https://{host}{p.path.rstrip('/')}?{urlencode(query)}"
    return hashlib.sha256(clean.encode()).hexdigest()

seen = set()

def is_new(url):
    key = url_key(url)
    if key in seen:
        return False
    seen.add(key)
    return True

This only catches the same URL arriving twice. Across outlets every URL is different, so it will not catch the Muse Wonder case. It is a floor, not a dedup system.

Step 4: hand it to the model

Once you have one Markdown record per story, there are three ways to get it to a model.

• Receive a webhook. A server pushes each new article to your endpoint. This suits an n8n or Make workflow, or a small service that calls a model on arrival.
• Poll a feed. You fetch a JSON Feed or Atom document on your schedule and take what is new. This suits a daily digest or a batch job.
• Batch to an index. You load the Markdown into a vector store or a search index for retrieval later. Either of the first two shapes can feed this.

Whichever you pick, keep the metadata with the text. Store the url, the source and the published_at with every chunk. Without them a model cannot cite its answer, and you cannot expire stale news from the index. Chunk by heading: Markdown gives you ## boundaries for free, which raw HTML does not.

If you receive webhooks, verify the signature before you trust the body. Feedmarkdown's webhook signs the raw body with HMAC-SHA256 and sends it in the X-Feedmarkdown-Signature header as sha256=<hex>:

import hashlib, hmac

def verify(secret: str, body: bytes, header: str) -> bool:
    digest = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(f"sha256={digest}", header)

# verify("YOUR_WEBHOOK_SECRET", raw_request_body, request_headers["X-Feedmarkdown-Signature"])

The stable payload fields are title, url, source, author, published_at and markdown. For a full workflow that turns those into an email, see how to build a daily news newsletter from feeds, which pulls a JSON Feed into n8n and an LLM node.

Where n8n, Make or Zapier fit

These tools do not replace any of the four steps. They are the glue after them. A webhook trigger node receives the article. An HTTP node polls a feed. An LLM node takes the Markdown. The cleaning and deduplication have to happen before the trigger, or you pay for them inside the workflow, one node run at a time. We have not checked the task or execution limits of n8n or Make for this post. For Zapier, we measured the RSS trigger's polling gaps and task use on a Pro trial in what the Zapier RSS trigger can't do with news feeds. Look at your plan before you wire in a high-volume feed. A working n8n example that verifies the signature and posts each story to Slack is in n8n RSS feed to Slack with full article text.

When a news API is the better choice

A Google News based pipeline gives you what Google News indexes, which is recent coverage from a very wide set of outlets, found by search query. It does not give you a full historical archive, sentiment scores, entity tagging, or a contract that guarantees coverage. If you need those, a dedicated news data vendor is the better pick. We did not check any vendor's current pricing or features for this post. Our separate comparison of Newscatcher and NewsData.io has dated prices; compare vendors on your own query volume.

If you only need recent stories on a topic as clean text for a model, the pipeline above is enough.

Skip the plumbing

Feedmarkdown runs these four steps for you. It takes Google News and Google Alerts as sources, resolves the redirect links, drops duplicates by headline before it scrapes and by content after, converts each article to Markdown, and sends it as an HMAC-signed webhook, or serves it as a JSON Feed 1.1 or Atom feed you pull. It does not read plain RSS or arbitrary site URLs yet. If your sources are not on Google News or Google Alerts, it will not fit today. Plans and check intervals are on the pricing section.