What it actually takes to read a Xiaohongshu post from a server
Every claim here was checked against a live link. Where something doesn't work, it says so.
Paste a Xiaohongshu link into an AI coding agent and it sees nothing. Same for Douyin. The usual answer — "just use yt-dlp" — is half true in a way that wastes an afternoon, so here is the whole picture.
yt-dlp only reads half of Xiaohongshu
XiaoHongShuIE reads note.video.media.stream. That is a video note. Image
notes — 图文, a caption plus a stack of photos — are the majority of the
platform and usually the ones worth reading, and the extractor has nothing to
say about them.
Getting those means parsing the note payload out of the page itself: the state
blob the page ships, and the imageList inside it. Reusing yt-dlp's own
js_to_json and traverse_obj for that keeps you aligned with upstream when
the page shape shifts, which it does.
The page is different depending on who you say you are
This one cost me an embarrassing amount of time.
Request a note with a mobile user agent and you get 202KB of app-download
shell whose <title> is just the site name. Request the same URL with a
desktop user agent and you get 85KB containing the real note.
Both are HTTP 200. Nothing tells you that you got the wrong one except that the content isn't there.
Cold requests are rejected
Fetching a note URL directly, with no prior session, does not work. Fetch the
/explore feed first, keep the cookie jar it gives you — acw_tc,
abRequestId — and then the note loads.
No account, no credentials, no API key. Just the same two-step a browser performs without you noticing.
Douyin does not negotiate
Douyin refuses anonymous requests outright: captcha on the web page, 403 from the APIs. There is no user-agent trick here. That path needs cookies from a logged-in session, and yt-dlp does not support the platform at all.
YouTube blocks your server, not your laptop
The single most common "it worked locally and broke in production" report. YouTube blocks datacenter IP ranges, so a yt-dlp fetch that is perfect on your machine fails from EC2, App Runner, Lambda or anywhere else you deploy — and no amount of configuration fixes an IP-range block.
The workable fallback is a model that watches the video and returns a transcript, which costs meaningfully more than parsing captions and is worth measuring separately.
Where it stops
Being honest about this saves everyone time:
- Bilibili returns HTTP 412 to a datacenter address. Needs a residential proxy.
- Instagram and Facebook need a logged-in session for most posts.
- TikTok resolves fine but rate-limits under load; a frame fetch can come back 403 mid-job.
The part nobody mentions: failure has to be loud
The bug that taught me the most had nothing to do with extraction.
A note was digested during a provider rate-limit storm. Every vision batch
failed, the per-batch error handler swallowed each one, and the job returned
images: 0, ocr: 0 — a perfectly well-formed, completely empty result. It was
then written to a shared cache with a 30-day TTL.
Long after the underlying problem was fixed, that link still returned nothing, because a cached answer is cheaper to serve than to recompute. The pipeline was healthy. The cache was serving a fossil, and nothing downstream could tell the difference between an empty result and an easy one.
Two things came out of that:
- A step that loses content has to say so. The result now carries what failed, so a thin answer is distinguishable from a genuinely short post.
- Thin results get a short TTL, not the full one. Not "don't cache" — a link that fails every time would then re-run the full paid pipeline on every request forever. An hour bounds the staleness and the spend together.
If you are building anything that caches derived content, that second one is the trap. The obvious fix is the expensive one.
I maintain LinkDigest, which does the above as a hosted service with an MCP server for Claude Code and Cursor. The findings are the same whether you use it or write your own.