LinkDigest ·

Did the agent read it? Measuring whether AI reads social links

Short answer: often it doesn't. We hid codes in 7 social posts of our own and asked five AI assistants to find them: 27 answers, 3 codes found, and none from ChatGPT. Across 145 public posts on nine platforms, a plain HTTP fetch opened 25% of the link forms people paste. The full study is an open preprint: doi:10.5281/zenodo.24422487. The harness, the benchmark and every result are on GitHub and Hugging Face.

Disclosure: the author operates LinkDigest. ReadProof, the harness and benchmark below, is independent of it, and no LinkDigest production route, account or customer data is used in any reported result.

The same Douyin share link read two ways: a plain fetch gets a 72,914-byte verification page with no visible text; ReadProof reads the post through Douyin's own player and returns channel-located records with verified quotes.

Why measure the read

Search agents are now near the ceiling of open-web text benchmarks. On social media they often fail before any reasoning starts:

  • The link returns a verification page. For Douyin, that's a 72,914-byte page with zero visible characters.
  • It returns a login wall or an expired share token.
  • When the post is served, the answer often sits in speech, burned-in text, carousel images or comments, which a text pipeline never reads.

Benchmarks that hand the system a fixed corpus or the video itself can't see any of this, so we measured the read itself. (The short version of why a chat assistant can't open these links is here.)

What we measured (21 November 2026)

  • Track A, access. 145 public posts on Douyin, Xiaohongshu/RedNote, Bilibili, YouTube, TikTok, Weibo, Toutiao, X and WeChat, in up to four link forms each: desktop, mobile, the app's share text, and a platform variant. 522 link forms in all, scored per form, because the same post can open from one form and fail from another.
  • Track K, planted canaries. Seven posts on our own accounts. Each has random codes hidden in exactly one channel: spoken only, on screen only, in the caption, in the first comment, or inside an image. A correct code proves a channel was read; a different code proves a fabrication. The answer key was committed by its SHA-256 hash before anything ran.

Results

System Link forms served Canary codes (of 28)
ReadProof, user-initiated profile 81.8% 9
Headless browser 66.5% 3
Reader service (Jina Reader) 64.9% 5
ReadProof, strict robots.txt profile 55.1% 8
Plain HTTP fetch 25.4% 3

Served counts machine-translated pages of the post. One of each system's correct canary answers is an absent-channel control, a question whose right answer is "there is none".

The commercial assistants, asked in their default web modes on the same day:

Assistant Posts answered Codes found
ChatGPT 7 0
Gemini 7 2, both on YouTube
Tencent Yuanbao 7 1, a Bilibili caption
Perplexity 5 0
Qwen 1 0

Four findings

  1. The failures come before reasoning. With the model held fixed, the systems differed by up to 56 points of coverage, and every gap was about what each one obtained.
  2. No single route reads every platform. The declared harness won on Douyin (through Douyin's own player), X and Xiaohongshu. A headless browser won on Bilibili, which challenges a declared bot but serves browsers an English translation.
  3. Honesty has a price, and it is uneven. Obeying robots.txt strictly cost 26.7 points of coverage, and four of nine platforms dropped to zero.
  4. Nobody invented a code, but almost nobody read the video. Among the open systems, spoken and on-screen text were reached only through the platforms' own players. On YouTube, only Google's own assistant read them. Comment codes were read only on X.

If you build agents

  • Check what your agent actually received before it answers. A 200 OK is not a post.
  • Treat a wall as a typed result, not as text. Then a login page never becomes an answer.
  • Read speech and on-screen text where the platform serves them. Otherwise that evidence doesn't exist for your agent.
  • Verify quotes in code. A prompt can ask for exact quotes; only code can refuse an inexact one.

Cite

Chew, L. N. (2026). Did the Agent Read It? ReadProof: Access-Aware Reading, Channel-Located Evidence and Verified Quotes for Agentic Search over Social Media. Preprint. doi:10.5281/zenodo.24422487

Related


This is the write-up behind LinkDigest, which does the above as a hosted service — with an MCP server for Claude Code and Cursor, a REST API, and 10 free credits to try it.