feat: agent discovery (Link headers, markdown negotiation, Content-Signal)#263
feat: agent discovery (Link headers, markdown negotiation, Content-Signal)#263sriramveeraghanta merged 8 commits intomasterfrom
Conversation
Brainstormed spec for three agent-discovery fixes: Content-Signal in robots.txt, Link response headers, and markdown content negotiation. Ephemeral — to be removed after implementation.
Task-by-task implementation plan for the three fixes in the companion spec. Ephemeral — removed as the final task.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughVitePress build configuration now includes a Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@vercel.json`:
- Around line 3-13: The Link response header rule for the catch‑all source
"/(.*)" needs to also emit a Vary: Accept header so caches distinguish HTML vs
Markdown responses; update the headers array entry for source "/(.*)" (the
object containing the Link header) to include another header with key "Vary" and
value "Accept" alongside the existing Link header so caches will vary by the
Accept request header.
- Around line 108-113: Add an explicit root rewrite entry before the existing
catch-all rule in the "rewrites" array so requests to "/" with Accept:
text/markdown are rewritten to "/index.md"; specifically, add a rule with
"source": "/" and the same "has" header condition (key: "accept", value matching
".*text/markdown.*") and "destination": "/index.md" placed above the existing
rule that maps "source": "/:path*" to "/:path*.md".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 963f30db-fd69-4548-9ee1-d87b11c74ebe
📒 Files selected for processing (3)
docs/.vitepress/config.mtsdocs/public/robots.txtvercel.json
| "headers": [ | ||
| { | ||
| "source": "/(.*)", | ||
| "headers": [ | ||
| { | ||
| "key": "Link", | ||
| "value": "</llms.txt>; rel=\"describedby\"; type=\"text/plain\", </api-reference/introduction>; rel=\"service-doc\"; type=\"text/html\", </sitemap.xml>; rel=\"sitemap\"; type=\"application/xml\"" | ||
| } | ||
| ] | ||
| } | ||
| ], |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
: "${PREVIEW_URL:?Set PREVIEW_URL to the Vercel preview origin, e.g. https://example.vercel.app}"
curl -fsSI "$PREVIEW_URL/api-reference/introduction" | tr -d '\r' | grep -i '^vary:.*accept'
curl -fsSI -H 'Accept: text/markdown' "$PREVIEW_URL/api-reference/introduction" | tr -d '\r' | grep -i '^vary:.*accept'Repository: makeplane/developer-docs
Length of output: 177
🏁 Script executed:
git ls-files | grep -i vercelRepository: makeplane/developer-docs
Length of output: 79
🏁 Script executed:
cat -n vercel.json | head -120Repository: makeplane/developer-docs
Length of output: 4281
Add Vary: Accept response header for content negotiated Markdown responses.
The rewrite rule at lines 108–113 serves different content (HTML vs. Markdown) based on the Accept header. Without Vary: Accept, caches may serve the wrong representation to clients requesting a different format.
Proposed fix
"headers": [
{
"key": "Link",
"value": "</llms.txt>; rel=\"describedby\"; type=\"text/plain\", </api-reference/introduction>; rel=\"service-doc\"; type=\"text/html\", </sitemap.xml>; rel=\"sitemap\"; type=\"application/xml\""
+ },
+ {
+ "key": "Vary",
+ "value": "Accept"
}
]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "headers": [ | |
| { | |
| "source": "/(.*)", | |
| "headers": [ | |
| { | |
| "key": "Link", | |
| "value": "</llms.txt>; rel=\"describedby\"; type=\"text/plain\", </api-reference/introduction>; rel=\"service-doc\"; type=\"text/html\", </sitemap.xml>; rel=\"sitemap\"; type=\"application/xml\"" | |
| } | |
| ] | |
| } | |
| ], | |
| "headers": [ | |
| { | |
| "source": "/(.*)", | |
| "headers": [ | |
| { | |
| "key": "Link", | |
| "value": "</llms.txt>; rel=\"describedby\"; type=\"text/plain\", </api-reference/introduction>; rel=\"service-doc\"; type=\"text/html\", </sitemap.xml>; rel=\"sitemap\"; type=\"application/xml\"" | |
| }, | |
| { | |
| "key": "Vary", | |
| "value": "Accept" | |
| } | |
| ] | |
| } | |
| ], |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@vercel.json` around lines 3 - 13, The Link response header rule for the
catch‑all source "/(.*)" needs to also emit a Vary: Accept header so caches
distinguish HTML vs Markdown responses; update the headers array entry for
source "/(.*)" (the object containing the Link header) to include another header
with key "Vary" and value "Accept" alongside the existing Link header so caches
will vary by the Accept request header.
| "rewrites": [ | ||
| { | ||
| "source": "/:path*", | ||
| "has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }], | ||
| "destination": "/:path*.md" | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
: "${PREVIEW_URL:?Set PREVIEW_URL to the Vercel preview origin, e.g. https://example.vercel.app}"
curl -fsSI -H 'Accept: text/markdown' "$PREVIEW_URL/" | tr -d '\r' | grep -Ei '^HTTP/|^content-type:'
curl -fsS -H 'Accept: text/markdown' "$PREVIEW_URL/" | head -n 20Repository: makeplane/developer-docs
Length of output: 177
🏁 Script executed:
cat -n vercel.json | sed -n '100,120p'Repository: makeplane/developer-docs
Length of output: 566
🏁 Script executed:
# Search for all rewrite rules in vercel.json to understand the full context
rg -A2 '"source"' vercel.json | head -60Repository: makeplane/developer-docs
Length of output: 1888
🏁 Script executed:
# Check if there's any rewrite rule for the root path "/" already
rg '"source".*"/"' vercel.jsonRepository: makeplane/developer-docs
Length of output: 50
Add explicit root rewrite for Markdown content negotiation.
The catch-all rule /:path* → /:path*.md fails for the root path / because the path variable captures an empty string, resulting in /.md instead of /index.md. Add an explicit root rewrite before the catch-all to handle requests to / with Accept: text/markdown header:
Proposed fix
"rewrites": [
+ {
+ "source": "/",
+ "has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }],
+ "destination": "/index.md"
+ },
{
"source": "/:path*",
"has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }],
"destination": "/:path*.md"
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "rewrites": [ | |
| { | |
| "source": "/:path*", | |
| "has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }], | |
| "destination": "/:path*.md" | |
| } | |
| "rewrites": [ | |
| { | |
| "source": "/", | |
| "has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }], | |
| "destination": "/index.md" | |
| }, | |
| { | |
| "source": "/:path*", | |
| "has": [{ "type": "header", "key": "accept", "value": ".*text/markdown.*" }], | |
| "destination": "/:path*.md" | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@vercel.json` around lines 108 - 113, Add an explicit root rewrite entry
before the existing catch-all rule in the "rewrites" array so requests to "/"
with Accept: text/markdown are rewritten to "/index.md"; specifically, add a
rule with "source": "/" and the same "has" header condition (key: "accept",
value matching ".*text/markdown.*") and "destination": "/index.md" placed above
the existing rule that maps "source": "/:path*" to "/:path*.md".
2 participants
Summary
Content-Signal: search=yes, ai-train=yes, ai-input=yestorobots.txtLinkresponse headers on all paths advertisingllms.txt(describedby),/api-reference/introduction(service-doc), and/sitemap.xml(sitemap)Accept: text/markdownis sent — static.mdfiles copied intodistby a VitePressbuildEndhook, routed via a Vercelrewritesheader matcherTest plan
Local:
pnpm check:formatpassespnpm buildsucceeds and produces 337.mdfiles alongside.htmlindist/docs/.vitepress/dist/robots.txtcontains theContent-SignallinePreview deploy:
curl -I https://<preview>.vercel.app/shows aLink:header with all three relationscurl https://<preview>.vercel.app/still returns HTML (regression check)curl -H "Accept: text/markdown" https://<preview>.vercel.app/api-reference/introductionreturns markdown body andContent-Type: text/markdown; charset=utf-8— if returned astext/plain, add an explicit header override in a follow-upSummary by CodeRabbit
Chores