-
Notifications
You must be signed in to change notification settings - Fork 45
feat(mcp): MCP server (search + publish) & fix empty English search index #365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Crokily
wants to merge
2
commits into
main
Choose a base branch
from
feat/Crokily/mcp-server
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| import { getPublicOrigin } from "mcp-handler"; | ||
|
|
||
| const corsHeaders = { | ||
| "Access-Control-Allow-Origin": "*", | ||
| "Access-Control-Allow-Methods": "GET, OPTIONS", | ||
| "Access-Control-Allow-Headers": "authorization, content-type", | ||
| }; | ||
|
|
||
| export function GET(request: Request): Response { | ||
| return Response.json( | ||
| { | ||
| resource: `${getPublicOrigin(request)}/api/mcp`, | ||
| bearer_methods_supported: ["header"], | ||
| scopes_supported: ["publish"], | ||
| }, | ||
| { headers: corsHeaders }, | ||
| ); | ||
| } | ||
|
|
||
| export function OPTIONS(): Response { | ||
| return new Response(null, { status: 204, headers: corsHeaders }); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| import { createMcpHandler } from "mcp-handler"; | ||
| import { protectMcpHandler } from "@/lib/mcp/auth"; | ||
| import { createMcpPostHandler } from "@/lib/mcp/request"; | ||
| import { registerMcpTools } from "@/lib/mcp/tools"; | ||
|
|
||
| export const runtime = "nodejs"; | ||
| export const maxDuration = 60; | ||
|
|
||
| const mcpHandler = protectMcpHandler( | ||
| createMcpHandler( | ||
| registerMcpTools, | ||
| { | ||
| serverInfo: { name: "involutionhell", version: "0.1.0" }, | ||
| instructions: | ||
| "Use this server to search the InvolutionHell documentation community and publish lightweight Markdown posts. Search is public and supports Chinese or English article indexes. Publish writes to the authenticated user's InvolutionHell account and requires an Authorization Bearer satoken.", | ||
| }, | ||
| { | ||
| basePath: "/api", | ||
| disableSse: true, | ||
| sessionIdGenerator: undefined, | ||
| maxDuration, | ||
| }, | ||
| ), | ||
| ); | ||
|
|
||
| export const GET = (request: Request): Promise<Response> => mcpHandler(request); | ||
| export const POST = createMcpPostHandler(mcpHandler); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| # MCP Auth 升级方案 | ||
|
|
||
| ## 当前基础 | ||
|
|
||
| MCP 的公开 `search` 不需要登录,`publish` 使用 `Authorization: Bearer <satoken>`。Next.js 侧已经准备好两处稳定边界: | ||
|
|
||
| 1. `verifyToken` 是唯一 token 校验入口,当前调用 Spring Boot `/auth/me`; | ||
| 2. `/.well-known/oauth-protected-resource` 已存在,当前声明 resource 与 `publish` scope,等授权服务器上线后补 `authorization_servers`。 | ||
|
|
||
| 工具层不判断 token 类型,只使用鉴权层给出的已验证身份和 backend headers。因此 satoken 与 OAuth 可以并行,迁移时不改 MCP tool schema。 | ||
|
|
||
| ## 路线 A:Spring Boot 在 sa-token 上实现 OAuth 2.1 | ||
|
|
||
| 这条路线由现有后端同时承担 Authorization Server 与 Resource Server。用户登录、账号关联、权限判断继续复用当前 sa-token 数据。 | ||
|
|
||
| ### Endpoint checklist | ||
|
|
||
| | Endpoint | 责任 | | ||
| | --------------------------------------------- | ------------------------------------------------------------------- | | ||
| | `GET /.well-known/oauth-authorization-server` | 发布 issuer、authorization/token/registration endpoint、PKCE 能力 | | ||
| | `GET /oauth/authorize` | 校验 client、redirect URI、resource、scope、state、PKCE challenge | | ||
| | `POST /oauth/authorize` | 用户确认授权后签发一次性 authorization code | | ||
| | `POST /oauth/token` | 用 code + PKCE verifier 换 access token,可选签发 refresh token | | ||
| | `POST /oauth/introspect` | 给 MCP resource server 校验 token、scope、subject、audience、expiry | | ||
| | `POST /oauth/revoke` | 撤销 access/refresh token | | ||
| | `POST /oauth/register` | 如果要支持动态客户端注册,登记 redirect URIs 与 client metadata | | ||
| | `GET /oauth/jwks` | 使用 JWT access token 时发布签名公钥 | | ||
|
|
||
| 最低安全要求: | ||
|
|
||
| - 只允许 Authorization Code + PKCE S256; | ||
| - 不提供 implicit flow 和 password grant; | ||
| - redirect URI 精确匹配,不接受 wildcard; | ||
| - code 单次使用且短 TTL; | ||
| - access token 必须绑定 `https://involutionhell.com/api/mcp` resource/audience; | ||
| - scope 至少区分 `publish`,默认不能扩大权限; | ||
| - refresh token rotation,并检测 reuse; | ||
| - state 必须原样校验,浏览器授权流程同时防 CSRF; | ||
| - token、code、verifier 不进入应用日志、错误响应和 tracing attribute。 | ||
|
|
||
| 如果继续签发 opaque token,Next.js `verifyToken` 调 introspection;如果签发 JWT,Next.js 可用 issuer/JWKS 本地验证,但 Java `/api/posts` 也必须验证同一 issuer、audience、expiry 与 `publish` scope。 | ||
|
|
||
| ## 路线 B:第三方 IdP,后端只做 access-token validation | ||
|
|
||
| Stytch、WorkOS 或 Auth0 承担登录、consent、authorize、token、refresh、revoke、JWKS 和可选动态客户端注册。Spring Boot 不签发 OAuth token,只把自己作为 Resource Server。 | ||
|
|
||
| ### Endpoint checklist | ||
|
|
||
| IdP 侧必须提供: | ||
|
|
||
| | Endpoint | 责任 | | ||
| | ----------------------------------------------------- | --------------------------- | | ||
| | `/.well-known/openid-configuration` 或 OAuth metadata | issuer 与能力发现 | | ||
| | `/authorize` | Authorization Code + PKCE | | ||
| | `/token` | code/refresh token exchange | | ||
| | `/jwks.json` | JWT 签名公钥 | | ||
| | `/revoke` | token 撤销 | | ||
| | `/register`(如启用 DCR) | MCP client 动态注册 | | ||
|
|
||
| Java 后端需要增加: | ||
|
|
||
| | Endpoint/组件 | 责任 | | ||
| | ----------------------------------------- | ------------------------------------------------------------------ | | ||
| | OAuth bearer filter | 读取 `Authorization: Bearer`,与 legacy `satoken` filter 并行 | | ||
| | JWT validator 或 IdP introspection client | 校验 issuer、签名、audience/resource、expiry、not-before、scope | | ||
| | subject-to-user mapper | 把 IdP `sub` 稳定映射到 `user_accounts`,禁止按可变 email 临时认人 | | ||
| | `GET /auth/me` | 同时接受 satoken 与 OAuth token,返回同一 `UserView` | | ||
| | `POST /api/posts` | 两种 token 都走同一用户身份与 `publish` 权限检查 | | ||
|
|
||
| IdP tenant 必须把 MCP production resource 配成独立 audience,不要接受发给其他 API 的 access token。账号关联、停用用户、角色变化和 token 撤销后的生效时间要在后端明确。 | ||
|
|
||
| ## 迁移与共存 | ||
|
|
||
| 建议分四阶段: | ||
|
|
||
| 1. **准备**:保持 `satoken` header 与现有 `Authorization: Bearer <satoken>` MCP 用法;后端测试双 filter 的优先级和冲突处理。 | ||
| 2. **并行**:`/auth/me`、`/api/posts` 同时接受 satoken 和 OAuth access token。一个请求只采用一种身份;如果两个 header 指向不同用户,必须拒绝,不能猜优先级。 | ||
| 3. **MCP 切换**:Next.js `verifyToken` 先识别/验证 OAuth,legacy satoken 作为 fallback。protected-resource metadata 增加真实 `authorization_servers`,客户端逐步重连 OAuth。 | ||
| 4. **收口**:观察 legacy 使用量,给出明确下线日期;撤掉 MCP 的 satoken 支持前,网页现有 satoken 登录不必同步下线。 | ||
|
|
||
| 并行期要保持两条 header 路径: | ||
|
|
||
| ```http | ||
| satoken: <legacy-token> | ||
| Authorization: Bearer <oauth-access-token> | ||
| ``` | ||
|
|
||
| 不要把 OAuth token 复制进 `satoken` header 来假装兼容。两类 token 的 issuer、audience、scope、撤销语义不同,后端应该显式验证。 | ||
|
|
||
| ## Backend 测试义务 | ||
|
|
||
| 每个安全不变量都必须有成对测试:一个合法请求通过,一个只破坏该不变量的请求失败。至少覆盖: | ||
|
|
||
| - PKCE S256 正确 verifier / 错误 verifier; | ||
| - redirect URI 精确匹配 / 相似域名或 wildcard 拒绝; | ||
| - state 保持 / state 缺失或篡改; | ||
| - code 首次兑换 / 重放; | ||
| - 正确 resource/audience / 错 audience; | ||
| - `publish` scope 存在 / 缺 scope; | ||
| - 未过期 token / expired、not-before、revoked token; | ||
| - 正确 issuer/签名 / 错 issuer、未知 key、算法降级; | ||
| - 已绑定 `sub` / 未绑定或停用用户; | ||
| - satoken 单独请求、OAuth 单独请求、双 header 同用户、双 header 冲突用户; | ||
| - introspection/JWKS 超时、5xx、malformed payload 时 fail closed; | ||
| - 日志和错误响应不包含 token、code、refresh token 或 verifier。 | ||
|
|
||
| 所有新增后端代码必须执行: | ||
|
|
||
| ```bash | ||
| ./mvnw verify | ||
| ``` | ||
|
|
||
| JaCoCo line coverage 不低于 80%。`SECURITY.md` 中新增或修改的每条 invariant 都要同时落一组正向/反向测试;不能只用 controller happy-path 把覆盖率刷到 80%。集成测试还要真实经过 security filter chain,不能全部 mock 掉 token validator。 | ||
|
|
||
| 上线前至少再做一次跨服务 E2E:客户端 authorize → token → MCP initialize → `search` → `publish` → Java 创建 PostView,并验证错误 audience、缺 scope、过期 token 都在写数据库前被拒绝。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| # MCP Server | ||
|
|
||
| ## 目标 | ||
|
|
||
| 本站在 Next.js 内提供 `/api/mcp`,让支持 MCP Streamable HTTP 的客户端直接完成两件事: | ||
|
|
||
| - 搜索 `content/docs/**/*.mdx` 中的站内文章; | ||
| - 使用已登录用户的凭证发布一篇轻量 Markdown 帖子。 | ||
|
|
||
| MCP route 与文档站一起部署到 Vercel,不需要增加独立 Node 服务,也不会复制一套内容同步流程。它直接读取构建时生成的 Fumadocs source,并通过 `BACKEND_URL` 调用现有 Spring Boot API。 | ||
|
|
||
| ## Transport 与运行模型 | ||
|
|
||
| 服务端使用 `mcp-handler@1.1.0` 和 `@modelcontextprotocol/sdk@1.26.0`: | ||
|
|
||
| - 只开放 Streamable HTTP; | ||
| - `sessionIdGenerator` 为 `undefined`,每个请求创建独立 MCP server/transport; | ||
| - 关闭 legacy SSE endpoint; | ||
| - 不保存 session,不依赖 Redis,可运行在 Vercel serverless; | ||
| - `GET /api/mcp` 在 stateless 模式返回 405,`POST /api/mcp` 处理 JSON-RPC。 | ||
|
|
||
| `POST` 会在进入 `mcp-handler` 前读取并校验原始 body:超过 1 MB 返回 HTTP 413, | ||
| 非法 JSON 返回 HTTP 400 / JSON-RPC `-32700`。MCP 2025-06-18 起不支持 batch, | ||
| 所以顶层数组返回 HTTP 400 / JSON-RPC `-32600`,不会交给旧版 SDK 执行。 | ||
|
|
||
| 仓库使用 zod 4。`mcp-handler` README 仍写 zod 3,但它配套的 SDK 1.26.0 已明确兼容 zod 3.25 和 zod 4。本实现固定 SDK 1.26.0 以满足 `mcp-handler@1.1.0` 的精确 peer dependency,并用 zod 4 schema 实测通过了 TypeScript、`tools/list` schema 生成和 `tools/call` 运行时校验,因此没有降级 zod,也没有改用 raw JSON Schema。 | ||
|
|
||
| ## Tool contract | ||
|
|
||
| ### `search` | ||
|
|
||
| 输入: | ||
|
|
||
| | 字段 | 类型 | 默认值 | 约束 | | ||
| | -------- | ------------ | ------ | ------------------------- | | ||
| | `query` | string | 无 | 去除首尾空白后 1–200 字符 | | ||
| | `locale` | `"zh"\|"en"` | `zh` | 选择中文或英文索引 | | ||
| | `limit` | integer | `8` | 1–20 | | ||
|
|
||
| 输出的 `structuredContent.results` 是: | ||
|
|
||
| ```ts | ||
| Array<{ | ||
| title: string; | ||
| description: string; | ||
| url: string; | ||
| snippet: string; | ||
| }>; | ||
| ``` | ||
|
|
||
| `url` 固定为 `https://involutionhell.com` 下的绝对地址。`snippet` 从页面 `structuredData.contents` 中选择与 query 最相关的段落,并截断到 300 字符以内。文本 `content` 同时提供同一批结果,兼容不读取 structured output 的客户端。 | ||
|
|
||
| 搜索索引沿用现有 `/search.zh.json` 和 `/search.en.json` 的数据管线: | ||
|
|
||
| 1. `source.getPages()`; | ||
| 2. 按 `isEnglishPage()` 分中文、英文 shard;`lang: en` 或源文件名以 `.en.md` / `.en.mdx` 结尾都视为英文; | ||
| 3. 使用 `pageToIndex()` 取得 title、description、URL 和 structured data; | ||
| 4. 中文 shard 使用 Orama Mandarin tokenizer,英文 shard 使用 English language; | ||
| 5. 查询保持 `threshold: 0.3`、`tolerance: 1`,并按 page 聚合结果; | ||
| 6. exact phrase 命中的页面优先按 title、description、heading、content 加权重排, | ||
| 再补 Orama fuzzy hits,避免短中文词在编辑距离为 1 时被近形词排到前面。 | ||
|
|
||
| 搜索实现和两个 shard 都是 module-level lazy load。同一 lambda 实例第一次搜索时才加载内容 chunk,第一次搜索某种语言时建对应索引,后续请求复用;构建失败会清除 promise,让下次请求重试。 | ||
|
|
||
| ### `publish` | ||
|
|
||
| 输入: | ||
|
|
||
| | 字段 | 类型 | 必填 | 说明 | | ||
| | ------------- | -------- | ---- | -------------------------------- | | ||
| | `title` | string | 是 | 帖子标题,最多 200 字符 | | ||
| | `content_md` | string | 是 | Markdown 正文,最多 100,000 字符 | | ||
| | `description` | string | 否 | 摘要,最多 500 字符 | | ||
| | `tags` | string[] | 否 | 最多 10 个,每个最多 50 字符 | | ||
| | `slug` | string | 否 | 最多 100 字符;不传时由后端生成 | | ||
|
|
||
| 成功输出 `structuredContent`: | ||
|
|
||
| ```ts | ||
| { | ||
| title: string; | ||
| slug: string; | ||
| url: string; | ||
| } | ||
| ``` | ||
|
|
||
| MCP route 将字段映射成 `{ title, contentMd, description?, tags?, slug? }`,调用 `POST ${BACKEND_URL}/api/posts`,超时为 10 秒。后端 409 会提示更换 slug;401/403 会提示重新登录取得 satoken;超时和其他错误不会把后端响应体或凭证回显给客户端。 | ||
|
|
||
| ## 鉴权边界 | ||
|
|
||
| `search` 完全匿名可用。`publish` 要求 MCP HTTP 请求携带: | ||
|
|
||
| ```http | ||
| Authorization: Bearer <satoken> | ||
| ``` | ||
|
|
||
| 所有凭证读取、`${BACKEND_URL}/auth/me` 校验、MCP `AuthInfo` 生成和后端 header 生成都集中在 `lib/mcp/auth.ts`。只有解析后的 `publish` tool call 才校验 bearer token;initialize、tools/list、search 等请求即使带 bearer header 也匿名执行,不访问鉴权后端。工具只接收“已验证身份 + 后端 headers”,不解析 Authorization,也不记录 token。 | ||
|
|
||
| 当前行为: | ||
|
|
||
| - 匿名调用 `search`:正常执行; | ||
| - 匿名调用 `publish`:HTTP 401,并返回 `WWW-Authenticate`; | ||
| - `publish` 携带无效或过期 token:HTTP 401,并返回同一 `WWW-Authenticate` challenge; | ||
| - `publish` 携带有效 token:先调用 `/auth/me`,再执行 `publish`; | ||
| - `/auth/me` 超时、网络失败或返回 5xx:HTTP 503 通用错误,提示稍后重试,不把有效凭证误报为无效。`/auth/me` 超时为 10 秒。 | ||
|
|
||
| `/.well-known/oauth-protected-resource` 已声明 MCP resource、header bearer method 和 `publish` scope。当前后端尚无 OAuth issuer,因此 metadata 不虚构 `authorization_servers`;接入 OAuth 时按升级文档补齐。 | ||
|
|
||
| sa-token 当前有效期是 30 天。网页端暂时没有安全的 OAuth 授权流程,浏览器类 MCP 客户端默认只使用匿名 `search`;需要 `publish` 时由用户在可信本地客户端显式配置 token。 | ||
|
|
||
| ## 限流 | ||
|
|
||
| `search` 使用独立的 Upstash sliding window:每 IP 每 60 秒 30 次,key prefix 为 `ratelimit:mcp:search`。环境变量读取顺序与站内现有 limiter 一致: | ||
|
|
||
| - `UPSTASH_REDIS_REST_URL` / `UPSTASH_REDIS_REST_TOKEN`; | ||
| - Vercel prefix 变体; | ||
| - `KV_REST_API_URL` / `KV_REST_API_TOKEN`。 | ||
|
|
||
| 本地没有 Upstash 环境变量时允许全部请求,并且每个模块实例只打印一次提示。`publish` 依赖登录身份和后端自身保护,不占 search 的额度。 | ||
|
|
||
| ## 本地开发 | ||
|
|
||
| ```bash | ||
| corepack pnpm check:pnpm-version | ||
| BACKEND_URL=http://localhost:8080 corepack pnpm dev | ||
| ``` | ||
|
|
||
| 匿名连接 Claude Code: | ||
|
|
||
| ```bash | ||
| claude mcp add --transport http involutionhell https://involutionhell.com/api/mcp | ||
| ``` | ||
|
|
||
| 需要发布时,将 satoken 放进环境变量,避免写进 shell history: | ||
|
|
||
| ```bash | ||
| read -s INVOLUTIONHELL_SATOKEN | ||
| export INVOLUTIONHELL_SATOKEN | ||
| claude mcp remove involutionhell | ||
| claude mcp add --transport http involutionhell https://involutionhell.com/api/mcp \ | ||
| --header "Authorization: Bearer ${INVOLUTIONHELL_SATOKEN}" | ||
| ``` | ||
|
|
||
| 本地 endpoint 可替换为 `http://localhost:3000/api/mcp`。 | ||
|
|
||
| ## 已知限制 | ||
|
|
||
| - satoken 最长 30 天,过期后需要重新登录并更新客户端配置; | ||
| - 当前没有 OAuth refresh token,不能静默续期; | ||
| - web MCP client 在 OAuth 上线前只应开放匿名 search; | ||
| - shard 缓存只在单个 lambda 实例内有效,冷启动会重新构建; | ||
| - 内容更新随下一次部署进入 MCP index,不是运行时增量索引; | ||
| - `publish` 只处理轻量帖子,不上传 cover 或正文图片。 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For browser-hosted MCP clients, tool calls are cross-origin POSTs with
Content-Type: application/jsonand oftenAuthorization/Mcp-Protocol-Version, so the browser preflightsOPTIONS /api/mcpand then requiresAccess-Control-Allow-Originon the POST response. This route only exports GET/POST and returns handler responses as-is, while only the metadata route has CORS, so those clients are blocked before even anonymoussearchcan run. Please export OPTIONS and wrap the MCP responses with CORS headers.Useful? React with 👍 / 👎.