Long-running tool calls starve the stdio worker pool #23

New issue

Closed

opened 2026-09-19 13:57:31 +00:00 by Phil · 0 comments

Phil commented

2026-09-19 13:57:31 +00:00

Owner

Symptom

On 2026-09-19 every call through the shared hub (mcp-hub-light, ~50 Claude sessions on one stdio backend) timed out twice within an hour, including get_forgejo_mcp_server_version. The process was alive and idle. A restart healed it each time.

Root cause

mcp-go v0.44.0 runs tools/call on a fixed pool of 5 workers (server/stdio.go:403 workerPoolSize: 5); ServeStdio is called without WithWorkerPoolSize (operation/operation.go:133).
wait_for_run blocks a worker for up to 1800 s by default (operation/actions/wait_for_run.go:31).
Abandoned calls are never cancelled: mcp-proxy's Python SDK does not forward notifications/cancelled to the backend when a client gives up, and mcp-go v0.44 does not handle it either. A caller that times out and retries adds another blocked worker.
Journal: at both restarts (15:22:33 and 15:53:27 CEST) 7 and 6 wait_for_run calls were in progress, each logging wait cancelled by caller (context done) as stdin closed. That is more than 5 workers, so every short call was stuck in the queue.
In addition, the SDK client has no HTTP timeout (pkg/forgejo/forgejo.go forgejo.NewClient without SetHTTPClient), so one stalled TCP read could also hold a worker indefinitely.

Acceptance criteria

Long-running tools (wait_for_run, full-scan list_workflow_runs, list_run_jobs, run/job log fetches) share a bounded number of slots. When the slots are full they fail immediately with a clear message instead of queueing.
Short calls always have free workers: the pool is sized well above the long-call slots.
Every Forgejo HTTP request has a client timeout.
Test: with all long-call slots held, get_forgejo_mcp_server_version still answers in under 2 s.

Refs Philflow/auto-issue-orchestrator#215

## Symptom On 2026-09-19 every call through the shared hub (`mcp-hub-light`, ~50 Claude sessions on one stdio backend) timed out twice within an hour, including `get_forgejo_mcp_server_version`. The process was alive and idle. A restart healed it each time. ## Root cause - mcp-go v0.44.0 runs `tools/call` on a fixed pool of **5** workers (`server/stdio.go:403` `workerPoolSize: 5`); `ServeStdio` is called without `WithWorkerPoolSize` (`operation/operation.go:133`). - `wait_for_run` blocks a worker for up to 1800 s by default (`operation/actions/wait_for_run.go:31`). - Abandoned calls are never cancelled: mcp-proxy's Python SDK does not forward `notifications/cancelled` to the backend when a client gives up, and mcp-go v0.44 does not handle it either. A caller that times out and retries adds another blocked worker. - Journal: at both restarts (15:22:33 and 15:53:27 CEST) 7 and 6 `wait_for_run` calls were in progress, each logging `wait cancelled by caller (context done)` as stdin closed. That is more than 5 workers, so every short call was stuck in the queue. - In addition, the SDK client has no HTTP timeout (`pkg/forgejo/forgejo.go` `forgejo.NewClient` without `SetHTTPClient`), so one stalled TCP read could also hold a worker indefinitely. ## Acceptance criteria - [ ] Long-running tools (`wait_for_run`, full-scan `list_workflow_runs`, `list_run_jobs`, run/job log fetches) share a bounded number of slots. When the slots are full they fail immediately with a clear message instead of queueing. - [ ] Short calls always have free workers: the pool is sized well above the long-call slots. - [ ] Every Forgejo HTTP request has a client timeout. - [ ] Test: with all long-call slots held, `get_forgejo_mcp_server_version` still answers in under 2 s. Refs Philflow/auto-issue-orchestrator#215

Phil referenced this issue from a commit

2026-09-19 14:00:26 +00:00

fix(server): keep long tool calls from starving the worker pool

Phil referenced this issue from a pull request that will close it,

2026-09-19 14:00:31 +00:00

Keep long tool calls from starving the worker pool #24

Phil closed this issue

2026-09-19 14:00:37 +00:00

Phil referenced this issue from a commit

2026-09-19 14:03:09 +00:00

fix(setup): build forgejo-mcp from main, not feat/run-logs