You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The openshell-router creates its reqwest::Client with a 60-second total request timeout (lib.rs: .timeout(Duration::from_secs(60))). For streaming inference responses (/v1/chat/completions with stream: true), this timeout covers the entire request lifecycle — from initial send through the last streamed chunk. When model generation takes longer than 60 seconds (common with large models at ~8-15 tok/s), reqwest aborts the connection mid-stream.
This was introduced alongside the streaming fix in #260 / PR #261, which correctly switched from buffered to chunked streaming but inherited the existing 60s total timeout.
Reproduction
Configure inference.local to route to a self-hosted vLLM instance serving a large model (e.g., Nemotron-3-Super-120B)
Send a chat completion request that produces a long response (500+ tokens at ~8-15 tok/s = 30-60+ seconds of generation)
After exactly 60 seconds, the streaming response is truncated
In crates/openshell-router/src/lib.rs, the reqwest client is configured with:
.timeout(Duration::from_secs(60))
reqwest's .timeout() is a total request timeout — it does not reset when data is received. For streaming SSE responses, this means the timeout fires even though the upstream is actively sending data.
...breaks out of the chunk-reading loop and then writes the chunked terminator (0\r\n\r\n), causing the downstream client to see a truncated-but-technically-valid HTTP response.
Suggested Fix
Replace the total .timeout() with a read timeout that resets on each chunk:
// Before (total timeout — kills long-running streams):
reqwest::Client::builder().timeout(Duration::from_secs(60)).build()// After (read timeout — only fires if upstream goes silent):
reqwest::Client::builder().read_timeout(Duration::from_secs(120)).connect_timeout(Duration::from_secs(30)).build()
This preserves the protection against hung upstreams (fires if no data arrives for 120s) while allowing arbitrarily long streaming generations.
Reproduced on NemoClaw + DGX Spark GB10 with Ollama backend
Confirmed this issue on a different backend (Ollama serving Nemotron-3-Super-120B at Q4_K_M quantization) on a DGX Spark GB10, OpenShell 0.0.16.
Repro:
NemoClaw configured with local Ollama provider (http://host.openshell.internal:11434/v1)
Ask the agent to generate a long document (500+ tokens)
Response truncates mid-sentence at exactly 60 seconds
Evidence from journalctl -u ollama -f:
[GIN] 2026/03/28 - 09:52:54 | 500 | 1m0s | 172.20.0.2 | POST "/v1/responses"
[GIN] 2026/03/28 - 09:53:54 | 200 | 1m0s | 172.20.0.2 | POST "/v1/responses"
[GIN] 2026/03/28 - 10:20:42 | 200 | 1m0s | 172.20.0.2 | POST "/v1/responses"
Exact 60-second intervals on every long generation request, regardless of prompt. Short requests (under 60s of generation) complete fine with normal timings (e.g., 17.7s for a one-sentence answer).
The suggested fix (replacing .timeout() with .read_timeout() + .connect_timeout()) looks correct. This makes local large-model inference with NemoClaw essentially unusable for any non-trivial generation task. Bumping for prioritization — this affects the NemoClaw use case directly, not just vLLM deployments.
Environment: DGX Spark GB10, aarch64, OpenShell 0.0.16, NemoClaw, Ollama 0.17.1, Nemotron-3-Super-120B Q4_K_M
Activity
dennisdarrow commentedon Mar 28, 2026
Reproduced on NemoClaw + DGX Spark GB10 with Ollama backend
Confirmed this issue on a different backend (Ollama serving Nemotron-3-Super-120B at Q4_K_M quantization) on a DGX Spark GB10, OpenShell 0.0.16.
Repro:
NemoClaw configured with local Ollama provider (http://host.openshell.internal:11434/v1)
Ask the agent to generate a long document (500+ tokens)
Response truncates mid-sentence at exactly 60 seconds
Evidence from journalctl -u ollama -f:
[GIN] 2026/03/28 - 09:52:54 | 500 | 1m0s | 172.20.0.2 | POST "/v1/responses"
[GIN] 2026/03/28 - 09:53:54 | 200 | 1m0s | 172.20.0.2 | POST "/v1/responses"
[GIN] 2026/03/28 - 10:20:42 | 200 | 1m0s | 172.20.0.2 | POST "/v1/responses"
Exact 60-second intervals on every long generation request, regardless of prompt. Short requests (under 60s of generation) complete fine with normal timings (e.g., 17.7s for a one-sentence answer).
The suggested fix (replacing .timeout() with .read_timeout() + .connect_timeout()) looks correct. This makes local large-model inference with NemoClaw essentially unusable for any non-trivial generation task. Bumping for prioritization — this affects the NemoClaw use case directly, not just vLLM deployments.
Environment: DGX Spark GB10, aarch64, OpenShell 0.0.16, NemoClaw, Ollama 0.17.1, Nemotron-3-Super-120B Q4_K_M