You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(router): add inference API translation between protocols #90
Today the router requires an exact protocol match between the client SDK and the backend route. If an agent uses the OpenAI SDK (POST /v1/chat/completions), only routes with openai_chat_completions in their protocols list are eligible. Similarly, an Anthropic SDK client can only hit anthropic_messages routes.
This means:
Route operators must provision backends matching every SDK their agents might use
Agents are constrained to SDKs that match available backends
Adding a new backend provider requires all agents to adopt its SDK, or the operator must run parallel routes
Proposed Solution
Add an API translation layer in the router so that a request detected as one protocol (e.g., openai_chat_completions) can be transparently translated and forwarded to a route speaking a different protocol (e.g., anthropic_messages), and the response translated back.
Key design considerations
Route selection: When no exact-match route exists for the detected protocol, the router should attempt translation to a compatible protocol that does have a route
Request translation: Convert request body schema between protocols (e.g., OpenAI messages array → Anthropic messages format, max_tokens semantics, tool/function calling schemas)
Response translation: Convert response body back to the client's expected schema (e.g., Anthropic response → OpenAI ChatCompletion shape)
Feature parity: Some features don't map 1:1 (e.g., OpenAI logprobs, Anthropic thinking). Define a strategy for unsupported features (error, best-effort, drop silently)
Header translation: Already partially exists (auth header rewriting per protocol). Extend to cover other protocol-specific headers (e.g., anthropic-version)
Scope of initial translation pairs
At minimum:
openai_chat_completions ↔ anthropic_messages
Nice to have:
openai_responses → other protocols
openai_completions → chat-based protocols
Where this fits
The translation should live in navigator-router, likely as a new module between detect_inference_pattern() (in navigator-sandbox) and proxy_with_candidates(). The router already rewrites headers and model IDs in backend.rs — translation extends this to full schema transformation.
Acceptance Criteria
A client using the OpenAI SDK can be routed to an Anthropic-compatible backend (and vice versa) without the client changing SDKs
Streaming responses are translated correctly
Non-translatable features produce clear errors rather than silent data loss
Existing exact-match routing is unaffected (no performance regression for same-protocol routes)
Translation pairs are extensible for future protocols
Instead of building an N×M matrix of direct translators, use an IR pattern powered by the genai crate.
Flow:
Ingest: Accept raw JSON, deserialize using async-openai crate
Map to IR: Convert async-openai structs → genai::chat::ChatRequest / ChatMessage (handles structural normalization, e.g., extracting system prompts for Anthropic)
Execute: Pass IR to genai client, which handles provider-specific HTTP protocols
Map to Egress: Map genai response → async-openai response struct, serialize to JSON, return to client
Deterministic Routing (No Model-Name Guessing)
genai by default infers the provider from the model string (e.g., claude-3 → Anthropic). We bypass this since the same model name may exist across providers.
Solution: Configure genai::Client with a custom ModelMapper using a routing key syntax: "{provider_name}|{model_name}" (e.g., "anthropic|claude-3-5-sonnet-latest"). The mapper explicitly assigns the AdapterKind (protocol) and passes the raw model name to the provider.
Custom Endpoints
For routing to custom infrastructure (e.g., local vLLM cluster with OpenAI-compatible API), use genai's ServiceTargetResolver alongside the ModelMapper to dynamically rewrite Base URL and Auth headers per-request.
Open Implementation Items
IR Mapper: Translation functions mapping async_openai::types::ChatCompletionRequestMessage variants → genai::chat::ChatMessage
SSE Streaming (high complexity): When stream: true, consume genai's ChatStreamEvent, map to async_openai::types::CreateChatCompletionStreamResponse, push to client via SSE
Provider-Specific Quirks ("lossy" edge cases): Strategy for features that don't map cleanly (e.g., Anthropic's explicit prompt caching cache_control). Decide between auto-injection, stripping, or lowest-common-denominator API
Activity
pimlock commentedon Mar 6, 2026
Exploration Notes: Architectural Approach
Intermediate Representation (IR) Pattern
Instead of building an N×M matrix of direct translators, use an IR pattern powered by the
genaicrate.Flow:
async-openaicrateasync-openaistructs →genai::chat::ChatRequest/ChatMessage(handles structural normalization, e.g., extracting system prompts for Anthropic)genaiclient, which handles provider-specific HTTP protocolsgenairesponse →async-openairesponse struct, serialize to JSON, return to clientDeterministic Routing (No Model-Name Guessing)
genaiby default infers the provider from the model string (e.g.,claude-3→ Anthropic). We bypass this since the same model name may exist across providers.Solution: Configure
genai::Clientwith a customModelMapperusing a routing key syntax:"{provider_name}|{model_name}"(e.g.,"anthropic|claude-3-5-sonnet-latest"). The mapper explicitly assigns theAdapterKind(protocol) and passes the raw model name to the provider.Custom Endpoints
For routing to custom infrastructure (e.g., local vLLM cluster with OpenAI-compatible API), use
genai'sServiceTargetResolveralongside theModelMapperto dynamically rewrite Base URL and Auth headers per-request.Open Implementation Items
async_openai::types::ChatCompletionRequestMessagevariants →genai::chat::ChatMessagestream: true, consumegenai'sChatStreamEvent, map toasync_openai::types::CreateChatCompletionStreamResponse, push to client via SSEcache_control). Decide between auto-injection, stripping, or lowest-common-denominator APIMerge pull request ai-openshell#90 from opendatahub-io/fix/konflux-ru…