Enhancement: Automatic Rate Limit Handling (429 & standard rate limit headers)

[dougborg]

Summary

Provide optional automatic handling of API rate limiting for generated clients by detecting 429 responses and standard rate limit headers, then delaying & retrying appropriately.

Motivation

APIs commonly enforce quotas; naive clients either hammer endpoints or force manual retry logic scattered throughout user code. Centralizing logic increases reliability and reduces boilerplate.

Goals

• Detect rate limit signals via:
  • HTTP 429 status codes
  • Standard headers: Retry-After, X-RateLimit-Remaining, X-RateLimit-Reset, RateLimit-Remaining, RateLimit-Reset (IETF draft), and GitHub-style variants.
• Compute sleep duration: prefer Retry-After (seconds or HTTP-date). Fallback to time until reset if remaining == 0.
• Integrate with (or layer atop) general retry logic (Enhancement: Built-in Retry Strategy for Generated Clients #102) without double-counting attempts.
• Allow configurable max cumulative wait and max attempts for rate-limit retries separately from transient retries.
• Expose hook for user to observe events (e.g. on_rate_limit(callback)).

Non-Goals (initial)

• Token bucket / leaky bucket emulation across processes.
• Central shared distributed limiter.

Proposed Approach

1. Extend APIConfig with rate_limit settings: enabled, max_attempts, max_total_sleep, respect_retry_after (bool), header_priority list.
2. After a response (status 429 or remaining==0), compute delay:
   • If Retry-After (int) -> seconds.
   • If Retry-After (HTTP-date) -> parse to seconds.
   • Else if reset epoch header -> delay = reset - now (clamped >=0).
   • Else fallback exponential backoff base (shared with generic retry or separate base).
3. Sleep (async sleep for aiohttp/httpx async, time.sleep for sync) then re-issue request.
4. Ensure body re-serialization per attempt and idempotency safeguards (warn if non-idempotent & user did not explicitly allow).
5. Compose with general retry: rate-limit branch occurs first; afterward non-429 errors go through normal retry predicate.

Telemetry / Hooks

• Optional callback: on_rate_limited(attempt, delay, headers, method, path).
• Could integrate simple logging (debug) for visibility.

Testing

• Simulated responses: sequence [429 with Retry-After:1, 200]. Assert single sleep.
• Header precedence: prefer Retry-After over RateLimit-Reset when both present.
• Exceed max_total_sleep -> surface custom RateLimitExceeded exception (subclass HTTPException?).
• Async variant tests for aiohttp/httpx.

Open Questions

1. Separate exception type for exhaustion vs reuse HTTPException with extra context?
2. Should we automatically downgrade concurrency or just sleep?
3. Provide pluggable policy object instead of config fields?

Future Extensions

• Global limiter shared across client instances (thread-safe & async safe) to smooth bursts.
• Adaptive jitter for thundering herd avoidance.

Next Steps

• Align on config surface & exception type.
• Implement minimal version tied into retry helper (Enhancement: Built-in Retry Strategy for Generated Clients #102) with dedicated tests.

Related: #102 (retry strategy).