S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active

Environment

Python 3.14
Linux aarch64
sentry-sdk==2.66.0
ddtrace==4.12.1 (Datadog admission-controller library injection)
FastAPI
boto3/botocore with AWS SigV4

Description

An S3 HeadObject request against an existing, accessible object returns 403 Forbidden only when it executes inside sentry_sdk.start_span(...) while Datadog's botocore integration is active. The same pod identity, bucket, key, client configuration, and credentials return 200 OK when either the Sentry span is removed or Datadog's botocore integration is disabled.

This also reproduces through our real FastAPI endpoint. The example below is read-only and reproduces on its first request.

Minimal reproducer

import boto3
import sentry_sdk
from botocore.config import Config
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sentry_sdk.integrations.fastapi import FastApiIntegration

sentry_sdk.init(
    dsn=None,
    traces_sample_rate=1.0,
    integrations=[FastApiIntegration()],
)

s3 = boto3.client(
    "s3",
    config=Config(
        signature_version="s3v4",
        s3={
            "addressing_style": "virtual",
            "us_east_1_regional_endpoint": "regional",
        },
    ),
)

app = FastAPI()

@app.post("/reproduce")
def reproduce():
    with sentry_sdk.start_span(op="dependency.resolve", name="FileStorage"):
        response = s3.head_object(
            Bucket="YOUR_BUCKET",
            Key="YOUR_EXISTING_KEY",
        )
    return {"status": response["ResponseMetadata"]["HTTPStatusCode"]}

response = TestClient(app).post("/reproduce")
print(response.status_code, response.text)

Run under Datadog auto-instrumentation, for example:

DD_TRACE_BOTOCORE_ENABLED=true ddtrace-run python reproducer.py

Actual result

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Control matrix

Datadog botocore integration	Nested `sentry_sdk.start_span`	Result
enabled	present	`403 Forbidden`
enabled	absent	`200 OK`
disabled	present	`200 OK`

The confirmed workaround is:

DD_TRACE_BOTOCORE_ENABLED=false

A corresponding Datadog report exists at DataDog/dd-trace-py#19477. This appears to be an interoperability issue; we are reporting it to both projects because the trigger requires both instrumentations.

PY-2667

@andrey-berenda

Thanks @andrey-berenda, we'll take a look.

I had Claude code verify whether ddtrace 4.12.0 exposed this preexisting issue and it was confirmed.

Confirmed it — checked ddtrace/propagation/http.py directly. The _BaggageHeader class (lines ~1000-1109) extracts the inbound baggage header into Context._baggage verbatim, and on inject, serializes whatever's in span_context._baggage straight back out to a baggage header (line 1043: headers[_HTTP_HEADER_BAGGAGE] = header_value) — no filtering by vendor prefix, since W3C baggage is meant to be opaque/pass-through.

That closes the loop on your question whether it was a preexisting sentry issue. Here's the mechanism:

Yes — this was a preexisting Sentry issue that ddtrace 4.12.0 exposed, not a ddtrace-introduced bug in isolation. Sentry's config (unrestricted trace_propagation_targets) has presumably always been willing to inject a baggage header on outgoing AWS calls via its httplib patch. But that injection happens at the wire layer, strictly after SigV4 signing — so pre-4.12.0, it was harmless. AWS never validated it because it was never part of SignedHeaders.

4.12.0 is what changed: it added the first-ever pre-sign header injection point in this dependency chain (the before-sign botocore hook). That handler, using _BaggageHeader.inject, faithfully carries forward whatever's in the ddtrace span's _baggage dict — which itself was populated by extracting the inbound request's baggage header (the sentry-* keys your upstream service wrote, extracted generically since ddtrace treats baggage as opaque pass-through, not Sentry-specific). So for the first time, a baggage header — containing inherited Sentry-authored content — got baked into the pre-sign headers and thus into SignedHeaders.

Sentry's unrelated, unchanged, post-sign injection then appends its own freshly-serialized copy (with a newly computed sample_rand) on the same request. Two baggage: header instances hit the wire; AWS comma-joins same-named headers before validating, producing exactly the duplicated content you saw — which no longer matches what was actually signed.

So: latent Sentry misconfiguration (present indefinitely) + a genuinely new ddtrace 4.12.0 mechanism (first pre-sign injection point, incidentally forwarding inherited baggage) = signature-breaking duplication. Neither side alone explains it; the version bump is the trigger, Sentry's targets config is the enabling condition. Your Sentry trace_propagation_targets fix is the right primary remediation since it removes the enabling condition regardless of ddtrace version; a ddtrace cap is a reasonable secondary belt-and-suspenders.

fix(stdlib): Move trace header generation into `endheaders()`

fix(stdlib): Replace `return set()` with `continue` to avoid skipping…

fix(stdlib): Move `get_aws_sigv4_signed_headers` to `sentry_sdk.utils…

fix(stdlib): Make mypy happy

fix(stdlib): Capture internal exceptions in `endheaders()`

test(utils): Add test for `get_aws_sigv4_signed_headers`

fix(aiohttp): Preserve SigV4 signed headers

test(aiohttp): Add tests related to SigV4 issues

test(boto3): Add signed header assertions to tests

test(httplib): Add tests for SigV4 signed headers

fix(aiohttp): Do `request.rel_url` instead of `request.url` so py3.7-…

fix(aiohttp): Use `raw_path` instead of `rel_url` to avoid encoding i…

chore(stdlib): Track headers during request construction

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace pro…

Navigation Menu

Repository navigation

S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active #7031

Environment

Description

Minimal reproducer

Actual result

Control matrix

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active #7031

Description

Environment

Description

Minimal reproducer

Actual result

Control matrix

Activity

linear-code commented on Aug 4, 2026

sentrivana commented on Aug 4, 2026

jian-wu commented on Aug 4, 2026

8 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions