Allow the developer to inject some metadata to the target process when they use sys.remote_exec at the same time

Feature or enhancement

Proposal:

For now, we emit an audit event when we execute the remote-injected script

    if (0 != PySys_Audit("remote_debugger_script", "O", path)) {
        PyErr_FormatUnraisable(
            "Audit hook failed for remote debugger script %U", path);
        return;
    }

I think maybe it's worth emitting the remote PID at the same time. It will be more helpful when we want collect the info related with security

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

run_remote_debugger_script() is called in the debugged process, the pid would the the process pid which should already be known. I don't think that logging the PID would be useful.

@vstinner

run_remote_debugger_script() is called in the debugged process, the pid would the the process pid which should already be known. I don't think that logging the PID would be useful.

@vstinner For me, I would like to record the pid itself when debugger process attach to the debugged process. And the debugged process can emit the debugger PID.

@pablogsal

cc @pablogsal

The debugged process doesn't know the debugger PID so it cannot emit it. For the debugged process perspective something writes to the interface but it doesn't know who

The debugged process doesn't know the debugger PID so it cannot emit it. For the debugged process perspective something writes to the interface but it doesn't know who

Yes, This is the issue I want to solve. I propose to inject the debugger PID when we inject the script at the same time. So the debugged process can know the original debugger process and know where the script from

The debugged process doesn't know the debugger PID so it cannot emit it. For the debugged process perspective something writes to the interface but it doesn't know who

Yes, This is the issue I want to solve. I propose to inject the debugger PID when we inject the script at the same time. So the debugged process can know the original debugger process and know where the script from

While I understand the motivation, I’d prefer not to include the debugger PID for the time being. The debugged process can’t verify it, and the debugger could lie—intentionally or not. I think it’s better to keep the debugged side minimal and avoid logging unverifiable data. If needed, the debugger side can log that information more reliably.

The debugged process doesn't know the debugger PID so it cannot emit it. For the debugged process perspective something writes to the interface but it doesn't know who

Yes, This is the issue I want to solve. I propose to inject the debugger PID when we inject the script at the same time. So the debugged process can know the original debugger process and know where the script from

While I understand the motivation, I’d prefer not to include the debugger PID for the time being. The debugged process can’t verify it, and the debugger could lie—intentionally or not. I think it’s better to keep the debugged side minimal and avoid logging unverifiable data. If needed, the debugger side can log that information more reliably.

I think we split this discussion into two parts:

verified/known-source tools/debugger
attack

For part2, yes, The debugged process can’t verify it. But the unverified message means a lot itself. For example, if the audit emit -1 PID and the log has been collected. But at the same time, we can not find a PID in host agent history. OK we know here's an unexpected attack.

BTW at the same time, here's a scenario in the real production environment. The enterprise may wrapped a developer tool be on the remote_exec API, and we just allow specific process to attach to the process. So maybe the code can be used

import sys
import json

def audit_callback(script, metadata):
	metadata = json.loads(metadata)
	if "version" is not in metadata:
		sys.exit(120)

So I propose to extend the remote_exec API

Allow the developer to inject some metadata at the same time. The metadata length can be defined at the compile time or bootstrap time(maybe default 256?)
We will emit the metadata in audit event when the process has been attached

@pablogsal

The debugged process doesn't know the debugger PID so it cannot emit it. For the debugged process perspective something writes to the interface but it doesn't know who

Yes, This is the issue I want to solve. I propose to inject the debugger PID when we inject the script at the same time. So the debugged process can know the original debugger process and know where the script from

While I understand the motivation, I’d prefer not to include the debugger PID for the time being. The debugged process can’t verify it, and the debugger could lie—intentionally or not. I think it’s better to keep the debugged side minimal and avoid logging unverifiable data. If needed, the debugger side can log that information more reliably.

I think we split this discussion into two parts:

verified/known-source tools/debugger

attack

For part2, yes, The debugged process can’t verify it. But the unverified message means a lot itself. For example, if the audit emit -1 PID and the log has been collected. But at the same time, we can not find a PID in host agent history. OK we know here's an unexpected attack.

BTW at the same time, here's a scenario in the real production environment. The enterprise may wrapped a developer tool be on the remote_exec API, and we just allow specific process to attach to the process. So maybe the code can be used

import sys
import json

def audit_callback(script, metadata):
metadata = json.loads(metadata)
if "version" is not in metadata:
sys.exit(120)
So I propose to extend the remote_exec API

Allow the developer to inject some metadata at the same time. The metadata length can be defined at the compile time or bootstrap time(maybe default 256?)

We will emit the metadata in audit event when the process has been attached

@pablogsal WDYT

I don't think that it's a good idea to send the debugger PID. Pablo gave a good rationale for not doing that. I suggest closing this issue.

@Zheaoli

Thanks @Zheaoli for bringing this up and for the thoughtful discussion about security audit improvements.

I agree with @vstinner and I think I unfortunately will decline for now so I am closing this as not planned for a couple of key reasons:

Passing additional metadata through the audit system introduces risk - metadata is provided by an external process and if that metadata is interpreted as a Python object and is malformed, it could potentially crash the process.
For the time being we prefer to keep the debugged side minimal and avoid adding complexity that could introduce new failure modes. I want to gain some experience from people using it to know how to extend it but for now is very early to do it.

While I understand the security monitoring use case you've outlined, the potential risks of extending the API in this way outweigh the benefits at this time. As mentioned earlier in the discussion, the debugger side can log this information more reliably if needed.

Thanks again for the contribution and for thinking about Python's security posture!

Sorry one correction: I reviewed more your proposal and I noticed that passing the metadata as raw bytes actually could work and is actually not unsafe. But this still feels a bit heavyweight for now because I can see people wanting the metadata not only on the audit system but also on the debugged process.

Sorry one correction: I reviewed more your proposal and I noticed that passing the metadata as raw bytes actually could work and is actually not unsafe. But this still feels a bit heavyweight for now because I can see people wanting the metadata not only on the audit system but also on the debugged process.

Thanks for the explanation. Would you mind that I open a discussion on discuss.python.org? Maybe we can collect more requirement about this.

I modify the issue title.

In my final thought, the metadata will be just a string(or bytes). The developer who develope the debugger decide what will be inject into the process. The target process developer decide how to use it.

Hummmm, the protocol is made so the process being debugged has no need to sync with the debugger. It can do it but fundamentally they are independent. That's partially why I am not convinced that this is going to be a good idea. But I understand how it could be beneficial so I am happy to discuss it a bit more

It can do it but fundamentally they are independent.

Yes, can't agree more. The independent is a very important point

BTW I think the metadata injection does not change the independent principle in most cases. We just can let people have more choice to decide the final behavior

I am more happy to consider this (I am reopening the issue) if we talk about some opaque metadata but I am still not 100% convinced so I want more opinions. @godlygeek WDYT?

I'm skeptical that anyone would use this feature. I don't think it's totally unreasonable, but using audit hooks to conditionally abort execution of some function based on the arguments passed to the hook is, to the best of my knowledge, an extremely rarely used feature.

From the point of view of preventing attacks, note that an attacker with arbitrary memory write permissions to your process can uninstall or replace whatever audit hook you've installed, bypassing it and then going on to leverage the remote execution feature.

I'm inclined to think that, from a security point of view, the only reasonable approach is to block processes before they start writing the the remote memory, instead of trying to block them after they've had a chance to write to arbitrary memory.

And I don't see much point to this proposal if not for security. We don't allow passing arbitrary information to other audit hooks, why should this one be special?

note that an attacker with arbitrary memory write permissions to your process can uninstall or replace whatever audit hook you've installed

Quite easily, in fact - the attacker can just write a 0 to the ob_size field of the PyListObject referred to by interp->audit_hooks, so that the list of audit hooks appears to be empty whenever the interpreter tries to loop over them to see if any should be called. (Or similarly write NULL to runtime->audit_hooks.head to bypass all hooks added with PySys_AddAuditHook instead of sys.addaudithook)

I am more happy to consider this (I am reopening the issue) if we talk about some opaque metadata but I am still not 100% convinced so I want more opinions. @godlygeek WDYT?

I'm skeptical that anyone would use this feature. I don't think it's totally unreasonable, but using audit hooks to conditionally abort execution of some function based on the arguments passed to the hook is, to the best of my knowledge, an extremely rarely used feature.

From the point of view of preventing attacks, note that an attacker with arbitrary memory write permissions to your process can uninstall or replace whatever audit hook you've installed, bypassing it and then going on to leverage the remote execution feature.

I'm inclined to think that, from a security point of view, the only reasonable approach is to block processes before they start writing the the remote memory, instead of trying to block them after they've had a chance to write to arbitrary memory.

And I don't see much point to this proposal if not for security. We don't allow passing arbitrary information to other audit hooks, why should this one be special?

I appreciate the feedback, but I think there might be some misunderstanding about the core intent of this proposal.
The primary goal isn't attack prevention, but enabling flexible injection of additional contextual information during remote execution. The conditional abort functionality I demonstrated was just one possible use case to show the flexibility of the approach. The real value lies in allowing developers to inject relevant metadata when performing remote execution operations.
This addresses legitimate compliance scenarios that are becoming increasingly common:

Production debugging compliance: In enterprise environments, developers often need to attach debuggers to production processes for troubleshooting critical issues. Compliance frameworks require detailed logging of who performed these operations, when, and why. Currently, there's no standardized way to inject and capture this metadata at the Python level during remote execution.
Future enterprise tooling ecosystem: I believe we'll see more and more internal debugging and monitoring tools leveraging sys.remote_exec for non-intrusive process inspection in the future. These tools will need a way to inject identifying metadata (tool name, operator ID, ticket number, etc.) that can be captured and correlated across different logging systems.
Cross-system audit trails: The ability to inject metadata during remote execution enables correlation between debugger logs and target service logs. For example, when a debugging session is initiated, both the debugging tool and the target service can log the same correlation information, making it possible to trace the complete interaction during compliance reviews.

You're absolutely right that this wouldn't stop a determined attacker with arbitrary memory write access. But that's not the threat model here - we're addressing legitimate operational scenarios where the ability to inject contextual information during remote execution is essential for compliance and accountability.
The reason this audit hook might deserve special treatment compared to others is that remote execution represents a fundamentally different category of operation - one that crosses process boundaries and often involves multiple stakeholders (the tool developer, the operator, the target service owner). Having a standardized way to inject metadata makes these complex scenarios more manageable and auditable.

we're addressing legitimate operational scenarios where the ability to inject contextual information during remote execution is essential for compliance and accountability.

OK, fair enough. Am I missing some reason why it wouldn't be acceptable to inject that contextual information into the file itself, rather than passing it separately? That is, it seems like all of the use cases you're suggesting today would work just fine in 3.13 if instead of requiring the debugger to pass two byte strings (a file path and some metadata about that file) it continued passing one byte string (a file path), and the audit hook itself enforced that the contents of that file must include the metadata in some particular format (say, that the first non-shebang line is a comment containing the metadata in JSON format, or whatever).

we're addressing legitimate operational scenarios where the ability to inject contextual information during remote execution is essential for compliance and accountability.

OK, fair enough. Am I missing some reason why it wouldn't be acceptable to inject that contextual information into the file itself, rather than passing it separately? That is, it seems like all of the use cases you're suggesting today would work just fine in 3.13 if instead of requiring the debugger to pass two byte strings (a file path and some metadata about that file) it continued passing one byte string (a file path), and the audit hook itself enforced that the contents of that file must include the metadata in some particular format (say, that the first non-shebang line is a comment containing the metadata in JSON format, or whatever).

That's a fair point, and you're right that we could technically achieve similar functionality by embedding metadata as comments in the injected file. In fact, I initially explored that approach when testing on this for 3.14.

However, the file-based approach introduces significant additional complexity that becomes particularly problematic in enterprise environments.

The core issue is that it requires a much more complex contract between the debugging tool and the target process. Instead of a simple agreement that "the debugger will inject some metadata," both sides now need to agree on:

Exactly which line the metadata should appear on
The precise comment format and JSON structure
How to handle edge cases (what if there's already a shebang? what about multi-line metadata?)

In enterprise scenarios, this complexity becomes a real barrier. For example, in our environment:

Our common service framework is developed and maintained by the business units
Our debugging tools are developed by the SRE team
These teams operate somewhat independently

With my proposed approach, the SRE team only needs to inject a metadata string through the extended parameter, while the business units only need to log the audit event (maybe with some JSON parsing, but that's optional). The contract between teams is minimal: just the understanding that "debuggers will inject metadata."

With the file-based approach, both teams would need to coordinate on file format specifications, parsing logic, and handle various edge cases. This significantly increases the integration complexity and makes the feature much harder to adopt in practice.

The separate metadata parameter keeps the concerns cleanly separated and reduces the coordination overhead between different teams and tools.

both sides now need to agree on:

1. Exactly which line the metadata should appear on

Sure, but this seems trivial? You can simply require it be the first line, or the last line, or whatever. You don't need to get tricky with it at all. You can take any valid Python file and concatenate

f"\n# {json.dumps(metadata)}\n"

to the end of it and it's still a valid Python file, for instance.

2. The precise comment format and JSON structure

You need to define the structure in both cases, regardless of whether the metadata is passed in the file or out of the file. If the precise comment format is just "the JSON structure prepended with # " that seems like a trivial difference.

3. How to handle edge cases (what if there's already a shebang? what about multi-line metadata?)

There don't need to be any edge cases depending on how you define (1). If you make the rule be that the last line of the file is a JSON payload preceded by literally # then there's no worries about whether there's already a shebang, and there's no risk of multi-line metadata (JSON can't contain newline characters).

With my proposed approach, the SRE team only needs to inject a metadata string through the extended parameter,

I'm not sure what you mean by "only" - the SRE team would need to patch every debugging tool to collect and inject this extra metadata, right? Not just the ones using the high level sys.remote_exec function, but also any that are using the low level PEP 768 interface directly.

With the file-based approach, both teams would need to coordinate on file format specifications, parsing logic, and handle various edge cases. This significantly increases the integration complexity and makes the feature much harder to adopt in practice.

Both teams already need to be in agreement about what format the metadata will be supplied in, and what contents it will have. How the metadata is passed seems like one tiny extra detail on top of the already complex coordination that the producer and consumer need to have. For instance, schema evolution aspects will need to be managed - what happens when you decide that there's a new piece of information that needs to be audited? Who's responsible for ensuring that the audit hook is installed? Who is responsible for deciding what to do if the audit hook fails? Where does the audit hook log to? Etc, etc.

If you don't like embedding it in the file, you could also just write it to a sibling file on the file system. You can make the audit hook require that, if a remote exec arrives for /tmp/tmp1234/foo.py, that there exists a /tmp/tmp1234/foo.py.metadata containing the metadata for that execution.

There don't need to be any edge cases depending on how you define (1). If you make the rule be that the last line of the file is a JSON payload preceded by literally # then there's no worries about whether there's already a shebang, and there's no risk of multi-line metadata (JSON can't contain newline characters).

No, JSON is just an example, actually here's other multi-line circumstance, like xml or custom procotol

If you don't like embedding it in the file, you could also just write it to a sibling file on the file system. You can make the audit hook require that, if a remote exec arrives for /tmp/tmp1234/foo.py, that there exists a /tmp/tmp1234/foo.py.metadata containing the metadata for that execution.

You're absolutely right that the sibling file approach would avoid the complexity of embedding metadata in the source file. However, I think there's a fundamental difference in design philosophy here.

It's not that I dislike embedding metadata in the file specifically - it's that I want to avoid adding unnecessary complexity around metadata acquisition altogether. In my design, I want to focus purely on handling the metadata itself, rather than having to deal with the mechanics of how to obtain it (though passing a file path would technically fall within the scope of this design, I'd prefer to avoid that complexity).

Regarding the bilateral audit aspect, we had a discussion about this at #135362 (comment), where we mentioned adding a separate audit event for sys.remote_exec(). In my design, we would also emit the injected metadata in this audit event. From my perspective, this makes the overall API much more user-friendly - in most cases, users only need to care about the injected metadata in the audit event, without worrying about additional concerns like file reading operations.

The sibling file approach, while clever, introduces an implicit contract that both the debugging tool and the target process need to understand: "if a remote exec arrives for /tmp/tmp1234/foo.py, there must exist a /tmp/tmp1234/foo.py.metadata". This kind of implicit convention goes against the principle that "explicit is better than implicit."

Even in cases where our metadata is very complex and exceeds the default metadata length limit (256 bytes) in my proposal, explicitly passing a file path would be better than having two different parties rely on an implicit convention about sibling file existence.

The direct parameter approach keeps the API clean and explicit, making it easier to reason about and implement across different tools and teams.

The sibling file approach, while clever, introduces an implicit contract that both the debugging tool and the target process need to understand: "if a remote exec arrives for /tmp/tmp1234/foo.py, there must exist a /tmp/tmp1234/foo.py.metadata". This kind of implicit convention goes against the principle that "explicit is better than implicit."

Tips: I acknowledge that explicitly passing a file path is also a form of implicit contract. We can't completely avoid these fundamental implicit agreements - otherwise many features simply wouldn't work. However, this type of contract is much simpler and clearer compared to conventions like "metadata exists at some specific location on the local machine" or "metadata is a sibling file of another file." Similarly, the assumption embedded in "we're passing metadata" is much more straightforward than contracts like "metadata is on a specific line in the file."

I want to focus purely on handling the metadata itself, rather than having to deal with the mechanics of how to obtain it (though passing a file path would technically fall within the scope of this design, I'd prefer to avoid that complexity).

My point is that how to obtain it seems like one trivial additional detail on top of all of the other things that the producer and consumer need to agree upon in advance - what format the metadata will be in, how it will be handled if it's not found or if it's missing a required field, how to handle schema evolution to cope with the addition of fields being added over time, etc.

And, as I said, the producer side needs to patch all of the tools that want to do remote execution to collect and inject this extra metadata. There's non-trivial engineering work involved in passing this extra metadata, regardless of whether it's passed in the file, or in the file name, or in a sibling file, or any other way.

I acknowledge that explicitly passing a file path is also a form of implicit contract.

And so is the agreement about what the contents of that file would be, and so is the mere agreement that metadata must be supplied.

We're getting stuck on how the audit hook can retrieve the metadata, but to me it seems like the least interesting part of the whole thing. There's tons of basically trivial strategies that could easily be used.

Even in cases where our metadata is very complex and exceeds the default metadata length limit (256 bytes) in my proposal, explicitly passing a file path would be better than having two different parties rely on an implicit convention about sibling file existence.

The relies on a buffer of a fixed size is a very good reason not to go with the approach you're proposing, as now the producer has an additional concern that they need to cope with (what to do when the metadata overflows) and potentially the consumer has an additional concern that they need to cope with as well (if the producer handles overflowing by writing to a file instead, as you suggest, the consumer needs to deal with the possibility that the received metadata is not actually the metadata but a path to the metadata).

Moreover, if we go with an approach where there are two buffers, writes to the two buffers wouldn't be synchronized, and it's possible that after one debugger has populated the script path buffer and the metadata buffer and then set the flag to tell the interpreter loop to execute it, another debugger comes along and overwrites the script path before before the interpreter loop executes it, and now you're running a script and auditing the wrong metadata.

Putting the metadata in the file might be slightly more complex, but it's also significantly more robust, and it works today in 3.14.

@godlygeek

Moreover, if we go with an approach where there are two buffers, writes to the two buffers wouldn't be synchronized, and it's possible that after one debugger has populated the script path buffer and the metadata buffer and then set the flag to tell the interpreter loop to execute it, another debugger comes along and overwrites the script path before before the interpreter loop executes it, and now you're running a script and auditing the wrong metadata.

This is a good point. I agree with it. I don't want to this feature become a risk for the interpreter. But this seems maybe.

Putting the metadata in the file might be slightly more complex, but it's also significantly more robust,

Yes, totally agreed. I will close this issue first and reopen it if one day the debugger community need another simple way to inject metadata

Thanks for your patience and time. Have a nice day! @godlygeek @pablogsal

Navigation Menu

Repository navigation

Allow the developer to inject some metadata to the target process when they use `sys.remote_exec` at the same time #135360

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow the developer to inject some metadata to the target process when they use sys.remote_exec at the same time #135360

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Activity

vstinner commented on Jun 11, 2025

Zheaoli commented on Jun 11, 2025

vstinner commented on Jun 11, 2025

pablogsal commented on Jun 11, 2025

Zheaoli commented on Jun 11, 2025

pablogsal commented on Jun 11, 2025

Zheaoli commented on Jun 11, 2025

Zheaoli commented on Jun 12, 2025

vstinner commented on Jun 12, 2025

pablogsal commented on Jun 12, 2025

pablogsal commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

7 remaining items

Zheaoli commented on Jun 12, 2025

pablogsal commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

godlygeek commented on Jun 12, 2025

Zheaoli commented on Jun 12, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Allow the developer to inject some metadata to the target process when they use `sys.remote_exec` at the same time #135360