Skip to content

Add MetricsFileDirectory parameter to ScriptExecutor - #809

Merged
imadityaa merged 3 commits into
microsoft:mainfrom
rhch1309:users/rchaturvedi/scriptexecutor-metrics-file-directory
Sep 30, 2026
Merged

imadityaa merged 3 commits into
microsoft:mainfrom
rhch1309:users/rchaturvedi/scriptexecutor-metrics-file-directory

Conversation

@rhch1309

Copy link
Copy Markdown
Contributor

Description

ScriptExecutor expects the script under execution to emit its metrics as test-metrics.json in the script directory:

this.MetricsFilePath = this.Combine(this.ExecutableDirectory, ScriptExecutor.MetricsFileName);
if (this.fileSystem.File.Exists(this.MetricsFilePath))
{
	this.fileSystem.File.Delete(this.MetricsFilePath);
}

When several Virtual Client instances run concurrently on the same machine against a shared package directory (a common lab/harness setup), every instance resolves the same metrics file path. Each instance then deletes that file during InitializeAsync while clearing stale results, so one experiment can remove the metrics file belonging to another experiment that is mid-run.

There is currently no way to scope the metrics file per instance. LogPaths can already be scoped by supplying a relative prefix, but the metrics path is fixed.

Change

Adds an optional MetricsFileDirectory parameter that supplies a directory path, relative to the script directory, in which the script writes the metrics file:

"MetricsFileDirectory": "{experimentId}"

Since profile parameters are resolved by EvaluateParametersAsync before the path is composed, placeholders such as {experimentId} work as expected, giving each concurrent instance its own metrics file.

Notes on scope:

  • The metrics file name is deliberately left unchanged (test-metrics.json), since that is the contract with the script. Only the containing directory is configurable.
  • When the parameter is not defined, MetricsFilePath resolves exactly as before, so existing profiles are unaffected.
  • No change to log capture, error handling, or cancellation behavior.

The example profile (EXAMPLE-EXECUTE-SCRIPT.json) and the extensions guide are updated to document the parameter.

Validation

  • dotnet build src/VirtualClient/VirtualClient.sln -c Debug — succeeds, 0 warnings, 0 errors.
  • VirtualClient.Actions.UnitTests filtered to ScriptExecutor — 32/32 passing.
  • EXAMPLE-EXECUTE-SCRIPT.json validated as well-formed JSON.

ScriptExecutor expects the script to emit its metrics as 'test-metrics.json'
in the script directory. When multiple Virtual Client instances run
concurrently against a shared package directory, every instance resolves the
same metrics file path. Each instance then deletes that file during
initialization as part of clearing stale results, which can remove the metrics
file belonging to another experiment.

This adds an optional 'MetricsFileDirectory' parameter that supplies a
directory path, relative to the script directory, in which the script writes
the metrics file (e.g. {experimentId}). The metrics file name itself is
unchanged, so the contract with the script is preserved. When the parameter is
not defined the behavior is exactly as before.

The example profile and the extensions documentation are updated to describe
the parameter.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@imadityaa
imadityaa enabled auto-merge (squash) September 30, 2026 10:17
@imadityaa
imadityaa merged commit 29ddd6d into microsoft:main Sep 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants