Skip to content

[BREAKING] FEAT: Let each deployment configure its finding fields - #3655

Open
Adrian Gavrila (adrian-gavrila) wants to merge 45 commits into
microsoft:mainfrom
adrian-gavrila:adrian-gavrila-configurable-finding-fields
Open

Adrian Gavrila (adrian-gavrila) wants to merge 45 commits into
microsoft:mainfrom
adrian-gavrila:adrian-gavrila-configurable-finding-fields

Conversation

@adrian-gavrila

@adrian-gavrila Adrian Gavrila (adrian-gavrila) commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

This is the follow-up promised in #3066 (Operations, findings, evidence). #3066 fixed a finding's severity, harm type and description. This PR lets each deployment define its own finding fields in Python instead. A deployment that configures nothing keeps the #3066 fields (Critical through Informational, the HarmCategory list, a description), and its findings migrate in place.

02-default-edit-dialog

Breaking change and migration

#3066 shipped in 1.2.0. Its fixed columns can't hold fields a deployment defines, so a finding is now a severity string plus a field_values object. The PyRIT 1.2 API keeps working, with deprecation warnings, until 1.5.0:

PyRIT 1.2 Use instead
FindingSeverity The option keys of FindingFields.severity, such as "critical"
description field_values.get("description", "")
harm_type and harm_type_other field_values.get("harm_type"), which holds the category or its Other text
severity_other severity, which holds Other text itself
finding.severity.value finding.severity
GET /api/operations/finding-options GET /api/operations/{operation_id}/finding-fields
  • An optional field that isn't set has no key in field_values, so read it with get.
  • Old fields sent on create or update move where the migration moves saved values. A deployment whose fields don't use the default keys rejects them.
  • Finding responses still include the four old keys, marked deprecated in the OpenAPI schema.
  • Three things break without a transition. Other severity text no longer equals FindingSeverity.OTHER, because the severity is the text itself. A severity is a str, so it isn't a FindingSeverity instance. Finding.model_dump() and model_dump_json() no longer include the old fields.

What changes

  • Fields are code. A deployment subclasses FindingsInitializer (pyrit/setup/initializers/findings.py) and returns a FindingFields from get_finding_fields().
    • The severity slot is always a required choice field. Its options run from highest to lowest, and findings sort in that order.
    • Other fields are choices (with optional cascade levels and Other text), single-line text, multi-line text, or numbers (a plain decimal such as 9.8). They can carry hint and section.
    • DEFAULT_FINDING_FIELDS reproduces FEAT: Findings dashboard with persistent Operations in the GUI #3066.
    • Keys are permanent; labels can change. Changing a key retires the old field, and saved values under it stay visible and read-only.
    • One field set applies to every operation. Sub-findings (FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) #4111) keep their own severity from it, and edits keep a finding's parent and demonstrated flag.
03-contoso-edit-dialog
  • Storage. One migration, d43b1bcf9c56, after 3e55dd099e38.

    • It drops the three CHECK constraints.
    • It stores Other severity text as the severity itself, and moves harm type and description into a JSON field_values column.
    • Finding IDs, Operation links, creation times and evidence links don't change.
    • severity becomes unbounded Unicode, with a binary collation on SQL Server so sorting and counting match SQLite.
    • Downgrade moves the default fields' values back into the 1.2 columns. It refuses offline, or while any finding has a value under another key, and names those keys.
    • Memory never reads the active FindingFields: the service passes it the severity order. Reads without one use the default fields' order, as in 1.2.
  • API. GET /api/operations/{operation_id}/finding-fields returns the fields that label and check an operation's findings. GET /api/operations/finding-fields returns the deployment's fields, for views across operations. Both return the one active set today, so per-operation field sets can come later without another breaking change.

    • Create and update check values against the active fields and answer 422 with a message naming the field.
    • Unchanged saved values that no longer match an option are kept.
    • Changed Other text that matches an option's key or label is rejected, with a 422 naming the option.
    • Updates keep retired values unless the request lists them in remove_retired_keys. Keys listed in keep_retired_keys keep their stored values even if a live apply configured the field again.
    • Request fields follow the existing request limits (TextStr, IdentifierStr, MAX_ITEMS).
  • Setup. The live-apply preflight rejects invalid finding fields, or two FindingsInitializers, before changing anything. A scenario run can't run a FindingsInitializer.

  • Export. The Word and Markdown export follows the fields: multi-line fields go in the Finding column, choice, text and number fields in Assessment, and retired values follow in the Finding column, labelled by their keys. finding_report_entry in pyrit.output.finding_report.report builds each report entry from the fields, and build_finding_report (from FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) #4111) now takes fields.

  • CoPyRIT. The finding dialog, cards, Operations counts, the Chat evidence picker and the export dialog all render from the finding fields.

    • Card values are a description list, so screen readers read each label with its value.
    • Retired values show read-only, with Remove (after a confirmation) and Undo.
    • The fields reload when the runtime generation changes.
    • An open edit dialog keeps the fields it opened with and lists its kept retired values by key, so a live apply during an edit never clears a value.
05-contoso-finding-card 01-operations-other-count
  • Docs. A new doc/code/setup/finding_fields.md (in doc/myst.yml) explains how to define fields, how changing them affects existing findings, and how to migrate code written for PyRIT 1.2. doc/gui/0_gui.md describes deployment-defined finding fields.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…values

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@adrian-gavrila Adrian Gavrila (adrian-gavrila) changed the title [DRAFT] FEAT: Let each deployment configure its finding fields FEAT: Let each deployment configure its finding fields Oct 29, 2026
@adrian-gavrila
Adrian Gavrila (adrian-gavrila) marked this pull request as ready for review October 29, 2026 19:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved defects can drop custom-field data, repeat or skip findings after severity changes, misvalidate replacement environments, and conflate distinct severities in SQL Server.

5 open issues
What changed in this PR

This follow-up to #3066 makes finding fields deployment-configurable across PyRIT’s storage, API, setup, and CoPyRIT interface, while retaining the existing fields by default.

Changes:

  • Adds initializer-based field configuration, validation, and live-apply preflight checks.
  • Migrates finding values to JSON storage and adapts ordering, updates, and exports.
  • Replaces fixed frontend fields with configurable controls, retired-value handling, and supporting tests and documentation.
File Description
tests/​unit/​setup/​test_reinitialization.py Tests finding-field preflight failures.
tests/​unit/​setup/​test_initialization.py Tests field resets and setup validation.
tests/​unit/​setup/​test_findings_initializer.py Tests initializer execution and discovery.
tests/​unit/​setup/​test_finding_fields.py Tests active-field state management.
tests/​unit/​setup/​conftest.py Adds custom-field fixtures.
tests/​unit/​models/​test_finding.py Tests flexible finding values.
tests/​unit/​memory/​test_operations_and_findings_migration.py Preserves earlier migration tests.
tests/​unit/​memory/​test_findings.py Tests persistence and configurable ordering.
tests/​unit/​memory/​test_finding_field_values_migration.py Tests migration data and database schema changes.
tests/​unit/​conftest.py Adds field-state cleanup fixture.
tests/​unit/​backend/​test_operations.py Tests field-aware routes and updates.
tests/​unit/​backend/​test_operation_service_findings.py Tests retired-value preservation and removal.
tests/​unit/​backend/​test_finding_export.py Tests configurable export projection.
tests/​unit/​backend/​test_common_models.py Tests finding request limits.
tests/​unit/​backend/​mocks.py Adds persisted-finding read helper.
tests/​unit/​backend/​conftest.py Adds configured and legacy finding fixtures.
pyrit/​setup/​initializers/​findings.py Introduces deployment field initializer.
pyrit/​setup/​initializers/​__init__.py Exports the initializer lazily.
pyrit/​setup/​initialization.py Resets fields during setup passes.
pyrit/​setup/​finding_fields.py Holds active deployment fields.
pyrit/​setup/​configuration_loader.py Adds field-definition preflight checks.
pyrit/​models/​finding.py Replaces fixed classifications with field values.
pyrit/​models/​__init__.py Updates public finding exports.
pyrit/​memory/​memory_models.py Stores severity and JSON field values.
pyrit/​memory/​memory_interface.py Accepts caller-defined severity ordering.
pyrit/​memory/​alembic/​versions/​d43b1bcf9c56_store_finding_field_values.py Migrates existing finding values and constraints.
pyrit/​backend/​services/​operation_service.py Validates values and preserves retired fields.
pyrit/​backend/​services/​finding_export.py Projects configured, non-internal fields.
pyrit/​backend/​routes/​operations.py Exposes fields and validation responses.
pyrit/​backend/​models/​operations.py Defines bounded configurable-field requests.
frontend/​src/​utils/​findingSeverity.ts Removes fixed severity helpers.
frontend/​src/​utils/​findingSeverity.test.ts Removes fixed severity tests.
frontend/​src/​utils/​findingFields.ts Adds field, cascade, and badge helpers.
frontend/​src/​utils/​findingFields.test.ts Tests configurable-field helpers.
frontend/​src/​types/​index.ts Defines configurable finding types.
frontend/​src/​test-utils/​findingFieldsFixtures.ts Adds default and custom field fixtures.
frontend/​src/​test-utils/​findingDialog.ts Adds shared dialog test helpers.
frontend/​src/​services/​api.ts Updates finding API methods.
frontend/​src/​services/​api.test.ts Tests revised API requests.
frontend/​src/​hooks/​useFindingFields.ts Loads fields and handles runtime refreshes.
frontend/​src/​hooks/​useFindingFields.test.tsx Tests loading, retries, and refreshes.
frontend/​src/​components/​Operations/​RetiredFieldValues.tsx Adds removal confirmation and undo controls.
frontend/​src/​components/​Operations/​RetiredFieldValues.test.tsx Tests retired-value controls and focus.
frontend/​src/​components/​Operations/​RetiredFieldValues.styles.ts Styles retired-value panels.
frontend/​src/​components/​Operations/​OperationsPage.tsx Renders configured severity counts.
frontend/​src/​components/​Operations/​OperationsPage.test.tsx Tests counts and field-loading failures.
frontend/​src/​components/​Operations/​Operations.styles.ts Styles configurable and retired values.
frontend/​src/​components/​Operations/​OperationDetailPage.tsx Integrates configurable finding displays and dialogs.
frontend/​src/​components/​Operations/​FindingSeverityBadge.tsx Uses configured severity labels and colors.
frontend/​src/​components/​Operations/​FindingFieldInput.tsx Adds text, choice, and cascade controls.
frontend/​src/​components/​Operations/​FindingFieldInput.styles.ts Styles configurable inputs.
frontend/​src/​components/​Operations/​FindingExportDialog.tsx Warns about excluded retired values.
frontend/​src/​components/​Operations/​FindingExportDialog.test.tsx Tests retired-value export warnings.
frontend/​src/​components/​Operations/​FindingDialog.tsx Builds drafts from configured fields.
frontend/​src/​components/​Operations/​FindingDialog.styles.ts Adds field-section styling.
frontend/​src/​components/​Operations/​FindingCardValues.tsx Displays configured and retired values.
frontend/​src/​components/​Chat/​FindingEvidenceDialog.tsx Uses configurable fields in evidence selection.
frontend/​src/​components/​Chat/​FindingEvidenceDialog.test.tsx Tests field-aware evidence workflows.
frontend/​e2e/​operations.spec.ts Updates requests and tests retired-value interactions.
doc/​myst.yml Adds finding-field documentation to navigation.
doc/​gui/​0_gui.md Describes deployment-defined finding fields.
doc/​code/​setup/​finding_fields.md Documents configuration, rules, and migration.

Review effort: Balanced

Comment thread frontend/src/utils/findingFields.ts Resolved
Comment thread pyrit/memory/memory_models.py Resolved
Comment thread pyrit/setup/configuration_loader.py Resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hannahwestra25 hannahwestra25 self-assigned this Nov 2, 2026
Re-point d43b1bcf9c56 after e7c4a2d9f013 from microsoft#3644, so the chain keeps a single head.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t reloads

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… flag

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adrian-gavrila

Copy link
Copy Markdown
Contributor Author

Done in 5e196af. The internal flag is gone, and exports now include retired values in the Finding column, labelled by their stored keys. The projection is now finding_report_entry in pyrit/output/finding_report/report.py, built on FindingReportField and FindingReportEntry, and the backend export calls it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@hannahwestra25

Copy link
Copy Markdown
Collaborator

Not done reviewing yet, but heads up: #3066 shipped in v1.2.0, so removing FindingSeverity, the old Finding fields and /finding-options now breaks public APIs. Should we add deprecation shims with removed_in=, or call this out explicitly as a breaking change with migration notes?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Deprecate the released findings API before removing it

The new export behavior addresses my earlier review: retired values are included, and the report projection is reusable from the library.

Following up on Hannah's release question: I checked the published 1.2.0 and 1.2.1 wheels. Both export FindingSeverity, accept the old finding fields, and serve GET /api/operations/finding-options. With this PR in 1.3, an existing client can no longer import FindingSeverity, an unchanged create payload with description is rejected, and the old route is gone. Migrating stored rows does not migrate those clients.

Please keep the configurable fields and deprecate the old Python and REST contracts with a compatibility layer scheduled for removal in 1.5.0. A [BREAKING] title and code migration notes would document the break if maintainers explicitly make a release-policy exception, but the label alone does not give clients a transition. Please also update the PR description's premise that #3066 has not shipped.

Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread frontend/src/types/index.ts Resolved

@hannahwestra25 hannahwestra25 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of broader points:

  1. Conflicts with main. #4111 (hierarchical findings) has since landed and added parent_finding_id and demonstrated to FindingCreate. This branch drops both, and GitHub shows it as conflicting. The rebase needs decisions on how sub-findings work with configured fields (including whether sub-findings keep their own severity), and on the alembic chain.

  2. One field set per deployment. Every operation shares the single active config, so changing it re-labels findings in past operations: old keys become Retired and the sort order changes. The docs only say "one FindingsInitializer per deployment" and don't mention that. That's my main feedback for this pr specifically--to add this in the docs.

but also as a follow up did we considered letting a deployment define several named field sets, with one default for new operations, and having each operation record the set it was created with? Even if that's out of scope here, should the API be operation-scoped (e.g. GET /operations/{id}/finding-fields) so we can add it later without another breaking change?

I think we talked about this shared finding template before but idk if we had landed on the initializers at that point

Comment thread pyrit/models/finding_fields.py Resolved
Comment thread pyrit/models/finding_fields.py Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…firmation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adrian-gavrila Adrian Gavrila (adrian-gavrila) changed the title FEAT: Let each deployment configure its finding fields [BREAKING] FEAT: Let each deployment configure its finding fields Nov 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated code review at 4611845. The deprecation path and the [BREAKING] title resolve the earlier inclusion request. The inline comments below list new defects. Replies in two earlier threads say what is still open.

Follow-up to the deprecation request in #3655 (review) (also see #3655 (comment)). Two released 1.2 behaviors still change without a warning:

  1. [P2] Default order of memory finding reads. In 1.2, get_findings and the related memory reads always sorted by severity (highest first, then newest). Now severity_order defaults to () (pyrit/memory/memory_interface.py:1192, also 1251, 1305, 1334, 1477), and _finding_order then returns newest first only. A 1.2 caller that does not pass the new argument gets a different order and no error. Make the default DEFAULT_FINDING_FIELDS.severity_order (or let None mean that order), and add a memory test for the default call. If the change is intentional, list it under Upgrading in doc/code/setup/finding_fields.md.

  2. [P3] Finding.model_dump() and model_dump_json() no longer include the 1.2 keys description, severity_other, harm_type, and harm_type_other, and severity is now a plain string instead of a FindingSeverity. The deprecated properties still work, but code that serializes a finding and reads those keys gets a KeyError or a missing value. Add this to the list of checks that cannot keep working (doc/code/setup/finding_fields.md:115), or include the deprecated keys in the dump during the deprecation period.

Comment thread doc/code/setup/finding_fields.md Resolved
Comment thread pyrit/models/finding.py Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated follow-up code review at 9ca49a7. The fixes for the earlier comments are verified at this head: the default severity order, the model_dump notes, the number input, the migration table, the OTHER check, the attack details fields error, the Other label, the guard test, the export reload and the combined key limit test. The inline comment below is about a layout problem in the new fields error bar.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants