Repository navigation
[BREAKING] FEAT: Let each deployment configure its finding fields - #3655
Adrian Gavrila (adrian-gavrila) wants to merge 45 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…values Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects can drop custom-field data, repeat or skip findings after severity changes, misvalidate replacement environments, and conflate distinct severities in SQL Server.
5 open issues
What changed in this PR
This follow-up to #3066 makes finding fields deployment-configurable across PyRIT’s storage, API, setup, and CoPyRIT interface, while retaining the existing fields by default.
Changes:
- Adds initializer-based field configuration, validation, and live-apply preflight checks.
- Migrates finding values to JSON storage and adapts ordering, updates, and exports.
- Replaces fixed frontend fields with configurable controls, retired-value handling, and supporting tests and documentation.
| File | Description |
|---|---|
| tests/unit/setup/test_reinitialization.py | Tests finding-field preflight failures. |
| tests/unit/setup/test_initialization.py | Tests field resets and setup validation. |
| tests/unit/setup/test_findings_initializer.py | Tests initializer execution and discovery. |
| tests/unit/setup/test_finding_fields.py | Tests active-field state management. |
| tests/unit/setup/conftest.py | Adds custom-field fixtures. |
| tests/unit/models/test_finding.py | Tests flexible finding values. |
| tests/unit/memory/test_operations_and_findings_migration.py | Preserves earlier migration tests. |
| tests/unit/memory/test_findings.py | Tests persistence and configurable ordering. |
| tests/unit/memory/test_finding_field_values_migration.py | Tests migration data and database schema changes. |
| tests/unit/conftest.py | Adds field-state cleanup fixture. |
| tests/unit/backend/test_operations.py | Tests field-aware routes and updates. |
| tests/unit/backend/test_operation_service_findings.py | Tests retired-value preservation and removal. |
| tests/unit/backend/test_finding_export.py | Tests configurable export projection. |
| tests/unit/backend/test_common_models.py | Tests finding request limits. |
| tests/unit/backend/mocks.py | Adds persisted-finding read helper. |
| tests/unit/backend/conftest.py | Adds configured and legacy finding fixtures. |
| pyrit/setup/initializers/findings.py | Introduces deployment field initializer. |
| pyrit/setup/initializers/__init__.py | Exports the initializer lazily. |
| pyrit/setup/initialization.py | Resets fields during setup passes. |
| pyrit/setup/finding_fields.py | Holds active deployment fields. |
| pyrit/setup/configuration_loader.py | Adds field-definition preflight checks. |
| pyrit/models/finding.py | Replaces fixed classifications with field values. |
| pyrit/models/__init__.py | Updates public finding exports. |
| pyrit/memory/memory_models.py | Stores severity and JSON field values. |
| pyrit/memory/memory_interface.py | Accepts caller-defined severity ordering. |
| pyrit/memory/alembic/versions/d43b1bcf9c56_store_finding_field_values.py | Migrates existing finding values and constraints. |
| pyrit/backend/services/operation_service.py | Validates values and preserves retired fields. |
| pyrit/backend/services/finding_export.py | Projects configured, non-internal fields. |
| pyrit/backend/routes/operations.py | Exposes fields and validation responses. |
| pyrit/backend/models/operations.py | Defines bounded configurable-field requests. |
| frontend/src/utils/findingSeverity.ts | Removes fixed severity helpers. |
| frontend/src/utils/findingSeverity.test.ts | Removes fixed severity tests. |
| frontend/src/utils/findingFields.ts | Adds field, cascade, and badge helpers. |
| frontend/src/utils/findingFields.test.ts | Tests configurable-field helpers. |
| frontend/src/types/index.ts | Defines configurable finding types. |
| frontend/src/test-utils/findingFieldsFixtures.ts | Adds default and custom field fixtures. |
| frontend/src/test-utils/findingDialog.ts | Adds shared dialog test helpers. |
| frontend/src/services/api.ts | Updates finding API methods. |
| frontend/src/services/api.test.ts | Tests revised API requests. |
| frontend/src/hooks/useFindingFields.ts | Loads fields and handles runtime refreshes. |
| frontend/src/hooks/useFindingFields.test.tsx | Tests loading, retries, and refreshes. |
| frontend/src/components/Operations/RetiredFieldValues.tsx | Adds removal confirmation and undo controls. |
| frontend/src/components/Operations/RetiredFieldValues.test.tsx | Tests retired-value controls and focus. |
| frontend/src/components/Operations/RetiredFieldValues.styles.ts | Styles retired-value panels. |
| frontend/src/components/Operations/OperationsPage.tsx | Renders configured severity counts. |
| frontend/src/components/Operations/OperationsPage.test.tsx | Tests counts and field-loading failures. |
| frontend/src/components/Operations/Operations.styles.ts | Styles configurable and retired values. |
| frontend/src/components/Operations/OperationDetailPage.tsx | Integrates configurable finding displays and dialogs. |
| frontend/src/components/Operations/FindingSeverityBadge.tsx | Uses configured severity labels and colors. |
| frontend/src/components/Operations/FindingFieldInput.tsx | Adds text, choice, and cascade controls. |
| frontend/src/components/Operations/FindingFieldInput.styles.ts | Styles configurable inputs. |
| frontend/src/components/Operations/FindingExportDialog.tsx | Warns about excluded retired values. |
| frontend/src/components/Operations/FindingExportDialog.test.tsx | Tests retired-value export warnings. |
| frontend/src/components/Operations/FindingDialog.tsx | Builds drafts from configured fields. |
| frontend/src/components/Operations/FindingDialog.styles.ts | Adds field-section styling. |
| frontend/src/components/Operations/FindingCardValues.tsx | Displays configured and retired values. |
| frontend/src/components/Chat/FindingEvidenceDialog.tsx | Uses configurable fields in evidence selection. |
| frontend/src/components/Chat/FindingEvidenceDialog.test.tsx | Tests field-aware evidence workflows. |
| frontend/e2e/operations.spec.ts | Updates requests and tests retired-value interactions. |
| doc/myst.yml | Adds finding-field documentation to navigation. |
| doc/gui/0_gui.md | Describes deployment-defined finding fields. |
| doc/code/setup/finding_fields.md | Documents configuration, rules, and migration. |
Review effort: Balanced
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Re-point d43b1bcf9c56 after e7c4a2d9f013 from microsoft#3644, so the chain keeps a single head. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t reloads Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… flag Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Done in 5e196af. The |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Not done reviewing yet, but heads up: #3066 shipped in v1.2.0, so removing |
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
[P2] Deprecate the released findings API before removing it
The new export behavior addresses my earlier review: retired values are included, and the report projection is reusable from the library.
Following up on Hannah's release question: I checked the published 1.2.0 and 1.2.1 wheels. Both export FindingSeverity, accept the old finding fields, and serve GET /api/operations/finding-options. With this PR in 1.3, an existing client can no longer import FindingSeverity, an unchanged create payload with description is rejected, and the old route is gone. Migrating stored rows does not migrate those clients.
Please keep the configurable fields and deprecate the old Python and REST contracts with a compatibility layer scheduled for removal in 1.5.0. A [BREAKING] title and code migration notes would document the break if maintainers explicitly make a release-policy exception, but the label alone does not give clients a transition. Please also update the PR description's premise that #3066 has not shipped.
There was a problem hiding this comment.
A couple of broader points:
-
Conflicts with main. #4111 (hierarchical findings) has since landed and added
parent_finding_idanddemonstratedtoFindingCreate. This branch drops both, and GitHub shows it as conflicting. The rebase needs decisions on how sub-findings work with configured fields (including whether sub-findings keep their own severity), and on the alembic chain. -
One field set per deployment. Every operation shares the single active config, so changing it re-labels findings in past operations: old keys become Retired and the sort order changes. The docs only say "one
FindingsInitializerper deployment" and don't mention that. That's my main feedback for this pr specifically--to add this in the docs.
but also as a follow up did we considered letting a deployment define several named field sets, with one default for new operations, and having each operation record the set it was created with? Even if that's out of scope here, should the API be operation-scoped (e.g. GET /operations/{id}/finding-fields) so we can add it later without another breaking change?
I think we talked about this shared finding template before but idk if we had landed on the initializers at that point
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…firmation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated code review at 4611845. The deprecation path and the [BREAKING] title resolve the earlier inclusion request. The inline comments below list new defects. Replies in two earlier threads say what is still open.
Follow-up to the deprecation request in #3655 (review) (also see #3655 (comment)). Two released 1.2 behaviors still change without a warning:
-
[P2] Default order of memory finding reads. In 1.2,
get_findingsand the related memory reads always sorted by severity (highest first, then newest). Nowseverity_orderdefaults to()(pyrit/memory/memory_interface.py:1192, also 1251, 1305, 1334, 1477), and_finding_orderthen returns newest first only. A 1.2 caller that does not pass the new argument gets a different order and no error. Make the defaultDEFAULT_FINDING_FIELDS.severity_order(or letNonemean that order), and add a memory test for the default call. If the change is intentional, list it under Upgrading indoc/code/setup/finding_fields.md. -
[P3]
Finding.model_dump()andmodel_dump_json()no longer include the 1.2 keysdescription,severity_other,harm_type, andharm_type_other, andseverityis now a plain string instead of aFindingSeverity. The deprecated properties still work, but code that serializes a finding and reads those keys gets aKeyErroror a missing value. Add this to the list of checks that cannot keep working (doc/code/setup/finding_fields.md:115), or include the deprecated keys in the dump during the deprecation period.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated follow-up code review at 9ca49a7. The fixes for the earlier comments are verified at this head: the default severity order, the model_dump notes, the number input, the migration table, the OTHER check, the attack details fields error, the Other label, the guard test, the export reload and the combined key limit test. The inline comment below is about a layout problem in the new fields error bar.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Description
This is the follow-up promised in #3066 (Operations, findings, evidence). #3066 fixed a finding's severity, harm type and description. This PR lets each deployment define its own finding fields in Python instead. A deployment that configures nothing keeps the #3066 fields (Critical through Informational, the HarmCategory list, a description), and its findings migrate in place.
Breaking change and migration
#3066 shipped in 1.2.0. Its fixed columns can't hold fields a deployment defines, so a finding is now a
severitystring plus afield_valuesobject. The PyRIT 1.2 API keeps working, with deprecation warnings, until 1.5.0:FindingSeverityFindingFields.severity, such as"critical"descriptionfield_values.get("description", "")harm_typeandharm_type_otherfield_values.get("harm_type"), which holds the category or its Other textseverity_otherseverity, which holds Other text itselffinding.severity.valuefinding.severityGET /api/operations/finding-optionsGET /api/operations/{operation_id}/finding-fieldsfield_values, so read it withget.FindingSeverity.OTHER, because the severity is the text itself. A severity is astr, so it isn't aFindingSeverityinstance.Finding.model_dump()andmodel_dump_json()no longer include the old fields.What changes
FindingsInitializer(pyrit/setup/initializers/findings.py) and returns aFindingFieldsfromget_finding_fields().severityslot is always a required choice field. Its options run from highest to lowest, and findings sort in that order.levelsand Other text), single-line text, multi-line text, or numbers (a plain decimal such as9.8). They can carryhintandsection.DEFAULT_FINDING_FIELDSreproduces FEAT: Findings dashboard with persistent Operations in the GUI #3066.Storage. One migration,
d43b1bcf9c56, after3e55dd099e38.field_valuescolumn.severitybecomes unbounded Unicode, with a binary collation on SQL Server so sorting and counting match SQLite.FindingFields: the service passes it the severity order. Reads without one use the default fields' order, as in 1.2.API.
GET /api/operations/{operation_id}/finding-fieldsreturns the fields that label and check an operation's findings.GET /api/operations/finding-fieldsreturns the deployment's fields, for views across operations. Both return the one active set today, so per-operation field sets can come later without another breaking change.remove_retired_keys. Keys listed inkeep_retired_keyskeep their stored values even if a live apply configured the field again.TextStr,IdentifierStr,MAX_ITEMS).Setup. The live-apply preflight rejects invalid finding fields, or two
FindingsInitializers, before changing anything. A scenario run can't run aFindingsInitializer.Export. The Word and Markdown export follows the fields: multi-line fields go in the Finding column, choice, text and number fields in Assessment, and retired values follow in the Finding column, labelled by their keys.
finding_report_entryinpyrit.output.finding_report.reportbuilds each report entry from the fields, andbuild_finding_report(from FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) #4111) now takesfields.CoPyRIT. The finding dialog, cards, Operations counts, the Chat evidence picker and the export dialog all render from the finding fields.
doc/code/setup/finding_fields.md(indoc/myst.yml) explains how to define fields, how changing them affects existing findings, and how to migrate code written for PyRIT 1.2.doc/gui/0_gui.mddescribes deployment-defined finding fields.