Repository navigation
FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) - #4111
Conversation
Add a shared DecompositionPath/DecompositionTree shape (pyrit.models.decomposition) and use it on SeedObjective: children name their parent and the parent lists ordered paths of direct children. YAML datasets nest sub_objectives with local keys. Memory persists the parent link and paths, stores and deduplicates each tree as one unit, and reads a stored subtree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Identical sibling sub-objectives tied in the canonical sort, so path step indexes followed input or read order and an equivalent tree was stored again. Break ties by each sibling's first use in the parent's paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Also import SeedEntry from pyrit.memory in the decomposition tests, as the public-import check now requires. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Findings now decompose like SeedObjectives, using the shared
pyrit.models.decomposition shape without being linked to objectives:
a sub-finding names its parent, and a parent lists DecompositionPaths,
each an ordered chain of its direct sub-findings. Every finding keeps its
own severity and a demonstrated flag; a path is complete only when every
step was demonstrated, and that status is derived, never stored.
Memory stores the parent link, demonstrated flag, and paths on
FindingEntries (migration 7e1c3a5b9d20), validates parents, cycles, and
path steps under a per-operation lock, deletes whole subtrees, and
rejects moving or deleting a finding its parent's paths still use. The
API adds parent/top-level list filters, child counts, path completeness,
and PUT .../findings/{id}/paths. Markdown and Word exports nest numbered
sub-findings with their paths and missing steps, and the CoPyRIT
operation page shows the finding tree, sub-finding creation, a
demonstrated toggle, a paths editor, and subtree export selection.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A decomposition is usually written step by step, so a parent's direct sub-findings now read oldest first: in the parent-filtered list API (and so the CoPyRIT tree, paths panel, and paths editor), the export dialog's nesting, and Markdown/Word report numbering. Path chains therefore read 1.1 -> 1.2 -> 1.4. Top-level findings keep the dashboard order (severity, then newest). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
main now type-checks indexed access as possibly undefined, so the paths editor guards a moved step and the tree falls back to the deepest heading. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a diagram of a finding, its sub-findings, and their paths to the Operations section of the GUI guide: what the operator records, how memory stores and guards the tree, and how Markdown and Word reports show it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent tree updates can cause valid exports to fail because reads lack a consistent database snapshot, while recursive dialog rendering can exhaust the JavaScript call stack on deep hierarchies.
2 open issues
What changed in this PR
Adds hierarchical, operator-authored findings to PyRIT, reusing the decomposition shape from #4110 without linking findings to seed objectives.
Changes:
- Adds parent relationships, demonstrated status, ordered paths, persistence checks, and API support.
- Introduces expandable finding trees and path editing in the GUI.
- Extends Markdown and Word exports with numbered subtrees, path completeness, and subtree-aware selection limits.
| File | Description |
|---|---|
| tests/unit/output/finding_report/test_markdown.py | Tests nested headings, paths, status, and escaping. |
| tests/unit/models/test_seed_decomposition.py | Tests stacked seed decomposition behavior. |
| tests/unit/models/test_finding.py | Tests finding relationships and path completeness. |
| tests/unit/models/test_decomposition.py | Tests shared tree validation and traversal. |
| tests/unit/memory/test_seed_decomposition.py | Tests stacked tree persistence and deduplication. |
| tests/unit/memory/test_seed_decomposition_migration.py | Tests seed migration compatibility. |
| tests/unit/memory/test_operations_and_findings_migration.py | Uses revision-specific table definitions. |
| tests/unit/memory/test_finding_decomposition_migration.py | Tests finding migration preservation and SQL generation. |
| tests/unit/backend/test_operations.py | Updates finding response expectations. |
| tests/unit/backend/test_finding_export.py | Tests subtree exports, numbering, and limits. |
| pyrit/output/finding_report/report.py | Adds hierarchical report entries and paths. |
| pyrit/output/finding_report/markdown.py | Renders numbered subtrees and path status. |
| pyrit/models/seeds/seed_record.py | Exposes stored objective relationships. |
| pyrit/models/seeds/seed_objective.py | Adds stacked objective decomposition support. |
| pyrit/models/seeds/seed_dataset.py | Flattens nested objectives and resolves paths. |
| pyrit/models/seeds/__init__.py | Exports the objective tree builder. |
| pyrit/models/finding.py | Adds finding hierarchy and completeness helpers. |
| pyrit/models/decomposition.py | Defines shared paths and validated trees. |
| pyrit/models/__init__.py | Exports decomposition and finding helpers. |
| pyrit/memory/memory_models.py | Maps finding and objective decomposition fields. |
| pyrit/memory/alembic/versions/7e1c3a5b9d20_add_finding_decomposition.py | Adds finding hierarchy columns and constraints. |
| pyrit/memory/alembic/versions/5d7f9b1c3e2a_add_seed_decomposition.py | Adds stacked objective decomposition columns. |
| pyrit/memory/_seed_decomposition.py | Handles whole-tree identity and deduplication. |
| pyrit/memory/_finding_decomposition.py | Implements locking, validation, and subtree deletion. |
| pyrit/backend/services/operation_service.py | Supports hierarchy operations and derived path status. |
| pyrit/backend/services/finding_export.py | Builds and renders hierarchical exports. |
| pyrit/backend/routes/operations.py | Exposes hierarchy filters and path updates. |
| pyrit/backend/models/operations.py | Defines hierarchy request and response fields. |
| frontend/src/utils/findingTree.ts | Provides tree traversal and path helpers. |
| frontend/src/utils/findingTree.test.ts | Tests ordering, traversal, and completeness. |
| frontend/src/types/index.ts | Adds hierarchy and path API types. |
| frontend/src/services/api.ts | Supports hierarchy filters and path replacement. |
| frontend/src/services/api.test.ts | Tests hierarchy API requests. |
| frontend/src/components/Operations/OperationDetailPage.tsx | Integrates tree actions and dialogs. |
| frontend/src/components/Operations/FindingTreeItem.tsx | Displays expandable findings and loads children. |
| frontend/src/components/Operations/FindingTreeItem.styles.ts | Styles nested findings and status badges. |
| frontend/src/components/Operations/FindingPathsPanel.tsx | Displays ordered paths and completeness. |
| frontend/src/components/Operations/FindingPathsPanel.styles.ts | Styles path summaries and steps. |
| frontend/src/components/Operations/FindingPathsDialog.tsx | Edits paths and their ordered steps. |
| frontend/src/components/Operations/FindingPathsDialog.test.tsx | Tests path editing and save behavior. |
| frontend/src/components/Operations/FindingPathsDialog.styles.ts | Styles the path editor. |
| frontend/src/components/Operations/FindingExportDialog.tsx | Adds subtree-aware export selection. |
| frontend/src/components/Operations/FindingExportDialog.test.tsx | Tests nested selection and export limits. |
| frontend/src/components/Operations/FindingExportDialog.styles.ts | Styles nested export choices. |
| frontend/src/components/Operations/FindingDialog.tsx | Shows parent context and demonstrated status. |
| frontend/src/components/Operations/FindingDialog.test.tsx | Tests sub-finding creation and draft protection. |
| frontend/src/components/Chat/FindingEvidenceDialog.test.tsx | Updates fixtures and creation expectations. |
| doc/gui/0_gui.md | Explains sub-findings, paths, and exports. |
| doc/code/framework.md | Clarifies shared decomposition responsibilities. |
| doc/code/datasets/3_dataset_writing.md | Documents stacked objective decomposition syntax. |
| doc/code/datasets/0_dataset.md | Links to objective decomposition guidance. |
Review effort: Balanced
- Reject a decomposition tree whose objectives or grouped prompts are not all in the root's dataset; matching is scoped by the root's dataset, so a member in another dataset could reuse a stored row from the wrong dataset. - When an equivalent tree is already stored, give each caller seed its stored counterpart's identifiers, including absent group IDs. - Add root_id to build_objective_decomposition_tree so a subtree read from memory below the root can be walked without dropping its parent link. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bring the reviewed seed-decomposition repairs from the lower stacked PR forward: one-dataset decomposition trees, stored group IDs on reload, and walking a stored subtree from any objective. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…inding actions - Read an export's finding trees in one transaction under the operation's finding lock, then attach evidence to those findings without re-reading them, so a concurrent sub-finding and path change cannot leave a path step outside the report. - Render the export dialog as one flat list in tree order, indenting sub-findings and naming their parent for screen readers, so a deep decomposition no longer exhausts the call stack. - Let a finding's Edit / Add sub-finding / Delete actions wrap so rows fit at 390px. - E2E: expect numbered Markdown headings and cover a nested sub-finding row at 390px. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Both issues from this review are fixed in 478bafb. Exports read each finding tree in one locked transaction and attach evidence without reading the findings again. The export dialog renders a flat list, so deep trees no longer recurse. The same commit fixes the two Frontend E2E failures: the Markdown heading check now expects numbered headings, and a finding's Edit / Add sub-finding / Delete buttons wrap so rows, including nested sub-finding rows, fit at 390px. It also merges #4110's head 55d365b. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reject partial-tree deletions before writes, deduplicate trees by execution and structure while updating descriptive metadata, and order reloads deterministically. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep assertions on the actual delete filter while allowing the new integrity query to report no partial tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep behavioral template declarations distinct through direct and prepared ingestion, with parameter-filtered retrieval regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate recorded dependency head 72b507e without rewriting PR history. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main 614768c while retaining finding decomposition, validation tests, and seed technique metadata. Join the two Alembic heads with an empty merge revision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep draft path verdicts library-owned and move workflow details out of framework guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Verified both reported causes at Verified on published commit |
Avoid current ORM defaults before their columns exist; retain branch upgrade and preservation assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
|
Left some comments and this isn't blocking at all but would be helpful for some images or a demo/notebook etc for changes! |
jbolor21
left a comment
There was a problem hiding this comment.
left a few small comments & haven't had time to play around with the UI enough for more comments but I think other changes I'll comment if I see them as I interact more with the UI! (Or ofc can make future PRs).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Main enables RUF100 for new Alembic revisions, so the TC003 suppression in this PR's unmerged revision is now unused. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Number root findings and sub-findings the way a full export numbers them, and reuse those numbers in path steps and the Edit paths dialog. - Drop the stale Complete/Partial expects from the Edit paths test and the dialog intro; keep step controls on one line at narrow widths. - Compute report path completeness with is_finding_path_complete. - Test promoting a real sub-finding in memory and through the API. - Replace the GUI guide diagram with a user-facing one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Thanks! Here are images from the GUI at this commit (a seeded operation, captured with Playwright), plus the simplified diagram now in the GUI guide. Finding tree with numbered sub-findings and paths: Edit paths dialog, with steps labelled by number: The same dialog at 390px, where step controls now stay on the step's line: I also replaced I didn't add a notebook: findings are written by operators in the GUI and the REST API rather than from Python code, and the GUI guide ( Verified on published commit |
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.






Description
Lets operators record a finding as a tree of sub-findings, each demonstrated or not, plus the ordered paths through them, and carries that structure into Markdown and Word reports. Findings use the same
pyrit.models.decompositionshape as decomposedSeedObjectives, but stay manual and aren't linked to objectives.The stack
Model:
FindingCreategainsparent_finding_idanddemonstrated(default true).Findinggainspaths: tuple[DecompositionPath, ...], validated withvalidate_own_decomposition. Each path is an ordered chain of direct sub-findings. A path is complete only when every step was demonstrated;is_finding_path_complete()derives this on read and it's never stored. Every finding keeps its own severity.Memory and migration: migration
7e1c3a5b9d20adds three columns toFindingEntries:parent_finding_id: a self foreign key, indexeddemonstrated: not null, server default truepaths: JSONEvery change takes a per-operation lock (the SQLite writer lock, or
UPDLOCK, HOLDLOCKon SQL Server). Under that lock, memory checks that a parent is in the same operation, that moves create no cycle, and that path steps are direct sub-findings. Deleting a finding deletes its whole subtree and the subtree's evidence. Moving or deleting a finding that its parent's paths use is rejected until those paths change. New queries list a parent's sub-findings, list top-level findings only, and count sub-findings.API:
parent_finding_idanddemonstrated. A missing parent returns 404; conflicts with the finding tree return 409.PUT /api/operations/{operation_id}/findings/{finding_id}/pathsreplaces a finding's paths.parent_finding_idandtop_levelfilters, and each item now carrieschild_countand per-pathcomplete. With no filter it still returns every finding.Reports:
1. **Credential theft** (partial): 1.1 Recon → 1.2 Obtain creds (not demonstrated) → 1.4 Pull records.GUI:
Docs: the Operations section of the GUI guide explains sub-findings and paths and includes the diagram above.
doc/code/framework.mdnotes that findings share the decomposition shape without being linked to objectives.Behavior choices:
1,1.2, …), including flat reports, so path steps can refer to sub-findings.PUTon a finding replaces it in full: omittingparent_finding_idmoves it to the top level. A finding used as a path step is still protected by the 409, and the GUI always sends the parent.