Skip to content

FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) - #4111

Merged
Richard Lundeen (richlundeen) merged 23 commits into
microsoft:mainfrom
richlundeen:richlundeen-hierarchical-findings
Nov 17, 2026
Merged

Richard Lundeen (richlundeen) merged 23 commits into
microsoft:mainfrom
richlundeen:richlundeen-hierarchical-findings

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #4110; review only the commits after 0b2ddc4.

Description

Lets operators record a finding as a tree of sub-findings, each demonstrated or not, plus the ordered paths through them, and carries that structure into Markdown and Word reports. Findings use the same pyrit.models.decomposition shape as decomposed SeedObjectives, but stay manual and aren't linked to objectives.

The stack

  1. FEAT: Decompose seed objectives into children and ordered paths (part 1) #4110 Decomposed seed objectives
  2. this PR

A finding with four sub-findings, numbered 1.1 to 1.4 in the order they were added; 1.2 Obtain creds was not demonstrated. Two paths order the direct sub-findings: Credential theft is partial because of 1.2, and Phishing is complete. Memory stores one row per finding with parent, demonstrated, and paths columns; deleting a finding deletes its sub-findings, and a path step can't be deleted or moved. Markdown and Word exports number and nest the sub-findings and mark each path complete or partial.

  • Model: FindingCreate gains parent_finding_id and demonstrated (default true). Finding gains paths: tuple[DecompositionPath, ...], validated with validate_own_decomposition. Each path is an ordered chain of direct sub-findings. A path is complete only when every step was demonstrated; is_finding_path_complete() derives this on read and it's never stored. Every finding keeps its own severity.

  • Memory and migration: migration 7e1c3a5b9d20 adds three columns to FindingEntries:

    • parent_finding_id: a self foreign key, indexed
    • demonstrated: not null, server default true
    • paths: JSON

    Every change takes a per-operation lock (the SQLite writer lock, or UPDLOCK, HOLDLOCK on SQL Server). Under that lock, memory checks that a parent is in the same operation, that moves create no cycle, and that path steps are direct sub-findings. Deleting a finding deletes its whole subtree and the subtree's evidence. Moving or deleting a finding that its parent's paths use is rejected until those paths change. New queries list a parent's sub-findings, list top-level findings only, and count sub-findings.

  • API:

    • Create and update accept parent_finding_id and demonstrated. A missing parent returns 404; conflicts with the finding tree return 409.
    • The new PUT /api/operations/{operation_id}/findings/{finding_id}/paths replaces a finding's paths.
    • Listing findings takes optional parent_finding_id and top_level filters, and each item now carries child_count and per-path complete. With no filter it still returns every finding.
  • Reports:

    • Each selected finding brings its whole subtree; a sub-finding inside a selected finding's subtree isn't listed a second time.
    • A sub-finding selected without its parent appears at the top level with Part of.
    • Markdown nests sub-findings under deeper headings (up to h6), adds Status: Not demonstrated, and lists paths such as 1. **Credential theft** (partial): 1.1 Recon → 1.2 Obtain creds (not demonstrated) → 1.4 Pull records.
    • Word writes rows depth-first: paths go in the Finding column, status in the Assessment column.
    • The export limit counts sub-findings.
  • GUI:

    • The operation page lists findings as an expandable tree, with Sub-findings (n) and a Not demonstrated badge.
    • Add sub-finding opens the finding dialog, which shows the parent and a Demonstrated switch.
    • Each expanded finding shows a paths panel with Complete and Partial badges. Edit paths adds, renames, reorders, and removes paths, choosing steps from the finding's direct sub-findings.
    • In the export dialog, checking a finding includes its subtree.
  • Docs: the Operations section of the GUI guide explains sub-findings and paths and includes the diagram above. doc/code/framework.md notes that findings share the decomposition shape without being linked to objectives.

  • Behavior choices:

    • Exports number every finding (1, 1.2, …), including flat reports, so path steps can refer to sub-findings.
    • Sub-findings are listed oldest first, in the order they were written, so paths read 1.1 → 1.2 → 1.4. Top-level findings keep the dashboard order: severity, then newest.
    • PUT on a finding replaces it in full: omitting parent_finding_id moves it to the top level. A finding used as a path step is still protected by the 409, and the GUI always sends the parent.
    • Moving a finding to another parent is API-only; the GUI shows the parent but doesn't change it.

Add a shared DecompositionPath/DecompositionTree shape (pyrit.models.decomposition)
and use it on SeedObjective: children name their parent and the parent lists
ordered paths of direct children. YAML datasets nest sub_objectives with local
keys. Memory persists the parent link and paths, stores and deduplicates each
tree as one unit, and reads a stored subtree.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Identical sibling sub-objectives tied in the canonical sort, so path step
indexes followed input or read order and an equivalent tree was stored again.
Break ties by each sibling's first use in the parent's paths.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Also import SeedEntry from pyrit.memory in the decomposition tests, as the
public-import check now requires.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Findings now decompose like SeedObjectives, using the shared
pyrit.models.decomposition shape without being linked to objectives:
a sub-finding names its parent, and a parent lists DecompositionPaths,
each an ordered chain of its direct sub-findings. Every finding keeps its
own severity and a demonstrated flag; a path is complete only when every
step was demonstrated, and that status is derived, never stored.

Memory stores the parent link, demonstrated flag, and paths on
FindingEntries (migration 7e1c3a5b9d20), validates parents, cycles, and
path steps under a per-operation lock, deletes whole subtrees, and
rejects moving or deleting a finding its parent's paths still use. The
API adds parent/top-level list filters, child counts, path completeness,
and PUT .../findings/{id}/paths. Markdown and Word exports nest numbered
sub-findings with their paths and missing steps, and the CoPyRIT
operation page shows the finding tree, sub-finding creation, a
demonstrated toggle, a paths editor, and subtree export selection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A decomposition is usually written step by step, so a parent's direct
sub-findings now read oldest first: in the parent-filtered list API (and
so the CoPyRIT tree, paths panel, and paths editor), the export dialog's
nesting, and Markdown/Word report numbering. Path chains therefore read
1.1 -> 1.2 -> 1.4. Top-level findings keep the dashboard order
(severity, then newest).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
main now type-checks indexed access as possibly undefined, so the paths
editor guards a moved step and the tree falls back to the deepest heading.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a diagram of a finding, its sub-findings, and their paths to the
Operations section of the GUI guide: what the operator records, how memory
stores and guards the tree, and how Markdown and Word reports show it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 9, 2026 16:21
@richlundeen Richard Lundeen (richlundeen) changed the title FEAT: Hierarchical findings with sub-findings and ordered paths FEAT: Hierarchical findings with sub-findings and ordered paths (part 2) Nov 9, 2026
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as draft November 9, 2026 16:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent tree updates can cause valid exports to fail because reads lack a consistent database snapshot, while recursive dialog rendering can exhaust the JavaScript call stack on deep hierarchies.

2 open issues
What changed in this PR

Adds hierarchical, operator-authored findings to PyRIT, reusing the decomposition shape from #4110 without linking findings to seed objectives.

Changes:

  • Adds parent relationships, demonstrated status, ordered paths, persistence checks, and API support.
  • Introduces expandable finding trees and path editing in the GUI.
  • Extends Markdown and Word exports with numbered subtrees, path completeness, and subtree-aware selection limits.
File Description
tests/​unit/​output/​finding_report/​test_markdown.py Tests nested headings, paths, status, and escaping.
tests/​unit/​models/​test_seed_decomposition.py Tests stacked seed decomposition behavior.
tests/​unit/​models/​test_finding.py Tests finding relationships and path completeness.
tests/​unit/​models/​test_decomposition.py Tests shared tree validation and traversal.
tests/​unit/​memory/​test_seed_decomposition.py Tests stacked tree persistence and deduplication.
tests/​unit/​memory/​test_seed_decomposition_migration.py Tests seed migration compatibility.
tests/​unit/​memory/​test_operations_and_findings_migration.py Uses revision-specific table definitions.
tests/​unit/​memory/​test_finding_decomposition_migration.py Tests finding migration preservation and SQL generation.
tests/​unit/​backend/​test_operations.py Updates finding response expectations.
tests/​unit/​backend/​test_finding_export.py Tests subtree exports, numbering, and limits.
pyrit/​output/​finding_report/​report.py Adds hierarchical report entries and paths.
pyrit/​output/​finding_report/​markdown.py Renders numbered subtrees and path status.
pyrit/​models/​seeds/​seed_record.py Exposes stored objective relationships.
pyrit/​models/​seeds/​seed_objective.py Adds stacked objective decomposition support.
pyrit/​models/​seeds/​seed_dataset.py Flattens nested objectives and resolves paths.
pyrit/​models/​seeds/​__init__.py Exports the objective tree builder.
pyrit/​models/​finding.py Adds finding hierarchy and completeness helpers.
pyrit/​models/​decomposition.py Defines shared paths and validated trees.
pyrit/​models/​__init__.py Exports decomposition and finding helpers.
pyrit/​memory/​memory_models.py Maps finding and objective decomposition fields.
pyrit/​memory/​alembic/​versions/​7e1c3a5b9d20_add_finding_decomposition.py Adds finding hierarchy columns and constraints.
pyrit/​memory/​alembic/​versions/​5d7f9b1c3e2a_add_seed_decomposition.py Adds stacked objective decomposition columns.
pyrit/​memory/​_seed_decomposition.py Handles whole-tree identity and deduplication.
pyrit/​memory/​_finding_decomposition.py Implements locking, validation, and subtree deletion.
pyrit/​backend/​services/​operation_service.py Supports hierarchy operations and derived path status.
pyrit/​backend/​services/​finding_export.py Builds and renders hierarchical exports.
pyrit/​backend/​routes/​operations.py Exposes hierarchy filters and path updates.
pyrit/​backend/​models/​operations.py Defines hierarchy request and response fields.
frontend/​src/​utils/​findingTree.ts Provides tree traversal and path helpers.
frontend/​src/​utils/​findingTree.test.ts Tests ordering, traversal, and completeness.
frontend/​src/​types/​index.ts Adds hierarchy and path API types.
frontend/​src/​services/​api.ts Supports hierarchy filters and path replacement.
frontend/​src/​services/​api.test.ts Tests hierarchy API requests.
frontend/​src/​components/​Operations/​OperationDetailPage.tsx Integrates tree actions and dialogs.
frontend/​src/​components/​Operations/​FindingTreeItem.tsx Displays expandable findings and loads children.
frontend/​src/​components/​Operations/​FindingTreeItem.styles.ts Styles nested findings and status badges.
frontend/​src/​components/​Operations/​FindingPathsPanel.tsx Displays ordered paths and completeness.
frontend/​src/​components/​Operations/​FindingPathsPanel.styles.ts Styles path summaries and steps.
frontend/​src/​components/​Operations/​FindingPathsDialog.tsx Edits paths and their ordered steps.
frontend/​src/​components/​Operations/​FindingPathsDialog.test.tsx Tests path editing and save behavior.
frontend/​src/​components/​Operations/​FindingPathsDialog.styles.ts Styles the path editor.
frontend/​src/​components/​Operations/​FindingExportDialog.tsx Adds subtree-aware export selection.
frontend/​src/​components/​Operations/​FindingExportDialog.test.tsx Tests nested selection and export limits.
frontend/​src/​components/​Operations/​FindingExportDialog.styles.ts Styles nested export choices.
frontend/​src/​components/​Operations/​FindingDialog.tsx Shows parent context and demonstrated status.
frontend/​src/​components/​Operations/​FindingDialog.test.tsx Tests sub-finding creation and draft protection.
frontend/​src/​components/​Chat/​FindingEvidenceDialog.test.tsx Updates fixtures and creation expectations.
doc/​gui/​0_gui.md Explains sub-findings, paths, and exports.
doc/​code/​framework.md Clarifies shared decomposition responsibilities.
doc/​code/​datasets/​3_dataset_writing.md Documents stacked objective decomposition syntax.
doc/​code/​datasets/​0_dataset.md Links to objective decomposition guidance.

Review effort: Balanced

- Reject a decomposition tree whose objectives or grouped prompts are not all
  in the root's dataset; matching is scoped by the root's dataset, so a member
  in another dataset could reuse a stored row from the wrong dataset.
- When an equivalent tree is already stored, give each caller seed its stored
  counterpart's identifiers, including absent group IDs.
- Add root_id to build_objective_decomposition_tree so a subtree read from
  memory below the root can be walked without dropping its parent link.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bring the reviewed seed-decomposition repairs from the lower stacked PR
forward: one-dataset decomposition trees, stored group IDs on reload,
and walking a stored subtree from any objective.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…inding actions

- Read an export's finding trees in one transaction under the operation's finding lock, then
  attach evidence to those findings without re-reading them, so a concurrent sub-finding and
  path change cannot leave a path step outside the report.
- Render the export dialog as one flat list in tree order, indenting sub-findings and naming
  their parent for screen readers, so a deep decomposition no longer exhausts the call stack.
- Let a finding's Edit / Add sub-finding / Delete actions wrap so rows fit at 390px.
- E2E: expect numbered Markdown headings and cover a nested sub-finding row at 390px.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

Both issues from this review are fixed in 478bafb. Exports read each finding tree in one locked transaction and attach evidence without reading the findings again. The export dialog renders a flat list, so deep trees no longer recurse. The same commit fixes the two Frontend E2E failures: the Markdown heading check now expects numbered headings, and a finding's Edit / Add sub-finding / Delete buttons wrap so rows, including nested sub-finding rows, fit at 390px. It also merges #4110's head 55d365b.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jbolor21 jbolor21 self-assigned this Nov 10, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reject partial-tree deletions before writes, deduplicate trees by execution and structure while updating descriptive metadata, and order reloads deterministically.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep assertions on the actual delete filter while allowing the new integrity query to report no partial tree.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep behavioral template declarations distinct through direct and prepared ingestion, with parameter-filtered retrieval regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review November 13, 2026 01:20
Integrate recorded dependency head 72b507e without rewriting PR history.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate main 614768c while retaining finding decomposition, validation tests, and seed technique metadata. Join the two Alembic heads with an empty merge revision.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep draft path verdicts library-owned and move workflow details out of framework guidance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

review:5829031739

Verified both reported causes at 0f7b64dc297f506e0f6000797d4648634f3f3f9d after integrating the dependency and recorded main: exports retain one subtree snapshot, and the export dialog renders iteratively. The existing fixes were preserved; the new repair also keeps saved path descriptions in both download formats and fixes microsecond sibling ordering.

Verified on published commit 0f7b64dc297f506e0f6000797d4648634f3f3f9d.

Avoid current ORM defaults before their columns exist; retain branch upgrade and preservation assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@jbolor21

jbolor21 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Left some comments and this isn't blocking at all but would be helpful for some images or a demo/notebook etc for changes!

@jbolor21 jbolor21 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a few small comments & haven't had time to play around with the UI enough for more comments but I think other changes I'll comment if I see them as I interact more with the UI! (Or ofc can make future PRs).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Main enables RUF100 for new Alembic revisions, so the TC003 suppression
in this PR's unmerged revision is now unused.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Number root findings and sub-findings the way a full export numbers them,
  and reuse those numbers in path steps and the Edit paths dialog.
- Drop the stale Complete/Partial expects from the Edit paths test and the
  dialog intro; keep step controls on one line at narrow widths.
- Compute report path completeness with is_finding_path_complete.
- Test promoting a real sub-finding in memory and through the API.
- Replace the GUI guide diagram with a user-facing one.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen

Copy link
Copy Markdown
Contributor Author

conversation:6953867379

Thanks! Here are images from the GUI at this commit (a seeded operation, captured with Playwright), plus the simplified diagram now in the GUI guide.

Finding tree with numbered sub-findings and paths:

Finding 1 with two numbered paths and sub-findings 1.1 to 1.4

Edit paths dialog, with steps labelled by number:

Edit paths dialog listing numbered steps for two paths

The same dialog at 390px, where step controls now stay on the step's line:

Edit paths dialog at 390px wide

I also replaced doc/assets/finding-decomposition.svg with a usage-only diagram (tree, paths and export, without internal classes, storage or locking):

Finding decomposition diagram showing a finding, its sub-findings, paths and the numbered export

I didn't add a notebook: findings are written by operators in the GUI and the REST API rather than from Python code, and the GUI guide (doc/gui/0_gui.md) walks through the workflow.

Verified on published commit 71fb572c452f119139c7ad7e24189a09eaeaa362.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Nov 17, 2026
Merged via the queue into microsoft:main with commit e729728 Nov 17, 2026
64 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-hierarchical-findings branch November 17, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants