Repository navigation
feat(agents): add a Gator review team and review replay harness - #5779
Open
johntmyers wants to merge 2 commits into
Open
johntmyers wants to merge 2 commits into
johntmyers wants to merge 2 commits into
Conversation
Add bin/review-team, which runs the generalist reviewer plus focused members chosen by review mode and changed code, merges their findings, and checks that every member finding is accounted for. Networking and contracts members run by default; correctness and security are opt-in. Add a replay harness that reruns a reviewer on pinned past PRs with known later fixes and grades the findings, to measure reviewer changes. Gator's review step is unchanged until the team is wired in. Closes #5778 Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
johntmyers
force-pushed
the
feat/5778-gator-review-team/johntmyers
branch
from
November 21, 2026 00:24
6bdfcd7 to
81127e6
Compare
…reviews Add networking checklist rules: control-plane changes such as policy reloads, approvals, and credential rotations disrupt only the connections they affect, and a targeted check that replaces a conservative one reads consistent, fully published state with versions that change on every transition. The contracts member now compares the pull request description's claims with the behavior the diff changes. Add replay cases for #2373, which closed every relay on each policy reload, and #5361, whose credential revalidation raced publication. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a Gator review team (
bin/review-team) and a replay harness that measures reviewer recall on past PRs with known later fixes. Gator's review step is unchanged; a follow-up wires the team into gator-gate.Related Issue
Closes #5778
Changes
bin/review-teamruns the generalist reviewer plus focused members chosen by review mode and changed code, in parallel through the harness subagent adapter. A merge subagent combines findings, and a deterministic check requires every member finding to be merged or dropped with a reason. A member or merge failure (after one retry) fails the run without output.review-networkingruns when changed paths or added code touch networking, and must give a verdict on each of the 35 rules innetworking-checklist.md.review-contractsruns in initial and follow-up reviews.review-correctnessandreview-securityare opt-in throughREVIEW_TEAM_MEMBERS.highreasoning.OPENSHELL_SUBAGENT_REASONING, so the team does not depend on a specific harness's reasoning setting.evals/holds 27 pinned cases (Gator review misses and networking escapes).run-replaychecks out each PR head without later history, builds the reviewer task from the gator-gate skill, runs the reviewer or the team, and grades findings with a judge model.scorecompares runs.agent.yamladds the member prompts (prompts/review-*.md), member rules, checklist, and team config to the payload.payload_versionis unchanged because nothing uses them yet.On the 19 small and medium replay cases (21 known bugs), each graded by the same judge:
Each configuration ran once, so differences of one or two catches are within run-to-run noise.
Networking checklist rules 34 and 35 and the contracts member's claims-against-the-diff check come from #2373, which closed every relay on each policy reload, and #5361, whose credential revalidation raced policy publication. Three runs of the default team on each, before and after these rules:
Testing
review_team_test.sh,replay_test.sh, Codexexec_test.sh,run_test.sh,launch_test.sh, andimage_test.shpassreview_team_test.sh,replay_test.sh)Checklist