Coverage for src/ai_jury/ballots.py: 100%
406 statements
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-20 18:15 +0000
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-20 18:15 +0000
1"""Per-reviewer ballots: each panelist's own stance, with vendor provenance (issue #663).
3The consolidated ``findings``/``consensus`` views answer *what the panel found*.
4They do not answer *who said what*, which is the question a downstream gate asks
5when it wants the panel to **be** its review: one head-pinned verdict per
6panelist, attributed to the vendor and model that produced it. Until now that
7existed only as prose in the markdown report's vote block.
9Everything here is pure and deterministic — a function of the outcome, the config
10and the (optional) vote. No I/O, no wall-clock, no randomness. Two renderings are
11built on it:
13* :func:`reviewer_ballots` — the ``reviewers`` array of the JSON report.
14* :func:`keel_reviews` — the ``--format keel-reviews`` bundle, shaped for a
15 consumer that accepts a JSON array of ``{reviewer, verdict, scope, findings,
16 testing, vendor, model}`` records (keel's ``keel review --reviews``).
18Only legitimate, already-reported fields are emitted: severities, locations,
19claims, vendor/model provenance and durations. Never raw diff text, prompt text
20or secrets. Free-text lifted out of an agent's reply (``scope``, ``testing``) is
21attacker-influenced, so it is flattened to a single line and length-capped before
22it leaves this module.
24**A ballot has to name something** (issue #700). Every ballot of a real run read
25``scope: "Reviewed the supplied diff; named no specific file."``, ``testing:
26"not stated"``, ``model: ""`` — and on a tier whose review *is* the panel, those
27ballots were the whole review. The consumer refuses a hand-posted verdict shaped
28like that (:data:`_SCOPE_ANCHORS` mirrors the rule: a path, a ``path:line``, a
29backticked symbol, a called identifier, or a "checked …" clause), precisely
30because a verdict naming nothing cannot be told apart from one never performed.
32**And the name has to exist** (issue #710). Mirroring the consumer's *shapes* was
33not enough on its own: ``describe_scope`` backticked whatever the reviewer wrote
34on its ``Checked:`` line, so ``Checked: nothing`` was rendered as an anchor,
35passed the shape test and counted as a review — the form of naming something with
36none of the substance, satisfiable by an agent that read nothing. Every stated
37token is now resolved against the change the panel was shown
38(:class:`ai_jury.largediff.ChangeIndex`, carried on the outcome), and the shape
39test is what the *resolved* tokens then have to pass. This module is therefore
40deliberately **stricter** than the consumer on that one path and never looser:
41a scope it accepts is one the consumer accepts.
42So the placeholder is gone in three directions:
44* the review prompt asks the reviewer for its own ``Checked:`` / ``Tested:``
45 lines, rather than this module inferring coverage from whatever prose landed;
46* a scope that still names nothing is not written — the ballot is reported as an
47 :data:`ABSTAIN` whose scope states *why*, because an agent that returned
48 nothing useful must not be counted as one that reviewed;
49* ``model`` is never blank: it is the model id actually requested of the agent
50 that answered, or a statement that the CLI's default was used and the CLI does
51 not report which model that was. A ballot naming its vendor but not its model
52 cannot answer whether the same model sat twice, which is the whole product of
53 a cross-vendor panel.
55**And a ballot that named nothing is not a review** (#700, round 2). Recording the
56abstention was only half of it: the abstaining ballot still counted toward the
57number of reviews the run announced and toward ``--min-reviews``, so a panel of
58three "Looks good to me, no concerns." replies satisfied a gate that exists to
59refuse exactly that. :func:`ai_jury.panel.is_review` is now the single definition
60— a ``panelist`` record with a substantive scope and a voting verdict — every
61ballot carries the two fields it reads (``scope_substantive`` and ``verdict``)
62plus the answer itself (``counts_as_review``), and three further consequences
63follow from it:
65* every seat that ran gets a record, including one that returned nothing at all.
66 Dropping it left the bundle unable to say *which* agent fell silent; the record
67 is an abstention naming the seat and the reason, and the count excludes it.
68* a finding attached to no file is a scope only under ``--issue``, where a
69 finding carries no file by construction. In code-review mode a claim is not a
70 place in the code, and a ballot with nothing but claims abstains.
71* ``model_source`` rides along in the ``keel-reviews`` projection too, so a
72 machine consumer of that shape can tell a requested id from a statement about
73 the CLI's default without parsing English.
75**And what it says about itself has to be true of the run** (#709/#710, round 2).
76Two claims here were still stated more strongly than the evidence behind them:
78* the ``Checked:`` line was split into tokens **lexically**, on whitespace, so a
79 changed file whose name contains a space arrived as two tokens that name
80 nothing and the ballot abstained under ``not_in_change`` — the rule for
81 catching a review of nothing, refusing a real review over a space. Tokens are
82 now cut against the change index itself, and a span the reviewer quoted is one
83 token whatever is inside it;
84* a ``model`` id this module *derived* went out under ``model_source:
85 requested``, whose whole claim is that the id was on the wire. Deriving one is
86 still right where no invocation recorded one, but it is labelled
87 :data:`MODEL_RECOMPUTED` and it is never the answer for a stale result-cache
88 entry — the record format changed, so the cache refuses it rather than
89 recomputing over the gap.
90"""
92from __future__ import annotations
94import re
95from typing import Any
97from . import redaction
98from .findings import flatten_inline
99from .panel import (
100 ABSTAIN,
101 ADAPTER_FAILED,
102 CAUSE_FIELD,
103 CHAIR_ROLE,
104 NAMED_NOTHING,
105 NOT_IN_CHANGE,
106 PANELIST_ROLE,
107 REFUSED,
108 SILENT,
109 adapter_failure,
110 ballot_seats,
111 bundle_records,
112 is_review,
113 responded,
114 review_count,
115)
116from .voting import is_abstention, tally_votes
118# ``ABSTAIN`` is the stance recorded for a panelist that did not actually review
119# — nothing at all, an empty reply, a refusal, an adapter that failed, or a reply
120# naming nothing checkable. Counting such a seat as the "clear" stance
121# (APPROVE/READY) is precisely the bug :mod:`ai_jury.voting` refuses to have
122# (issue #251): a non-answer is not an approval. The vote tally drops those
123# reviewers entirely; a ballot list has to name every seat that ran, so it names
124# the abstention instead of inventing a stance for it. It is *defined* in
125# :mod:`ai_jury.panel` and re-exported here for the module's own callers, because
126# :func:`ai_jury.panel.is_review` — the one definition of what counts as a review
127# — has to test for it, and two copies of the token are two places for the count
128# and the record to disagree.
130#: ``reviewer`` / ``name`` used for the chair's entry.
131CHAIR_NAME = "chair"
133#: The ``mode`` that selects the issue-review vocabulary (``--issue``), and with
134#: it the single exception to "a scope must name a file": there a finding carries
135#: no file by construction, so the claims a reviewer raised are what it named.
136ISSUE_MODE = "issue"
138#: What ``testing`` says when the reviewer named no verification at all. It says
139#: it plainly rather than shrugging: "not stated" reads like a field nobody
140#: filled in, and a reader cannot tell that apart from a reviewer that ran
141#: nothing and said so (#700). Both are "no verification was run"; only one of
142#: them used to be legible.
143NOT_STATED = "Nothing run: this reviewer named no command, test run or reproduction."
145#: ``model`` for an agent whose CLI was invoked with no model id pinned. Filled
146#: in from the agent's command so the field states the situation instead of
147#: being blank — "which model answered" has an honest answer here, and it is
148#: "the CLI's default, which the CLI does not report".
149_CLI_DEFAULT_MODEL = "{command} default (the CLI does not report which model answered)"
151#: Where a ballot's ``model`` came from, as one machine token, so a consumer can
152#: tell a real model id from a statement about one without parsing prose.
153MODEL_REQUESTED = "requested" # an id was pinned (or mapped from `effort`) and sent
154MODEL_RECOMPUTED = "recomputed" # no invocation recorded one; derived from the config
155MODEL_CLI_DEFAULT = "cli_default" # nothing pinned; the CLI chose, and does not say
156MODEL_UNKNOWN = "unknown" # the answering slot has no spec in this config
157MODEL_NONE = "none" # there is no agent in this slot at all (an unchaired run)
159#: Caps on free text lifted out of agent output. A reviewer's reply is
160#: attacker-influenced, so a forged 100 kB "scope" must not become the bulk of a
161#: bundle posted downstream.
162_CLAUSE_MAX = 240
163_SCOPE_CLAUSES = 3
164_FILES_LISTED = 8
166# Sentence boundary: end punctuation followed by whitespace. Lines are split
167# first, so a bullet list yields one clause per bullet.
168_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+")
170#: Clauses that describe what a reviewer looked at (folded into ``scope``).
171#: Anchored on word boundaries, which is load-bearing rather than tidy: the most
172#: common finding wording in this tool's own fixtures is "**un**checked return
173#: value", and a bare substring test folds that claim into the coverage summary
174#: as though the reviewer had said it checked something.
175_COVERAGE_RE = re.compile(r"\b(checked|examined|inspected|reviewed|covered)\b", re.IGNORECASE)
177#: Clauses that describe what a reviewer did to verify a claim (``testing``).
178_TESTING_RE = re.compile(
179 r"\b(ran\s+(?:the\s+)?tests?|test\s+suite|unit\s+tests?|i\s+ran|reproduced|"
180 r"verified|verification|tested)\b",
181 re.IGNORECASE,
182)
184#: The two lines :mod:`ai_jury.prompts` asks every reviewer to open with. Matched
185#: after the line has been flattened and stripped of list/heading/emphasis
186#: markers, so ``- **Checked:** src/a.py`` and ``Checked: src/a.py`` both land.
187_STATED_SCOPE_RE = re.compile(r"^checked\s*\**\s*:\s*\**\s*(.+)$", re.IGNORECASE)
188_STATED_TESTING_RE = re.compile(r"^tested\s*\**\s*:\s*\**\s*(.+)$", re.IGNORECASE)
190#: A concrete thing a review points at, as the downstream consumer defines it —
191#: a path, a ``path:line``, a backticked symbol, or a called identifier. This is
192#: a test for *structure*, never a judgement about whether the review was any
193#: good; it distinguishes a review from a receipt. Kept deliberately identical
194#: to the consumer's own rule (keel's ``review-verdict-insubstantial``): a scope
195#: this tool is happy with but the consumer rejects is the defect in #700, and
196#: the only way the two cannot drift is for the same shapes to be listed here.
197#: Drift in the other direction is fine and intended: since #710 a stated
198#: ``Checked:`` token must also *resolve* against the change before it is
199#: rendered as one of these shapes, so this tool is stricter there and never
200#: looser — a scope it accepts is one the consumer accepts.
201_SCOPE_ANCHORS = (
202 re.compile(r"[\w./-]+\.[A-Za-z0-9]{1,5}:\d+"), # path/to/file.py:42
203 re.compile(r"[\w-]+/[\w./-]+\.[A-Za-z0-9]{1,5}\b"), # src/ai_jury/thing.py
204 re.compile(r"`[^`\n]{2,}`"), # `a_symbol`, `--a-flag`
205 re.compile(r"\b\w+\.\w+\(\)"), # module.function()
206)
208#: The escape hatch the consumer's rule keeps, and this one keeps with it: a
209#: genuinely clean review ("checked X, Y and Z; found nothing") is a real
210#: outcome and must not be forced to invent a file reference.
211_CHECKED_CLAUSE_RE = re.compile(r"\bchecked\b[^.\n]{8,}", re.IGNORECASE)
213# --- Resolving a stated `Checked:` line against the change (issue #710) -----
214#
215# `_tick` backticks any non-empty stated value, and a backticked token is an
216# anchor, so `Checked: nothing` — or `everything`, or `the diff` — was rendered
217# as an anchor, passed :func:`scope_is_substantive`, and made the ballot a
218# review. That is the *shape* of an anchor with none of the substance, and a rule
219# satisfied by an agent that read nothing is #700's own failure one layer up.
220#
221# So a stated token is resolved against the change the panel was shown — the
222# paths and symbols of :class:`ai_jury.largediff.ChangeIndex`, carried on the
223# outcome. The check is local and cheap: the jury holds the diff at that moment.
225#: List punctuation: a **hard** token boundary, kept apart from whitespace,
226#: which is not one. See :func:`_scope_tokens` — a changed file whose name
227#: contains a space is one token spelled with whitespace in the middle of it,
228#: and a lexical split on ``[\s,;]+`` cut it in half (#710, round 2).
229_LIST_SEP_SPLIT = re.compile(r"[,;]+")
231#: Whitespace: a *candidate* boundary, resolved against the change.
232_WHITESPACE_SPLIT = re.compile(r"\s+")
234#: A span the reviewer itself marked as one token — backticks, or double quotes,
235#: straight or curly. Whatever is inside is one token even when it contains a
236#: space or a comma: the reviewer drew the boundary, and re-splitting it is the
237#: same defect the joining below exists to fix. Single quotes are deliberately
238#: absent: ``'`` and ``’`` are also apostrophes, so ``the reviewer's own file``
239#: would open a span at ``'s`` and swallow the rest of the line.
240_QUOTED_SPAN_RE = re.compile(r"`([^`\n]+)`|\"([^\"\n]+)\"|“([^”\n]+)”")
242#: The same span shapes, anchored: a run that is *entirely* one mark, marks
243#: included. See :func:`_unmarked` — the marks a reviewer nested inside its own
244#: span come off too, so a path that is quoted *and* backticked resolves the way
245#: either alone does.
246_NESTED_MARK_RE = re.compile(r"\A`([^`\n]+)`\Z|\A\"([^\"\n]+)\"\Z|\A“([^”\n]+)”\Z")
248#: How many nested layers of marks are peeled off a span. Two, because two is as
249#: deep as nesting goes: there are three mark shapes and none can sit inside
250#: itself (``[^`\n]+`` cannot hold a backtick), so once :func:`_marked_spans` has
251#: taken the pair the span opened with, at most two remain. Anything a deeper
252#: line left on can only fail to resolve, which is reported.
253_MAX_NESTED_MARKS = 2
255#: How many whitespace-separated pieces may be joined while looking for a path.
256#: A cap rather than the whole line: the search is quadratic in the window, the
257#: line is attacker-influenced, and a filename with five spaces in it is already
258#: an outlier. Exceeding it can only make a token fail to resolve, which is the
259#: safe direction — an unresolved token is reported as unresolved.
260_MAX_JOINED_PIECES = 6
262#: Punctuation a token may be **wrapped** in: brackets and quotes, plus the
263#: markdown emphasis mark. Stripped from either end, so ``(src/a.py)`` and
264#: ``src/a.py`` are one token. Nothing here can begin a file name, which is why
265#: taking it off either end is unconditional. See :func:`_edge_stripped`.
266_WRAP_EDGE = "`'\"“”‘’()[]{}<>*"
268#: Sentence punctuation a token may be **trailed** by, stripped from the end
269#: only — a full stop ends a sentence far more often than it opens a filename,
270#: but a *leading* dot is part of the name and is never touched. A trailing
271#: ``:`` goes too, but a ``:42`` does not — that is a line. ``_`` is in neither
272#: set: it is part of the identifier, and stripping it turned ``_stated_line``
273#: into a symbol the change does not have.
274_TRAIL_EDGE = ".,;:!?"
276#: ``path:line`` / ``path:line-line`` — the line part is dropped before the path
277#: is looked up, because a diff index knows files, not line numbers.
278_PATH_LINE_RE = re.compile(r"^(?P<path>.+?):(?P<line>\d+(?:-\d+)?)$")
280#: A token that *claims* to name something: a path, a ``path:line``, a call, a
281#: dotfile, or an identifier (an ``_`` or an interior case change). Ordinary
282#: connective prose in the same sentence — "lines", "and", "the tests" — claims
283#: nothing, so it is neither counted as an anchor nor reported as a broken one. A
284#: token the reviewer backticked counts too: the reviewer marked it as a name.
285#:
286#: The **leading-dot** alternative is what makes ``.gitignore`` a name (#710,
287#: round 4). The trailing-extension one never reached it — ``gitignore`` is nine
288#: characters, not a suffix — so an *absent* dotfile was dropped as connective
289#: prose and the ballot said the line named nothing, for a path the reviewer
290#: named exactly. Prose is untouched by it: a full stop that ends a sentence
291#: sits at the end of the piece before it, never at the start of the next.
292# A bare dotted word — ``foo.proto``, ``e.g``, ``Ph.D`` — is *not* name-shaped.
293# Rounds 10 and 11 tried to tell a file from a Latin aside by the shape of the
294# suffix and by the length of the segments, and each rule had a family it
295# missed. The reviewer's own punctuation settles it instead: a slash, a leading
296# dot, or marks around the token make it a claim that is reported when absent;
297# a bare dotted word is read as prose. (#711 round 12)
298_NAME_SHAPED_RE = re.compile(r"[/\\]|\A\.[A-Za-z0-9]|\(\)$|_|[a-z0-9][A-Z]")
300#: Anchor-forming characters, removed before an unresolved token is quoted in an
301#: abstention. See :func:`_deanchor`.
302_DEANCHOR = str.maketrans(dict.fromkeys("`/\\:()[]{}<>", " "))
304#: ...and the one word that forms an anchor without punctuation, under
305#: :data:`_CHECKED_CLAUSE_RE`. Elided rather than dropped silently.
306_CHECKED_WORD_RE = re.compile(r"\bchecked\b", re.IGNORECASE)
309def _deanchor(text: str) -> str:
310 """A reviewer's token, quoted so it cannot pass for an anchor (pure).
312 An unresolved token is named in the abstention — a reader has to see that
313 the reviewer claimed ``src/made/up.py`` — but the abstention is
314 *deliberately anchorless* (see :func:`abstention_scope`): a consumer applying
315 the same substance rule has to reach the same conclusion this module did,
316 and it would not if the sentence explaining that nothing was checked itself
317 contained a path. So the shapes that make an anchor are removed: the
318 punctuation of :data:`_SCOPE_ANCHORS` and the ``checked …`` clause. The
319 reviewer's words survive; only their ability to masquerade as evidence does.
320 """
321 flat = _CHECKED_WORD_RE.sub("…", flatten_inline(text))
322 return " ".join(flat.translate(_DEANCHOR).split())[:_CLAUSE_MAX]
325def _marked_spans(value: str) -> list[tuple[str, bool]]:
326 """*value* as ``(text, quoted)`` runs (pure).
328 ``quoted`` runs are the spans the reviewer wrapped in backticks or double
329 quotes; the rest is everything between them. Splitting on the marks first is
330 what lets a quoted ``docs/my file.py`` survive as one token: a reviewer that
331 quoted a name has already said where it begins and ends, and no rule below
332 is entitled to a second opinion about that.
333 """
334 runs: list[tuple[str, bool]] = []
335 pos = 0
336 for match in _QUOTED_SPAN_RE.finditer(value):
337 if match.start() > pos:
338 runs.append((value[pos : match.start()], False))
339 # Exactly one alternative of :data:`_QUOTED_SPAN_RE` can have matched.
340 runs.append((match.group(1) or match.group(2) or match.group(3), True))
341 pos = match.end()
342 if pos < len(value):
343 runs.append((value[pos:], False))
344 return runs
347def _unmarked(text: str) -> str:
348 """One marked span's content, with nested marks peeled off (pure).
350 :func:`_marked_spans` removes the pair the span *opened* with, and a
351 reviewer that both quoted and backticked a path leaves the inner pair on:
352 ``resolve_stated_scope("“`docs/my file.py`”", changed)`` looked up the
353 backticked string, matched nothing, and reported the reviewer's own
354 ``docs/my file.py`` as a claim that failed (#710, round 3). Quoting a
355 backticked path is the same claim about the same file as either mark alone,
356 so the marks come off before the lookup — as many layers as
357 :data:`_MAX_NESTED_MARKS` allows, whichever mark each layer used.
358 """
359 inner = text.strip()
360 for _ in range(_MAX_NESTED_MARKS):
361 match = _NESTED_MARK_RE.match(inner)
362 if match is None:
363 break
364 inner = (match.group(1) or match.group(2) or match.group(3)).strip()
365 return inner
368def _path_base(token: str) -> str:
369 """*token* with a trailing ``:line`` / ``:line-line`` dropped (pure)."""
370 match = _PATH_LINE_RE.match(token)
371 return match.group("path") if match else token
374#: A call marker ``()`` that may sit inside wrapping punctuation and before
375#: trailing sentence punctuation: ``(module.env()),`` names the call
376#: ``module.env()``. Group 1 is everything before the marker. The trailing class
377#: is *derived* from the same two edge sets the plain strip uses, so the two
378#: cannot disagree about what counts as wrapping (#711 round 9: the hand-written
379#: class lacked ``<>`` and the curly quotes, which ``_WRAP_EDGE`` has).
380_WRAPPED_CALL_RE = re.compile(r"^(.*?)\(\)[" + re.escape(_WRAP_EDGE + _TRAIL_EDGE) + r"]*$")
383def _edge_stripped(piece: str) -> str:
384 """*piece* with its edge punctuation removed (pure).
386 Wrapping punctuation — brackets and quotes, :data:`_WRAP_EDGE` — comes off
387 either end; trailing sentence punctuation (:data:`_TRAIL_EDGE`) comes off
388 the end. **A character that begins a name is never removed**: a dot followed
389 by a letter or a digit, a letter, a digit and ``_`` are in neither set, so
390 ``(.gitignore).`` is ``.gitignore``, ``src/a.py,`` is ``src/a.py``, and
391 ``all.`` is ``all``.
393 A leading dot is part of a file name, and the earlier rounds of #710 stripped
394 it with everything else and then tried to put it back: the trims that
395 restored the stripped characters were offered to the change index,
396 most-stripped first, and the first trim the index confirmed was the token.
397 That asked the diff a question the token had already answered — and the diff
398 answered it wrongly whenever it happened to contain the stripped remnant.
399 With ``env`` (or ``bin/env``) changed, ``Checked: .env`` resolved on the
400 fully stripped ``env``, matching on a component boundary in a *different*
401 file, and the ballot counted as a review of a file the reviewer never named;
402 ``.gitignore`` did the same against a changed ``src/gitignore`` (#709/#710,
403 round 5). Never stripping the dot removes the question rather than adding a
404 case to it: ``.env`` against a changed ``env`` is a name this change does not
405 have, reported unresolved under ``not_in_change`` — and one deterministic
406 strip is all a token needs, so no trim is enumerated against the index at
407 all.
408 """
409 # A trailing ``()`` is a call marker, not wrapping punctuation: it is what
410 # tells :func:`_token_resolves` that ``module.function()`` is a symbol claim
411 # rather than a file (#711 round 7), so it is kept whole.
412 # The marker is looked for *inside* any wrapping, not at the raw end: on
413 # ``(module.env()),`` the last characters are wrap and trail punctuation,
414 # and a check at the raw end missed the call and then ate its parentheses
415 # as wrapping (#711 round 8).
416 marked = _WRAPPED_CALL_RE.match(piece)
417 if marked:
418 stripped = marked.group(1).lstrip(_WRAP_EDGE)
419 return stripped + "()" if stripped else ""
420 return piece.lstrip(_WRAP_EDGE).rstrip(_WRAP_EDGE + _TRAIL_EDGE)
423def _joined_against_change(pieces: list[str], changed: Any) -> list[str]:
424 """One whitespace-separated run's pieces, with spaced paths rejoined (pure).
426 **The change index is the tokeniser, not the whitespace.** ``Checked:
427 docs/my file.py`` used to split into ``docs/my`` and ``file.py``, neither of
428 which is in a diff that changes ``docs/my file.py``: the ballot came back
429 ``scope_substantive: false``, ``counts_as_review: false``,
430 ``abstention_cause: not_in_change`` — a real review of a real file, refused
431 for a space in its name, by the rule that exists to catch reviews of nothing.
433 So adjacent pieces are offered to the index joined, longest window first, and
434 a join that names a changed path *is* the token. Longest-first matters: with
435 both ``my file.py`` and ``docs/my file.py`` changed, the reviewer named the
436 second. A join is only ever accepted when the index confirms it, so this can
437 turn a non-token into a token but never the reverse — the failure direction
438 is an unresolved token, which is reported as one.
439 """
440 tokens: list[str] = []
441 index = 0
442 while index < len(pieces):
443 joined = ""
444 width = 1
445 for size in range(min(_MAX_JOINED_PIECES, len(pieces) - index), 1, -1):
446 candidate = _edge_stripped(" ".join(pieces[index : index + size]))
447 if changed.has_path(_path_base(candidate)):
448 joined, width = candidate, size
449 break
450 tokens.append(joined or _edge_stripped(pieces[index]))
451 index += width
452 return tokens
455def _scope_tokens(value: str, changed: Any) -> list[str]:
456 """The distinct candidate tokens of one stated ``Checked:`` value (pure).
458 Tokenised **against the change**, not lexically (#710, round 2). Three rules,
459 in this order: a span the reviewer quoted is one token whatever is inside it;
460 list punctuation is a hard boundary; and whitespace is a boundary only where
461 joining across it does not name a changed path.
462 """
463 seen: list[str] = []
464 for text, quoted in _marked_spans(flatten_inline(value or "")):
465 if quoted:
466 # No edge punctuation is stripped: inside the reviewer's own marks
467 # there is none to strip, and stripping it would cost the leading
468 # dot of a `.gitignore` the reviewer took care to quote. Only marks
469 # the reviewer nested inside its own span come off — see
470 # :func:`_unmarked`.
471 candidates = [_unmarked(text)]
472 else:
473 candidates = [
474 token
475 for segment in _LIST_SEP_SPLIT.split(text)
476 for token in _joined_against_change(
477 [p for p in _WHITESPACE_SPLIT.split(segment) if p], changed
478 )
479 ]
480 for token in candidates:
481 if token and token not in seen:
482 seen.append(token)
483 return seen
486def _is_name_shaped(token: str, value: str) -> bool:
487 """Does *token* claim to name something? (pure — see :data:`_NAME_SHAPED_RE`)
489 A token the reviewer *marked* counts too, whichever mark it used: quoting a
490 token is the reviewer saying it is a name, and a claim that failed has to be
491 reported as one rather than dropped as connective prose.
492 """
493 if _NAME_SHAPED_RE.search(token):
494 return True
495 return any(f"{o}{token}{c}" in (value or "") for o, c in (("`", "`"), ('"', '"'), ("“", "”")))
498def _path_shaped(base: str) -> bool:
499 """Does *base* claim to be a path rather than a symbol? (pure)
501 A directory, a dotfile, or any dotted name written without ``()``. The
502 extension list this replaced could not be complete — ``foo.proto`` fell
503 through to the symbol index and matched a ``proto()`` call (#711 round 7) —
504 so the rule is now the reviewer's own punctuation: ``module.function()``
505 names code, ``module.function`` and ``foo.proto`` name a file.
506 """
507 if "/" in base or base.startswith("."):
508 return True
509 return "." in base and not base.endswith("()")
512def _token_resolves(token: str, changed: Any) -> bool:
513 """Is *token* a path or symbol that is actually in the change? (pure)
515 A path-shaped token — a directory, a dotfile, or a dotted name without
516 ``()`` — is a path claim and resolves only as a path. It is never split and
517 asked of the symbol index: ``.env`` against a hunk that adds ``env(1)`` used
518 to resolve on the remnant ``env`` and count as a review of a file the
519 reviewer never named (#711 round 6). A symbol claim is a token with no dot,
520 or one ending in ``()``; for ``module.function()`` the member ``function``
521 is asked, because the qualification is the reviewer's and the member is
522 the claim.
523 """
524 base = _path_base(token)
525 if changed.has_path(base):
526 return True
527 if _path_shaped(base):
528 return False
529 member = base.rstrip("()").rsplit(".", 1)[-1]
530 return bool(member) and changed.has_symbol(member)
533def resolve_stated_scope(value: str, changed: Any) -> tuple[list[str], list[str]]:
534 """Split one ``Checked:`` value into ``(resolved, unresolved)`` tokens (pure).
536 ``resolved`` are the tokens that name a path or a symbol present in
537 *changed*; they are what makes the scope substantive. ``unresolved`` are the
538 tokens that are shaped like a name and are not in the change — reported, so
539 a reader sees the claim that failed, but never anchoring anything.
541 **The split into tokens is itself made against the change** (see
542 :func:`_scope_tokens`), because a lexical one gets a changed file whose name
543 contains a space wrong in the direction that costs a real review.
545 **A mixed line is carried by its real tokens.** ``Checked:
546 src/ai_jury/ballots.py, src/made/up.py`` is a review of
547 ``src/ai_jury/ballots.py``, with the second path listed in the scope as not
548 in the change: the reviewer demonstrably read something a reader can go and
549 check, and abstaining over the extra token would discard a real review to
550 punish a typo. A line with *no* resolving token is not a review, whatever
551 else it says.
552 """
553 resolved: list[str] = []
554 unresolved: list[str] = []
555 for token in _scope_tokens(value, changed):
556 if _token_resolves(token, changed):
557 resolved.append(token)
558 elif _is_name_shaped(token, value):
559 unresolved.append(token)
560 return resolved, unresolved
563def normalize_verdict(verdict: str) -> str:
564 """Fold a display verdict into a single machine token.
566 ``REQUEST CHANGES`` → ``REQUEST_CHANGES``, ``NEEDS-INFO`` → ``NEEDS_INFO``,
567 ``NO QUORUM`` → ``NO_QUORUM``. The markdown report keeps the spaced form for
568 humans; a machine consumer keys on one word, and a verdict that changes shape
569 between the two renderings is a verdict that gets matched wrong.
570 """
571 token = flatten_inline(verdict or "").strip().upper()
572 return re.sub(r"[\s\-]+", "_", token)
575def _prose_lines(text: str) -> list[str]:
576 """The agent's prose with fenced blocks removed.
578 A review's fenced ``json`` block is the *structured* findings — it is already
579 parsed into :class:`~ai_jury.findings.Finding` objects and rendered as the
580 ballot's ``findings``. Left in, its serialized claim/evidence text is by far
581 the longest thing in the reply and swamps every prose clause after it.
582 Unterminated fences swallow the remainder, which is the fail-safe direction:
583 less lifted text, never more.
584 """
585 lines: list[str] = []
586 in_fence = False
587 for raw in (text or "").splitlines():
588 if raw.lstrip().startswith(("```", "~~~")):
589 in_fence = not in_fence
590 continue
591 if not in_fence:
592 lines.append(raw)
593 return lines
596def _clauses(text: str) -> list[str]:
597 """Split agent output into flattened, capped candidate clauses (pure)."""
598 out: list[str] = []
599 for raw in _prose_lines(text):
600 line = flatten_inline(raw).strip().lstrip("-*#> ").strip()
601 if not line:
602 continue
603 # The split pattern consumes the whitespace after the end punctuation and
604 # ``line`` is already stripped, so no part can be blank — hence no guard.
605 for part in _SENTENCE_SPLIT.split(line):
606 out.append(part[:_CLAUSE_MAX])
607 return out
610def _first_matching(clauses: list[str], pattern: re.Pattern[str]) -> str:
611 for clause in clauses:
612 if pattern.search(clause):
613 return clause
614 return ""
617def _matching(clauses: list[str], pattern: re.Pattern[str], limit: int) -> list[str]:
618 hits: list[str] = []
619 for clause in clauses:
620 if pattern.search(clause) and clause not in hits:
621 hits.append(clause)
622 if len(hits) >= limit:
623 break
624 return hits
627def _files_named(findings: list) -> list[str]:
628 """Distinct file paths a reviewer named, in first-reported order."""
629 seen: list[str] = []
630 for f in findings:
631 path = flatten_inline(getattr(f, "file", "") or "").strip()
632 if path and path not in seen:
633 seen.append(path)
634 return seen
637def scope_is_substantive(scope: str) -> bool:
638 """Does this scope name something a reader could go and check? (pure)
640 The gate between a ballot and an abstention. Note what it does *not* do: it
641 never asks whether the review was correct, thorough or agreeable — it cannot,
642 and trying would make this a critic. It asks only whether the text points at
643 anything, which is the single question separating "this agent reviewed the
644 diff" from "this agent returned a string".
645 """
646 text = (scope or "").strip()
647 if not text:
648 return False
649 return bool(_CHECKED_CLAUSE_RE.search(text)) or any(p.search(text) for p in _SCOPE_ANCHORS)
652def _tick(text: str) -> str:
653 """One concrete token, backticked for a scope line.
655 Two jobs, and the second is why this is not an f-string at the call site.
656 Backticks make the token an anchor under :data:`_SCOPE_ANCHORS`, so a scope
657 built from a bare filename (``notes.md`` — no directory, so it matches none
658 of the path shapes) still names something checkable. And the token is
659 attacker-influenced, so its own backticks are stripped first: otherwise a
660 crafted path could close the quoting and forge structure around it.
661 """
662 return "`" + flatten_inline(text).replace("`", "").strip() + "`"
665def _stated_line(result: Any, pattern: re.Pattern[str]) -> str:
666 """The value of the first ``Checked:``/``Tested:`` line in the reply (pure).
668 The reviewer's own statement of its coverage, which beats anything this
669 module can infer from prose — inference is exactly how a ballot that named
670 nothing still shipped a scope sentence (#700). Fenced blocks are skipped as
671 everywhere else here, and the value is flattened and capped before use.
672 """
673 for raw in _prose_lines(getattr(result, "output", "")):
674 line = flatten_inline(raw).strip().lstrip("-*#> ").strip()
675 match = pattern.match(line)
676 if match:
677 value = match.group(1).strip().strip("*").strip()
678 if value:
679 return value[:_CLAUSE_MAX]
680 return ""
683def _free_clauses(result: Any) -> list[str]:
684 """Candidate prose clauses with the reviewer's own stated lines removed.
686 ``Checked: …`` matches the coverage pattern, so without this the stated line
687 is folded in twice — once quoted as the reviewer's statement and once again
688 as an inferred clause — and the second copy is unquoted, which is how a
689 crafted path gets into the scope without :func:`_tick` seeing it.
690 """
691 return [
692 clause
693 for clause in _clauses(getattr(result, "output", ""))
694 if not _STATED_SCOPE_RE.match(clause) and not _STATED_TESTING_RE.match(clause)
695 ]
698def _claims_named(findings: list) -> list[str]:
699 """Distinct claims a reviewer raised, for a review that attached no file.
701 Not decoration, and **scoped to ``--issue``**: there, every finding carries
702 ``file: ""`` by construction — the panel is reading an issue's prose, not a
703 diff, so there is no file for a finding to name — and the file list is empty
704 for a reviewer that did real work. Its claims are the only thing it *can*
705 name, and naming them keeps an issue-mode ballot out of the abstention branch
706 it does not belong in. That is the one legitimate exception to keel's rule
707 that a scope must name a file, line or symbol, or carry a ``Checked …``
708 clause, and :func:`describe_scope` applies it in issue mode only.
710 In code-review mode the same fallback was a hole (#700, round 2): a finding
711 with ``file: ""`` is a claim about a diff that failed to say *where*, and
712 :func:`_tick` backticked it into an anchor, so a scope naming no place in the
713 code passed the substance test and the ballot cast a voting verdict.
714 """
715 seen: list[str] = []
716 for f in findings:
717 claim = flatten_inline(getattr(f, "claim", "") or "").strip()
718 if claim and claim not in seen:
719 seen.append(claim[:_CLAUSE_MAX])
720 return seen
723def _stated_scope_sentence(stated: str, changed: Any) -> str:
724 """The scope sentence for a reviewer's own ``Checked:`` line (pure).
726 ``""`` when the line resolved to nothing in the change — the caller then has
727 no sentence to add from this source, and the ballot falls through to the
728 next one exactly as a reply with no ``Checked:`` line does. The unresolved
729 tokens are not silently dropped: :func:`_no_scope_reason` names them in the
730 abstention, where they cannot be mistaken for evidence.
732 With ``changed`` as ``None`` there is nothing to resolve against — a
733 hand-built outcome, a caller that never had a diff — and the pre-#710
734 structural rule applies. "Not verifiable here" is not "does not exist".
735 """
736 if changed is None:
737 return f"Checked, as stated by the reviewer: {_tick(stated)}."
738 resolved, unresolved = resolve_stated_scope(stated, changed)
739 if not resolved:
740 return ""
741 sentence = f"Checked, as stated by the reviewer: {', '.join(_tick(t) for t in resolved)}."
742 if unresolved:
743 listed = ", ".join(_deanchor(t) for t in unresolved[:_FILES_LISTED])
744 sentence += (
745 f" The same line also named {listed} — not in this change, and so"
746 f" anchoring nothing; the rest of the line is what this ballot rests on."
747 )
748 return sentence
751def describe_scope(result: Any, findings: list, *, mode: str = "code", changed: Any = None) -> str:
752 """What this panelist named that it read — or ``""`` when it named nothing.
754 Pure and deterministic. Four sources, most authoritative first:
756 1. the reviewer's own ``Checked:`` line, **resolved against the change** —
757 a token counts only when it names a path or a symbol that is actually in
758 the diff (#710), so ``Checked: nothing`` contributes nothing;
759 2. the distinct files it attached to its structured findings;
760 3. up to three "checked / examined / reviewed" clauses from its prose;
761 4. **in ``--issue`` mode only**, failing a file, the claims it raised.
763 An empty return is the meaningful case and the reason this no longer emits a
764 sentence unconditionally: with nothing from any of the four, the honest
765 output is *nothing*, and the caller turns that into an abstention. The old
766 fallback — "Reviewed the supplied diff; named no specific file." — asserted
767 coverage from the absence of evidence for it, and read identically whether
768 the agent had reviewed all 17 files or returned an empty string.
770 ``changed`` is what the first source is resolved against (#710). Until then
771 :func:`_tick` backticked whatever the reviewer wrote and
772 :func:`scope_is_substantive` accepted any backticked token, so ``Checked:
773 nothing`` was rendered as an anchor and the ballot counted as a review — the
774 shape of naming something, satisfiable by an agent that read nothing. The
775 ``Checked:`` path is now held to the standard the findings-derived path
776 already met: it must name a place that exists in the change.
778 ``mode`` gates the fourth source, and that gate is the whole of #700's third
779 round. keel's rule is that a scope must name a file, line or symbol, or carry
780 a ``Checked …`` clause; a backticked *claim* is none of those, and letting one
781 stand as a scope meant a code-review ballot raising one ``major`` finding
782 against ``file: ""`` cast ``REQUEST_CHANGES`` while naming no place in the
783 code. Issue mode is the one legitimate exception — see :func:`_claims_named`
784 — because there a finding genuinely has no file to name.
785 """
786 parts: list[str] = []
787 stated = _stated_line(result, _STATED_SCOPE_RE)
788 if stated:
789 sentence = _stated_scope_sentence(stated, changed)
790 if sentence:
791 parts.append(sentence)
792 files = _files_named(findings)
793 if files:
794 listed = ", ".join(_tick(f) for f in files[:_FILES_LISTED])
795 more = len(files) - _FILES_LISTED
796 suffix = f" (+{more} more)" if more > 0 else ""
797 parts.append(f"Named {len(files)} file(s): {listed}{suffix}.")
798 parts.extend(_matching(_free_clauses(result), _COVERAGE_RE, _SCOPE_CLAUSES))
799 if not files and mode == ISSUE_MODE:
800 claims = _claims_named(findings)
801 if claims:
802 listed = ", ".join(_tick(c) for c in claims[:_FILES_LISTED])
803 more = len(claims) - _FILES_LISTED
804 suffix = f" (+{more} more)" if more > 0 else ""
805 parts.append(f"Raised {len(claims)} finding(s) against no file: {listed}{suffix}.")
806 scope = " ".join(parts)
807 return scope if scope_is_substantive(scope) else ""
810def _named_only_absent(result: Any, changed: Any) -> bool:
811 """Did this seat state a ``Checked:`` line that names only absent things? (pure)"""
812 if changed is None:
813 return False
814 stated = _stated_line(result, _STATED_SCOPE_RE)
815 if not stated:
816 return False
817 resolved, unresolved = resolve_stated_scope(stated, changed)
818 return not resolved and bool(unresolved)
821#: Longest adapter error a ballot carries. The full text is in the report's seat
822#: status line; the ballot only needs enough for a consumer to say why.
823BALLOT_ERROR_MAX = 200
826def _ballot_error(result: Any) -> str:
827 """The adapter's short, redacted, single-line error for the ballot (#975).
829 Empty when the adapter reported none. Redacted again here (the adapters
830 already redact stderr; a result built elsewhere may not have been), flattened
831 like every untrusted field that reaches a record, and capped.
832 """
833 err = getattr(result, "error", None)
834 if not err or getattr(result, "ok", False):
835 return ""
836 text = flatten_inline(redaction.redact(str(err))[0])
837 code = getattr(result, "error_code", None)
838 text = f"[{code}] {text}" if code else text
839 return text[:BALLOT_ERROR_MAX]
842def abstention_cause(result: Any, changed: Any = None) -> str:
843 """Which of :data:`ai_jury.panel.ABSTENTION_CAUSES` this seat's ballot records.
845 **The** classification, and the only one: the ballot carries its answer under
846 :data:`ai_jury.panel.CAUSE_FIELD`, the two abstention sentences below are
847 written from it, and :func:`ai_jury.panel.abstention_buckets` counts by it. A
848 second reading of the raw result is a second place for the count and the
849 prose to part company, which is how a seat that named a file and then refused
850 came to be counted as one that named nothing (#700, round 5).
852 ``changed`` is the change under review, and it separates the last two
853 causes: without it a seat that named only absent things is indistinguishable
854 from one that named nothing, so the cause degrades to ``named_nothing``
855 rather than being guessed.
857 An allowance failure (``quota_exhausted`` / ``rate_limited``) with no output
858 is tested **first** (#975): a seat whose CLI exited on a usage limit produced no output, but
859 "silent" hides the one fact an operator acts on, so it is ``adapter_failed``
860 and the redacted error rides on the ballot's ``error``. ``silent`` therefore
861 means "no output and no failure reported" — exactly
862 :func:`ai_jury.panel.silent_seat` — so the metadata's ``silent`` is the same
863 number whether it is taken from the results or from the ballots.
865 Only ever asked of a ballot that did not review; a seat that reviewed has no
866 cause, and :func:`reviewer_ballots` records an empty string for it.
867 """
868 if adapter_failure(result):
869 return ADAPTER_FAILED
870 if not responded(result):
871 return SILENT
872 if not getattr(result, "ok", False):
873 return ADAPTER_FAILED
874 if is_abstention(getattr(result, "output", "")):
875 return REFUSED
876 # The two shapes of "its scope did not stand", kept apart because they send
877 # their reader to opposite places (#710). A seat whose ``Checked:`` line
878 # named `src/made/up.py` did not fail to say what it read — it said it read
879 # something this change does not contain, and "named nothing checkable"
880 # printed over that ballot is a description its own scope contradicts.
881 if _named_only_absent(result, changed):
882 return NOT_IN_CHANGE
883 return NAMED_NOTHING
886#: The sentence each cause contributes to the scope of a ballot that could not
887#: state one. Keyed by :func:`abstention_cause` so the reason and the bucket are
888#: one classification; ``named_nothing`` is refined below by whether the seat
889#: raised findings it attached to nothing.
890_NO_SCOPE_REASONS = {
891 SILENT: "it ran and returned nothing at all — an empty reply from the CLI",
892 ADAPTER_FAILED: "its adapter reported failure and what came back named nothing",
893 REFUSED: "it returned a refusal rather than a review",
894}
897def _no_scope_reason(result: Any, findings: list | None = None, changed: Any = None) -> str:
898 """Why nothing checkable could be lifted from this seat (pure).
900 Six reasons, kept apart because they ask for different fixes: a broken
901 adapter, a CLI that answered with nothing at all, a refusal, a ``Checked:``
902 line naming only things this change does not contain, a ``Checked:`` line
903 naming nothing at all, a code review whose findings named no location, and a
904 reply that reviewed nothing. The first three are :func:`abstention_cause`'s,
905 read from the one classifier rather than re-tested here; the rest are the
906 shapes of ``named_nothing`` and ``not_in_change``, and the splits matter
907 because "said nothing" sends its reader somewhere different from "named a
908 file that is not in the diff" and from "raised findings and attached none of
909 them to a file".
911 Every quoted token passes through :func:`_deanchor` first: this sentence
912 lands in the ballot's ``scope``, and a scope explaining that nothing was
913 checked must not itself read as an anchor to the consumer applying the same
914 rule.
915 """
916 cause = abstention_cause(result, changed)
917 if cause == ADAPTER_FAILED and adapter_failure(result) and not responded(result):
918 # Nothing came back because the CLI was refused on its allowance (#975):
919 # say that, and quote the CLI's own short error so a consumer reading
920 # only the scope (keel) sees why. De-anchored like every quoted token.
921 return (
922 "its CLI failed on its usage allowance (usage limit or rate limit) and "
923 f"returned nothing — {_deanchor(_ballot_error(result)).rstrip('.')}"
924 )
925 stated = _NO_SCOPE_REASONS.get(cause)
926 if stated:
927 return stated
928 named = _stated_line(result, _STATED_SCOPE_RE)
929 if cause == NOT_IN_CHANGE:
930 listed = ", ".join(_deanchor(t) for t in resolve_stated_scope(named, changed)[1])
931 return (
932 f"it stated it read {listed}, and no such path or symbol is in this change — "
933 f"so the ballot names a place a reader cannot go to, which is not the same as "
934 f"naming none"
935 )
936 if named:
937 return (
938 f"it stated it read {_deanchor(named)}, which names no path, line or symbol in "
939 f"this change at all — the shape of a scope with nothing in it"
940 )
941 raised = len(findings or [])
942 if raised:
943 return (
944 f"it raised {raised} finding(s) but attached none of them to a file, and its "
945 f"reply named no file, symbol or coverage clause either, so the ballot points "
946 f"at no place in the code a reader could go and check"
947 )
948 return "its reply named no file, symbol, coverage clause or finding"
951def abstention_scope(result: Any, findings: list | None = None, changed: Any = None) -> str:
952 """The scope of a ballot that could not state one: the reason, in the field.
954 Deliberately anchorless — no path, no backticked symbol, no "checked …"
955 clause — so a consumer applying the same substance rule reaches the same
956 conclusion this module did instead of being talked past it. The record is
957 here to say *nothing was reviewed*; dressing it up to survive the gate would
958 reinstate the defect with better prose.
960 ``findings`` is this reviewer's own findings, and it is passed so the reason
961 can tell "said nothing" from "said something that named nowhere" — the second
962 is a reviewer that worked and skipped the locations, and a reason that called
963 it "named no finding" would send its author looking for the wrong problem.
964 """
965 name = flatten_inline(getattr(result, "agent", "") or "").strip() or "this reviewer"
966 return (
967 f"Abstention: no scope can be stated for '{name}' because "
968 f"{_no_scope_reason(result, findings, changed)}. Recorded as an abstention rather than an "
969 f"approval — an agent that named nothing did not review, and counting it "
970 f"as one that did is the difference between a panel and a receipt."
971 )
974def describe_testing(result: Any) -> str:
975 """What this panelist ran to verify its claims, or :data:`NOT_STATED`.
977 The reviewer's ``Tested:`` line first, then any verification clause in its
978 prose. Both are lifted verbatim (flattened and capped) rather than
979 summarized: a testing claim carried downstream as evidence must be the
980 reviewer's words, not this module's paraphrase of them. With neither, the
981 field says plainly that nothing was run — which is a statement about the
982 review, where "not stated" was a statement about the field.
983 """
984 stated = _stated_line(result, _STATED_TESTING_RE)
985 if stated:
986 return f"Tested, as stated by the reviewer: {stated}"
987 clause = _first_matching(_free_clauses(result), _TESTING_RE)
988 return clause or NOT_STATED
991def _spec_for(config: Any, agent_name: str) -> Any:
992 for spec in getattr(config, "agents", []) or []:
993 if spec.name == agent_name:
994 return spec
995 return None
998def sent_model(result: Any) -> str:
999 """The model id this seat's invocation recorded having sent (``""`` if none).
1001 :attr:`ai_jury.adapters.AgentResult.model`, stamped by the path that ran the
1002 adapter from :meth:`ai_jury.adapters.Adapter.resolved_model` — the same call
1003 that put the id in the argv or the request payload. Reading it back is the
1004 whole of #709's fix: the ballot quotes the id the run sent instead of
1005 computing a second one that can differ from it.
1007 Empty for a record no invocation produced — a hand-built result, a chair
1008 slot with no round-1 seat — and :func:`requested_model` answers for those
1009 instead, under :data:`MODEL_RECOMPUTED` rather than :data:`MODEL_REQUESTED`.
1010 A stale result cache is deliberately *not* on that list: the record format
1011 gained this field, so :data:`ai_jury.cache.CACHE_SCHEMA` refuses an entry
1012 written without it rather than serving a recomputation in its place.
1013 """
1014 return (getattr(result, "model", "") or "").strip()
1017def requested_model(spec: Any) -> str:
1018 """The model id this agent's CLI would be asked for, from the spec alone.
1020 The fallback for a record that carries no sent id (:func:`sent_model`), and
1021 it computes the id the way the invocation path computes it — with
1022 :func:`ai_jury.config.spec_adapter`, **not** ``spec.vendor``. What it returns
1023 ships under :data:`MODEL_RECOMPUTED`: it is derived here, and the one thing
1024 it cannot be called is the id that was sent.
1026 That distinction is #709. ``spec.model`` is what the operator wrote down, but
1027 it is not always what is sent: where reasoning effort is encoded *in the
1028 model id*, the ``effort`` knob rewrites it. How effort is expressed is a
1029 property of the protocol the seat is invoked through, so every adapter keys
1030 :func:`ai_jury.adapters.effort_args` on the adapter; this keyed it on the
1031 vendor, and since #705 the two can differ. A seat configured
1032 ``vendor = google, adapter = cli, model = gemini-3-pro, effort = high`` was
1033 invoked with ``gemini-3-pro`` and balloted ``gemini-3-pro-high``, under a
1034 ``model_source: requested`` that claims to be the id actually sent.
1036 One thing this cannot see, and the reason the sent id is preferred over it:
1037 the invocation may consult the vendor's live model listing and fall back when
1038 the mapped id is not offered. That is I/O, and this module is pure.
1040 An effort level :func:`ai_jury.adapters.effort_args` rejects degrades to the
1041 configured id — a bad config value is ``validate_config``'s to refuse, never
1042 a ballot's to crash on.
1043 """
1044 from .adapters import effort_args
1045 from .config import spec_adapter
1047 configured = (getattr(spec, "model", "") or "").strip()
1048 try:
1049 plan = effort_args(spec_adapter(spec), getattr(spec, "effort", None), configured)
1050 except ValueError:
1051 return configured
1052 return (getattr(plan, "model", "") or configured or "").strip()
1055def describe_model(config: Any, agent_name: str, result: Any = None) -> tuple[str, str]:
1056 """``(model, source)`` for the agent that answered in this slot.
1058 ``result`` is that seat's own round-1 result when there is one, and it is
1059 the first source: it carries the id its invocation sent (#709), so
1060 ``model_source: "requested"`` names the string that was actually on the wire
1061 rather than a second derivation of it.
1063 Never the empty string for a slot that has an agent. An empty ``model`` was
1064 the provenance half of #700: a ballot naming ``vendor: "openai"`` and
1065 ``model: ""`` cannot answer "was that the same model as the other seat", and
1066 provenance is the entire product of a cross-vendor panel. Where the CLI was
1067 invoked with no id pinned there *is* an honest answer — the CLI's own
1068 default, which the CLI does not report back — so the field says that instead
1069 of going blank and letting a reader guess which of the two it meant.
1071 ``source`` is the same fact as one machine token, so a consumer can tell an
1072 id from a statement about one without parsing English.
1074 **A derived id is never labelled as a sent one** (#709, round 2). Where no
1075 invocation recorded an id — a hand-built result, a chair slot with no
1076 round-1 seat, a library caller — :func:`requested_model` still answers, and
1077 that answer goes out under :data:`MODEL_RECOMPUTED` rather than
1078 :data:`MODEL_REQUESTED`. It is a real id and worth quoting, but it is this
1079 module's arithmetic over the config, not a reading of the wire: for a Google
1080 seat at ``effort = high`` whose live model listing forced the adapter back to
1081 ``gemini-3-pro``, this returns ``gemini-3-pro-high``. Under ``requested``
1082 that is #709 restated — a model the run did not send, under a token whose
1083 whole claim is that it did.
1084 """
1085 name = (agent_name or "").strip()
1086 if not name:
1087 return "", MODEL_NONE
1088 spec = _spec_for(config, name)
1089 if spec is None:
1090 return f"unknown (no agent named '{name}' in this run's config)", MODEL_UNKNOWN
1091 sent = sent_model(result)
1092 if sent:
1093 return sent, MODEL_REQUESTED
1094 recomputed = requested_model(spec)
1095 if recomputed:
1096 return recomputed, MODEL_RECOMPUTED
1097 command = (getattr(spec, "command", "") or getattr(spec, "vendor", "") or name).strip()
1098 return _CLI_DEFAULT_MODEL.format(command=command), MODEL_CLI_DEFAULT
1101def _vendor_for(config: Any, agent_name: str) -> str:
1102 for spec in getattr(config, "agents", []) or []:
1103 if spec.name == agent_name:
1104 return spec.vendor or ""
1105 return ""
1108def participating(outcome: Any) -> list:
1109 """Every round-1 seat that ran, in the stable panel order — one ballot each.
1111 It used to be "seats that returned output at all", which quietly dropped a
1112 silent agent from the bundle: an `alpha` result with empty output left no
1113 `alpha` entry, and the report could not say which seat had returned nothing
1114 (#700, round 2). It is recorded as an abstention naming the seat and the
1115 reason instead, and :func:`ai_jury.panel.is_review` — not this function's
1116 length — is what decides whether it counts as a review.
1118 That is the change from #699, where the length of this *was* the count.
1119 Recording a seat and counting it as a review are now two different questions,
1120 because a seat can ballot without reviewing; the count lives in
1121 :mod:`ai_jury.panel` and reads the produced records.
1122 """
1123 return ballot_seats(getattr(outcome, "reviews", []) or [])
1126def _stance_by_reviewer(outcome: Any, names: list[str], mode: str) -> dict[str, str]:
1127 """Per-panelist stance, derived exactly as the vote tally derives ballots.
1129 :func:`ai_jury.voting.tally_votes` is the single source of truth for turning
1130 "the worst supported finding this reviewer raised" into a stance, so it is
1131 called rather than reimplemented — a second copy of that mapping is a second
1132 place for the severity thresholds to drift.
1133 """
1134 result = tally_votes(getattr(outcome, "groups", []) or [], names, mode=mode)
1135 return {b.reviewer: normalize_verdict(b.vote) for b in result.ballots}
1138def _verdict_for(result: Any, stances: dict[str, str], *, scoped: bool) -> str:
1139 """This panelist's stance, or :data:`ABSTAIN` (pure).
1141 ``scoped`` is the third way to abstain and the one #700 added: a seat that
1142 exited 0, said something, and named nothing checkable — including, since
1143 round 2, one whose only "scope" was a claim raised against no file in a
1144 code review. The other two — a failed adapter, an empty reply or a refusal —
1145 were already here, and this is the same principle applied one step further
1146 out. A reviewer whose scope is empty raised no findings *with a location*
1147 either, so the tally would have handed it the clear stance
1148 (``APPROVE``/``READY``): an approval inferred from silence, which is
1149 precisely what :mod:`ai_jury.voting` refuses to do (#251).
1150 """
1151 if not getattr(result, "ok", False):
1152 return ABSTAIN
1153 if is_abstention(getattr(result, "output", "")):
1154 return ABSTAIN
1155 if not scoped:
1156 return ABSTAIN
1157 return stances.get(getattr(result, "agent", ""), ABSTAIN)
1160def chair_verdict(outcome: Any, vote: Any = None) -> str:
1161 """The run's final verdict as one machine token.
1163 The panel vote when voting; otherwise the label the chair opened its
1164 synthesis with. The headline lift is
1165 :func:`ai_jury.report._verdict_headline` — reused rather than duplicated, so
1166 the JSON verdict and the markdown TL;DR can never disagree. The headline is a
1167 label plus a sentence (``REQUEST CHANGES — one confirmed major issue.``); only
1168 the label is a verdict, so the sentence is dropped.
1169 """
1170 from .report import _verdict_headline
1172 headline = _verdict_headline(getattr(outcome, "synthesis", None), vote)
1173 if not headline:
1174 return ABSTAIN
1175 label = re.split(r"[—–:.]| - ", headline, maxsplit=1)[0]
1176 return normalize_verdict(label) or ABSTAIN
1179#: The clause each cause contributes to a *scoped* abstention's sentence. Only
1180#: two causes can reach it: a seat whose scope stands named something, so it was
1181#: neither silent nor a reply that named nothing.
1182_SCOPED_ABSTENTION_REASONS = {
1183 ADAPTER_FAILED: "its adapter reported failure",
1184 REFUSED: "it returned a refusal rather than a review",
1185}
1188def _abstained_because(result: Any) -> str:
1189 """Why a seat that *did* name something checkable still abstained (pure).
1191 The companion to :func:`_no_scope_reason`, and deliberately not that
1192 function: this ballot's scope is substantive, so every reason phrased around
1193 "it named nothing" would be false of it. Only two of :func:`_verdict_for`'s
1194 three gates can fire while the scope stands — a failed adapter and a refusal
1195 — and the other two causes never reach here, because a seat with no scope
1196 gets :func:`abstention_scope` instead. Read from
1197 :func:`abstention_cause` all the same, so this sentence and the bucket the
1198 same seat is counted in cannot name two different things.
1199 """
1200 return _SCOPED_ABSTENTION_REASONS.get(abstention_cause(result), "it cast no vote")
1203def _chaired_ballot_sentence(result: Any, ballot: dict) -> str:
1204 """What the chairing agent's own ballot is, read off the record (#700, round 3).
1206 Said on the ballot as well as on the chair record, because the two are read
1207 in different places: a consumer that posts one verdict per review shows this
1208 text on its own, with no chair record beside it.
1210 The sentence is a function of ``counts_as_review`` — the answer
1211 :func:`ai_jury.panel.is_review` already gave for this record — and never of
1212 "did a scope come back". Round 2 keyed it on the scope alone and appended it
1213 before the verdict was in, so a chaired seat that named a file and then
1214 refused shipped ``verdict: ABSTAIN``, ``counts_as_review: false`` and a scope
1215 telling the human reading it that this ballot was one of the panel's reviews.
1216 The count and the prose beside it are one statement, or the prose is a second
1217 definition of a review that nothing keeps honest.
1218 """
1219 lead = " This reviewer also chaired the run (verification and synthesis);"
1220 tail = " the chair record is the panel's consensus rather than a further review."
1221 if ballot.get("counts_as_review"):
1222 return f"{lead} this ballot is one of the panel's reviews, and{tail}"
1223 return (
1224 f"{lead} this ballot abstained — {_abstained_because(result)} — so it is"
1225 f" NOT one of the panel's reviews, and{tail}"
1226 )
1229def _verified_count(outcome: Any, name: str) -> int:
1230 """Consensus groups this reviewer contributed to that the verifier upheld."""
1231 return sum(
1232 1
1233 for g in getattr(outcome, "groups", []) or []
1234 if name in (getattr(g, "reviewers", []) or [])
1235 and (getattr(g, "status", "") or "") == "verified"
1236 )
1239def reviewer_ballots(
1240 outcome: Any, config: Any, *, vote: Any = None, mode: str = "code"
1241) -> list[dict]:
1242 """The JSON report's ``reviewers`` array: one ballot per seat, then the chair.
1244 Panelist entries carry ``name``, ``role: "panelist"``, ``chaired``,
1245 ``vendor``, ``model``, ``model_source``, ``verdict``, ``scope``,
1246 ``scope_substantive``, ``counts_as_review``, ``testing``, ``findings``
1247 (indexes into the report's top-level ``findings`` array), ``round1_ok``,
1248 ``verified_count`` and ``duration_s``. The chair's entry is the one carrying
1249 ``role: "chair"`` and is always last.
1251 ``role`` is what a consumer splits on: the ``chair`` entry is the panel's
1252 consensus record and every other entry is a ballot (#699). But a ballot is
1253 not automatically a review — ``scope_substantive`` and ``verdict`` are the
1254 two facts :func:`ai_jury.panel.is_review` reads, and ``counts_as_review``
1255 carries its answer so the consumer need not re-derive it. Every seat that ran
1256 gets an entry, a silent one included, because the report has to be able to
1257 say *which* agent returned nothing; the count is what excludes it.
1259 ``chaired`` and the chair entry's ``agent``/``ballot_counted`` exist because
1260 the chairing agent reviews too: without them a reader cannot tell that the
1261 ``claude`` ballot and the ``chair`` record are the same agent, nor whether
1262 that agent contributed a review at all — and a reader who guesses drops a
1263 review the panel cast.
1265 ``scope`` and ``testing`` live here rather than only in the bundle (#700) so
1266 that the two renderings are the same text by construction: the JSON report
1267 said who voted and the bundle said what they read, and a reader comparing
1268 them had no guarantee the second described the first.
1269 """
1270 seats = participating(outcome)
1271 # What the panel was actually shown (#710), so a reviewer's `Checked:` line
1272 # is resolved against the change rather than accepted on its shape. ``None``
1273 # on an outcome that was not built from a diff; the rule then falls back to
1274 # the structural test, because "not verifiable" is not "does not exist".
1275 #
1276 # ``--issue`` is the one mode that supplies no change to resolve against: the
1277 # panel is reading an issue's prose, and a reviewer naming "the acceptance
1278 # criteria section" has named exactly what it was asked to name. That is the
1279 # same exception :func:`_claims_named` documents — there a finding carries no
1280 # file by construction — and applying a diff rule to a document with no diff
1281 # would abstain over a clean triage that did its job.
1282 changed = None if mode == ISSUE_MODE else getattr(outcome, "changed", None)
1283 names = [getattr(r, "agent", "") for r in seats]
1284 stances = _stance_by_reviewer(outcome, names, mode)
1285 all_findings = list(getattr(outcome, "findings", []) or [])
1286 chair_name = getattr(outcome, "chair", "") or ""
1288 entries: list[dict] = []
1289 for r in seats:
1290 name = getattr(r, "agent", "")
1291 chaired = bool(chair_name) and name == chair_name
1292 indexes = [i for i, f in enumerate(all_findings) if f.reviewer == name]
1293 own_findings = [all_findings[i] for i in indexes]
1294 scope = describe_scope(r, own_findings, mode=mode, changed=changed)
1295 scoped = bool(scope)
1296 if not scoped:
1297 scope = abstention_scope(r, own_findings, changed)
1298 model, model_source = describe_model(config, name, r)
1299 entry = {
1300 "name": name,
1301 "role": PANELIST_ROLE,
1302 "chaired": chaired,
1303 "vendor": getattr(r, "vendor", "") or "",
1304 "model": model,
1305 "model_source": model_source,
1306 "verdict": _verdict_for(r, stances, scoped=scoped),
1307 "scope": scope,
1308 # The two facts the count is made of, stated structurally rather than
1309 # left to be inferred from the prose in ``scope`` — a consumer that
1310 # had to parse English for this is a consumer that will get it wrong.
1311 "scope_substantive": scoped,
1312 "testing": describe_testing(r),
1313 "findings": indexes,
1314 "round1_ok": bool(getattr(r, "ok", False)),
1315 "error": _ballot_error(r),
1316 "verified_count": _verified_count(outcome, name),
1317 "duration_s": round(float(getattr(r, "duration_s", 0.0) or 0.0), 3),
1318 }
1319 # Derived, never asserted: the answer this record carries is the one the
1320 # gate and the announcements use, computed by the same function.
1321 entry["counts_as_review"] = is_review(entry)
1322 # And, when the answer is no, *why* — the fact a reader of the record
1323 # alone cannot recover, because a seat that fell silent and one that
1324 # answered without naming anything leave the same two fields behind. It
1325 # travels with the ballot so every renderer counts and describes the same
1326 # seat the same way instead of subtracting one bucket from another
1327 # (#700, round 5). Empty on a ballot that reviewed: it has no cause.
1328 entry[CAUSE_FIELD] = "" if entry["counts_as_review"] else abstention_cause(r, changed)
1329 # After the answer, never before it: the chair's sentence *reports* that
1330 # answer, so it cannot be written while the answer is still unknown.
1331 if scoped and chaired:
1332 entry["scope"] += _chaired_ballot_sentence(r, entry)
1333 entries.append(entry)
1335 # The chair's own round-1 seat, when it has one: the chairing agent reviews
1336 # too, and every phase of it runs through the same adapter, so the id its
1337 # ballot recorded is the id its synthesis was produced with.
1338 chair_result = next((r for r in seats if getattr(r, "agent", "") == chair_name), None)
1339 chair_model, chair_model_source = describe_model(config, chair_name, chair_result)
1340 verify = getattr(outcome, "verify", None)
1341 chair_scope = _chair_scope(outcome, entries)
1342 entries.append(
1343 {
1344 "name": CHAIR_NAME,
1345 "role": CHAIR_ROLE,
1346 "agent": chair_name,
1347 # Whether the chairing agent's own ballot is one of the counted
1348 # reviews — the fact the bundle never stated (#699). A chairing agent
1349 # that ran and abstained has a ballot in the bundle and is not a
1350 # review, so "did it ballot" is the wrong question to answer here.
1351 "ballot_counted": any(e["counts_as_review"] for e in entries if e["chaired"]),
1352 "reviews_supplied": review_count(entries),
1353 "vendor": _vendor_for(config, chair_name),
1354 "model": chair_model,
1355 "model_source": chair_model_source,
1356 "verdict": chair_verdict(outcome, vote),
1357 "scope": chair_scope,
1358 # Measured, not asserted: the chair record is a verdict too, and a
1359 # consumer applying one substance rule applies it here as well.
1360 "scope_substantive": scope_is_substantive(chair_scope),
1361 # This synthesis record is NOT one of the reviews, whatever its
1362 # scope says: the consumer reads it as the panel's consensus.
1363 "counts_as_review": False,
1364 # Empty, and present rather than absent: the chair is not a seat that
1365 # failed to review, it is the record carried alongside the seats, and
1366 # a key missing here would read as a cause nobody wrote down.
1367 CAUSE_FIELD: "",
1368 "error": "",
1369 "testing": describe_testing(verify) if verify is not None else NOT_STATED,
1370 }
1371 )
1372 return entries
1375def _keel_finding(f: Any) -> dict:
1376 """One finding in the consumer's shape: ``file`` → ``path``, ``claim`` → ``message``."""
1377 line = getattr(f, "line", None)
1378 return {
1379 "severity": getattr(f, "severity", "") or "",
1380 "path": getattr(f, "file", "") or "",
1381 "line": line if isinstance(line, int) else None,
1382 "message": flatten_inline(getattr(f, "claim", "") or ""),
1383 }
1386def _chair_findings(outcome: Any) -> list:
1387 """The chair's surviving evidence: group representatives the verifier did not reject."""
1388 return [
1389 g.representative
1390 for g in getattr(outcome, "groups", []) or []
1391 if (getattr(g, "status", "") or "") != "unsupported"
1392 ]
1395def _chair_role_sentence(chair_agent: str, chaired_ballot: dict | None) -> str:
1396 """State, in the record itself, whether this chair's ballot is counted (#699).
1398 A chair that reviewed and a chair that only synthesised produce records that
1399 are otherwise identical, and a reader who cannot tell them apart guesses —
1400 which is how a panel's ballots get handed on short. So the record says which
1401 agent chaired and where (or whether) its own ballot is in the bundle, and it
1402 says plainly that this synthesis record is not itself one of the reviews.
1403 """
1404 who = f"`{chair_agent}`" if chair_agent else "This run's chair"
1405 if chaired_ballot is not None and chaired_ballot.get("counts_as_review"):
1406 return (
1407 f"The chair is {who}, which also sat on the panel: its ballot is the "
1408 f"'{chaired_ballot['name']}' review in this bundle and counts as one "
1409 f"of the reviews. This synthesis record does not — it is the panel's "
1410 f"consensus, not a ballot."
1411 )
1412 if chaired_ballot is not None:
1413 # It balloted and the ballot is not a review. Saying "no ballot from it"
1414 # would be false and saying "its ballot counts" would be the defect, so
1415 # the record says both facts. It does *not* say why: "named nothing
1416 # checkable" was one of the three ways to abstain asserted as if it were
1417 # the only one, and it is plainly false of a seat that named a file and
1418 # then refused. The ballot's own scope carries the cause (#700, round 3).
1419 return (
1420 f"The chair is {who}, which also sat on the panel but abstained: its "
1421 f"'{chaired_ballot['name']}' ballot is in this bundle and is NOT counted "
1422 f"as a review — that ballot's own scope says why. This "
1423 f"synthesis record is not counted either — it is the panel's consensus, "
1424 f"not a ballot."
1425 )
1426 return (
1427 f"The chair is {who}, which returned no panel review of its own, so this "
1428 f"bundle carries no ballot from it. This synthesis record is not counted "
1429 f"as a review either — it is the panel's consensus, not a ballot."
1430 )
1433def _chair_scope(outcome: Any, panelists: list[dict]) -> str:
1434 """What the chair's synthesis covered (pure).
1436 Backticked file names, like every other scope here (#700): the chair record
1437 is a verdict too, and a consumer applying one substance rule applies it to
1438 this record as well. The chairing agent's name is already backticked by
1439 :func:`_chair_role_sentence`, so a chaired run anchors either way.
1441 The two numbers it quotes are deliberately different (#700, round 2): how
1442 many of the ballots are **reviews**, and how many **records** the bundle
1443 carries. They used to be the same integer, which is exactly how an abstaining
1444 ballot got announced as a review.
1445 """
1446 groups = getattr(outcome, "groups", []) or []
1447 files = _files_named([g.representative for g in groups])
1448 listed = ", ".join(_tick(f) for f in files[:_FILES_LISTED]) if files else "no specific file"
1449 more = len(files) - _FILES_LISTED
1450 suffix = f" (+{more} more)" if more > 0 else ""
1451 chair_agent = getattr(outcome, "chair", "") or ""
1452 chaired = next((p for p in panelists if p.get("chaired")), None)
1453 reviews = review_count(panelists)
1454 # "Further" than the chair's own ballot, which the sentence before this one
1455 # has already accounted for — and no cause is named, because these seats
1456 # abstained for whichever of the three reasons applied to each and their own
1457 # scopes say which (#700, round 3).
1458 abstained = sum(1 for p in panelists if not p.get("chaired") and not is_review(p))
1459 abstained_clause = (
1460 f" {abstained} further ballot(s) abstained; they are recorded here but"
1461 f" are not reviews, and each one's scope says why."
1462 if abstained
1463 else ""
1464 )
1465 return (
1466 f"Chair synthesis over {reviews} panel review(s) and "
1467 f"{len(groups)} consensus group(s), across {listed}{suffix}. "
1468 f"{_chair_role_sentence(chair_agent, chaired)}{abstained_clause} This bundle carries "
1469 f"{reviews} review(s) plus this record, "
1470 f"{bundle_records(len(panelists))} records in all."
1471 )
1474def keel_reviews(outcome: Any, config: Any, *, vote: Any = None, mode: str = "code") -> list[dict]:
1475 """The ``--format keel-reviews`` bundle: one record per seat, plus the chair.
1477 Each record is ``{reviewer, verdict, scope, findings, testing, vendor, model,
1478 model_source, counts_as_review, error}`` where ``findings`` are ``{severity, path,
1479 line, message}`` objects — the shape a consumer of head-pinned per-reviewer
1480 verdicts accepts. Pure: the caller serializes it.
1482 A projection of :func:`reviewer_ballots`, not a second derivation of the same
1483 facts (#700). Every field but ``findings`` is renamed or copied straight
1484 across, so the verdict the JSON report shows for a panelist and the verdict
1485 the consumer is handed for it cannot disagree — and neither can the scope
1486 that is supposed to justify it.
1488 Two of those fields are the projection catching up with the ballot (#700,
1489 round 2). ``model_source`` because ``model`` here changed meaning in the same
1490 release — a CLI default is now an English sentence — and a machine consumer
1491 of *this* shape could not tell a requested id from a default without parsing
1492 prose; the ``reviewers`` array grew the discriminator and this one did not.
1493 ``counts_as_review`` because the bundle now carries abstention records for
1494 seats that returned nothing, and a consumer counting the array would count
1495 them.
1496 """
1497 ballots = reviewer_ballots(outcome, config, vote=vote, mode=mode)
1498 all_findings = list(getattr(outcome, "findings", []) or [])
1500 records: list[dict] = []
1501 for b in ballots:
1502 chair = b.get("role") == CHAIR_ROLE
1503 own = _chair_findings(outcome) if chair else [all_findings[i] for i in b["findings"]]
1504 records.append(
1505 {
1506 "reviewer": b["name"],
1507 "verdict": b["verdict"],
1508 "scope": b["scope"],
1509 "findings": [_keel_finding(f) for f in own],
1510 "testing": b["testing"],
1511 "vendor": b["vendor"],
1512 "model": b["model"],
1513 "model_source": b["model_source"],
1514 "counts_as_review": b["counts_as_review"],
1515 # Why a seat that did not review failed (#975): without it the
1516 # consumer sees an abstention and not that the CLI hit its limit.
1517 "error": b["error"],
1518 }
1519 )
1520 return records