Coverage for src/ai_jury/panel.py: 100%
65 statements
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-20 18:15 +0000
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-20 18:15 +0000
1"""Panel arithmetic: how many reviews a consumer actually receives (#699, #700).
3A downstream gate does not ask "how many agents ran"; it asks "how many reviews
4did I get". Those two numbers were never the same and nothing in the tool ever
5said so, which is how a healthy three-CLI bench was reported as ``cross-vendor
6ready: yes``, exited 0, and then failed at the consumer with *supplied 2
7review(s) but tier requires at least 3*.
9Four facts settle the arithmetic, and all four live here so that every renderer,
10``--doctor`` and the run itself quote the same one:
12* **A review is a ballot that reviewed.** :func:`is_review` is the definition,
13 and it is the only one: a ``panelist`` record whose scope names something a
14 reader could go and check, and whose verdict is a vote rather than
15 :data:`ABSTAIN`. Every seat that ran gets a ballot record — that is how the
16 report can say *which* seat returned nothing — but a ballot that named nothing
17 is the record of a seat that did **not** review, and counting it supplies the
18 consumer a review it will refuse. The consumer's own
19 ``review-verdict-insubstantial`` rule rejects exactly that shape, so a count
20 including it would be #699's mismatch wearing better prose: a number ai-jury
21 states and the consumer does not find.
22* **The chair's synthesis is not a review.** The consumer splits the report's
23 ``reviewers`` array on ``role``: the entry carrying ``role: "chair"`` becomes
24 the panel's consensus record, and only a panelist entry can be a review. The
25 chair's synthesis is carried *alongside* the reviews, never as one of them.
26* **The chair sits on the panel.** :func:`ai_jury.orchestrator.resolve_chair`
27 only ever picks a chair from the *usable* agents, and round 1 runs every usable
28 agent, so the chairing agent reviews on every code path. That ballot is an
29 ordinary panelist record carrying no chair role, and the consumer counts it
30 like any other. What was missing was any statement of that: a reader of the
31 bundle could not tell whether the chairing agent had also voted, and one who
32 guesses drops a review the panel actually cast.
33* **A seat can ballot without reviewing.** An agent that returns nothing, refuses,
34 fails, or answers with prose naming no file, line or symbol is recorded as an
35 abstention naming *which* seat and *why* — visible in the bundle, and excluded
36 from the count. So the number of reviews is knowable only as an upper bound
37 before the run, which is why the gate is checked on both sides of it.
38* **Every seat is in exactly one bucket, and the bucket is the cause.**
39 :func:`abstention_buckets` splits the non-reviewing seats five ways — silent,
40 named nothing, named only what is not in the change, refused, adapter failed —
41 by reading each ballot's own recorded cause. It replaces a subtraction (``ballots - silent - supplied``) that was
42 arithmetically right and descriptively false: once a ballot could carry a
43 substantive scope and still abstain, the remainder held seats that had named a
44 file and then refused, and every renderer billed them as having named nothing
45 checkable — a cause the ballot beside the number contradicted (#700, round 5).
47Everything here is pure: no I/O, no clock, no randomness.
48"""
50from __future__ import annotations
52from collections.abc import Mapping
53from typing import Any
55#: Stance recorded for a seat that did not review. Defined here rather than in
56#: :mod:`ai_jury.ballots` because :func:`is_review` — the one definition of what
57#: gets counted — has to test for it, and a second copy of the token is a second
58#: place for the count and the record to disagree. ``ballots`` re-exports it.
59ABSTAIN = "ABSTAIN"
61#: The two values of a ``reviewers`` entry's ``role``. A consumer splits on this.
62PANELIST_ROLE = "panelist"
63CHAIR_ROLE = "chair"
65#: Records the bundle carries that are **not** reviews: the chair's synthesis.
66#: Named rather than inlined as ``+ 1`` so that prose about the bundle's *shape*
67#: and prose about its *review count* cannot quietly become the same number.
68CHAIR_SYNTHESIS_RECORDS = 1
70#: Why a seat that balloted did not review. Five causes, because five different
71#: things happen to a seat and they ask for five different fixes.
72SILENT = "silent"
73NAMED_NOTHING = "named_nothing"
74#: The seat named something and none of it is in the change (#710). Split out of
75#: ``named_nothing`` because the two send their reader to opposite places: a
76#: reviewer that named nothing did not say what it read, while one that named
77#: `src/made/up.py` said it read a file that is not in the diff — a claim about
78#: coverage that the change itself contradicts, and a much louder signal.
79NOT_IN_CHANGE = "not_in_change"
80REFUSED = "refused"
81ADAPTER_FAILED = "adapter_failed"
83#: The causes in the order every renderer lists them, and the key set of
84#: :func:`abstention_buckets` — so a renderer cannot iterate a cause the buckets
85#: do not carry, nor miss one they do.
86ABSTENTION_CAUSES = (SILENT, NAMED_NOTHING, NOT_IN_CHANGE, REFUSED, ADAPTER_FAILED)
88#: The clause each cause contributes to a count sentence, subject ``"N "``. One
89#: table, because the markdown report and :func:`shortfall` describe the same
90#: seats and used to phrase them apart — and the phrasing was where they lied.
91#: Every clause is true of *every* ballot in its bucket; in particular none of
92#: them says the seat named nothing, except the one bucket where that is what
93#: happened. "N named nothing checkable" asserted over a seat that named a file
94#: and *then* refused is the defect this table exists to make impossible
95#: (#700, round 5).
96CAUSE_PHRASES = {
97 SILENT: "returned nothing at all",
98 NAMED_NOTHING: "answered but named nothing checkable",
99 NOT_IN_CHANGE: "named only things that are not in this change",
100 REFUSED: "returned a refusal rather than a review",
101 ADAPTER_FAILED: "reported an adapter failure",
102}
104#: The run-metadata key each cause is published under. Kept here because three
105#: modules read those keys and one of them is not spelled like its cause:
106#: ``named_nothing`` ships as ``insubstantial``, the name it has had since #700's
107#: second round, and renaming a published key to tidy a table would break every
108#: consumer reading it. A renderer iterating this mapping cannot leave a cause
109#: out, which is what a hand-written pair of ``if`` statements did.
110PANEL_METADATA_KEYS = {
111 SILENT: "silent",
112 NAMED_NOTHING: "insubstantial",
113 NOT_IN_CHANGE: "not_in_change",
114 REFUSED: "refused",
115 ADAPTER_FAILED: "adapter_failed",
116}
118#: The key a ballot record carries its cause under. Read from the record, never
119#: recomputed from it: whether a seat fell silent or refused is a fact about the
120#: result that produced the ballot, and no predicate over the record alone can
121#: tell those two apart — both are ``scope_substantive: false`` when the refusal
122#: named nothing, which is why subtraction was reached for in the first place.
123CAUSE_FIELD = "abstention_cause"
126def responded(result) -> bool:
127 """Did this round-1 seat return any output at all?
129 "Returned output", not "exited 0": adapters fail soft, so a nonzero exit can
130 still carry a complete review on stdout. A seat with nothing at all still
131 gets a ballot — an abstention naming it, so the report can say which seat
132 fell silent (#700, round 2) — but this predicate is what lets that record
133 state *which* kind of nothing came back.
134 """
135 return bool((getattr(result, "output", "") or "").strip())
138#: Adapter error codes that say *why* a seat produced nothing, in terms an
139#: operator can act on: the account's allowance is spent (#975). A seat that fell
140#: silent on one of these did not fall silent, it was told no. Literal strings
141#: (``adapters.ERR_QUOTA_EXHAUSTED`` / ``ERR_RATE_LIMITED``) because this module
142#: stays free of the adapters; a test pins the two together.
143ALLOWANCE_ERROR_CODES = frozenset({"quota_exhausted", "rate_limited"})
146def adapter_failure(result) -> bool:
147 """Did the adapter report an allowance failure (usage limit / rate limit)?
149 Narrower than "any typed failure" on purpose: a timeout or a no-review banner
150 that left nothing behind is still reported as ``silent`` (the established
151 reading). A usage limit is different: the CLI *said* why, and "silent" hid it
152 from the one consumer that could act on it (#975).
153 """
154 return not getattr(result, "ok", True) and (
155 getattr(result, "error_code", None) in ALLOWANCE_ERROR_CODES
156 )
159def silent_seat(result) -> bool:
160 """No output at all **and** no allowance failure the adapter reported (#975)."""
161 return not responded(result) and not adapter_failure(result)
164def ballot_seats(reviews) -> list:
165 """The round-1 seats that produce a ballot, in the stable panel order.
167 **Every** seat that ran. A seat that returned nothing used to be dropped
168 here, which left the bundle unable to say which agent had gone silent: the
169 report simply had one fewer entry than the bench, and a reader comparing the
170 two had to guess which. It is recorded as an abstention instead, and
171 :func:`is_review` is what keeps it out of the count.
172 """
173 return list(reviews or [])
176def is_review(ballot: Mapping[str, Any]) -> bool:
177 """**The** definition: does this ballot record count as a review? (pure)
179 Three conditions, and each one is a defect this project has already shipped:
181 * ``role == "panelist"`` — the chair's synthesis record is the panel's
182 consensus, not an *n+1*-th review (#699).
183 * ``scope_substantive`` — the ballot names something a reader could go and
184 check. A prose-only reply (*"Looks good to me, no concerns."*) names
185 nothing, and the consumer refuses a verdict shaped like that (#700).
186 * ``verdict != ABSTAIN`` — the seat actually cast a vote. An abstention is
187 not an approval (#251), and it is not a review either.
189 Read from the record rather than recomputed from the agent result, because
190 the record is what the consumer is handed: if the two could disagree, the
191 number ai-jury announces would once again describe something other than the
192 document it shipped.
193 """
194 if not isinstance(ballot, Mapping): # pragma: no cover - defensive
195 return False
196 return (
197 (ballot.get("role") or "") == PANELIST_ROLE
198 and bool(ballot.get("scope_substantive"))
199 and (ballot.get("verdict") or "") != ABSTAIN
200 )
203def review_count(ballots) -> int:
204 """Reviews a consumer receives from these ballot records.
206 The one count. Every announcement, both halves of the ``min_reviews`` gate,
207 the markdown report, the chair record's ``reviews_supplied`` and ``--doctor``
208 resolve to this, so the number ai-jury states and the number the consumer
209 finds in the bundle are equal by construction.
211 Takes the **ballots**, not the raw round-1 results: whether a seat reviewed
212 is a fact about the record it produced — its scope and its verdict — and no
213 predicate over the raw result can see it.
214 """
215 return sum(1 for b in ballots or [] if is_review(b))
218def panelist_ballots(ballots) -> list:
219 """The ballot records that are *seats* — everything but the chair's synthesis.
221 The unit the buckets and the count are both taken over, so "how many seats
222 balloted" and "how many of them reviewed" can never be measured against two
223 different populations.
224 """
225 return [
226 b
227 for b in ballots or []
228 if isinstance(b, Mapping) and (b.get("role") or "") == PANELIST_ROLE
229 ]
232def abstention_cause(ballot: Mapping[str, Any]) -> str:
233 """Why this ballot is not a review — one of :data:`ABSTENTION_CAUSES` (pure).
235 **Read** from :data:`CAUSE_FIELD` rather than recomputed. Which of the four
236 happened is a fact about the result that produced the ballot: a seat that
237 fell silent and one that answered with nothing checkable leave the same two
238 structural fields behind, so a record that does not carry its cause has lost
239 it. The ballot is what the consumer is handed, so the cause travels with it.
241 A record written by an older build — or by hand, as tests do — carries no
242 cause, and is read from the fields every ballot has rather than refused: a
243 failed adapter says so in ``round1_ok``, and an empty ``scope_substantive``
244 means nothing checkable was named. That reading cannot see silence, which is
245 exactly why the field exists; it errs toward ``named_nothing``, which is true
246 of a silent seat as well.
247 """
248 if not isinstance(ballot, Mapping): # pragma: no cover - defensive
249 return NAMED_NOTHING
250 recorded = (ballot.get(CAUSE_FIELD) or "").strip()
251 if recorded in ABSTENTION_CAUSES:
252 return recorded
253 if not ballot.get("round1_ok", True):
254 return ADAPTER_FAILED
255 if not ballot.get("scope_substantive"):
256 return NAMED_NOTHING
257 return REFUSED
260def abstention_buckets(ballots) -> dict[str, int]:
261 """The seats that balloted without reviewing, counted by cause (pure).
263 The five counts and :func:`review_count` partition the panelist ballots:
264 every seat lands in exactly one, because each is classified by what happened
265 to it. Subtraction is what this replaces, and what subtraction shipped: with
266 ``insubstantial`` derived as ``ballots - silent - supplied``, every seat that
267 was neither silent nor a counted review was billed as "named nothing
268 checkable" — including, since a ballot could carry a substantive scope and
269 still abstain, the seat that named a file and *then* refused, and the one
270 whose adapter died holding a file name. The number was right and the cause
271 printed beside it was contradicted by the ballot it described (#700, round 5).
272 """
273 counts = dict.fromkeys(ABSTENTION_CAUSES, 0)
274 for ballot in panelist_ballots(ballots):
275 if not is_review(ballot):
276 counts[abstention_cause(ballot)] += 1
277 return counts
280def bundle_records(ballots: int) -> int:
281 """Total records in the bundle: the ballots, plus the chair's synthesis.
283 Deliberately distinct from the review count, and now distinct from it in two
284 directions: the chair record is one more record than there are ballots, and
285 a ballot is not necessarily a review. It exists for prose describing the
286 bundle's *shape* ("this bundle carries N records") and must never answer
287 "how many reviews", which is what :func:`review_count` is for.
288 """
289 return int(ballots) + CHAIR_SYNTHESIS_RECORDS
292def describe(reviews: int, *, available: int | None = None) -> str:
293 """One line stating the number a consumer will actually receive.
295 Rendered before the panel runs, from the agents that are reachable — which
296 can only be an **upper bound**, and now says so: every seat is a seat that
297 *might* review, and one that returns nothing or names nothing checkable is
298 recorded as an abstention rather than a review. Passing ``available``
299 selects that pre-run wording; without it the number is the count that
300 actually materialised.
302 The chair record is named in the same breath, and named as *not* a review,
303 because this line exists to forestall exactly the arithmetic that would add
304 it.
305 """
306 tail = f"plus {CHAIR_SYNTHESIS_RECORDS} chair synthesis record (not counted as a review)"
307 if available is not None:
308 return (
309 f"panel: {available} available agent(s) → at most {available} review(s) for a "
310 f"downstream consumer (one per seat that names what it read and votes; a seat "
311 f"that returns nothing, or names nothing checkable, abstains and is not one), "
312 f"{tail}"
313 )
314 return f"panel: {reviews} review(s) for a downstream consumer, {tail}"
317def shortfall(
318 supplied: int,
319 required: int,
320 *,
321 stage: str,
322 silent: int = 0,
323 insubstantial: int = 0,
324 not_in_change: int = 0,
325 refused: int = 0,
326 adapter_failed: int = 0,
327) -> str | None:
328 """Why this run cannot supply ``required`` reviews, or ``None`` if it can.
330 ``stage`` is the phrase naming when the check fired, so the three call sites
331 ("before the panel runs" / "after the panel ran" / "on this machine") read as
332 one message with the timing filled in rather than as unrelated errors.
334 The five counts are :func:`abstention_buckets` — the ways a seat that ran
335 produces no review — and they are named separately because the remedies
336 differ: a silent agent is usually a CLI that broke or a budget that ran out,
337 a seat that answered and named nothing is a reviewer that did not review, a
338 seat that named only things absent from the change reviewed something that
339 is not this diff, a refusal is a model declining the task, and a failed
340 adapter is an invocation to fix. ``insubstantial`` keeps its name for
341 ``named_nothing`` and means exactly that. Pass them as the buckets report
342 them: a caller that folds causes into one prints a sentence the ballots
343 contradict.
344 """
345 required = int(required or 0)
346 if required <= 0:
347 return None
348 supplied = int(supplied)
349 if supplied >= required:
350 return None
351 counted = {
352 SILENT: int(silent),
353 NAMED_NOTHING: int(insubstantial),
354 NOT_IN_CHANGE: int(not_in_change),
355 REFUSED: int(refused),
356 ADAPTER_FAILED: int(adapter_failed),
357 }
358 reasons = [
359 f"{counted[cause]} {CAUSE_PHRASES[cause]}" for cause in ABSTENTION_CAUSES if counted[cause]
360 ]
361 missing = sum(counted.values())
362 clause = (
363 f" {missing} seat(s) ran without producing a review "
364 f"({'; '.join(reasons)}), so they are recorded as abstentions "
365 f"and are not counted."
366 if missing
367 else ""
368 )
369 return (
370 f"panel too small {stage}: {supplied} review(s) available "
371 f"(the chair's synthesis record is not one of them, and neither is an "
372 f"abstaining ballot) but min_reviews requires {required}.{clause} "
373 f"Enable another agent, or lower `[jury.ci] min_reviews` / `--min-reviews`."
374 )