CodeEvolver Memory

memory last updated at iteration number: 26

Run id: 20260617092234 Seed branch: codeevolver-20260617092234-main Seed architecture: DSPy 3-Hop ColBERT Retrieval Pipeline with Resource-Penalized Evaluation

Overall Progress

Macro Approaches

Seed: Summarize-then-Query 3-Hop Retrieval

Iteration 1: Entity-Focused Query Generation (IdentifyNextTarget) + Deduplication

Iteration 2: 4-Hop Architecture + Improved Title-Only Coverage Check (No previous_queries)

Iteration 3: Fruitless-Query Feedback + Incremental Deduplication + Stronger Prompt (Incremental)

Iteration 4: All-Queries Tracking + Round-Robin Interleaving (Bug-fix Structural)

Iteration 5 was an incremental prompt-only change to IdentifyNextTarget (no architectural changes). It is excluded from the Macro Approaches list as it does not introduce a new solution structure.

Iteration 6: Multi-Entity Pre-Planning Before Retrieval (PlanAllTargets)

Iteration 6 Round 2 (programmatic query-retry via _get_query_with_retry) is an incremental structural bug-fix. It is excluded from the Macro Approaches list as it does not change the retrieval architecture.

Iteration 7 (title-list context format + strengthened false-presence Step 2 prompt) is an incremental prompt/formatting change. Excluded from Macro Approaches. Key lessons: (1) "RETRIEVED ARTICLE TITLES" prepended list is structurally incompatible with round-robin interleaving — creates eviction-mismatch failures, see Observations; (2) strengthened false-presence Step 2 with concrete worked examples provides modest net gain (+3 fixes vs –7 regressions when combined with the bad context format in Round 1; +3 fixes vs –6 regressions in Round 2). The Step 2 rewrite is likely net-positive in isolation but was masked by stochastic regressions.

Iteration 8: 5-Hop Architecture + no_results_queries Feedback

Iteration 9 (programmatic retry mechanism via _get_query_with_retry + Step 2 concrete false-presence example + concurrent Step 4 body scan with PRIORITY RULE) is an incremental prompt and structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lesson: same SHA scored +2.7% and −2.7% vs parent in consecutive subsample runs (gpt-5.4-nano at reasoning_effort=low has ±3% per-run variance). Small incremental changes cannot be reliably evaluated with a single subsample run. The PRIORITY RULE in Step 4 ("prefer implied entities over peripheral claim entities such as major cities, broad organisations, or entities from false-premise clauses") adds dangerous ambiguity — the LM over-applies it in unpredictable ways and should be replaced with a direct action directive stating explicitly that identified implied entities must become the query.

Iteration 10 (retry mechanism + normalized duplicate detection + removed no_results_queries + Step 2 false-presence example + Step 4 "OUTPUT IMMEDIATELY" action directive, all applied to candidate 5 / 5-hop base) is an incremental prompt and structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) The "OUTPUT THAT IMPLIED ENTITY AS YOUR QUERY IMMEDIATELY" language in Step 4 is as dangerous as the PRIORITY RULE — it caused ex35 (Queens) to fail by triggering premature/wrong Step 4 activation; the effect is identical to the iter5 PRIORITY RULE regression. (2) The false-presence example (Person A / Person B) and retry mechanism did not prevent any of the targeted failures. (3) The 5-hop architecture continues to amplify the negative impact of any LM behavioral change. (4) Candidate 5 (5-hop) is definitively not a productive base for incremental changes.

Iteration 11 (retry mechanism cherry-picked onto candidate 4 + Step 1 comparison-subject nuance + Step 1/3 phantom-entity guard + Step 2 false-presence example + Step 4 expanded pattern examples) is an incremental prompt and structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Retry mechanism confirmed working on 4-hop architecture — fired in 10/50 examples, 9 passed including the direct fix of ex76 (Delmer Daves). Only ex85 (Flex club / Ronnie Flex) failed with retry — the article simply does not exist under either name the LM tries. (2) Phantom-entity guard (Round 2: "skip descriptive claim phrases; ONLY check Step 1 entities in Step 3") fixed ex100 (20th Century Fox "Chinese film studio" loop) but failed to fix ex65 (Ancient Egyptian religion — LM still invented "Weighing of the Heart" in Step 3 despite the guard). The guard is insufficient alone; code-level fruitless detection is needed. (3) Step 4 expanded pattern examples (doubles partner, adapted song, stunt-performer film, broader topic article) improved structure but no single confirmed fix attributable to them in the trace. (4) The +2.6% subsample gain is within the ±3% noise floor — valset impact is uncertain. (5) 11 failure cases in the iter11 child are overwhelmingly inherited persistent failures; only 1 net improvement vs parent (ex100 fixed, no new regressions).

Iteration 12 Round 2 (context trimming in forward() + em-dash/hyphen normalization in _normalize_query) is an incremental structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Context trimming is a principled, mechanistic fix for the "false coverage" bug caused by round-robin interleaving: with k=7 and 4 equal hops, rank-7 docs from hop1 always land at position 25 in interleaved order and are evicted. Passing hop1_new[:6] (not all 7) to context2 prevents the LM from treating these evicted docs as "covered." Confirmed fix: ex31 (Sojourner Truth at rank-7 of hop1 — now correctly identified as uncovered and queried directly). (2) The valset confirmation (+3.8% relative, 0.7968→0.8267) proves context trimming generalizes beyond the subsample and is not noise. (3) em-dash normalization (-/–/— → spaces) is verified correct; its specific valset impact is not isolable but is low-risk and structurally sound. (4) Iteration 12 Round 1 (soft pre-planning via advisory PlanEntities) failed and was reverted — see Macro Approaches Iteration 6 entry for updated analysis.

Iteration 13 (code-level fruitless-query tracking + Step 4 named-pattern strengthening + Step 1 entity-type extension for routes/genera/concepts) is an incremental prompt and structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Step 4 named-pattern wording is effective when concrete — specific song title ("A World Without Love"), specific TV show ("Punchlines"), musical adaptation ("Stranger in Paradise") patterns fixed 2 subsample examples and represent generalizable categories. (2) Step 1 entity-type extension to include routes and named scientific concepts is confirmed useful (fixed NH Route 124). (3) Fruitless-query tracking adds useful architecture but didn't visibly change outcomes in this subsample because most failure cases have ColBERT returning WRONG articles (not zero articles). Isatis is the clearest example: querying "Isatis" returns Isatis tinctoria (new unique doc) so it is NOT marked fruitless, but the genus article is still missing. Fruitless detection only catches true zero-result queries, not ranking gaps. (4) Valset regression of −1.2% (0.8267→0.8167) is within ±3% noise and should not be over-interpreted. (5) The Dinner Party / Sojourner Truth failure persists across seeds: context trimming (iter12) fixed the specific case where Sojourner Truth was at rank-7 of hop1; but with a different seed, the claim's garbled wording "Isabella (Belle) Isabella" causes the LM to literally search for that phrase rather than resolving to "Sojourner Truth." This is a claim-resolution failure distinct from the prior context-trimming fix.

Iteration 14: Conditional 5th Hop with Priority Interleaving

Iteration 22: LM-Based Final Reranker Replacing Round-Robin Selection

Iteration 15 (k=7→12, None-query guard in _get_query_with_retry, Step 4 director/actress/choreographer bullet in IdentifyNextTarget, all applied to candidate 9) is an incremental structural code + prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) k=12 confirmed fixes rank-8+ ColBERT failures: Jonathan Lynn (was persistent across 7+ iterations — article at rank 8+ with k=7, now captured with k=12), Mars Inc., Greatest Game Ever Played all confirmed rank-boundary fixes. This is structurally generalizable — many persistent ColBERT failures are caused by gold articles ranking at positions 8-12. (2) Step 4 director/actress pattern confirmed working for Felicity Jones (ex33): after retrieving Rogue One, the LM correctly output Felicity Jones in hop5. (3) All 5 regressions were confirmed purely stochastic — context2 is IDENTICAL between parent and child (both use hop1_new[:6]; the first-6 hop1 results don't change with k), so k=12 did not structurally cause any regressions. (4) Asymmetric k (k=7 for hop1, k=12 for hops 2-5) is the logical next step: hop1 uses a broad raw-claim query where gold docs rank 1-6 (extra ranks 7-12 add noise to seen_titles); hops 2-5 use targeted entity queries where rank-boundary failures occur. (5) None-query guard: low-impact (fired once in ex75), but structurally correct and zero-risk. (6) The 5 stochastic regressions confirm that single-subsample evaluations cannot reliably measure changes below ±5% effect size — the structural fixes from k=12 + Step 4 pattern are real but were masked by LM variance on this seed.

Iteration 16 (k=7 for hops 1-4, k=12 for hop5 only + None-query guard + Step 4 director/actress bullet, applied to candidate 9 via two rounds) is an incremental structural code + prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) seen_titles pollution confirmed: Round 1's asymmetric k (hop1=7, hops 2-5=12) caused −10.4% subsample regression because k=12 for intermediate hops retrieves 12 docs but only 4 per hop enter the round-robin output (4-hop priority interleaving: hop1→6, hop2→4, hop3→4, hop4→4 slots). The remaining 8 docs per hop land in seen_titles without appearing in output, silently blocking later hops from retrieving those articles. Round 2 correctly isolates k=12 to hop5 only (last hop — no subsequent hops to pollute), which is safe and provides extra coverage when the target article is at rank 4-12 because earlier hops have exhausted duplicates. (2) Subsample fixes appear largely stochastic: ex07 (Sojourner Truth) was fixed because the LM stochastically output the correct query in hop4 rather than through k=12 hop5; ex42 (Winnie the Pooh 2011 film) fixed via hop5 identifying the film title. The 1 regression (ex49 Beyond Rangoon) was also stochastic — same hop1-2 path as parent but hop3 diverged. (3) Valset regression (0.8450→0.8152, −3.5%) despite positive subsample: the Step 4 director/actress bullet is the most likely culprit. When multiple film articles are retrieved in earlier hops, the LM may now apply the pattern to the wrong film (e.g., extracting Johnny Depp from True Romance instead of querying Beyond Rangoon), wasting hop5 on incorrect targets. The pattern fires too broadly — it should only apply when the claim explicitly implies a specific person behind a specific retrieved film. (4) None-query guard is safe: Low-frequency and no negative effects observed in the subsample trace. Keep in architecture. (5) k=12 for hop5: Mechanistically sound. If the valset regression is indeed caused by the Step 4 director/actress pattern rather than k=12, then k=12 for hop5 should be retained in the next iteration. Consider an A/B test: iter16 code minus Step 4 director/actress bullet, to isolate whether that bullet alone caused the regression.

Iteration 17 (asymmetric k: hop1=6, hops2-4=5, hop5=12 + NONE_PATTERNS None-query guard + seen-titles guard for hop5, all applied to candidate 9) is an incremental structural code improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) k=6 for hop1 confirmed structural fix for seen_titles pollution: Jonathan Lynn (ex48) was at rank-7 of hop1 with k=7; it entered seen_titles but was evicted at position 25 in round-robin, permanently blocking later hops from finding it via get_new_unique(). With k=6, rank-7 was never retrieved, never added to seen_titles, and a later hop correctly retrieved and included it. This is mechanistic and reproducible. (2) k=5 for hops 2-4 is mathematically correct: with 4-hop equal round-robin (6+5+5+5=21 slots), all retrieved docs enter the output with zero eviction. Previously k=7 for hops 2-4 added ranks 6-7 to seen_titles without contributing to the output. (3) Subsample was neutral due to exactly-cancelling stochastic regression: ex39 (Greatest Game Ever Played) regressed purely stochastically — same retrieved context, different hop3 LM choice. The structural fix (Jonathan Lynn) and the stochastic regression had identical score magnitudes (+0.994 − 0.994 = 0). (4) Seen-titles guard for hop5 is conservative and low-risk: fires in edge cases where hop5 tries to re-query an article already in seen_titles from an earlier hop's retrieval. Did not visibly affect the subsample but adds structural soundness. (5) Hop5 fire rate: ~88% (44/50 examples used 5 searches). Still very high — virtually every example pays the 0.002 penalty. (6) No valset evaluation: since subsample was neutral (no improvement), valset was not computed. The structural changes (k=6/k=5 for hop1/hops2-4) are expected to benefit the valset (Jonathan Lynn was a persistent 10+ iteration failure) but this is unconfirmed.

Iteration 18 (asymmetric k from iter17 + NONE guard + seen-titles guard for hop5 + Step 2 sub-article false-coverage fix + Step 1 claim-verbatim self-check + Step 3 disambiguation suffix guidance, all applied to candidate 9) is an incremental structural code + prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Sub-article false-coverage fix is a generalizable structural prompt improvement: Blocking "X bibliography | ..." and "X filmography | ..." from counting as coverage of X fixed ex14 (Robert E. Howard) and likely generalizes to many prominent people with Wikipedia sub-pages. The wording must explicitly allow disambiguation suffixes ("Dave Evans (singer)" DOES cover Dave Evans) and name-form variants. Round 1's strict wording ("EXACT plain title without any qualifier") caused two regressions; Round 2's narrower wording fixed them. (2) k=12 for hop5 continues to fix rank-boundary failures: ex75 (Robert Jordan) captured at rank 8-12, confirming the mechanism from iter15. (3) Disambiguation suffix guidance has low practical uptake: Only ~6/250 queries in the subsample used type qualifiers despite Step 3 instruction. ex117 (Secret Agent TV series) was NOT fixed at hop1 (the LM generated "The Secret Agent" without suffix). To reliably fix this class of failure, a code-level intervention is needed. (4) Valset confirmed generalizes: +1.0% absolute (0.8450→0.8550) is a genuine improvement across diverse examples, consistent with structural changes that generalize. (5) The iter18 Round 1→Round 2 progression is instructive: multi-change bundles can introduce regressions even when each change is structurally sound; it is worth validating each change independently when regressions appear.

Iteration 19 (partner bullet clarification, multi-membership bullet for multi-org membership claims, anti-fame-bias instruction in Step 4 PRIORITY section, all applied to candidate 11 / iter18) is an incremental prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Multi-membership bullet is confirmed effective: Glenn Hughes (ex58) fixed — the LM correctly identified the implicit person described by membership across three bands (Deep Purple, Black Sabbath, Trapeze) and directly queried "Glenn Hughes" in hop3, freeing remaining hops for band articles. This generalizes to any claim describing a person via multiple affiliation mentions (music, sports, multi-role public figures). (2) Anti-fame bias helps for disambiguation compliance but NOT for strong hallucinations: The instruction "do not default to the most famous name; identify the entity filling the SPECIFIC ROLE described by the claim" did not fix Gene Kelly/Bob Fosse (ex97) — the LM has a strong prior associating Fosse with Liza Minnelli ("Liza with a 'Z'") that overrides the anti-fame-bias instruction even though Fosse was born 1927, not 1912 as the claim states. Anti-fame bias works for soft disambiguation (choosing between co-equally famous people) but not for confident hallucinations. (3) Disambiguation suffix coverage check bug DISCOVERED: When the LM uses "F.E.A.R. (video game)" to retrieve the article (successfully), Step 2's coverage check fails to recognize "F.E.A.R. (video game)" as covering the bare entity "F.E.A.R." in the claim — causing 2 wasted follow-up hops where the LM re-queries related entities thinking F.E.A.R. is still uncovered. The current sub-article false-coverage rule (iter18) was designed to block false sub-page coverage, but inadvertently also blocks VALID disambiguation-suffix coverage. Fix: Step 2 should explicitly state that "X (type qualifier)" DOES cover X (the qualifier form counts as the entity's article). This is an actionable, low-risk improvement. (4) Valset did not improve (0.8550 = tie with iter18 best): the +4.7% subsample gain did not translate to valset improvement. The Glenn Hughes multi-membership pattern may be relatively rare in the valset; anti-fame bias may not help consistently. Partial overfitting to this subsample seed is likely. (5) Air Lituanica failure mode (ex20): With only 4 searches used (hop5 didn't fire because the LM thought all entities were covered), all LM hops queried "Vilnius Airport" / "Vilnius International Airport" variants but NEVER queried "Air Lituanica." The LM identifies the airport codes (VNO/EYVI) as pointing to the airport article, but does not infer that a DEFUNCT AIRLINE based at that airport needs its own separate article. The claim structure ("airport that ceased operations in 2015") is the cue that a defunct entity (Air Lituanica) needs to be retrieved, not just the airport. (6) Hop5 fire rate ~96% (48/50 examples used 5 searches in iter19 trace). Higher than prior estimates. Tightening the firing condition remains the highest-priority structural refinement not yet tested.

Iteration 20 (disambiguation suffix type-qualifier fix in Step 2 + defunct/former/ceased entity pattern in Step 4, applied to candidate 12 / iter19) is an incremental prompt-only improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Disambiguation suffix type-qualifier fix is validated: Step 2 now explicitly allows "X (film/TV series/video game/artwork/etc.)" to count as covering X; only sub-page suffixes (bibliography, filmography, discography, early life, health) are blocked. Fixed ex31 (The Dinner Party artwork) — the LM had been looping on re-queries because the "(artwork)" suffix was treated like a sub-page suffix, preventing coverage recognition. This generalizes to any example where the LM correctly uses a type-qualifier suffix to retrieve an article (F.E.A.R. (video game), The Secret Agent (TV series), etc.). (2) Defunct entity Step 4 pattern did NOT fix Air Lituanica: The Vilnius Airport article does not appear to mention "Air Lituanica" prominently enough in its text for the LM to extract it via body scan. The pattern may benefit other valset examples where a defunct entity IS prominently named in a retrieved article — but this is unconfirmed. (3) Valset improvement is negligible (+0.0001): The tiny valset movement (0.8550→0.8551) confirms the type-qualifier class of failures is small on the valset. The subsample gain (+2.3%) primarily reflects the specific (artwork) fix. (4) Hop5 fire rate: 92% (46/50 examples used 5 searches in iter20 trace) — slightly down from iter19's 96%, largely due to different seed. Still very high.

Iteration 21 (Step 3 CRITICAL type-qualifier enforcement + three new Step 4 body-scan bullets: parent company/owner, town/location-of, event venue, applied to candidate 13 / iter20) is an incremental prompt-only improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) CRITICAL directive framing is effective for media-type disambiguation: The CRITICAL flag with an explicit bad→good example ("F.E.A.R." → "F.E.A.R. (video game)") drove genuine LM compliance where prior softer guidance had ~6/250 uptake. This is the opposite of the pattern seen with entity selection (where CRITICAL/STRONGLY PREFER language causes over-triggering) — the key difference is that type-qualifier insertion is a LOW-STAKES operation (the LM is not choosing WHICH entity to query, only HOW to format a known entity). The lesson: strong directive framing is safe when the claim itself provides the answer (the media type is stated in the claim) but dangerous when it requires entity selection/reasoning. (2) Location-of bullet generalizes broadly: Confirmed fix for Afonwen — railway/station articles consistently include location phrases ("located in X"). This class is likely common in the HoVer dataset. (3) Parent company bullet introduces ordering risk in multi-article ownership chains: When both the product (e.g., Skittles) AND the manufacturer (Mars Inc.) are required supporting facts, the bullet causes the LM to jump to Mars and skip Skittles. Needs a condition: "only output the parent company if the product's OWN article is already retrieved." (4) Valset +3.5% relative (0.8551→0.8748) confirms these patterns generalize broadly — the type-qualifier class of failures is more common on the valset than the subsample suggested (F.E.A.R. was just one example; the valset has many more media-type disambiguation cases). (5) 8 failures in trace vs ~6 in iter20 trace: The additional failures (ex32 partial fix, ex82 stochastic regression) are not structural regressions. Persistent failures (Guy Davis, Airlines of Africa, Ancient Egyptian religion, Flex club, Gene Kelly, On the Buses) remain unchanged.

Iteration 24 (four prompt-only changes to candidate 14 / iter21: full legal name variant coverage fix in Step 2 + CREATOR bullet + COMPARISON/RANKING bullet + parent company ORDERING RULE — no code logic changes) is an incremental prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Full-name variant coverage fix is a confirmed structural gain: Boris Becker covers "Boris Franz Becker" — directly freed a hop that was previously wasted re-querying the formal name form. Generalizes to any entity with a formal longer name in the claim but shorter Wikipedia article title. (2) CREATOR and COMPARISON/RANKING bullets persist as over-triggering risks: 5 examples shifted from 4-hop to 5-hop (soft regressions) after adding these bullets. This mirrors the iter23 finding — the hop5 penalty cost of new body-scan bullets is ~5/50 examples per bullet pair. (3) DEFUNCT ENTITY rule has a cross-entity contamination risk: The Tallinn Airport article (retrieved because AirBaltic flies there) mentioned "Estonian Air" as defunct in 2015, causing the rule to fire for the wrong airline. The rule is mechanically sound but any retrieved article can name multiple defunct entities. (4) Hop5 fire rate: 95.5% (42/44 passing examples) — highest measured rate, confirming near-universal hop5 triggering. The hop5 firing condition tightening remains the single highest-priority untested structural change. (5) Iter24 is effectively candidate 15 + full-name variant fix: Since iter23 (candidate 15) had valset 0.8550, iter24's expected valset would be ~0.855-0.860 (slightly above iter23 due to the full-name fix), still below best (0.8748).

Iteration 25 (three prompt-only changes to candidate 14 / iter21: formal-name variant fix in Step 2 + CRITICAL PERSON COVERAGE RULE in Step 2 + COMPARISON/RANKING Step 4 bullet — no code logic changes) is an incremental prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) CRITICAL PERSON COVERAGE RULE confirmed as over-triggering in its universal form: The rule "org/film/work article naming a person does NOT cover that person's biographical article" is mechanically correct for the Matthew Bailes failure class (the person IS a gold supporting fact but org coverage doesn't count). However, the universal form (no conditioning on claim relevance) caused the LM to query biographical articles for every person mentioned in retrieved articles, including non-gold people like directors, minor cast members, and co-authors. This wasted hops broadly across the valset. Valset dropped from 0.8748 to 0.8252 (−5.7%) despite a +7.2% subsample gain. (2) CRITICAL framing is safe for type disambiguation but dangerous for coverage scope: The CRITICAL label drives near-100% LM compliance. For type disambiguation (iter21: "CRITICAL: use '(video game)' qualifier"), this is safe because the rule is binary and claim-verified. For coverage scope rules (iter25: "CRITICAL: org article does NOT cover mentioned person"), the LM applies it universally regardless of whether the person is actually needed. The Matthew Bailes fix requires a CONDITIONED rule, not a universal one: "query a person's biographical article if they are directly referenced or implied by the claim's relationship — NOT merely mentioned as a role-holder in a retrieved article." (3) +7.2% subsample gain was misleading — the seed contained ex13 (UEA Creative Writing) as the exact target of the new rule. This is the strongest subsample overfitting example in this run. (4) Formal-name variant fix is safe and structural: The fix (short common-name article covers longer formal name in claim) did not cause regression. Should be retained in future iterations. (5) COMPARISON/RANKING bullet suffers from sequencing: For ex18 (Heathrow), the LM issues broad queries in Steps 1-3 (London, United Kingdom) before reaching Step 4 body-scan. The bullet only fires when the bridge article (Gatwick) is retrieved AND no other claim entities are queued. Step 1 extension for superlative entities would help more: "if the claim describes an entity by a unique superlative role without naming it, enumerate it for direct querying in Step 3." (6) Hop5 fire rate dropped to 76% — lowest measured rate. The CRITICAL PERSON COVERAGE RULE caused the LM to query people earlier (hops 2-4), completing more examples in 4 hops. This shows the rule DID work behaviorally (Matthew Bailes class would pass) but the broad application cost the valset.

Iteration 23 (five Step 4 prompt-only additions to candidate 14 / iter21: CREATOR bullet, HOST/CAST MEMBER bullet, CO-FOUNDER bullet, COMPARISON/RANKING bullet, parent company ORDERING RULE — no code logic changes) is an incremental prompt improvement. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) CREATOR and COMPARISON/RANKING bullets are confirmed effective in the subsample: Adam Guettel (ex12) fixed — the CREATOR bullet caused the LM to extract the composer from Elena Shaddow's retrieved article and query him directly (previously the LM fixated on the Tony Award category name). Heathrow Airport (ex18) fixed — the COMPARISON/RANKING bullet caused the LM to extract the ranked entity from the Gatwick article body ("busiest airport" → Heathrow). Both patterns target common Wikipedia text structures and are likely to generalize. (2) CO-FOUNDER bullet did NOT fire for ex114 (Charlotte Baldwin Allen, persistent 18+ iterations): Augustus Chapman Allen article was retrieved, but Charlotte Baldwin Allen was never queried. Either: the Augustus Chapman Allen article text doesn't prominently name Charlotte in the snippet returned by ColBERT, or the LM doesn't recognize the co-founding context at scan time. The failure mode is uncertain — would need to verify whether the Allen article body text actually mentions Charlotte Baldwin Allen. (3) HOST/CAST MEMBER bullet did NOT fix ex126 (On the Buses/Pat Ashton, persistent 10+ iterations): Thick as Thieves (TV series) WAS retrieved (confirmed in trace retrieved titles), but Pat Ashton was not extracted and On the Buses film was never queried. The bullet is in place but the LM didn't execute the host-extraction scan when Thick as Thieves appeared in context. Possible causes: the Thick as Thieves article text doesn't name Pat Ashton prominently, or the LM skipped the HOST/CAST MEMBER scan because other claim entities (Harry Booth's filmography) dominated attention. (4) Valset dropped to 0.8550 (−2.3% from best 0.8748): The subsample gained +2.5% but the valset regressed. This could be: (a) stochastic noise — 2.3% is at the edge of the ±3% noise band and the seed changed; (b) real regression from new bullets causing over-triggering on valset examples that previously passed (the CREATOR bullet might cause the LM to look for a composer from any retrieved article even when the article doesn't name one, wasting a hop); or (c) the subsample gains (Adam Guettel, Heathrow) may be rare in the valset while the over-triggering cost is more common. The valset verdict is negative: candidate 14 (0.8748) remains best. (5) ex129 (Stranger in Paradise, persistent) — Kismet retrieved but song not extracted: The Kismet (musical) article IS now in the retrieved titles, and presumably names "Stranger in Paradise" as an adapted song. Yet the LM does not extract this song as a query target — it continues pursuing Borodin → Kipchaks → Prince Igor. Neither the existing named-song Step 4 bullet (iter13) nor the new CREATOR bullet caused the extraction. This suggests a deep body-scan compliance failure for musical-adaptation chains where the connection (Kismet → Stranger in Paradise) requires multi-step musical knowledge beyond simple name-presence in the text. (6) Hop5 fire rate: ~84% (42/50 examples used 5 searches in iter23 trace; ex85, ex92 used 4 searches — hop5 didn't fire because LM declared all entities covered). Still very high; the 0.002 penalty is paid nearly universally.

Iteration 26 (removing CRITICAL PERSON COVERAGE RULE from candidate 16) is a targeted reversal of a prior harmful change. It does not introduce a new retrieval architecture. Excluded from Macro Approaches. Key lessons: (1) Removing a CRITICAL rule is NOT the same as restoring the baseline: Iter26 = candidate 14 + formal-name fix + COMPARISON/RANKING bullet. The expected valset should be ~0.875 (candidate 14 baseline minus the COMPARISON/RANKING bullet cost). Instead, valset came in at 0.8351 — suggesting the COMPARISON/RANKING bullet costs ~0.04 valset score on its own. This is substantially larger than the −0.02 estimated from iter23 (which bundled 5 bullets together). (2) Partial recovery confirmed: 0.8252 → 0.8351 (+0.0099) after removing the CRITICAL rule. The CRITICAL PERSON COVERAGE RULE alone caused ~0.04 of the iter25 regression; the retained COMPARISON/RANKING bullet is responsible for the remaining gap from candidate 14. (3) Conditioned person-coverage rule remains the primary untested fix for Matthew Bailes class: The reflection correctly identifies that a conditioned rule ("query a person's biographical article ONLY IF they are directly referenced or implied by the claim's core relationship") would fix Matthew Bailes without over-triggering. This is the highest-priority prompt-level improvement not yet tested. (4) Hop5 fire rate recovered to 86% (from 76% in iter25), confirming that the CRITICAL PERSON COVERAGE RULE was suppressing hop5 by driving early person queries in hops 2-4. With the rule removed, the LM reserves more queries for hop5 (k=12), restoring rank-boundary capture capability.

Observations

Failure Pattern Distribution: Iteration 17 Child Trace (9 failures from 50)

Subsample score: 0.8153 (neutral, Δ=0.0 vs parent 0.8153). Built on candidate 9 (iter14). Rejected. No valset score. Note: 1 structural fix (ex48 Jonathan Lynn) exactly cancelled by 1 stochastic regression (ex39 Greatest Game Ever Played). All other failures are persistent.

  1. Wrong entity chain / inference failure (4/9):

    • ex7 (Violent Restitution / Allan Goldstein / Charles Bronson): Used only 4 searches (hop5 didn't fire — LM proposed a repeat hop5 query). Hops 1-4 followed Charles Bronson → Leslie Nielsen → Naked Gun chain. Allan Goldstein never queried. Persistent across 15+ iterations. Notable: with k=5 for hops 2-4 (instead of k=7), exactly 6+5+5+5=21 slots filled from 4 hops — no room for hop5 docs even if hop5 had fired.
    • ex71 (Airlines of Africa / Comair / British Airways): Comair (South Africa) correctly retrieved in hop1. Hops 2-5 followed KLM → Air France-KLM chain. Airlines of Africa (Comair subsidiary) never queried. Persistent across 10+ iterations.
    • ex95 (Felicity Jones / Rachelle Beinart / Rogue One): Rogue One and Rachelle Beinart correctly retrieved. Step 4 body scan in hop5 did NOT extract Felicity Jones — the LM likely extracted a different entity from the Rogue One article. Persistent across 10+ iterations. Note: the Step 4 director/actress bullet was NOT included in iter17 (removed vs iter16), confirming that even without it, the failure persists.
    • ex137 (Aliens of the Deep / Bears film / Keith Scholey): Bears film retrieved. Hops looped on "African Cats" variants and "Fothergill" disambiguation articles. Aliens of the Deep never queried. Persistent.

  2. ColBERT retrieval gap / wrong disambiguation (2/9):

    • ex97 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM hallucinated Bob Fosse (born 1927) as the 1912 choreographer. All 5 hops pursued Fosse's collaborators. Gene Kelly and Best Foot Forward never queried. Persistent hallucination — the claim says "born in 1912" which should identify Gene Kelly, but the LM anchors on Fosse.
    • ex117 (Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved "The Secret Agent (film)" (1996 Bob Hoskins). All 5 hops followed wrong path: Bob Hoskins → Shane Meadows → etc. None of the 3 gold articles found. Persistent across 9+ iterations. Requires disambiguation suffix at hop1 or claim-level disambiguation.

  3. Bridge article retrieved but target not extracted (2/9):

    • ex126 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Harry Booth article likely retrieved. LM fixated on "Thick as Thieves" from the claim. On the Buses and Pat Ashton never queried. Persistent across 9+ iterations.
    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Wrong person identified (Dharmendra instead of Karan Kapoor as film star). Hop5 also fired. Persistent — claim disambiguation failure.

  4. Stochastic regression (1/9):

    • ex39 (Greatest Game Ever Played / Josh Flitter): Parent passed; child failed. Same context3 (Josh Flitter article) but LM chose "Nancy Drew" (first article in Flitter's list) instead of "The Greatest Game Ever Played" (the sports-context film). Pure LM variance — not caused by any code change.

Fix vs parent (candidate 9):

Key new observations from iteration 17:

Failure Pattern Distribution: Iteration 18 Child Trace (5 failures from 50)

Subsample score: 0.8948 (+9.7% vs parent 0.8154). Built on candidate 9 (iter14). Accepted as candidate 11. Valset: 0.8550 (NEW BEST, +1.0% absolute vs prior best 0.8450). Note: strong multi-example fix across different failure categories: sub-article fix (ex14), k=12 rank-boundary (ex75), asymmetric k from iter17 (ex5 recovered), disambiguation prompt guidance (ex123 Christian Poulsen partially). Round 2 narrowed sub-article wording to fix Round 1 regressions.

  1. Wrong entity chain / inference failure (3/5):

    • ex3 (Matthew Bailes / Houman Younessi / Swinburne): CAS article retrieved in hop1. Hops went geographic distractors (Melbourne, Hawthorn) — Matthew Bailes (CAS director) never queried. Persistent across 15+ iterations.
    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Wrong actor identified (Dharmendra instead of Karan Kapoor). Geoffrey Kendal and Karan Kapoor both missed. Persistent — first-hop disambiguation failure.
    • ex56 (Guy Davis comics / Crimson Peak / Burn Gorman): Burn Gorman found. Hops followed Guillermo del Toro → Pacific Rim chain; Guy Davis (comics) never queried. Persistent across 10+ iterations.

  2. Loop / entity fixation (1/5):

    • ex92 (Green Chair / Shim Ji-ho / Suh Jung): LM returned "2005 Sundance Film Festival" repeatedly despite retry mechanism. Shim Ji-ho never queried. Persistent across 9+ iterations. The retry fires but the anchor on the festival is overwhelming.

  3. Wrong disambiguation at hop1 (1/5):

    • ex117 (Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved "The Secret Agent (film)" (1996 Bob Hoskins). Despite disambiguation suffix guidance, the LM did NOT use "(TV series)" suffix in queries. Improvement from prior iterations: This Is England IS now found (via Shane Meadows chain), but Stephen Graham and the TV series article remain missing. Partial improvement. The disambiguation guidance fires in only ~6/250 total search queries across the full subsample — not reliable enough to fix this case.

Confirmed fixes vs parent (candidate 9):

Key new observations from iteration 18:

Failure Pattern Distribution: Iteration 19 Child Trace (6 failures from 50)

Subsample score: 0.8748 (+4.7% vs parent 0.8351). Built on candidate 11 (iter18). Accepted as candidate 12. Valset: 0.8550 (tied with iter18 best — no improvement). Changes: partner bullet clarification, multi-membership bullet, anti-fame-bias instruction in Step 4.

  1. Wrong entity chain / LM hallucination (2/6):

    • ex97 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): Queried Bob Fosse (×2, via "Liza with a 'Z'" reasoning), then Cabaret (musical), then "Liza with a 'Z': A Concert for Television." Never queried Gene Kelly or Best Foot Forward. The anti-fame-bias instruction did NOT fix this — the LM confidently hallucinates Fosse as the 1912 choreographer (Fosse was born 1927) because of his strong association with Liza Minnelli. Persistent hallucination across 15+ iterations — appears unfixable by prompt alone.
    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Queried Loha (1987 film) → Dharmendra → Shatrughan Sinha → Amrish Puri. The LM identifies the most famous Bollywood actors from that era, not the actual star Karan Kapoor. Anti-fame bias did not fix this because the LM doesn't know the correct answer — it requires external knowledge of the film's star. Persistent across 15+ iterations.

  2. ColBERT/chain failure — airline article never inferred (1/6):

    • ex20 (Air Lituanica / Vilnius Airport / AirBaltic): Only 4 searches fired (hop5 didn't fire — LM thought all entities were covered after retrieving Vilnius Airport). All LM calls queried "Vilnius International Airport" or "Vilnius Airport" variants. Never queried "Air Lituanica." The LM correctly identifies the airport from the IATA/ICAO codes but does NOT infer that a defunct airline (Air Lituanica) based at that airport needs its own separate article. The claim's "airport that ceased operations in 2015" cues the defunct airline, but the LM treats it as pointing to the airport itself. Persistent across 10+ iterations. The airline article itself is the key missing piece, not the airport. Root cause: the LM correctly identifies AirBaltic and Vilnius Airport as covered, declares all entities covered in Step 2, and hop5 does not fire.

  3. False-premise partner extraction failure (1/6):

    • ex38 (Renáta Tomanová / 1980 French Open Mixed Doubles / Larisa Neiland): Queried "Boris Diaw" (a famous basketball player!) → 1980 French Open Mixed Doubles → "Billy Martin (tennis)" → "Boris Diaw (tennis)." The claim falsely names Boris Diaw (NBA player) as the tennis doubles partner, which confuses the LM. Renáta Tomanová is the actual doubles partner but was never queried. The updated partner bullet did not fix this because: (a) the false-premise entity ("Boris Diaw") doesn't exist in the 1980 French Open context, and (b) after retrieving the Mixed Doubles article, the LM cannot identify Tomanová as Diaw's partner because Diaw never played tennis. Persistent false-premise failure — requires external knowledge to override the claim.

  4. Bridge article retrieved but specific sub-article not derived (1/6):

    • ex53 (Douglas MacArthur's escape / Boeing B-17 / Texas Raiders): Queried "Douglas MacArthur" (generic biography) → "Conroe North Houston Regional Airport" (×2 — retry failed to prevent this despite exact-string match, possibly different normalization) → Philippines → "Flight of Douglas MacArthur." Boeing B-17 Flying Fortress was presumably retrieved, Douglas MacArthur's escape article and Texas Raiders never queried. The LM fixates on the airport and generic biography rather than the SPECIFIC escape article. The retry fired but "Conroe North Houston Regional Airport" was re-issued anyway. Persistent across 10+ iterations.

  5. Bridge article retrieved but target not extracted from filmography (1/6):

    • ex126 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Queried "Harry Booth" → "Thick as Thieves (cooking show)" (×2, retry fired but produced same query) → "Thick as Thieves (TV series)" → "Harry Booth filmography." The Thick as Thieves TV series was correctly retrieved (hop4), but On the Buses (film) and Pat Ashton never queried. The Harry Booth filmography query was a good instinct (would surface "On the Buses") but the query wasn't issued early enough and the hop budget was exhausted. The LM fixates on the cooking show name from the claim and cannot pivot to the correct film. Persistent across 10+ iterations.

Confirmed fixes vs parent (candidate 11, different seed):

Key new observations from iteration 19:

Failure Pattern Distribution: Iteration 20 Child Trace (6 failures from 50)

Subsample score: 0.8748 (+2.3% vs parent 0.8550). Built on candidate 12 (iter19). Accepted as candidate 13. Valset: 0.8551 (new best, +0.0001 vs 0.8550 — essentially noise-level). Changes: disambiguation suffix type-qualifier fix in Step 2 (any "X (film/TV series/video game/artwork/etc.)" now counts as covering X) + defunct/former/ceased entity pattern in Step 4.

  1. Wrong entity chain / geographic distractor (1/6):

    • ex1 (Matthew Bailes / Houman Younessi / Swinburne): CAS article retrieved in hop1. Hops 2-5 went to geographic distractors (Melbourne, Hawthorn, Swinburne). Matthew Bailes (CAS director) never queried. Persistent across 16+ iterations.

  2. Defunct entity never inferred (1/6):

    • ex6 (Air Lituanica / AirBaltic / Vilnius Airport): Only 4 searches fired (hop5 didn't fire — LM declared all entities covered after retrieving Vilnius Airport and AirBaltic). "Air Lituanica" never queried. The new defunct entity Step 4 pattern did NOT fix this — Vilnius Airport article text doesn't prominently name Air Lituanica. Persistent across 10+ iterations.

  3. False-premise partner extraction (1/6):

    • ex13 (Renáta Tomanová / 1980 French Open Mixed Doubles / Larisa Neiland): Retrieved the Mixed Doubles article but failed to extract Renáta Tomanová as the correct partner. The claim's false-premise entity ("Boris Diaw") — an NBA player with no tennis history — confuses the LM's partner extraction. Persistent false-premise failure across 8+ iterations.

  4. Host extraction failure — correct show found but host not derived (1/6):

    • ex21 (Bill Cullen / Paul Melba / Punchlines): Punchlines correctly found (partial improvement vs prior iterations). Hop5 failed to extract Bill Cullen as the host from the Punchlines article. "Punchlines" is fixed; "Bill Cullen" remains missing. Persistent pattern — correct bridge article retrieved but leaf-node entity not extracted.

  5. LM hallucination — wrong entity identified from claim (1/6, 2 missing docs):

    • ex30 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM identified Bob Fosse (born 1927) as the 1912 choreographer instead of Gene Kelly (born 1912) — strong prior association with Liza Minnelli overriding the birth year clue. Both Gene Kelly and Best Foot Forward missed. Persistent hallucination across 16+ iterations. Unfixable by prompt alone.

  6. Bridge article retrieved but specific targets not derived (1/6, 2 missing docs):

    • ex42 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Harry Booth article retrieved. On the Buses (film) and Pat Ashton never queried — the LM fixated on "Thick as Thieves" from the claim and couldn't derive "On the Buses" from the Harry Booth filmography. Persistent across 10+ iterations.

Confirmed fix vs parent (candidate 12, different seed):

Key new observations from iteration 20:

Failure Pattern Distribution: Iteration 16 Child Trace (8 failures from 50)

Subsample score: 0.8351 (+2.4% vs parent 0.8152). Built on candidate 9 (iter14). Accepted as candidate 10. Valset: 0.8152 (−3.5% vs best 0.8450 — significant regression). Note: the subsample parent had 9 failures; the child has 8 — net +1 fix. Both ex07 (Sojourner Truth) and ex42 (Winnie the Pooh 2011 film) were fixed; ex49 (Beyond Rangoon) newly regressed. All three changes are largely stochastic. The valset regression is real and unexplained by the subsample signal.

  1. Wrong entity chain / inference failure (3/8):

    • ex00 (Matthew Bailes / Swinburne / CAS): CAS article retrieved in hop1. Hops 2-4 went to Swinburne → Melbourne → Hawthorn, Victoria (geographic distractors). Matthew Bailes (CAS director) never queried. Persistent across 12+ iterations.
    • ex27 (Green Chair / Shim Ji-ho / Suh Jung): Hops cycled to "2005 Sundance Film Festival" four times (hops 2-4 + hop5 each returning the festival or Park City, Utah). Retry mechanism fired but LM kept returning the festival. Shim Ji-ho (the actor) never queried. Park Chul-soo was found in hop3 but the chain never advanced to the actor. Persistent across iter4, iter9, iter12, iter13, iter14, iter15, iter16. Root cause: the Sundance Film Festival is the most prominent entity in the claim context after "Green Chair" is retrieved, and the LM anchors on it regardless of the repeat warning.
    • ex49 (Beyond Rangoon / Patricia Arquette / Victor Slezak): Queries went Victor Slezak → Johnny Depp → The Bridges of Madison County → The Bridges of Madison County (film). Both parent and child went to Johnny Depp in hop2 (plausible: True Romance had both Arquette and Depp). Parent then correctly queried Beyond Rangoon in hop3; child went to Bridges of Madison County instead. Stochastic divergence — not attributable to the iter16 changes.

  2. ColBERT retrieval gap — correct query issued but article not returned (2/8):

    • ex21 (Lion of Ireland / Morgan Llywelyn / Robert Jordan): Correctly queried "Robert Jordan" and "Robert Jordan (author)" — both in hop2 and hop5 via retry — but ColBERT did not return the Robert Jordan article in any of the top-7 results. Morgan Llywelyn and Lion of Ireland correctly retrieved. This is a fundamental ColBERT ranking gap for "Robert Jordan" (the fantasy author). Persistent.
    • ex33 (Asymptotic safety in quantum gravity / Sheldon Lee Glashow / Steven Weinberg): Steven Weinberg correctly retrieved in hop1. Queried "Asymptotic safety in quantum gravity" five times (hops 2, 3, 5 and two retries) plus "Asymptotic safety" and "Asymptotically safe gravity" — ColBERT never returned the article. Also Sheldon Lee Glashow never queried (wrong chains to David Gross, Hugh Politzer, Harvard University). Persistent ColBERT deep gap — the article title doesn't score well in the index.

  3. Wrong disambiguation at hop1 / wrong article chain (1/8):

    • ex41 (The Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved "The Secret Agent (film)" (1996 Bob Hoskins) instead of "The Secret Agent (TV series)." All 5 hops followed wrong path: Shane Meadows → Bob Hoskins → Christopher Hampton → Patricia Arquette. This Is England not found, Stephen Graham not found. Persistent across 8+ iterations. Requires disambiguation suffix at hop1 or claim-level disambiguation logic.

  4. Bridge article retrieved but target not extracted (2/8):

    • ex45 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Queries: Harry Booth → "Thick as Thieves (cooking show)" (queried 3 times, retry fired twice) → Dennis Price → "Harry Booth filmography". On the Buses and Pat Ashton never queried. The Harry Booth article was correctly retrieved but the LM could not identify "On the Buses" as Harry Booth's 1971 film — it fixated on "Thick as Thieves" from the claim and couldn't resolve the TV show name. Persistent.
    • ex46 (Stranger in Paradise song / Polovtsian Dances / Cór Cois Abhann): Queries: Alexander Borodin → Kipchaks → Cumans → Prince Igor. Kismet (musical) IS retrieved at position [2] in hop1, but the LM pursues the Borodin → opera → Kipchaks chain instead of extracting "Stranger in Paradise" from the Kismet article. Persistent across 8+ iterations. The Step 4 named-work pattern (added in iter13) has not fixed this — the connection between Kismet and "Stranger in Paradise" requires music-domain knowledge not surfaced by the current Step 4 patterns.

Fixes introduced vs parent (candidate 9, same seed):

New regression vs parent:

Key new observations from iteration 16:

Failure Pattern Distribution: Iteration 15 Child Trace (15 failures from 50)

Subsample score: 0.6960 (net neutral vs parent 0.6960). Built on candidate 9 (iter14). Rejected. Valset N/A. Note: 5 fixes and 5 stochastic regressions perfectly cancelled. The same seed was used for parent and child, so the +0.0 delta is a fair comparison — however stochasticity at this level is well within the ±3% noise band.

  1. Wrong entity chain / inference failure (6/15):

    • ex00 (Houman Younessi / Matthew Bailes / Swinburne): CAS article retrieved in hop1. Hops 2-5 went to Hawthorn → Swinburne → Melbourne (geographic distractors). Matthew Bailes (CAS director) never queried. Persistent across 10+ iterations.
    • ex03 (Allan Goldstein / Charles Bronson / Violent Restitution): Charles Bronson correctly queried in hop1, then hops went Leslie Nielsen → The Naked Gun → Charles Dennis Buchinsky. Allan Goldstein (director) never queried. Persistent.
    • ex07 (Richard Ford / Robert E. Howard): Claim says "more scope of profession than Robert E. Howard." LM retrieved The Lay of the Land but never queried Robert E. Howard. Persistent.
    • ex25 (Airlines of Africa / Comair): Hops went KLM → British Airways chain. Airlines of Africa (Comair subsidiary) never queried. Persistent.
    • ex40 (The Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved "The Secret Agent (film)" (1996 Bob Hoskins) instead of TV series. All hops followed wrong disambiguation path. Persistent across 6+ iterations.
    • ex47 (1974-75 NY Islanders season / Madison Square Garden / NY Rangers): LM correctly retrieved Madison Square Garden and NY Rangers, but never queried the specific historical season article. Persistent.

  2. Stochastic regressions (5/15 — were passing in parent; regressed due to LM variance, not k=12):

    • ex05 (NH Route 124): NH Route 125 retrieved instead; Route 124 missed. Context2 identical to parent (hop1 top-6 unchanged), so regression is pure LM variance.
    • ex09 (Air Lituanica): Parent passed by querying "Air Lituanica" in hop2; child's hop2 went to "Kogalymavia" (different query from identical context). Pure stochastic divergence.
    • ex15 (Zeitgeist Film Series): Retrieved "Zeitgeist (company)" but not "Zeitgeist Film Series." Parent succeeded; child failed due to LM variance.
    • ex36 (Charles Lane / Lucille Ball): Claim has typo "Charpes Lane." Child started with Jim Brochu but diverged to Isa Bowman, Zero Hour etc., never querying Lucille Ball or Charles Lane. Typo-anchoring + LM variance.
    • ex41 (Afonwen / Caerwys railway station): Child retrieved Caerwys and A55 road but never reached Afonwen or the specific Caerwys railway station article. Parent used 5 hops with Afonwen as one query; child's hop sequence diverged stochastically.

  3. ColBERT ranking gaps (2/15):

    • ex17 (Swoosh / Dan Wieden / Just Do It): Swoosh correctly queried but ColBERT never returns the Swoosh article. Persistent across 5+ iterations. k=12 did not help (article simply not ranked at positions 1-12 for this query).
    • ex23 (F.E.A.R. / Alma Wade / SAS: Secure Tomorrow): F.E.A.R. queried correctly but ColBERT returns disambiguation/series/album variants, not the base game article. Persistent.

  4. Hallucination (1/15):

    • ex34 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM identified Bob Fosse (born 1927) as the 1912 choreographer instead of Gene Kelly (born 1912). Despite Step 4 director/actress pattern, the LM hallucinated Fosse from the Liza with a Z article. Persistent — the Step 4 pattern guides extraction correctly in ex33 (Felicity Jones) but fails here because the LM hallucinates the wrong person.

  5. Morgan Llywelyn / Lion of Ireland (1/15):

    • ex28 (Lion of Ireland / Morgan Llywelyn / Robert Jordan): Both Lion of Ireland and Robert Jordan correctly retrieved in hops. But the Morgan Llywelyn article was not found — likely a ColBERT ranking issue where "Morgan Llywelyn" doesn't surface her own article at ranks 1-12.

Confirmed fixes introduced by iter15 changes:

Key new observations from iteration 15:

Failure Pattern Distribution: Iteration 14 Child Trace (15 failures from 50)

Subsample score: 0.6959 (+2.7% vs parent 0.6773 on same seed). Built on candidate 8 (iter13, fruitless tracking). Accepted as candidate 9. Valset: 0.8450 (NEW BEST, +2.2% absolute vs prior best 0.8267). Note: the 15 failures vs 9 in iter13 reflects a different random seed — the fair comparison is the +2.7% delta on the same seed.

  1. ColBERT retrieval gap — correct query issued but article not returned (4/15):

    • ex14 (Jonathan Lynn / Yes Minister / Ettore Scola): Jonathan Lynn correctly queried in hops 2, 3, and 5 but ColBERT never returns the Jonathan Lynn article (returns Wiley Lynn, George Lynn variants). Persistent across iter4, iter6, iter7, iter8, iter12, iter13, iter14. Fundamental ColBERT ranking gap.
    • ex15 (Swoosh / Dan Wieden / Just Do It): Swoosh correctly queried in hops 1, 3, and 5 but ColBERT never returns the Swoosh article (returns Nike-related near-misses). Persistent across iter4, iter11, iter12, iter13, iter14. The retry mechanism fires but surfaces only Nike-branded articles.
    • ex26 (Moonrunners / Briscoe Darling Jr. / Dukes of Hazzard): Moonrunners correctly queried in hop3 but ColBERT returns Sunrunner, Traitor's Moon, Jerry Rushing — not the film. Hops 4-5 wasted on Uncle Jesse and another variant. Persistent across 6+ iterations. ColBERT near-name match failure.
    • ex33 (Asymptotic safety in quantum gravity / Sheldon Glashow / Steven Weinberg): Hop5 retry correctly queried "asymptotic safety in quantum gravity" (exact article title) but the document is still missing from final 21. Likely a ColBERT ranking gap or slot eviction at boundary. Hops 1-4 cycled through "asymptotic safety" → "quantum gravity" → retry → "sheldon glashow" → retry → exact title in hop5.

  2. Wrong entity chain / second-hop inference failure (5/15):

    • ex1 (Matthew Bailes / Swinburne / CAS): CAS article retrieved in hop1. Hops 2-5 went Melbourne → Swinburne → Hawthorn → Swinburne (loop). Matthew Bailes (CAS director, visible in CAS article) never queried despite hop5 having an extra chance. Persistent across iter6, iter7, iter9, iter10, iter11, iter12, iter13, iter14.
    • ex4 (Allan Goldstein / Charles Bronson / Violent Restitution): Charles Bronson correctly queried in hop1, then hops went Leslie Nielsen → The Naked Gun → Charles Dennis Buchinsky (Bronson birth name). Allan Goldstein (director) never queried. The LM follows Bronson → co-star chain instead of Bronson → director chain. Persistent.
    • ex11 (Apple Inc. / Welcome to Macintosh / Vertex Pharmaceuticals): Welcome to Macintosh correctly retrieved in hop1. Hops 2-5 went Boston → Stephen Ives (documentary director) → New York City → (hop5). Apple Inc. — directly implied by the documentary's subject — never queried. Geographic distractors dominate. Persistent across multiple iterations.
    • ex17 (Guy Davis comics / Crimson Peak / Burn Gorman): Burn Gorman correctly retrieved in hop1. Hops 2-5 went Guillermo del Toro → Pacific Rim → Pacific Rim (2013 film) → (hop5). Never reached Crimson Peak or Guy Davis. LM follows del Toro's filmography in wrong direction. Persistent.
    • ex21 (Bill Cullen / Paul Melba / Punchlines): Hops 1-3 wasted on Val Doonican → The Val Doonican Show (loop). Hop4 correctly queried "Punchlines" — but Bill Cullen (the show's host) was not extracted from it. Hop5 went to Lennie Bennett instead of Bill Cullen. One step away from success but the host extraction failed. Persistent — correct show found but host not derived.

  3. Wrong disambiguation at hop1 / wrong article chain (2/15):

    • ex38 (The Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved "The Secret Agent (film)" (1996 Bob Hoskins film) instead of "The Secret Agent (TV series)" with Stephen Graham. All 5 hops followed wrong path: Bob Hoskins → Shane Meadows → This Is England → Patricia Arquette. Interestingly, This Is England WAS correctly queried (hop3) but the TV series and Stephen Graham were not reached. Persistent across iter8, iter10, iter12, iter13, iter14. The specific fix needed is a disambiguation suffix at hop1 or a claim-level disambiguation check.
    • ex30 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM hallucinated Bob Fosse as the choreographer (Fosse born 1927, claim says 1912 — Gene Kelly, born 1912, is correct). Hops went Bob Fosse → Chet Walker → "Liza with a 'Z'" → Bob Fosse (loop). Never queried Gene Kelly or Best Foot Forward. Persistent hallucination — same across multiple iterations.

  4. Claim resolution failure / typo loop (2/15):

    • ex31 (Lucille Ball / Charles Lane actor / Jim Brochu): Claim contains typo "Charpes Lane" (should be "Charles Lane"). Hops went Piper Laurie → "Charpes Lane" → Charles Lane (correct!) → "Charpes Lane" (back to typo) → "Lucy in the Afternoon" (Jim Brochu's Lucille Ball book — close but not the direct article). Lucille Ball never queried despite being the biography's subject. The retry mechanism fires but the LM reverts to the typo form. Typo-anchoring is a distinct failure mode — the claim's garbled spelling overrides correct reasoning.
    • ex45 (1974-75 NY Islanders / New York Rangers / Madison Square Garden): LM started with "New Jersey Devils" (wrong team) instead of New York Rangers, then went Madison Square Garden → New York metropolitan area → NHL. New York Rangers never queried, 1974-75 NY Islanders season never queried. The specific historical season article requires knowing the year from unstated context. Persistent across multiple iterations.

  5. Loop / fruitless query not suppressed (1/15):

    • ex42 (Aliens of the Deep / Bears film / Keith Scholey): Hops cycled: Alastair Fothergill → American Cats → American Cats (repeated) → Alastair Fothergill (repeated) → Alastair Fothergill (repeated) → American Cats (repeated). The retry mechanism should suppress these repeats, but since ColBERT returns at least some docs for "American Cats" and "Alastair Fothergill" (fruitless detection not triggered), the cycle persists. "Aliens of the Deep" — the other documentary co-directed by Scholey and Fothergill — never queried. The fruitless tracking mechanism is insufficient when wrong-but-non-empty results are returned.

  6. Partner extraction failure (1/15):

    • ex47 (Ellis Ferreira / Boris Becker / 1997 Stockholm Open Doubles): Boris Becker and 1997 Stockholm Open Doubles correctly retrieved (hops 1-2). Hops 3-5 kept querying the event article (loop) or Nicklas Kulti, Jonathan Stark — never Ellis Ferreira (Kulti's actual partner). The partner extraction Step 4 example added in iter11 still fails for this case. Persistent across iter11, iter12, iter13, iter14.

Net comparison vs iteration 13 child (9 failures, different seed):

Key new observations from iter14 trace:

Failure Pattern Distribution: Iteration 13 Child Trace (9 failures from 50)

Subsample score: 0.8167 (+5.1% vs parent 0.7769). Built on candidate 7 (4-hop, valset 0.8267). Accepted as candidate 8. Valset: 0.8167 (−1.2% vs best, within noise).

  1. Second-hop inference failure — implied entity visible in retrieved text but wrong entity selected (3/9):

    • ex0 (Allan Goldstein / Charles Bronson / Violent Restitution): Retrieved Violent Restitution and Charles Bronson correctly, but never queried Allan Goldstein (director). The Leslie Nielsen article is retrieved but the LM does not extract the director from it. Persistent (ex7 in iter13 reflection, ex1/ex2 in prior traces).
    • ex23 (Punchlines / Bill Cullen / Paul Melba): Retrieved Paul Melba article correctly. Step 4 was supposed to find "Punchlines" (the American TV show Paul Melba appeared on), but the LM extracted Val Doonican (a British artist mentioned in the Paul Melba article) instead of identifying Punchlines as the "American version of a TV show" and Bill Cullen as its host. Hops 2/3/4 all wasted on Val Doonican and Regis Philbin. Persistent failure — Step 4 named-show pattern still fails to identify the right show from ambiguous body text. Root cause: the claim says "an American version of a TV show" — the Paul Melba article mentions multiple shows, and the LM picks the wrong one.
    • ex33 (Charlotte Baldwin Allen / History of Texas / George Ruby): Retrieved George Ruby AND History of Texas correctly (both gold docs found), but Charlotte Baldwin Allen (female co-founder of Houston) never queried. Persistent co-founder omission — the LM retrieves Augustus Chapman Allen but not his wife. Persistent across iter5, iter6, iter7, iter8, iter11, iter12, iter13.

  2. ColBERT retrieval gap — correct query issued but wrong-level article returned (2/9):

    • ex21 (F.E.A.R. / Alma Wade / SAS: Secure Tomorrow): Alma Wade and SAS: Secure Tomorrow both correctly found. Hop queried F.E.A.R. but ColBERT returned F.E.A.R. (series), F.E.A.R. (disambiguation), F.E.A.R. (album), F.E.A.R. 3 — NOT the base game article. Persistent across iter6, iter8, iter11, iter12, iter13. The exact Wikipedia article F.E.A.R. is never ranked first by ColBERT for this query string.
    • ex46 (Crepis / Isatis / E.B. Babcock): E.B. Babcock and Crepis both correctly retrieved (hop1 on raw claim). Hop2 correctly queries Isatis but ColBERT returns Isatis tinctoria (species) not the Isatis (genus stub). Since Isatis tinctoria is a NEW unique doc, Isatis is NOT added to fruitless_queries — the mechanism can't help here. Hops 3 and 4 each begin by trying Isatis again (both in previous_queries), triggering the retry mechanism twice, and pivoting to Crepis (already retrieved) and Brassicaceae respectively. Key insight: fruitless detection is blind to "wrong article returned" cases — it only catches true zero-result queries. LM 2's own reasoning correctly identifies the gap ("not just a species like Isatis tinctoria") but can't fix it.

  3. 4-hop budget exhausted on wrong entities, core article unreached (2/9):

    • ex11 (The Dinner Party / Sojourner Truth / Barbe de Verrue): Barbe de Verrue retrieved in hop1. Hops 2/3/4 all searched for "Isabella (Belle)" or "Isabella (Belle) Isabella" (literal claim text), getting irrelevant Isabella articles. Never queried "Sojourner Truth" or "The Dinner Party." Root cause: the claim's garbled wording "Isabella (Belle) Isabella was the birth name" — the LM resolves this as a literal search query rather than recognizing the abolitionist as Sojourner Truth. Context trimming (iter12) fixed this under a specific seed where Sojourner Truth was at hop1 rank-7; with the new seed, Sojourner Truth is not in hop1's results at all. Persistent but seed-dependent.
    • ex18 (Douglas MacArthur's escape / Texas Raiders / Boeing B-17): Boeing B-17 correctly retrieved. Hops 2/3/4 went to Douglas C-74 Globemaster → Lone Star Executive Airport → Douglas MacArthur (general biography). The specific article "Douglas MacArthur's escape from the Philippines" and "Texas Raiders" (the specific B-17 at Conroe) never queried. Airport disambiguation failure persists despite hyphen normalization (different airport queried this time vs. iter12). Persistent across iter7, iter11, iter12, iter13.

  4. ColBERT chain gap — bridge article retrieved but next-hop entity not text-inferable (2/9):

    • ex39 (Stranger in Paradise / Polovtsian Dances / Cór Cois Abhann): Cór Cois Abhann and Alexander Borodin correctly retrieved (hop1). Hop2 correctly covers all named claim entities and goes to Step 4, but queries Prince Igor (Borodin's opera) instead of Stranger in Paradise (song) (the Kismet musical adaptation). Hop3 pivots to Kipchaks (descriptive entity). The Kismet (musical) article IS retrieved (appears in retrieved titles at position 9) but the LM doesn't extract Stranger in Paradise as the specific song from it. Persistent across iter5, iter6, iter7, iter8, iter11, iter12, iter13. The PRIORITY note in Step 4 ("prefer direct claim-relationship entities") partially helps but the LM still chose Prince Igor over the specific song. The specific song name requires music-domain knowledge to derive from Kismet.
    • ex4 (Adam Guettel / Elena Shaddow / How Glory Goes): Elena Shaddow and How Glory Goes both correctly retrieved. But "Adam Guettel" (the songwriter born in 1964) never queried — the claim gives no name, only "A songwriter who was born in 1964." No hop extracts Guettel from the How Glory Goes article. Deep inference failure — requires reading the retrieved article to find Guettel's name.

Net comparison vs iteration 12 child (8 failures):

Key new observations from iter13 trace:

Failure Pattern Distribution: Iteration 9 Child Trace (14 failures from 50)

Subsample score: 0.7171 (final re-eval) — approximately neutral vs parent 0.7370 (within ±3% noise).

  1. Wrong entity chain / background-entity distractor (7/14):

    • ex2 (Matthew Bailes): CAS article implicitly names Matthew Bailes as director, but LM queried Hawthorn, Victoria → Melbourne (geographic context) instead. Step 4 concurrent body scan did not extract Matthew Bailes from the CAS article as a query target.
    • ex3 (Robinsons Galleria): Misleading claim ("Metro Manila is in Turkmenistan") distracted LM to Metro Manila → Turkmenistan. Robinsons Galleria (the shopping mall beside Galleria Corporate Center) never queried.
    • ex20 (Apple Inc.): Claim explicitly identifies Apple Inc. as the company behind Welcome to Macintosh. LM retrieved the film correctly (hop 1) but then queried New York City → Insignia Films (geographic distractor). Apple Inc. — explicitly and unambiguously named in the claim — was never queried.
    • ex25 (Airlines of Africa): Correct chain is Comair → Airlines of Africa (subsidiary). LM went KLM → British Airways → Air France–KLM. Never reached Airlines of Africa.
    • ex37 (Bill Watts): Hops 1–2 wasted on Bobby Fulton → Scott Anton. Hop 3 correctly found Erik Watts. No hop left to query Bill Watts (his father, born May 5, 1939). 4-hop budget exhausted after early wrong turns.
    • ex49 (Beyond Rangoon): Claim mentions "True Romance" and "Johnny Depp" as surface distractors. LM pursued Johnny Depp in hop 2. Beyond Rangoon (the 1995 film with Victor Slezak) never queried. The concurrent Step 4 body scan of the Victor Slezak article apparently did not surface Beyond Rangoon.
    • ex32 (Rogue One / Felicity Jones): LM pursued Comedy of Errors → Oxford Dramatic Society → Teletubbies (completely wrong chain from hop 1). Neither Rogue One nor Felicity Jones queried. Rachelle Beinart article was retrieved but the LM did not derive the film from it.

  2. Second-hop inference failure (4/14): LM retrieves the bridge article but fails to extract and query the next entity from it:

    • ex15 (The Dinner Party): Barbe de Verrue and Sojourner Truth both retrieved, but "The Dinner Party" art installation never queried. Persistent across 5+ iterations (ex31 in iter4–8). The concurrent body-scan addition (iter9 Step 4 rewrite) did NOT fix this — the LM reads the bridge articles but does not extract the art installation name as a query target.
    • ex19 (Renáta Tomanová): 1980 French Open – Mixed Doubles article was retrieved in hop 2. LM then queried Billy Martin instead of extracting Renáta Tomanová as the doubles partner from the article text.
    • ex45 (Christian Poulsen): Pierre Womé and Denmark v Sweden (UEFA Euro 2008 qualifying) both retrieved. LM never derived Christian Poulsen from the qualifying match article. Complication: the claim calls Poulsen the "Nigerian midfielder" (he is Danish), so the LM was looking for a Nigerian player and couldn't identify Poulsen. Hop 3 repeated "Pierre Nlend Womé" — retry fired and returned same entity. Hop 4 queried "UEFA Euro 2008 qualifying Group F" (wrong). Christian Poulsen never queried.
    • ex46 (On the Buses film / Pat Ashton): Harry Booth article retrieved in hop 1. LM should read it to find "On the Buses" as his 1971 film, then query that. Instead, LM directly queried "Thick as Thieves" (wrong path) and "Thick as Thieves (cooking show)" (wrong suffix). On the Buses film and Pat Ashton never queried.

  3. ColBERT retrieval gap (1/14):

    • ex9 (Air Lituanica): Air Lituanica IS queried correctly in hop 2. ColBERT returns "Lituanica", "Lituanica SAT-1", "London Lituanica" — not the Air Lituanica article. Persistent across multiple iterations. Possible fix: query "Air Lituanica airline" or "Air Lituanica Latvia."

  4. Deep inference / completely wrong path from hop 1 (1/14):

    • ex18 (Loha 1987 film): Chain requires film→star→maternal grandfather (Geoffrey Kendal). LM identified Dharmendra as Loha's star (wrong — star is Dimple Kapadia, whose father is Karan Kapoor). All 3 gold articles missed. No in-claim anchor for the star's identity.

  5. Retry mechanism fires but fails (2/14):

    • ex28 (Flex club): Hop 3 repeated "Ronnie Flex." Retry fired and still returned "Ronnie Flex" (no Wikipedia article exists under this name, LM has no alternative). "Flex (club)" in Vienna never queried.
    • ex45 (Christian Poulsen): Retry fired for hop 3 (Pierre Womé repeat) and returned the same entity again. The misleading "Nigerian" label in the claim prevents the LM from finding the correct Danish player.

Retry mechanism statistics (iter9 child, 50 examples): Fired in 4 examples (ex28, ex39, ex40, ex45). ex39 and ex40 PASSED despite retry firing — the retry redirected to a new query that worked. ex28 and ex45 failed — retry returned same entity (no Wikipedia article / misleading claim). Overall: retry fires in 8% of examples (4/50), unlike the 46% in 5-hop iter8. At 4 hops, query repetition is rare and the retry rarely changes the outcome.

New key observations from iter9 trace:

Failure Pattern Distribution: Iteration 11 Child Trace (11 failures from 50)

Subsample score: 0.7769 (+2.6% vs parent 0.7570). Built on candidate 4 (4-hop, valset 0.7968). Accepted as candidate 6. Valset not yet evaluated.

  1. Wrong entity at hop 1 / wrong person identified (1/11):

    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): The claim says "star of the 1987 film Loha" — but the star is Karan Kapoor (not Dharmendra, who appeared in a different Loha). LM queried Dharmendra → Raj N. Sippy → Dharmendra Singh Deol. None of the 3 gold articles found. Persistent — was ex18 in iter9 trace. Pure first-hop disambiguation failure, not fixable by prompt alone without external disambiguation knowledge.

  2. Partner/collaborator extraction from event article (2/11):

    • ex38 (Renáta Tomanová / 1980 French Open Mixed Doubles): LM retrieved the mixed doubles article correctly in hop 2 but then queried Stanislav Birner instead of extracting Renáta Tomanová as the actual doubles partner. Persistent — same as ex19 in iter9 trace. The Step 4 examples added "doubles partner" but the LM still chose the wrong player from the article text.
    • ex146 (Ellis Ferreira / 1997 Stockholm Open Doubles): LM retrieved the doubles article in hop 2 but queried Nicklas Kulti (one of the players) instead of Ellis Ferreira (Kulti's actual partner). Same failure pattern as ex38. Appears in iter8 trace as well. The LM consistently reads a 2-player event article and then queries one of the two players named in it, not the specific partner of the claim's subject.

  3. Adaptation/derivative work not surfaced (1/11):

    • ex129 (Stranger in Paradise / Polovtsian Dances): Hop 2 retrieved Alexander Borodin, hop 3 queried Kipchaks, hop 4 queried Cumans. No retrieved article bridges to "Stranger in Paradise (song)" — the famous Kismet adaptation of Polovtsian Dances. Persistent across iter6, iter7, iter8, iter11. Classic ColBERT bridge gap — the connection between Polovtsian Dances and Stranger in Paradise is not text-inferable from any retrieved passage in this index.

  4. Explicitly-named entity in claim never queried (2/11):

    • ex50 (Swoosh / Dan Wieden): Hop 3 DID query "Swoosh" but ColBERT returned near-misses rather than the Swoosh Wikipedia article. Hop 4 wasted on "Nike logo." Persistent ColBERT retrieval gap. The article title is "Swoosh" (no disambiguation suffix needed) but ColBERT ranks it poorly relative to Nike-related articles.
    • ex95 (Felicity Jones / Rogue One): Hops 2+3 went to "The Comedy of Errors" and "Oxford University Dramatic Society." Hop 4 queried "Rogue One" (found). "Felicity Jones" — the actress explicitly implied — was never directly queried across all 4 hops. Persistent across iter4, iter5, iter6, iter7, iter8, iter9, iter11. The Step 4 "stunt performer film" example did not trigger the missing direct Felicity Jones query.

  5. Phantom-entity guard insufficient — LM still chases sub-concepts (1/11):

    • ex65 (Ancient Egyptian religion / Mehetweret / Damnation): Despite Round 2's phantom-entity guard ("skip descriptive phrases like feather of truth, lake of fire"), the LM still queried "Feather of truth" (hop 2), "Lake of fire" (hop 3), and "Weighing of the heart" (hop 4). Never queried "Ancient Egyptian religion." The guard instruction was directly violated — the LM invented these as "Step 1 entities" despite the guard. Persistent. Code-level enforcement (e.g., colbert zero-new-results → forced Step 4) is needed; prompt instruction alone is insufficient.

  6. Abbreviation expansion failure (1/11):

    • ex106 (Cork / National Maritime College of Ireland): Claim uses "NMIO" (slightly garbled abbreviation for NMCI). Hops went UCC → County Cork (found) → Cork College of Commerce. "National Maritime College of Ireland" never queried. The phantom guard correctly identified UCC/CoCC as real entities, but the garbled abbreviation left the LM unable to expand "NMIO" to its full Wikipedia title. Partial progress vs. parent (County Cork now found). Step 4 "broader topic article" hint partially helped.

  7. Wrong disambiguation path for entity with multiple articles (2/11):

    • ex85 (Flex club / High Contrast / Waves Vienna): Hops 2+4 both queried "Ronnie Flex" (the retry fired on hop 3 but returned the same entity). "Lincoln Barrett" (High Contrast's real name) was in previous_queries but never used as a Waves Vienna or Flex club clue. Persistent across iter4, iter5, iter6, iter7, iter8, iter11. The retry mechanism fires but produces no Wikipedia article under either name — the LM is stuck.
    • ex133 (The Company / Ehrlichman novel): Hops queried "The Company" → "The Company (book)" → "John Ehrlichman." Never reached "The Company (Ehrlichman novel)" (the exact Wikipedia disambiguation suffix). Persistent across iter5, iter7, iter11.

  8. Douglas MacArthur duplicate airport query (1/11):

    • ex53: Hops 2 and 4 both targeted the airport using slightly different punctuation. "Douglas MacArthur's escape from the Philippines" and "Texas Raiders" never queried. Persistent across iter7, iter11. The retry mechanism did NOT fire here — the two airport variants ("Conroe North Houston Regional Airport" vs "Conroe-North Houston Regional Airport") are normalized as different queries by _normalize_query, so no duplicate was detected. Fuzzy punctuation normalization would fix this.

Retry mechanism stats (iter11 child, 50 examples): Fired in 10 examples (ex22, 26, 51, 76, 84, 85, 93, 103, 110, 137). 9 passed. 1 failed (ex85 — Ronnie Flex/Flex club, no article under any queried name). Ex76 (Delmer Daves) confirmed direct fix by retry. Ex100 (20th Century Fox) fixed by phantom-entity guard in Round 2.

New observations from iter11 trace not in prior records:

Failure Pattern Distribution: Iteration 10 Child Trace (15 failures from 50)

Subsample score: 0.6958 — a −5.4% regression vs parent (candidate 5, 0.7356). 4 NEW failures introduced vs candidate 5's 11. Built on candidate 5 (5-hop, valset 0.7554).

All 11 candidate 5 parent failures persisted (ex0/Matthew Bailes, ex5/Robert E. Howard, ex18/Guy Davis, ex21/F.E.A.R., ex22/Ancient Egyptian religion, ex23/Jay T. Wright, ex24/Airlines of Africa, ex28/Moonrunners, ex30/Flex club, ex34/Gene Kelly, ex38/Asymptotic safety). 4 new regressions introduced:

New regressions introduced by iter10 changes:

Persistent failure categories (from the 11 carried-over parent failures):

  1. Second-hop inference failure (4/15): LM retrieves bridge article but fails to extract correct implied entity: ex0 (Matthew Bailes not extracted from CAS article → went to Melbourne geographic chain), ex24 (Airlines of Africa not extracted from Comair article → went to British Airways), ex28 (Moonrunners not extracted from James Mitchum article → went to Dukes of Hazzard/Uncle Jesse variants), ex45 (On the Buses not extracted from Harry Booth article → went to Harry Booth filmography).
  2. Explicitly-named entity in claim never queried (3/15): ex5 (Robert E. Howard — comparison entity in claim skipped; LM went Frank Bascombe → Hurricane Sandy), ex28 (Moonrunners — "1975 film" explicitly referenced in claim; query never issued), ex38 (Asymptotic safety in quantum gravity — explicitly named concept in claim; LM went Nobel Prize in Physics → Steven Weinberg → University of Texas).
  3. Sub-concept drill-down instead of parent article (2/15): ex22 (Ancient Egyptian religion — LM drills into Weighing of the Heart, Feather of Ma'at, Lake of fire, Book of the Dead but never queries the parent religion article), ex35 (Queens — LM queries census sub-concepts instead of the borough itself).
  4. Hallucination / wrong disambiguation (2/15): ex34 (Bob Fosse identified as 1912 choreographer instead of Gene Kelly — persistent), ex42 (The Secret Agent 1996 film instead of TV series).
  5. ColBERT entity name mismatch (2/15): ex21 (F.E.A.R. queried as hop 4 but article not returned in top-7), ex23 (Jay T. Wright queried but article missing).
  6. Misleading claim entity (1/15): ex44 (Christian Poulsen called "Nigerian midfielder" → LM finds Obafemi Martins instead).
  7. Entity has no Wikipedia article / retry loops (1/15): ex30 (Flex club never queried; Lincoln Barrett repeated twice via retry; Waves Vienna found but Flex club itself missed).

New structural insight from iter10 trace:

Failure Pattern Distribution: Iteration 21 Child Trace (8 failures from 50)

Subsample score: 0.8352 (+5.0% vs parent 0.7954). Built on candidate 13 (iter20). Accepted as candidate 14. Valset: 0.8748 (NEW BEST, +3.5% relative vs prior best 0.8551). Changes: Step 3 CRITICAL type-qualifier rule with bad→good example + three new Step 4 body-scan bullets (parent company, town/location-of, event venue).

  1. Intermediate article skipped in ownership chain (1/8):

    • ex32 (Darkened Skye / Mars, Incorporated / Skittles confectionery): Mars, Incorporated found (parent company bullet worked). Darkened Skye found in hop1. Skittles (confectionery) never queried. The chain requires querying the CANDY first (Skittles), then the manufacturer (Mars). The new parent company bullet caused the LM to jump directly to Mars Inc., skipping the intermediate Skittles article. New partial failure category: the parent company bullet may cause the LM to shortcut a required intermediate entity in ownership chains.

  2. Wrong entity chain / second-hop bridge failure (2/8):

    • ex56 (Guy Davis comics / Crimson Peak / Burn Gorman): Burn Gorman and Crimson Peak found. Hops followed Guillermo del Toro → Pacific Rim chain; Guy Davis (comics, production designer for Crimson Peak) never queried. Persistent across 10+ iterations.
    • ex71 (Airlines of Africa / British Airways franchise destinations / Comair South Africa): Comair (South Africa) found. Hops followed KLM → British Airways chain; Airlines of Africa (Comair subsidiary) never queried. Persistent across 10+ iterations.

  3. Phantom entity loop — claim sub-concepts queried, parent article never reached (1/8):

    • ex65 (Ancient Egyptian religion / Damnation / Mehetweret): Only 4 searches fired. Mehetweret and Damnation retrieved. The claim mentions "feather of truth" and "lake of fire" — the LM queried these phantom sub-concepts instead of the parent "Ancient Egyptian religion" article. The phantom-entity guard (in place since iter11) failed to prevent this. Persistent across 10+ iterations. Prompt guard is insufficient; code-level enforcement needed.

  4. Fruitless loop — entity name has no Wikipedia article (1/8):

    • ex85 (Flex club / High Contrast / Waves Vienna): High Contrast and Waves Vienna found. LM looped on "Ronnie Flex" / "Lincoln Barrett" variants (retry mechanism fired but LM returned same entity or repeated). "Flex (club)" in Vienna never queried. The venue extraction bullet did NOT fire because the Waves Vienna article was never retrieved early enough for a body scan. Persistent across 10+ iterations.

  5. LM hallucination — wrong entity identified from claim (1/8):

    • ex97 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM identified Bob Fosse (born 1927) as the "choreographer born in 1912" instead of Gene Kelly (born 1912). All hops followed Fosse's collaborators. Gene Kelly and Best Foot Forward never queried. Persistent hallucination across 16+ iterations — unfixable by prompt alone. Only 4 searches used (18 docs retrieved — probable slot-fill gap as well).

  6. Bridge article retrieved but leaf entities not derived (1/8):

    • ex126 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Harry Booth article presumably retrieved. LM fixated on "Thick as Thieves" from the claim; On the Buses (film) and Pat Ashton never queried. Persistent across 10+ iterations.

  7. Stochastic regression (1/8):

    • ex82 (Moonrunners / Dukes of Hazzard / Briscoe Darling Jr.): Parent (iter20 seed) passed. Child failed because the LM fell into Uncle Jesse disambiguation rabbit hole (hops 3-5) rather than following James Mitchum → Moonrunners chain. Context at hop2 is IDENTICAL between parent and child; divergence at hop3 is pure LM stochasticity. Not caused by iter21 changes.

Confirmed fixes vs parent (candidate 13, different seed):

Key new observations from iteration 21:

Failure Pattern Distribution: Iteration 22 Child Trace (11 failures from 50)

Subsample score: 0.7755 (−7.2% vs parent 0.8350). Built on candidate 9 (iter14, k=7 uniform). Rejected. Valset N/A. Note: this iteration was built on candidate 9 (not the best candidate 14). Two rounds: Round 1 reranker (−7.2% regression, reverted), Round 2 prompt-only cherry-pick from iter18-21 onto k=7 base (same score). The parent (candidate 9) baseline of 0.8350 passes ex31 (Sojourner Truth) as 0.994; the child regressed it to 0.0 due to Step 2 sub-article false-coverage wording interaction with k=7 context.

  1. Wrong entity chain / second-hop bridge failure (3/11):

    • ex3 (Matthew Bailes / Houman Younessi / Swinburne): CAS article in hop1. Hops followed geographic distractors (Melbourne, Hawthorn, Swinburne). Matthew Bailes (CAS director) never queried. Persistent across 16+ iterations.
    • ex7 (Allan Goldstein / Charles Bronson / Violent Restitution): Charles Bronson hop1, then Leslie Nielsen → Naked Gun chain. Allan Goldstein never queried. Persistent across 16+ iterations.
    • ex71 (Airlines of Africa / Comair / British Airways): Comair found, then KLM → British Airways chain; Airlines of Africa (Comair subsidiary) never queried. Persistent across 10+ iterations.

  2. Broad-topic-chain failure — specific entity inferrable from description but never queried directly (1/11):

    • ex18 (Heathrow Airport / Gatwick Airport / Coldwaltham): Query chain: raw claim → "London" → "List of busiest airports in the United Kingdom" → "United Kingdom" → "Busiest airports in the United Kingdom". Heathrow Airport and Gatwick Airport were NEVER directly queried — the LM pursued list/category articles instead of resolving the description "busiest airport outside London" to a specific entity name. Neither airport found. Likely new to this subsample seed; may appear in valset.

  3. Regression caused by porting prompt improvements to wrong architecture (1/11):

    • ex31 (The Dinner Party artwork / Sojourner Truth / Barbe de Verrue): Parent (candidate 9) passed with 0.994 (Sojourner Truth found). Child regressed to 0.0. The Round 2 sub-article false-coverage wording from iter18-21 — designed for asymmetric-k — interacted badly with the k=7 context window, causing the LM to either miscategorize some title as covered or fail to query Sojourner Truth. Query chain shows hop4 DID query "Sojourner Truth" but it was in seen_titles (k=7 for hop1 puts rank-7 docs into seen_titles). This failure mode confirms that iter18-21 prompt improvements are coupled to the asymmetric-k architecture.

  4. ColBERT loop — wrong disambiguation path from hop1 (2/11):

    • ex53 (Douglas MacArthur's escape / Boeing B-17 / Texas Raiders): Boeing B-17 found; hops went to Conroe airport variants + generic MacArthur biography. "Douglas MacArthur's escape from the Philippines" and "Texas Raiders" never queried. Persistent across 10+ iterations.
    • ex117 (Secret Agent TV series / Stephen Graham / This Is England): Hop1 retrieved 1996 film instead of TV series. Wrong chain throughout. Persistent across 10+ iterations.

  5. Multi-hop band-member chain without multi-membership bullet (1/11):

    • ex58 (Glenn Hughes / Dave Evans / From Now On): Hop chain: claim → Dave Evans → Deep Purple → Black Sabbath → Trapeze. Each band queried individually; Glenn Hughes (the person connecting all three bands) never directly queried. The multi-membership bullet (added in iter19 for candidate 14's architecture) is NOT present in candidate 9, so this failure is expected. Would be fixed by the multi-membership bullet that already exists in candidate 14.

  6. Phantom entity loop — claim sub-concepts queried instead of parent article (1/11):

    • ex65 (Ancient Egyptian religion / Damnation / Mehetweret): Only 4 hops used (hop5 didn't fire). LM queried "feather of truth" and "lake of fire" sub-concepts instead of "Ancient Egyptian religion." Persistent across 10+ iterations. Prompt guard is insufficient; code-level enforcement needed.

  7. Fruitless loop — entity has no Wikipedia article (1/11):

    • ex92 (Green Chair / Shim Ji-ho / Suh Jung): LM anchored on "2005 Sundance Film Festival" across all hops despite retry mechanism. Shim Ji-ho never queried. Persistent Sundance anchor failure across 10+ iterations.

  8. Film→performer extraction too late — film only in hop5 (1/11):

    • ex95 (Felicity Jones / Rachelle Beinart / Rogue One): Chain: claim → Oxford University Dramatic Society → The Comedy of Errors → Shakespeare → Rogue One. Rogue One only retrieved in hop5 (the LAST hop). The film→performer Step 4 bullet (Round 1) was supposed to extract Felicity Jones from the Rogue One article, but since Rogue One is in hop5, there is no hop6 to query Felicity Jones. Timing-dependent failure: the bullet only helps when the film is retrieved in hops 1-4. The fix for ex95 class requires the film being retrieved earlier.

  9. Partner extraction — wrong person identified from event article (1/11):

    • ex146 (Ellis Ferreira / Boris Becker / 1997 Stockholm Open): Boris Becker and event article correctly retrieved; Ellis Ferreira (Becker's doubles partner) never queried. Persistent across 10+ iterations.

Key new observations from iteration 22:

Failure Pattern Distribution: Iteration 23 Child Trace (9 zero-score examples from 50)

Subsample score: 0.8152 (+2.5% vs parent 0.7954). Built on candidate 14 (iter21). Accepted as candidate 15. Valset: 0.8550 (−2.3% vs best 0.8748 from candidate 14). Prompt-only changes: CREATOR, HOST/CAST MEMBER, CO-FOUNDER, COMPARISON/RANKING Step 4 bullets + parent company ORDERING RULE. Confirmed fixes: ex12 (Adam Guettel — CREATOR bullet), ex18 (Heathrow Airport — COMPARISON/RANKING bullet). Stochastic regression: ex63 (F.E.A.R. — LM issued bare query, pure LM variance).

  1. Wrong entity chain / inference failure (2/9):

    • ex56 (Guy Davis comics / Crimson Peak / Burn Gorman): Burn Gorman and Crimson Peak found. Hops followed Guillermo del Toro → Pacific Rim chain; Guy Davis (production designer, comics) never queried. Persistent across 10+ iterations.
    • ex97 (Gene Kelly / Best Foot Forward / Liza Minnelli discography): LM hallucinated Bob Fosse (born 1927) as the "choreographer born in 1912" instead of Gene Kelly (born 1912). All hops pursued Fosse's collaborators. Gene Kelly and Best Foot Forward never queried. Persistent hallucination across 17+ iterations — unfixable by prompt alone.

  2. Bridge article retrieved but leaf entity not extracted (4/9):

    • ex114 (Charlotte Baldwin Allen / History of Texas / George Ruby): George Ruby and History of Texas found. Augustus Chapman Allen retrieved. CO-FOUNDER bullet did NOT fire — Charlotte Baldwin Allen never queried. The August Chapman Allen article may not prominently name Charlotte, or the LM didn't apply the bullet. Persistent across 18+ iterations.
    • ex126 (On the Buses film / Pat Ashton / Thick as Thieves TV series): Thick as Thieves TV series WAS retrieved (confirmed in retrieved titles). HOST/CAST MEMBER bullet did NOT fire — Pat Ashton not extracted from the show article. On the Buses film never queried. LM remained fixated on Harry Booth filmography/disambiguation instead of extracting the star. Persistent across 10+ iterations.
    • ex129 (Stranger in Paradise / Cór Cois Abhann / Polovtsian Dances): Cór Cois Abhann and Polovtsian Dances found. Kismet (musical) IS in retrieved titles (confirming the bridge is reached). But "Stranger in Paradise (song)" never extracted from Kismet article — LM pursued Borodin → Kipchaks → Prince Igor chain. Neither the named-song bullet (iter13) nor CREATOR bullet triggered. Persistent across 10+ iterations. Deep song-adaptation extraction failure.
    • ex92 (Shim Ji-ho / Suh Jung / Green Chair): Green Chair and Suh Jung found. Only 4 searches used (hop5 didn't fire — LM declared all claim entities covered). Shim Ji-ho (co-lead actor) never queried. The LM retrieved Park Chul-soo (director) but not the actor. Persistent Sundance-anchor/actor-extraction failure across 10+ iterations.

  3. Fruitless loop / entity has no Wikipedia article (2/9):

    • ex85 (Flex club / High Contrast / Waves Vienna): High Contrast and Waves Vienna found. Only 4 searches used (hop5 didn't fire — LM declared all entities covered). Flex (club) never queried. LM retrieved Waves Vienna and High Contrast but did NOT fire a search for the nightclub. Persistent across 10+ iterations.
    • ex38 (Renáta Tomanová / 1980 French Open Mixed Doubles / Larisa Neiland): 1980 French Open Mixed Doubles and Larisa Neiland found. Renáta Tomanová never queried. False-premise partner claim (Boris Diaw is an NBA player with no tennis history) continues to confuse the LM. Persistent false-premise failure across 8+ iterations.

  4. ColBERT ranking gap + LM stochastic non-compliance (1/9):

    • ex63 (F.E.A.R. / Alma Wade / SAS: Secure Tomorrow): Alma Wade and SAS: Secure Tomorrow found. LM issued bare "F.E.A.R." (without CRITICAL type-qualifier "(video game)") at hop2, wasting a hop on series/disambiguation variants. Then all 21 slots filled before correct article retrieved. The CRITICAL TYPE QUALIFIER RULE was not followed on this seed. Stochastic non-compliance — same pattern fixed in iter21 subsample, regressed here. Confirms that CRITICAL directive has ~85% compliance, not 100%.

Confirmed fixes vs parent (candidate 14, different seed):

Key new observations from iteration 23:

Failure Pattern Distribution: Iteration 24 Child Trace (6 failures from 50)

Subsample score: 0.8748 (−0.0002 vs parent 0.8750, neutral). Built on candidate 14 (iter21, best valset 0.8748). Rejected. Valset: N/A (subsample not above parent threshold). Changes: full legal name variant coverage fix in Step 2 + CREATOR bullet + COMPARISON/RANKING bullet + parent company ORDERING RULE (latter three already in candidate 15 / iter23). Hop5 fire rate: 95.5% (42/44 passing examples). Score distribution: 2 × 0.996 (4-hop, ex10/ex18), 42 × 0.994 (5-hop), 6 × 0.0 (failure).

  1. Wrong entity chain / geographic distractor (2/6):

    • ex000 (Matthew Bailes / Houman Younessi / Swinburne): CAS and Houman Younessi correctly retrieved. LM declared all claim entities covered after retrieving the CAS article, then used remaining hops for geographic distractors (Melbourne, Swinburne context). Matthew Bailes (CAS director, presumably named in the CAS article) never queried. LM went through Melbourne → Swinburne sequence. Persistent across 20+ iterations — deepest unresolved failure.
    • ex002 (Allan Goldstein / Charles Bronson / Violent Restitution): Charles Bronson correctly retrieved in hop1. LM followed Leslie Nielsen → Naked Gun chain; Allan Goldstein (director) never queried. Persistent across 15+ iterations.

  2. Defunct entity never inferred / wrong DEFUNCT trigger (1/6):

    • ex007 (Air Lituanica / AirBaltic / Vilnius Airport): Vilnius Airport and AirBaltic retrieved. The DEFUNCT ENTITY rule fired — but for "Estonian Air" (mentioned in the Tallinn Airport excerpt as also ceasing in 2015) rather than "Air Lituanica." LM queried Estonian Air instead of Air Lituanica. New wrinkle on a persistent failure: the DEFUNCT ENTITY rule is triggering on the wrong defunct airline. Air Lituanica never queried. Persistent across 15+ iterations.

  3. Wrong person identified / first-hop disambiguation failure (1/6):

    • ex014 (Karan Kapoor / Geoffrey Kendal / Loha 1987 film): LM identified Dharmendra as the star of Loha instead of Karan Kapoor. Both Karan Kapoor and Geoffrey Kendal (his maternal grandfather) missed. Persistent across 15+ iterations — requires external Bollywood film knowledge.

  4. Second-hop inference gap — explicitly named entity never queried (1/6):

    • ex015 (Apple Inc. / Welcome to Macintosh film / Vertex Pharmaceuticals): Welcome to Macintosh correctly retrieved. LM pursued "company that made the documentary" → Insignia Films (production company) rather than Apple Inc. (the documentary's subject and the entity implicitly required by the claim). Stochastic regression from parent — the ORDERING RULE applied to parent company bullet may have affected the chain, but root cause is claim ambiguity ("the company that made the documentary" is genuinely ambiguous between Insignia Films vs Apple Inc.).

  5. LM hallucination — wrong entity from claim context (1/6):

    • ex035 (Gene Kelly / Best Foot Forward musical / Liza Minnelli discography): LM identified Bob Fosse (born 1927) as the "choreographer born in 1912" instead of Gene Kelly (born 1912). All 5 hops followed Fosse's collaborators (Fosse → "Liza with a 'Z'" → Cabaret etc.). Gene Kelly and Best Foot Forward never queried. Persistent hallucination across 20+ iterations — confirmed unfixable by prompt alone. The LM's strong Fosse-Minnelli prior overrides the birth year clue.

Confirmed structural fix vs parent:

Stochastic regression vs parent:

Soft regressions (penalty increases: 4-hop → 5-hop):

Key new observations from iteration 24:

Failure Pattern Distribution: Iteration 25 Child Trace (5 failures from 50)

Subsample score: 0.8951 (+7.2% vs parent 0.8352). Built on candidate 14 (iter21, best valset 0.8748). Accepted as candidate 16. Valset: 0.8252 (REGRESSION — −5.7% relative vs best 0.8748). Changes: formal-name variant fix in Step 2 (from iter24) + CRITICAL PERSON COVERAGE RULE in Step 2 + COMPARISON/RANKING Step 4 bullet. Hop5 fire rate: 76% (38/50 used 5 searches) — lowest measured rate. Score distribution: 12 × 0.996 (4-hop), 33 × 0.994 (5-hop), 5 × 0.0 (failure).

  1. Broad-topic-chain failure — busiest airport description never resolved to named entity (1/5):

    • ex18 (Heathrow Airport / Gatwick Airport / Coldwaltham): Queries: "London" → "Gatwick Airport" → "List of the busiest airports in the United Kingdom" → "United Kingdom". Gatwick WAS retrieved in hop2, but the COMPARISON/RANKING bullet never fired from the Gatwick article body to extract "Heathrow Airport." The LM issued broad-topic queries in Steps 1-3, exhausting the hop budget before reaching Step 4 body-scan. Persistent across 5+ iterations. The bullet is in place but the LM issues list/geo queries first, leaving no hop for body-scan extraction.

  2. Wrong person identified / first-hop disambiguation failure (1/5):

    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Queries: "Loha (1987 film)" → "Dharmendra" → "Raj N. Sippy" → "Dharmendra" → "Shatrughan Sinha". The LM identifies Dharmendra (not Karan Kapoor) as Loha's star. Both Karan Kapoor and Geoffrey Kendal missed. Persistent across 20+ iterations — requires external Bollywood knowledge.

  3. Phantom entity loop — claim sub-concepts queried, parent article never reached (1/5):

    • ex65 (Ancient Egyptian religion / Damnation / Mehetweret): Queries: "Weighing of the Heart" → "Feather of truth" → "Weighing of the Heart" → "Lake of fire" → "Book of the Dead". Only 5 searches used. The LM queried phantom sub-concepts despite the phantom-entity guard. "Ancient Egyptian religion" never queried. Persistent across 10+ iterations. Prompt guard is insufficient; code-level enforcement needed.

  4. Wrong disambiguation at hop1 / wrong article chain (1/5):

    • ex117 (Secret Agent TV series / Stephen Graham / This Is England): Queries: "Shane Meadows" → "Bob Hoskins" → "A Room for Romeo Brass" → "Patricia Arquette". Hop1 retrieved the 1996 film instead of the TV series. All hops followed wrong path. None of the 3 gold articles found. Persistent across 15+ iterations.

  5. Bridge article retrieved but specific song adaptation not extracted (1/5):

    • ex129 (Stranger in Paradise / Cór Cois Abhann / Polovtsian Dances): Queries: "Alexander Borodin" → "Kipchaks" → "Cumans" → "Prince Igor". Cór Cois Abhann and Polovtsian Dances found. Kismet (musical) IS in retrieved docs but "Stranger in Paradise (song)" never extracted. Persistent across 15+ iterations.

Confirmed fixes vs parent (candidate 14, different seed):

Notable observations from trace:

Key new observations from iteration 25:

Failure Pattern Distribution: Iteration 26 Child Trace (5 failures from 50)

Subsample score: 0.8949 (+7.1% vs parent 0.8352). Built on candidate 16 (iter25). Accepted as candidate 17. Valset: 0.8351 (per task context; iteration history says N/A — possible discrepancy between auto-evaluation and architect-recorded value). Single change: removed CRITICAL PERSON COVERAGE RULE from Step 2. Hop5 fire rate: 86% (43/50 used 5 searches). Score distribution: 7 × 0.996 (4-hop), 38 × 0.994 (5-hop), 5 × 0.0 (failure).

  1. Wrong actor identification / first-hop disambiguation failure (1/5):

    • ex12 (id=37 — Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Hop chain: raw claim → Loha (film) → Dharmendra → Amrish Puri → Shatrughan Sinha. The LM identified prominent Bollywood actors (Dharmendra, Amrish Puri, Shatrughan Sinha) instead of the actual star Karan Kapoor. Karan Kapoor and Geoffrey Kendal (his maternal grandfather) never queried. Persistent 20+ iterations — requires external Bollywood film knowledge to identify the correct star.

  2. Wrong entity chain / director's second-hop inference failure (1/5):

    • ex20 (id=56 — Guy Davis comics / Crimson Peak / Burn Gorman): Hop chain: raw claim → Burn Gorman → Guillermo del Toro → Pacific Rim → Pacific Rim (2013 film). Guy Davis (production designer and comics artist for Crimson Peak) never queried. The LM follows the director's other filmography (Pacific Rim) instead of inferring the production designer from the Crimson Peak article. Persistent 10+ iterations.

  3. Wrong entity chain / subsidiary never inferred from franchise carrier (1/5):

    • ex25 (id=71 — Airlines of Africa / Comair South Africa / British Airways franchise): Hop chain: raw claim → KLM → Comair Limited → Comair (South Africa) → OR Tambo International Airport. Airlines of Africa (a Comair subsidiary) never queried. The LM correctly finds Comair but doesn't infer the subsidiary. OR Tambo Airport is used as hop5 instead of Airlines of Africa. Persistent 10+ iterations.

  4. Sundance festival anchor / co-lead actor not extracted (1/5):

    • ex31 (id=92 — Green Chair / Shim Ji-ho / Suh Jung): Hop chain: raw claim → 2005 Sundance Film Festival (k=5) → Park Chul-soo → 2005 Sundance Film Festival again (k=5) [retry fired but LM re-issued same query] → Green Chair (k=12). Shim Ji-ho (co-lead actor) never queried. The LM finds Green Chair and Suh Jung but cannot extract Shim Ji-ho from those articles. NOTE: Retry mechanism fired for the Sundance repeat, warned the LM, LM still returned the same entity — confirming the Sundance anchor is stronger than any retry warning can overcome. Persistent 10+ iterations.

  5. Film→performer extraction failure — film in hop4, no hop6 (1/5):

    • ex33 (id=95 — Felicity Jones / Rachelle Beinart / Rogue One): Hop chain: raw claim → Oxford University Dramatic Society (k=5) → The Comedy of Errors (k=5) → Rogue One (film) (k=5) → Game of Thrones (k=12). Rogue One IS retrieved in hop4 (k=5), but hop5 pivoted to "Game of Thrones" instead of querying Felicity Jones from the Rogue One article. The Step 4 film→performer bullet is NOT in the current architecture (was removed after iter16 caused −3.5% valset regression). Without the bullet, the LM pursues a Rachelle Beinart → stunt work → Game of Thrones chain instead of Felicity Jones → Rogue One → actress. Persistent 10+ iterations. Felicity Jones requires the film→performer bullet to be reinstated with a tighter trigger condition (only when the claim's phrasing directly implies the actress from the retrieved film), and the film must be in hops 1-3 (not hop4) for a subsequent hop to query the person.

Confirmed fixes vs parent (candidate 16, different seed):

Key new observations from iteration 26:

Current State (Post Iteration 26)

Failure Pattern Distribution: Iteration 12 Child Trace (8 failures from 50)

Subsample score: 0.8366 (+2.4% vs parent 0.8167). Built on candidate 6 (4-hop, valset previously not evaluated). Accepted as candidate 7. Valset: 0.8267 (new best).

  1. Wrong entity chain / hops exhausted on wrong entities (3/8):

    • ex37 (Loha 1987 film / Karan Kapoor / Geoffrey Kendal): Model queried Loha (1987 film) and fixated on it (retry fired, issued same query twice). Never searched for Karan Kapoor (film's star) or Geoffrey Kendal (maternal grandfather). Persistent — same as ex37 in iter11. The retry cannot help here because the problem is wrong entity selection, not query repetition.
    • ex53 (Douglas MacArthur's escape / Boeing B-17 Flying Fortress / Texas Raiders): Hop1 serendipitously retrieved B-17. But hops 2–4 went to Conroe North Houston Regional Airport → Douglas MacArthur (generic biography) → Philippines (retry fired). The specific article "Douglas MacArthur's escape from the Philippines" and "Texas Raiders" (specific B-17 at Conroe) never queried. Hyphen normalization reduced the airport duplicate issue but could not fix the wrong chain pivot.
    • ex85 (Flex club / High Contrast / Waves Vienna): Hop1 retrieved High Contrast (Lincoln Barrett). Hops 2–4 queried Ronnie Flex → Lincoln Barrett (retry fired, repeated Lincoln Barrett). Flex (club) (Vienna nightclub) and Waves Vienna never queried. Persistent — ex85 across iter4, iter11, iter12.

  2. ColBERT retrieval gap — correct query issued but article not returned (3/8):

    • ex63 (F.E.A.R. / Alma Wade / SAS: Secure Tomorrow): Hop3 correctly queried F.E.A.R. but ColBERT returned F.E.A.R. (series), F.E.A.R. (disambiguation), F.E.A.R. (album), F.E.A.R. 3 — not the base article. Hop4 wasted on Jupiter EX. Persistent across iter6, iter8, iter11, iter12. The exact Wikipedia article title F.E.A.R. is systematically not returned by ColBERT for the query string "F.E.A.R." — an index ranking issue.
    • ex82 (The Dukes of Hazzard / Moonrunners / Briscoe Darling Jr.): Hop3 correctly queried Moonrunners but ColBERT returned Sunrunner, Traitor's Moon, Jerry Rushing, etc. — not the Moonrunners article. Persistent — ex82/Moonrunners across iter4, iter5, iter7, iter11, iter12. ColBERT returns near-name matches, not the actual film article.
    • ex143 (Crepis / Isatis / E.B. Babcock): Hop2 queried Isatis but ColBERT returned species articles (Isatis tinctoria, etc.) rather than the genus stub Isatis. Hop3 correctly pivoted to Crepis (found). Hop4 wasted on Roy Elwood Clausen (Babcock's coauthor). Isatis remains unretrieved despite correct query.

  3. Partner extraction failure (1/8):

    • ex146 (Ellis Ferreira / Boris Becker / 1997 Stockholm Open – Doubles): Model retrieved Boris Becker and 1997 Stockholm Open – Doubles correctly (hops 1-3), but hop4 queried Nicklas Kulti (one player in the final) rather than Ellis Ferreira (Kulti's actual partner). Persistent — ex146 in iter11, iter12. The Step 4 "doubles partner" example was added in iter11 but did not fix this — the LM extracts one player from the event article but does not specifically identify the opponent's partner.

  4. Explicitly-named entity in claim / ColBERT gap (1/8):

    • ex50 (Just Do It / Swoosh / Dan Wieden): Hop 4 queried Swoosh but ColBERT returned Nike-related near-misses, not the Swoosh article itself. Also Dan Wieden and Just Do It not found. Persistent across iter4, iter11, iter12. Multiple swoosh-related queries (Nike logo, Nike Swoosh) all fail to surface the base article.

Retry mechanism stats (iter12 child, 50 examples): Fired in ~10 examples. In failures: fired in ex37 (repeated same query), ex53 (pivoted to wrong entity), ex85 (repeated same query). In the other ~7 retry examples, the retry was net-positive (redirected to new productive query).

Confirmed fixed by context trimming (iter12 Round 2): ex31 (Dinner Party / Sojourner Truth). Sojourner Truth was at rank-7 of hop1 in the parent. Round-robin evicts hop1 rank-7 docs to position 25 (beyond 21-doc cap). With context2 showing only hop1_new[:6], the LM correctly identified Sojourner Truth as uncovered and issued a direct query. This fix is mechanistic and generalizes.

Context-trimming boundary analysis: With 4-hop round-robin, the slot allocation per hop is positions {1,5,9,13,17,21} for hop1 (6 slots), {2,6,10,14,18} for hop2 (5 slots), etc. The 7th doc from hop1 would land at position 25, definitively outside the 21-doc cap. The trimming to hop1_new[:6] and hop2_new[:5] and hop3_new[:5] exactly mirrors the round-robin eviction boundary. Any doc at rank ≥ 7 in hop1, rank ≥ 6 in hop2, rank ≥ 6 in hop3 would be evicted — and they're now also hidden from the LM's context. This is structurally correct and generalizes.

Hop 4 Slot Starvation: Mostly Resolved by Round-Robin Interleaving

Round-robin interleaving (iteration 4) successfully fixed the major slot starvation cases confirmed in iteration 3 (ex33 Rochester Hills, ex60 Ice Princess, ex93 Pablo Escobar, ex25 Warren Fu). 1 residual starvation case remains in the iteration 4 trace:

Conclusion: Round-robin interleaving is proven and effective. The remaining starvation case is actually a wrong-entity-chain failure in disguise.

previous_queries Mechanism: Improvement Over fruitless_queries But Not Fully Effective

The upgrade from fruitless_queries (zero-return only) to previous_queries (all queries) fixed several repetition cases (ex76 Delmer Daves, ex120 Afonwen). However, 3 query-repetition failures remain in the iteration 4 trace:

  1. ex92 (Shim Ji-ho / Green Chair): previous_queries IS populated with "Sundance Film Festival, 2005 Sundance Film Festival" at hop 4. The LM re-issues "2005 Sundance Film Festival" anyway — explicitly ignoring the no-repeat instruction. Shim Ji-ho never queried. 18 docs retrieved.

  2. ex125 (Jānis Strazdiņš / Bass voice type): Hops 2, 3, AND 4 all query "Jānis Strazdiņš". previous_queries at hop 4 literally shows "Jānis Strazdiņš, Jānis Strazdiņš". Key insight: this entity has NO Wikipedia article. When ColBERT returns nothing, the LM cannot distinguish "not yet retrieved" from "article doesn't exist" — it keeps retrying. Required article "Bass (voice type)" is never queried. Only 12 docs retrieved.

  3. ex126 (Harry Booth / On the Buses): Hops 3 and 4 both query the non-existent article "Thick as Thieves (1971 film)". "On the Buses" and "Pat Ashton" never queried.

Root cause: When a queried entity has no Wikipedia article (ColBERT returns nothing), the LM interprets the empty result as "I haven't found it yet" rather than "this article doesn't exist." The previous_queries field prevents repeating the exact string but not the underlying entity confusion. A possible fix: explicitly surface the retrieval result (0 docs) in the context so the LM can infer the article doesn't exist.

Dominant Failure Pattern: Wrong Entity Chains (9/15 failures in iter4 trace)

Wrong entity chain selection is now the single largest failure category. The LM correctly follows the chain for explicit entities in the claim, but frequently makes wrong turns:

Sub-patterns:

A. Manufacturer-instead-of-product disambiguation (1 case — ex54 Skittles): LM knows Skittles is the candy but queries "Mars, Incorporated" (manufacturer) instead of "Skittles (confectionery)". The required disambiguation suffix (confectionery) is never used.

B. Production-without-person (1 case — ex95 Felicity Jones / Rogue One): LM queries "The Comedy of Errors", "Oxford University Dramatic Society", "Rogue One" but never directly queries "Felicity Jones" herself. Claim requires her personal article.

C. Context entities instead of derived article (1 case — ex129 Stranger in Paradise): LM pursues Borodin → Kipchaks → Cumans (peoples mentioned in Polovtsian Dances) but never queries "Stranger in Paradise (song)" — the song adapted from Borodin's Polovtsian Dances.

D. Intermediate entity named in reasoning but not issued as query (1 case — ex82 Moonrunners): Hop 2 LM reasoning explicitly names "Moonrunners" but then queries something else. The required article is stated in the reasoning but abandoned.

E. Specific season/article not derivable from claim alone (1 case — ex141 NY Islanders season): Claim references NJ Devils + a building → LM queries "New Jersey Devils", "Madison Square Garden", "New York metropolitan area". The required article "1974-75 New York Islanders season" requires knowing specific historical facts not present in the claim text.

F. Hallucination — completely wrong entity identified (2 cases):

G. Wrong entity cascade filling all slots (3 cases — ex7, ex13, ex44): LM identifies plausible but non-gold entities and fills all 21 slots with wrong-chain documents before finding the required article.

Key insight: For chains requiring >1 hop of inference beyond what's in the claim text (e.g., "the person who X" when X's article is needed to identify the person), the current architecture fails because it processes entities greedily rather than pre-planning.

Second-Hop Scan Principle: Validated But Priority Overrule Is Dangerous

From iteration 5:

Failure Pattern Distribution: Iteration 5 Child Trace (13 failures from 50)

  1. Wrong entity chains / wrong article selection (~7/13): LM pursues plausible but non-gold entities:

    • ex3 (Elena Shaddow): Queries "Best Score and Best Orchestrations" awards category instead of "Elena Shaddow" directly
    • ex4 (Rosi Golan): Queries "Artsana" then "Mayaeni Strauss" (wrong Belgian co-composer) instead of "Rosi Golan"
    • ex18 (Guy Davis (comics)): After finding Crimson Peak, queries Guillermo del Toro → Pacific Rim instead of "Guy Davis (comics)"
    • ex24 (F.E.A.R.): Prioritizes "Jupiter EX engine" (second-hop) over "F.E.A.R." (primary game)
    • ex26 (Punchlines/Bill Cullen): Goes completely off-track querying Val Doonican articles; never finds Punchlines or Bill Cullen
    • ex33 (Felicity Jones): Queries Oxford Dramatic Society, The Comedy of Errors, Rogue One — but never directly queries "Felicity Jones" herself
    • ex40 (Charlotte Baldwin Allen): Queries "Augustus Chapman Allen" (co-founder of Houston) but skips "Charlotte Baldwin Allen" (the required article)

  2. Query repetition — LM ignores previous_queries (3/13):

    • ex30: Queries "Ronnie Flex" 3 times (all 3 hops). The artist's Wikipedia article is under "High Contrast" (Lincoln Barrett's stage name), not "Ronnie Flex" — LM can't distinguish "article doesn't exist under this name" from "not yet retrieved."
    • ex32: Queries "2005 Sundance Film Festival" in hops 2 AND 4 despite it being in previous_queries. Never queries "Shim Ji-ho."
    • ex43: Queries "The Broken Tower" and "The Broken Tower (novel)" in hops 2 AND 3. Never queries "Hart Crane."

  3. Wrong disambiguation suffix (2/13):

    • ex44: Queries "Space Cowboy (song)" — needs "Space Cowboy (musician)"
    • ex45: Queries "The Company (book)" — needs "The Company (Ehrlichman novel)"

  4. Sub-concept vs. parent article confusion (within above categories):

    • ex25 (Ancient Egyptian religion): Queries "Weighing of the Heart", "Lake of fire", "Book of the Dead" — all sub-concepts — but never "Ancient Egyptian religion" (the parent article). The second-hop scan was supposed to help here (it explicitly included "broader parent/overview article" as a case) but the LM still drilled into sub-concepts.

Key new observation from ex30 (Ronnie Flex / High Contrast): The claim contains the wrong name for the artist — it says "Ronnie Flex" but the Wikipedia article is "High Contrast" (real name Lincoln Barrett). This is a claim-error case: the LM should recognize that "Ronnie Flex" returns nothing and try "Lincoln Barrett" or "High Contrast". This class of failure (Wikipedia article uses a different name than the claim) is distinct from normal wrong-entity chains.

Failure Pattern Distribution: Iteration 7 Child Trace (Round 2, 14 failures from 50)

  1. Wrong entity chains / wrong article selection (7/14): LM pursues plausible but non-gold entities:

    • ex5 (Robinsons Galleria): Claim says "Metro Manila is in Turkmenistan" (intentionally false). LM queries Turkmenistan instead of pursuing Robinsons Galleria. Wrong entity introduced by a misleading claim — the LM partially follows the false context.
    • ex12 (Adam Guettel / How Glory Goes): Queries "Best Score and Best Orchestrations" (a Broadway award category) and "Elana Shaddow" (wrong spelling) → Audra McDonald. Never queries "Adam Guettel" (the songwriter) or "How Glory Goes" (the title song).
    • ex14 (Robert E. Howard): Correctly queries The Lay of the Land and Richard Ford. Hop 4 queries "California" instead of "Robert E. Howard." Wrong entity in hop 4.
    • ex30 (Additi Gupta / Ishqbaaaz): Queries Star Plus → Qubool Hai → Star India. Never queries the actress (Additi Gupta) or the required drama series (Ishqbaaaz).
    • ex39 (The Greatest Game Ever Played): Correctly queries Josh Flitter but then derives "Scalplock" and "The Iron Horse" (wrong films) instead of "The Greatest Game Ever Played."
    • ex50 (Swoosh / Just Do It / Dan Wieden): Queries Nike, Inc. → Wieden+Kennedy → Nike logo. Never issues the three required specific article names.
    • ex123 (Denmark v Sweden UEFA Euro 2008 / Christian Poulsen): Queries general "UEFA Euro 2008 qualification" and "Denmark national football team" instead of the specific match article and Christian Poulsen.

  2. Second-hop inference failure (3/14): LM retrieves the bridge article but fails to derive the next hop from reading it:

    • ex31 (The Dinner Party): Correctly retrieves Barbe de Verrue and Sojourner Truth but never derives "The Dinner Party" art installation from their text. This is the same failure as ex31 in iterations 5/6 — the second-hop scan principle should fix this.
    • ex82 (Moonrunners): Correctly retrieves Briscoe Darling Jr., The Dukes of Hazzard. Hop 4 queries "Uncle Jesse" (a character) instead of reading Dukes of Hazzard text to find "Moonrunners" (the 1975 film that inspired it). Persistent across iterations 4, 5, 6, 7.
    • ex122 (Winnie the Pooh 2011 film): Correctly retrieves LA Urban Rangers and Huell Howser. Hop 3 queries "Backson (Home on the Range)" — wrong disambiguation. Should query "Winnie the Pooh (2011 film)." Same failure across iterations 5, 6, 7.

  3. ColBERT retrieval gap (2/14): Correct entity queried but article not returned in top-7:

    • ex20 (Air Lituanica): "Air Lituanica" IS queried (hop 2) but the article isn't in the final 21 docs. Likely evicted by round-robin or ColBERT returned near-misses.
    • ex48 (Jonathan Lynn): "Jonathan Lynn" IS queried (hop 4) but ColBERT returns "Jonathan Griffiths", "George Lynn (composer)", "Wiley Lynn", "Uriel Lynn" — approximate-name matches — not the actual Jonathan Lynn article. Persistent across iterations 4, 5, 6, 7. This is a genuine ColBERT retrieval gap, not an LM failure.

  4. Ran out of hops after initial misstep (1/14):

    • ex85 (Flex club / Waves Vienna): Hops 2-3 wasted on "Ronnie Flex" (no Wikipedia article under this name). Hop 4 queries "Lincoln Barrett" → ColBERT returns "High Contrast" (found!). But "Flex (club)" and "Waves Vienna" are never queried — no 5th hop available. Also confirmed: retry mechanism fired in hop 3 but produced "Ronnie Flex" again (same entity, no Wikipedia article = LM's only candidate).

  5. Missing 2+ required articles, wrong chain from start (1/14):

    • ex53 (Douglas MacArthur's escape / Texas Raiders): Queries Douglas MacArthur (general biography, not the specific escape article) and Conroe North Houston Regional Airport twice (different punctuation). "Douglas MacArthur's escape from the Philippines" (specific article) and "Texas Raiders" (the specific B-17 at that airport) never queried. Duplicate-query on airport with slightly different punctuation bypassed the retry check.

Key new observation from ex53: The retry mechanism checks exact lowercase string match. "Conroe North Houston Regional Airport" vs "Conroe–North Houston Regional Airport" (with em-dash) are treated as different queries by the retry, even though they represent the same entity. The LM issued both forms, wasting a hop. Fuzzy/normalized duplicate detection could help.

Key new observation from ex48 (persistent): ColBERT for "Jonathan Lynn" consistently returns partial-name matches (Wiley Lynn, George Lynn, Uriel Lynn, Jonathans with different surnames) instead of the exact article. This is a ColBERT index/ranking issue. Possible fix: try "Jonathan Lynn (director)" or "Jonathan Lynn writer" as the query to improve precision.

Failure Pattern Distribution: Iteration 6 Child Trace (Round 2, 10 failures from 50)

  1. Wrong entity chains / wrong article selection (7/10): LM pursues plausible but non-gold entities:

    • ex1 (Matthew Bailes): Queries Swinburne University → Melbourne → Hawthorn, Victoria. Matthew Bailes (CAS director, a second-hop implied entity from the CAS article) never queried.
    • ex2 (Allan Goldstein): Queries Charles Bronson → Leslie Nielsen → The Naked Gun. Allan Goldstein (director of the comedy) never queried despite being derivable from the Naked Gun article.
    • ex24 (F.E.A.R.): Sierra Entertainment → F.E.A.R. (queried in hop 3) → Jupiter EX. F.E.A.R. WAS queried but ColBERT did not return its article in the top-7 results. Persistent failure across iterations.
    • ex25 (Airlines of Africa): KLM → OR Tambo International Airport → Sun Air (wrong BA franchise carrier in Denmark). "Airlines of Africa" (Comair subsidiary) never queried.
    • ex36 (Charlotte Baldwin Allen): Augustus Chapman Allen → Texas → History of Texas. Charlotte Baldwin Allen (female co-founder of Houston) never queried — the LM retrieves only the prominent co-founder.
    • ex47 (Isatis): Crepis (correctly queried) → Asteraceae → Brassicaceae. "Isatis" — an entity explicitly named in the claim — is never directly queried; LM drills into family articles instead.
    • ex48 (Beyond Rangoon): Victor Slezak → Johnny Depp → Ed Wood. The unnamed 1995 film (Beyond Rangoon) is never reached; LM pursues the "Johnny Depp" connection instead.

  2. Retry mechanism activated but entity not identifiable (1/10):

    • ex39 (Winnie the Pooh 2011 film): 5 LM calls (2 retries triggered). "Backson" queried 3 times before retries redirected. LM issues Backson → Backson → LA Urban Rangers → Backson → Huell Howser. "Winnie the Pooh (2011 film)" requires knowledge not derivable from the claim text and is never queried. Retry mechanism correctly fired on repeats but cannot fix the root entity-identification failure.

  3. Correct query issued but ColBERT does not return article (1/10):

    • ex16 (Jonathan Lynn): Hops 2-4 query Antony Jay → Yes, Prime Minister → Jonathan Lynn. Jonathan Lynn IS queried in hop 4 but "jonathan lynn" is not in the 21 retrieved docs. This may be a ColBERT retrieval gap (query "Jonathan Lynn" returns 7 docs but not the Jonathan Lynn article itself) or a round-robin slot displacement.

  4. Specific historical season/article not inferable from claim (1/10):

    • ex45 (1974-75 New York Islanders season): New Jersey Devils → Madison Square Garden → New York metropolitan area. The claim says "New York Islanders" generically but the required article is a specific historical season that cannot be derived from the claim alone.

Key new observations from iter6 trace:

Failure Pattern Distribution: Iteration 8 Child Trace (5-hop, 7 failures from 50)

  1. Reasoning-to-query conversion failure (2/7): LM reasoning correctly identifies the entity but issues a different query:

    • ex95 (Felicity Jones): After retrieving Rogue One (hop 4), hop 5 reasoning correctly identifies Felicity Jones as needed but issues "Game of Thrones" as the actual query — a reasoning-to-query conversion failure. Rachelle Beinart's article mentioned "Game of Thrones stunt teams" and distracted the LM. Gold articles (Rogue One retrieved, Felicity Jones and Rachelle Beinart not found).
    • ex146 (Ellis Ferreira / 1997 Stockholm Open): 1997 Stockholm Open Doubles article says "Galbraith partnered Ellis Ferreira, losing in the final." LM instead concluded Mikael Tillström was Kulti's 1997 partner (confusion with 1999 data). Ellis Ferreira was never queried as the correct doubles partner.

  2. Wrong disambiguation from hop 1 (1/7): Wrong article retrieved in hop 1, all subsequent hops follow wrong path:

    • ex117 (This Is England / The Secret Agent TV series): Hop 1 retrieved "The Secret Agent (film)" — the 1996 Christopher Hampton film starring Bob Hoskins — instead of "The Secret Agent (TV series)" with Stephen Graham. All 5 hops followed the wrong path. None of the 3 gold articles (Stephen Graham, This Is England, The Secret Agent TV series) were found.

  3. Deep ColBERT chain gap (2/7): Correct entities queryable, but no retrieved passage bridges to the gold article:

    • ex129 (Stranger in Paradise / Polovtsian Dances): LM queries Borodin → Kipchaks → Cumans → Prince Igor (all reasonable). No retrieved article mentions "Stranger in Paradise" (song from Kismet musical adapted from Polovtsian Dances). The bridge between Polovtsian Dances and "Stranger in Paradise" is not in any retrieved text.
    • ex141 (1974-75 New York Islanders season): LM queries New Jersey Devils → Madison Square Garden → NHL → NY metropolitan area. The specific historical season article requires inferring a year from context not present in the claim; the LM never narrows to "1974-75 New York Islanders season."

  4. Round-robin displacement + query repetition (1/7): Gold article retrieved in early hop but displaced from final 21 by repeated hop 5:

    • ex14 (Robert E. Howard / The Lay of the Land): Hops 1-3 retrieve correctly. Hops 4 and 5 both query "Frank Bascombe" (fictional character from The Lay of the Land), each returning 0 new docs. Robert E. Howard was at rank 7 of hop 1; with 5-hop round-robin each hop only gets 4 guaranteed slots, so hop 1's rank-7 doc is at position 25+ in round-robin order and falls outside the 21-doc cap. The model sees Robert E. Howard in retrieved_passages and (incorrectly) declares the entity covered in its coverage check, then wastes hops 4 and 5 on the fictional character.

  5. ColBERT entity name mismatch (1/7): Correct entity identified but wrong title form used:

    • ex32 (Mars, Incorporated / Darkened Skye / Skittles): LM correctly traces chain to the Mars company but queries "Mars, Inc." — ColBERT returns Mars Canada, Mars Music, Mars family, but NOT "Mars, Incorporated" (the exact Wikipedia article title). Abbreviation fails despite correct chain reasoning.

Key new observations from iter8 trace:

Key Technical Observations (updated post iteration 25)

Highest-Leverage Opportunities (ranked, post iteration 25)

CRITICAL: Build on candidate 14 (iter21, valset 0.8748 = current best). Do NOT build on candidate 17 (iter26, valset 0.8351), candidate 16 (iter25, valset 0.8252), or candidate 10 (iter16, valset 0.8152). Do NOT build on candidate 5 (iter8, unconditional 5-hop). Do NOT attempt pre-planning again without code-level entity validation. Do NOT use CRITICAL PERSON COVERAGE RULE in universal form. Do NOT add COMPARISON/RANKING bullet in isolation without further isolation testing — iter26 data suggests it causes ~0.04 valset regression.

  1. Conditional 5th hop (only run when needed) DONE in iteration 14 (candidate 9, valset 0.8450).

  2. Asymmetric k: hop1=6, hops2-4=5, hop5=12 DONE in iteration 18 (candidate 11, valset 0.8550).

  3. Sub-article false-coverage fix DONE in iteration 18 (Robert E. Howard bibliography class). ⚠️ Needs patch PATCHED in iteration 20: type-qualifier suffix coverage check fixed — "X (film/TV series/video game/artwork)" now correctly recognized as covering X. Full coverage check is now correct for sub-pages, type-qualifiers, name-form variants, and disambiguation suffixes.

  4. Step 1 claim-verbatim self-check DONE in iteration 18.

  5. Multi-membership bullet for multi-org membership claims DONE in iteration 19 (Glenn Hughes fix confirmed; generalizes to multi-org membership claims).

  6. Fix disambiguation suffix coverage check bug (iter19 discovery) DONE in iteration 20: Step 2 now explicitly allows type-qualifier parentheticals while blocking sub-page suffixes. Confirmed fix: The Dinner Party (artwork). Valset improvement: +0.0001 (negligible — the class of examples requiring rare type qualifiers is small in the valset).

  7. Fix parent company bullet ordering to prevent skipping intermediate product article ATTEMPTED in iteration 23 (ORDERING RULE added to parent company bullet in candidate 14). Result: test inconclusive — ex32 (Darkened Skye/Skittles) was not in the iter23 subsample seed, so the fix was not directly tested. The ORDERING RULE is in the current program (candidate 15) and candidate 14's architecture already has the standard parent company bullet. If Skittles/Mars class persists on the valset, consider reinforcing the ordering rule.

7b. Full legal name variant coverage fix (iter24 confirmed structural) DONE in iteration 25 (carried from iter24) and retained in iteration 26. The fix "Boris Becker covers Boris Franz Becker" is confirmed structural and low-risk. When building on candidate 14 in the next iteration, include this fix as a standalone addition. Note: iter26 retained this fix alongside the COMPARISON/RANKING bullet; the valset was 0.8351 vs candidate 14's 0.8748 — the delta is attributed to COMPARISON/RANKING, not the formal-name fix.

  1. Tighten hop5 firing condition to reduce ~95% fire rate: Hop5 fires in 95.5% of examples (iter24 trace: 42/44 passing used 5 searches — highest measured rate), paying the 0.002 penalty nearly universally. A tighter condition: only fire hop5 if len(hop4_new) == 0 (hop4 returned zero new unique docs) OR len(hop4_new) < 3. Targets genuine hop-exhaustion while skipping examples where 4 hops already retrieved all docs. Expected to reduce fire rate to ~20-30%, recovering ~0.001 avg score. Low risk (non-firing examples fall back to 4-hop round-robin). This remains the highest-priority untested structural refinement — never tested in 25 iterations.

  2. Code-level hop1 disambiguation fix for TV/film confusion (ex117 class): Disambiguation suffix guidance in Step 3 has minimal practical uptake. ex117 (Secret Agent TV series) is a persistent hop1 failure — ColBERT returns the 1996 film instead of the 2016 TV series. Consider a code-level post-hop1 check: if the claim implies a TV series and hop1 returned a film article for that title, retry hop1 with "(TV series)" appended. Medium complexity; high value for this class of failure.

  3. ColBERT disambiguation suffix strategy for persistent ranking failures: Moonrunners (8+ iterations), Swoosh (7+ iterations), Asymptotic safety in quantum gravity (persistent), Airlines of Africa (persistent) remain unresolved. F.E.A.R. was fixed by the CRITICAL type-qualifier enforcement in iter21 (but stochastically regressed in iter23 — compliance is ~85%). Consider a code-level fallback in _get_query_with_retry: when a query is fruitless (returns no new unique docs), automatically retry with a Wikipedia disambiguation suffix. Medium risk.

  4. Partner extraction Step 4 example refinement: Renáta Tomanová (false-premise claim, persistent 8+ iterations) and Ellis Ferreira remain unresolved. For false-premise partner claims (e.g., "Boris Diaw partner in mixed doubles"), the LM is confused because the named person (Boris Diaw the NBA player) doesn't exist in tennis context. Hard to fix. For legitimate partner extraction failures (Ellis Ferreira), explicitly specify "output the partner — find the TWO participants in the event article, output the one who is NOT already retrieved."

  5. Host/presenter extraction from show article (Bill Cullen / Pat Ashton pattern): HOST/CAST MEMBER bullet was added in iter23 to candidate 15, but did NOT fix ex126 (On the Buses/Pat Ashton) despite Thick as Thieves TV series being retrieved. The Thick as Thieves article may not prominently name Pat Ashton in the leading text snippet, or the LM deprioritizes the host-scan when other claim entities dominate. Options: (a) strengthen the HOST/CAST MEMBER bullet wording to require the LM to explicitly scan for star/host credits in any retrieved show article; (b) add a concrete example that exactly matches the "TV series names its star" pattern; (c) investigate whether the Thick as Thieves Wikipedia article actually names Pat Ashton prominently. Still unresolved after iter23 — bullet in place but not firing.

  6. Song/adaptation extraction from musical article (ex129 Stranger in Paradise): Kismet (musical) IS in retrieved titles across multiple seeds, but "Stranger in Paradise (song)" is never extracted. The LM pursues Borodin → Kipchaks → Prince Igor instead of reading the Kismet article to find the adapted song. Consider a new Step 4 bullet: "the specific SONG, ARIA, or MUSICAL PIECE named in a retrieved musical or theatrical article as an adaptation or standout number — e.g., if a retrieved musical article mentions 'Stranger in Paradise' as its hit song (adapted from Borodin), query that song's article directly." This is a distinct pattern from the CREATOR bullet (which targets people) and from the existing named-song bullet (which may not be specific enough). Not yet attempted — persistent 10+ iteration failure.

  7. Air Lituanica — defunct entity inference: The LM retrieves Vilnius Airport and AirBaltic but does NOT infer that a defunct airline (Air Lituanica) needs its own article. Hop5 never fires because the LM declares all entities covered (ex85 and ex92 both showed only 4 searches used — LM declared coverage complete). A Step 4 defunct entity pattern was added in iter20 but did NOT fix this — the Vilnius Airport article text doesn't appear to prominently mention Air Lituanica. Alternative approach: code-level check that fires hop5 when only 4 searches were used AND the claim explicitly mentions an entity that "ceased operations" or "was dissolved." Medium complexity.

  8. Charlotte Baldwin Allen (ex114) — CO-FOUNDER bullet non-compliance: CO-FOUNDER bullet added in iter23 but did not fire. Two hypotheses: (a) Augustus Chapman Allen's Wikipedia article body text doesn't mention Charlotte Baldwin Allen prominently enough in the retrieved snippet; (b) the LM doesn't recognize the co-founding context when reading the Allen article. If (a), no prompt fix is possible — need a different retrieval path (e.g., querying "Charlotte Baldwin Allen" directly if "Allen Houston" is retrieved). If (b), consider strengthening the CO-FOUNDER bullet with a more explicit trigger or concrete example. Would need to verify whether Charlotte Allen appears in the Augustus Allen article snippet.

  9. Shim Ji-ho (Sundance festival anchor) — systemic loop: The LM repeats "2005 Sundance Film Festival" 4+ times despite retry mechanism. In iter23, hop5 didn't even fire (only 4 searches). The Green Chair and Suh Jung articles are found but Shim Ji-ho (the co-lead actor) is never reached. Possible fixes: (a) code-level count check — when a normalized query appears in previous_queries 2+ times, force Step 4 body-scan mode only; (b) Step 4 bullet for co-lead actor extraction from a retrieved film article when the claim implies a second principal actor. Medium complexity.

  10. Misleading claim entity discrimination: "Charpes Lane" (typo), "Nigerian midfielder" (wrong nationality), "Boris Diaw" (wrong sport), garbled birth names cause systematic failures. Hard to fix without claim veracity understanding.

  11. ⚠️ Do NOT attempt pre-planning again without code-level validation: Confirmed failed twice (iter6 strong priority, iter12 soft advisory). The LM at reasoning_effort=low cannot reliably extract claim entities.

  12. ⚠️ Fruitless detection does NOT fix ColBERT ranking gaps: Isatis query returns Isatis tinctoria (new unique doc, not marked fruitless). For ranking gaps, disambiguation suffix retry is the correct fix.

  13. ⚠️ Do NOT increase k above the slot count for any hop: Confirmed by iter16 Round 1. Only the LAST hop that actually fires can use k=12. Correct limits: hop1=6, hops2-4=5, hop5=12.

  14. ⚠️ Do NOT reuse the Step 4 director/actress/choreographer bullet from iter16: Confirmed caused valset regression (0.8450→0.8152). Only re-introduce with a tighter trigger condition.

  15. ⚠️ Do NOT attempt LM reranker full 21-slot selection with gpt-5.4-nano at reasoning_effort=low: Confirmed failed (iter22). The model cannot reliably select 21 precise indices from 25-30. A conservative reranker (only last 2-3 slots) or code-level scoring heuristic is the correct approach.

  16. ⚠️ Always build on candidate 14 (iter21, asymmetric-k architecture) for next iteration targeting best valset: Candidate 15 (iter23, 0.8550) and candidate 16 (iter25, 0.8252) are both below best. Iter22 was built on candidate 9 (k=7 uniform) and caused regressions. Prompt improvements coupled to asymmetric-k CANNOT be ported to k=7 without re-validation.

  17. ⚠️ New Step 4 bullets carry cumulative over-triggering risk: Iter23 added 4 new Step 4 bullets and the valset dropped −2.3% despite a subsample gain. Iter24 added 2 new bullets (CREATOR + COMPARISON/RANKING) to candidate 14 and again caused ~5/50 soft regressions (examples shifting from 4-hop to 5-hop). The cumulative pattern is clear: each pair of new body-scan bullets costs ~5/50 examples in soft penalty regressions. When adding multiple new body-scan bullets, the LM may apply some spuriously when no matching entity exists in the retrieved article — wasting a hop. Future bullet additions should be targeted at classes confirmed to have the target entity prominently in retrieved article text. Avoid adding more than 1-2 bullets per iteration. Consider whether each bullet genuinely generalizes beyond subsample examples before including it — CREATOR and COMPARISON/RANKING bullets appear useful for specific confirmed cases but may be too broad for general use.

  18. Superlative/unique-descriptor entity identification (iter22 observation, partially addressed by COMPARISON/RANKING bullet in iter23): The COMPARISON/RANKING bullet fixed ex18 (Heathrow Airport) in iter23 subsample but FAILED to fix it in iter25 subsample (LM queried broad entities first, exhausting hop budget before body-scan). Step 1 extension would help more directly: "if the claim contains 'the busiest X', 'the only Y in Z', 'the largest W' describing an entity NOT explicitly named, identify the specific Wikipedia article for that entity by its own name." This puts the superlative resolution in Step 1/3 (where the LM directly queries "Heathrow Airport") rather than deferring to Step 4 body-scan (which may not execute in time). Medium-high confidence this would fix the Heathrow class of failures.

  19. ⚠️ Do NOT use CRITICAL PERSON COVERAGE RULE in universal form: Iter25 confirmed: the rule "org/film/work article naming a person does NOT cover that person's biographical article" causes −5.7% valset regression when labeled CRITICAL and applied universally. The Matthew Bailes failure class requires a CONDITIONED version: "query a person's biographical article ONLY IF the claim explicitly mentions that person by name or implies them through a direct claim relationship (e.g., 'the director of X' when X is retrieved) — NOT just any person mentioned as a role-holder in a retrieved article." The conditioned version could be tested as a non-CRITICAL rule with specific examples to guide activation.

  20. ⚠️ COMPARISON/RANKING bullet likely causes ~0.04 valset regression when used alone: Iter23 (candidate 14 + 5 bullets including COMPARISON/RANKING) got valset 0.8550 (−0.0198 vs candidate 14). Iter26 (candidate 14 + formal-name fix + COMPARISON/RANKING alone) got valset 0.8351 (−0.0397 vs candidate 14). The formal-name fix is expected ~neutral. This implies the COMPARISON/RANKING bullet alone costs approximately 0.04 valset points — likely because it triggers broadly on any comparison text in retrieved articles (not just the exact pattern the claim describes). If retested, must be paired with a stronger trigger condition: "ONLY if the retrieved article EXPLICITLY names the comparison entity using a superlative matching the claim's description (e.g., 'X is the busiest airport in the UK') — NOT general comparative mentions." Do NOT add as a standalone change to candidate 14 without first reformulating the trigger condition.

  21. Tightly-conditioned film→performer bullet (iter26 new observation): Iter26 trace confirms: Rogue One IS retrieved in hop4, hop5 is available for querying Felicity Jones, but hop5 chose "Game of Thrones" instead. The film→performer bullet (removed after iter16's −3.5% valset regression) is needed but must be conditioned: only fire when (a) the claim explicitly implies "the actress/actor who starred in [film]" AND (b) the specific film article is in retrieved_docs from hops 1-4. The iter16 version triggered too broadly on ANY retrieved film article. A conditioned version with a claim-relevance check could fix the Felicity Jones class without broad regression. Medium-high priority — the film (Rogue One) is now consistently found in hop4 across multiple seeds, making the timing correct for a hop5 person query.