CodeEvolver Memory

memory last updated at iteration number: 24

Run id: 20260615073201 Seed branch: codeevolver-20260615073201-main Seed architecture: DSPy ReAct Agent with ColBERT Retrieval over PhantomWiki Fictional QA Corpus

Overall Progress

Current Program Valset Score

0.6118 — iteration 24 ACCEPTED on subsample (parent 0.6463 → child 0.6994, +8.2%), but the valset scored 0.6118 — a major regression (−11.0%) from the all-time best 0.6867 and −7.6% from iteration 23. The current live program is candidate 18 (iteration 24), branch codeevolver-20260615073543-62729b. Changes: Phase 1 output deduplication + Phase 2 random sampling (random.sample(entities, 20) with per-question deterministic seed). The large subsample gain was driven by stochastic wins on unchanged entity/attribute paths (do not generalize) and a slightly net-negative structural count_pivot effect. Random sampling introduces too much per-question variance: correct pivots are missed just as often as spurious pivots. All next iterations should branch from candidate 15 (iteration 21), NOT candidate 18.

Best Program Valset Score

0.6867 (iteration 21, candidate 15). Branch: codeevolver-20260615073543-c7abb8. Previous best was 0.6526 (iteration 18, candidate 13).

Macro Approaches

Approach 1: Enriched DSPy Signature + Retrieval Parameter Tuning

Approach 2: Workspace Scratchpad Tools (note_finding / recall_findings) + Mandatory Enumeration Rules

Approach 3: Enriched BFS Docstring + Kinship Term Definitions + k=12 (Iteration 3 — Incremental)

Approach 4: Second Aunt/Uncle Fix + Conditional "How Many" Multi-Count Guidance + NEVER ABANDON + Generation Level Tracking (Iteration 4 — Incremental, REJECTED)

Approach 6: Exact-Match DOB Index Tool — search_by_date_exact (Iteration 8 — CURRENT BEST)

Approach 7: Two-Phase ReAct Architecture — EntityFinder → AnswerComputer (Iteration 10, Attempt 1 — FAILED)

Approach 8: Single-Phase with All Proven Wins Restored (Iteration 10, Attempt 2 — ACCEPTED, valset 0.5533)

Approach 9: Hybrid Two-Phase Architecture — Two-Phase ONLY for count_pivot, Single-Phase for All Other Types (Iteration 12 Round 2 — CURRENT BEST, valset 0.6417)

Approach 11: ANCHOR HINT Removal — Surgical Revert for Complex Chain Count Pivot (Iteration 14 REJECTED → Iteration 15 ACCEPTED, valset 0.6343)

Approach 12: DSPy InputField as Structured Anchor Data for Phase 1 (Iteration 15 Round 1 — CATASTROPHICALLY FAILED)

Approach 13: Exact Hobby/Occupation Index Tools Exposed to Phase 1 (EntityFinderSig) (Iteration 16 — VALSET REGRESSION despite subsample gain)

Approach 10: Python-Level Hobby/Occupation Index Routing — Simple Anchor Bypass + ANCHOR HINT (Iteration 13 — VALSET REGRESSION)

Approach 15: Entity-Question Routing Through Phase 1 (EntityFinderSig) (Iteration 18 — NEW BEST valset 0.6526; Iteration 19 — Prompt Incremental, Valset REGRESSED)

Approach 16: Additive Reasoning Rules in EntityFinderSig + CountComputerSig (Iteration 21 — NEW BEST valset 0.6867)

Iteration 24: Incremental — Phase 1 Deduplication + Phase 2 Random Sampling (MAJOR VALSET REGRESSION)

Iteration 23: Incremental — Phase 1 Deduplication + Phase 2 cap 20→50 (VALSET REGRESSION)

Iteration 22: Incremental — entity_finder_entity (max_iters=60) + Phase 2 cap 20→30 (VALSET REGRESSION)

Approach 14: Attribute_Deep Two-Phase Extension — Phase 1 (EntityFinderSig) + Phase 2 (AttributeAnswerSig) (Iteration 17 — REJECTED)

Approach 5: Dual Retrieval Tools (search_wiki_broad k=30) + HOW MANY Format Fix + Minimal Targeted Additions on Clean Iteration 1 Base (Iterations 5–6 — CURRENT BEST)

Iteration 6 extension (built on iteration 5, ACCEPTED, valset 0.5425→0.5530, +1.9%): Added MULTI-ENTITY ENUMERATION prompt section (9 lines) instructing model to search BOTH maternal/paternal branches for ancestor anchors, issue 2+ follow-up searches for DOB/hobby anchors, enumerate all friends/siblings before counting. Also increased retrieve_broad from k=30 to k=50. Key wins: second uncle of Rosina Robey (iteration 5 entry 16/iteration 6 entry 17) finally fixed at 1.0 — model searched both ancestor branches and found both Alvaro/Joesph Sharma. Female first cousin once removed of Sara Leyva (iteration 5 entry 39/iteration 6 entry 38) fixed at 1.0 — all 5 correct entities found. Friend-of-great-grandparent count (entry 4): 7 of 9 counts correct (0.875). Key regressions introduced: (a) Tautological DOB question (entry 27): "What is the DOB of person whose DOB is 0954-03-04?" → model now searches for all people with that DOB instead of returning the value directly; returned [] vs gold '0954-03-04'. (b) Over-counting when multiple grandfathers found (entry 42): model reports counts for EACH grandfather independently (['2','1'] vs gold ['2']), a new multi-entity enumeration failure mode where the model over-applies counting instructions to every resolved intermediate. k=50 for broad search showed no improvement for DOB fan-out failures (entries 12, 22, 35, 40) — ColBERT ceiling confirmed at k=30 already.

Iteration 7 extension (built on iteration 6, ACCEPTED on subsample, valset 0.5530→0.5464, −1.2% regression from best): Three code changes and three prompt changes: (1) Added search_by_date(date_str) tool that decomposes date into year, year-month, and full-date components and issues 5 query formats ("born YYYY", "date of birth YYYY", "YYYY-MM", "born YYYY-MM-DD", "YYYY-MM-DD") against retrieve_broad (k=30), deduplicating all results; (2) Reverted k=50→k=30 for search_wiki_broad (iteration 6 confirmed no benefit at k=50); (3) New ATTRIBUTE FAN-OUT section requiring 5+ varied query phrasings for attribute-anchored questions before computing any count; (4) Second cousin clarification (goes UP to grandparent level, NOT great-grandparent — avoid confusion with second uncle); (5) Tautological DOB short-circuit rule ("if question asks for DOB of person whose DOB is X, return X directly"). Subsample: +2.8% (0.5363→0.5611). Key subsample wins: example 64 (hobby fan-out: 0.22→1.0 — ATTRIBUTE FAN-OUT guidance worked, model issued multiple hobby queries and found all 4 hobbies); example 26 (0.0→1.0 — correct count computed). Key subsample regressions: example 58 (stochastic failure: 0.57→0.0 — model retrieved correct intermediate entities but drilled into irrelevant branch), example 20 (0.2→0.0 — deep kinship chain returned empty). Trace confirmed: search_by_date is called correctly (first tool call in DOB-anchored questions), but even with 5 varied query formats at k=30, ColBERT returns year-neighborhood documents ranked by semantic proximity — exact-date matches beyond top 2-3 are outranked by near-miss dates (e.g., "0918-01-16" crowding out "0918-01-17"). Example 27 still finds 2/7 people; example 79 still finds 1/9 — identical recall to k=30 without the tool. The valset regression despite subsample gain is consistent with either noise (subsample is 50 rows, seed changed between iterations 6→7) or mild overfitting to the specific subsample's DOB/hobby examples. The k=30 revert is a clean no-cost improvement (k=50 was confirmed worse, not just equal).

Observations

What Works Well

Generation-Level Confusion — Partially Resolved in Iteration 5

A systematic "off-by-one generation" error was identified in iteration 3 and persisted through iteration 4. Iteration 5 shows partial resolution:

Carlotta Freeland "How Many Great-Grandsons" — FIXED in Iteration 15

"How many great-grandsons does Carlotta Freeland have?" → gold=['1']. FIXED in iteration 15 (Row 7/ex_id=14: pred=['1'], 1.0). This was 0.0 in iterations 3–14 for the count form — the model found Frederic Koons as the great-grandson but then went one generation too deep looking for Frederic's children. The fix came naturally with the ANCHOR HINT removal and seed change; no targeted prompt change was needed. The question now routes through count_answer (single-phase), and the model correctly returns the count of 1. Note: "Who is the great-grandson of Carlotta Freeland?" (entity form) was also already working (1.0 since iteration 5). Both forms are now solved.

Iteration 9 Key Finding: "Enumerate-then-Collapse" Confirmed at Reasoning Level

Iteration 9 trace inspection on the iteration 8 parent program revealed: the model correctly lists all intermediate entities in its Chain-of-Thought (e.g., all 15 people born on a given date, all 4 friends of a person, all financial controllers) but then picks only one entity to process and calls finish(). This is the "enumerate-then-collapse" failure — the knowledge of the full entity set exists in the model's reasoning, but iteration over the full set does not happen. This confirms that the failure is NOT a retrieval problem (the model has the entities) but a control flow problem (the model doesn't loop over them all in the action phase).

Both MULTI-COUNT ITERATION section (−5.8%) and ATTRIBUTE FAN-OUT alone (−6.8%) caused regressions on the iteration 8 parent because they induced over-searching on simple single-count questions — the instructions cannot discriminate "implicit multi-entity anchor requiring N separate counts" from "simple single-entity anchor requiring 1 count." Every round of prompt instructions to iterate has either over-applied (false positives on simple questions) or under-applied (model ignores instruction for some cases).

"How Many" Questions — Two Distinct Sub-Problems (Updated Through Iteration 12)

Iteration 5 revealed a critical distinction between two "how many" failure modes:

Sub-problem A: FORMAT BUG (FIXED in iteration 5) — Model computes correct count but outputs entity names instead of the count string. HOW MANY FORMAT RULE fixed this. Entries 0, 22, 28, 44 all moved from 0.0→1.0. This is now resolved.

Sub-problem B: MULTI-PATH FAN-OUT (LARGELY UNRESOLVED through iteration 8) — Questions where there are N distinct intermediate entities, each requiring a separate count. The model computes 1 count for 1 entity and stops. This pattern persists through all iterations:

Intermediate Entity Answer Contamination (Persists Through Iteration 10)

When traversing multi-level kinship chains, the model sometimes returns attributes of INTERMEDIATE entities instead of the final target. This has been confirmed across iterations 3, 4, 5, and 6:

Date-of-Birth Lookup — STRUCTURALLY SOLVED by Exact-Match Index (Updated Through Iteration 8)

ColBERT semantic search cannot match exact date strings like "0946-07-14". After four failed semantic interventions (k=30, k=50, 5-format search_by_date, iteration 7), iteration 8 added search_by_date_exact(date_str) backed by a pre-built 494MB JSON exact-match index. This is now solved for direct DOB→attribute lookups.

Prior semantic failure history (iterations 5–7):

Iteration 8 — exact-match index results:

Remaining DOB challenges (post-exact-index):

Pattern summary: The exact-match index completely fixes DOB recall (the core structural failure). What remains is that DOB-anchored questions requiring 2+ downstream hops (DOB → friends → their attribute) still suffer from the same multi-count fan-out problem that affects all multi-hop questions.

Non-Standard Kinship Terms — Partially Fixed, Partially Still Failing (Updated Through Iteration 6)

Iteration 3 added kinship definitions. Iteration 4 fixed second aunt/uncle. Iteration 5 kept the second aunt/uncle definition and added IMPLICIT RELATIONSHIPS guidance. Iteration 6 added multi-entity enumeration with bilateral ancestor search. Results:

Phase 1 Budget Exhaustion on Deep Kinship Chains (New Failure Mode — Iteration 16)

When Phase 1 (entity_finder) has access to exact hobby/occupation tools, it gets many anchor persons (up to 30+ from the index). If the kinship chain from anchor to pivot is 3+ hops, Phase 1 exhausts its max_iters=40 budget attempting multi-hop traversal from too many anchors and returns empty target_entities. The system then falls back to single-phase, which also typically fails for these questions. This is a new failure mode introduced in iteration 16 that did not exist before (previously, Phase 1 found only 1-3 anchors via ColBERT, so traversal budget was not the bottleneck).

Confirmed in ex_id=102 (iteration 16 trace): "How many sisters does the niece of the great-granddaughter of the person whose hobby is gongoozling have?" Phase 1 correctly called search_by_hobby("gongoozling"), then issued 8+ compound ColBERT queries attempting to resolve the 3-hop chain (anchor→great-granddaughter→niece), exhausted its budget, called finish() with no arguments → empty target_entities → single-phase fallback → also returned [] → score=0.0.

The compound query problem: Phase 1 tries ColBERT queries like "niece of the great-granddaughter of gongoozling person" — which ColBERT cannot resolve as a multi-hop kinship query. The correct approach is one-hop-at-a-time: (1) get all gongoozling persons, (2) for each, find their great-granddaughters, (3) for each great-granddaughter, find her nieces. But 30+ anchors × 2 hops = up to 60+ individual lookups, far exceeding max_iters=40.

Implication: The max_iters=40 cap for Phase 1 is now too tight when the anchor population is large AND the chain is deep. Either increase max_iters or reduce the anchor population before Phase 1 traversal.

Entity Questions — No Longer Weakest Type After Iteration 18 Routing Fix

In the iteration 14 subsample (50 examples), entity questions scored dramatically lower than other types:

Entity question failures:

Iteration 15 trace (50 examples) — question type breakdown:

Iteration 16 trace (50 examples) — question type breakdown (seed changed again):

Iteration 18 trace (50 examples) — entity questions through Phase 1 (NEW):

Iteration 19 trace (50 examples) — GENDER QUALIFIER + SINGULAR FORM additions to EntityFinderSig:

DSPy Signature InputField Invasiveness — Critical Constraint (New Finding, Iteration 15)

Adding a new InputField to a DSPy Signature changes the LLM prompt structure for every call through that module, including calls where the field is empty. This caused −14.8% subsample regression in iteration 15 Round 1. Key lessons:

Output Format

Major Opportunities

  1. DONE: ANCHOR HINT freetext injection removed (iteration 15, candidate 11, valset 0.6343). The change is now in the live program. Partial recovery from 0.6153→0.6343. Gap to best: 0.6417−0.6343=0.0074. The current program (iteration 15) starts from iteration 13 candidate 10 with ANCHOR HINT removed. This is now the base for next iterations.

  2. TRIED (iteration 16) — VALSET REGRESSION: Exact hobby/occupation tools as Phase 1 tools (see Approach 13): Adding search_by_hobby/search_by_occupation to Phase 1's tool list caused valset regression (0.6343→0.6217, −2.0%) despite strong subsample gain (+4.51%). Root causes: (a) corpus/Prolog mismatch amplified by 100% anchor recall — exact index returns ALL hobby/occupation persons including those outside Prolog's closed-world, generating spurious Phase 2 counts; (b) Phase 1 budget exhaustion for deep chains — with 30+ anchor persons, Phase 1 exhausts max_iters=40 attempting multi-hop kinship traversal and returns empty target_entities. The structural idea remains sound; the implementation needs refinement. Priority next steps: (i) Python-level anchor pre-sampling in forward() — look up anchor persons via the exact index in Python BEFORE Phase 1 (not inside Phase 1), take a random sample of N=10-15, and pass only those to Phase 1 as a simple Python-controlled input (e.g., injecting a subset list into Phase 1's context without DSPy InputFields); (ii) increase Phase 1 max_iters (40→60 or 40→80) to give more budget for deep chain traversal; (iii) cap the tool's return size (return only first 20 matches) to prevent Phase 1 from being overwhelmed. Critical: do NOT implement as a DSPy InputField (iteration 15 catastrophic failure). Critical: do NOT expose these tools to the single-phase react (iteration 13 Round 1 failure). The current program (iteration 16) has these tools in Phase 1 — they may need to be REMOVED if the refinements cannot fix the regression.

  3. Corpus/Prolog Over-Enumeration in Phase 1 (count_pivot): Phase 1 sometimes finds MORE pivot entities than the Prolog used to generate the gold answer. E.g., Ex 20: found 4 great-grandfathers, gold has 2 — the other 2 exist in the text corpus but weren't in the Prolog query's closed-world scope. Ex 43: found ~20 clinical research associates from the corpus but only a subset appear in the Prolog query result. This produces spurious counts that dilute F1. Mitigation options: (a) deduplicate Phase 1 entity list (already done via set, not helpful here), (b) cap Phase 1 to entities that appear in ≥2 independent search queries (confidence filter), (c) note that this is a fundamental tension between the open-world text corpus and the closed-world Prolog ground truth — may not be fully fixable with the current approach. DON'T OVER-INVEST in fixing this specific sub-problem unless Opportunities 1–2 are already in place and this is the next main failure cluster.

  4. Attribute Question Fan-Out — Two-Phase Extension to Attribute Type (TRIED in iteration 17, FAILED — Phase 1 bottleneck confirmed): Iteration 17 added attribute_deep routing (attribute questions with ≥2 "of the" + no "person whose" anchor) dispatched to Phase 1 (EntityFinderSig) → Phase 2 (new AttributeAnswerSig). The architecture is correct but Phase 1 is the bottleneck: EntityFinderSig returned the same 3 wrong entities as single-phase for the Deon Gall question (gold=12). SIBLING CROSS-REFERENCE instruction in Phase 1 made it worse (0.0). The approach needs a fundamentally better Phase 1 entity traversal mechanism for attribute question chains, not just the same EntityFinderSig with new instructions. Next attempt options: (a) Add a concrete SIBLING CROSS-REFERENCE worked example to single-phase PhantomWikiQA docstring instead of Phase 1 — so the full-context single-phase can use it without the over-expansion risk; (b) Build a dedicated Phase 1 agent specifically for attribute questions with explicit step-by-step hop-by-hop traversal instructions (separate from EntityFinderSig which was designed for count_pivot); (c) Accept that attribute fan-out is ColBERT-limited (cannot enumerate all N children of a person) and focus gains elsewhere.

  5. count_answer Single-Phase Under-Counting (Low Priority, Hard Fix): Ex 0 ("How many second aunts does Alan Denney have?" → pred=['0'], gold=['2']) and Ex 6 ("How many male second cousins does Reinaldo Pettus have?" → pred=['4'], gold=['7']) are single-phase failures for simple named-person counting questions. Root cause: corpus dead-ends (some ancestor pages not in the wiki) cause the model to find fewer entities than exist in the Prolog. These are 2/3 count_answer examples failing (0.333 avg). No structural fix exists without a full-corpus traversal index. Not worth investing in — these are inherent corpus coverage gaps.

  6. Intermediate Entity Contamination — STILL UNSOLVED but lower impact now: The two-phase approach for count_pivot doesn't help attribute questions (single-phase path). Ex 41 ("What is the occupation of the cousin of the great-grandchild of the great-grandfather of Deon Gall?" → completely wrong occupations) continues to fail in iteration 13. This question has failed in EVERY iteration (8+ consecutive). The hybrid was architecturally designed to fix this for attribute questions too, but Round 1 of iteration 12 showed attribute Phase 1 causes regressions. The only way to fix contamination for attribute questions is a working two-phase for attributes — which requires Opportunity 2 (reliable attribute anchor mechanism) first to ensure Phase 1 entity quality. Record this as a known persistent failure; do not attempt without the attribute anchor mechanism in place.

  7. Entity Question Phase 1 Routing — DONE in iteration 18 (valset 0.6526, NEW BEST): Entity questions now route through Phase 1 (EntityFinderSig, question_type='entity'). Subsample +6.5%, valset +1.7% (0.6417→0.6526). ex_id=12 confirmed structural win. Entity avg improved from ~0.529 (iter 14, single-phase) to ~0.695 (iter 18, Phase 1). Entity questions are no longer the weakest type. Remaining entity sub-opportunities: (a) Gender-filter post-processing: ex_id=35 found both uncles and aunts (0.667); regex-detect gender qualifier in question + filter Phase 1 output would recover to 1.0; (b) Phase 1 max_iters increase for entity questions: ex_id=23 returned empty at max_iters=40; a separate entity_finder_entity module with max_iters=50 would help; (c) Phase 1 fan-out under-enumeration: ex_id=80 (Deon Gall great-grandchild chain) still returns 3/12 — same ColBERT ceiling, same as single-phase. No fix known.

  8. Always Branch from Current Best (Process Note — CONFIRMED THROUGH ITERATION 24): The best program is now iteration 21, candidate 15 (codeevolver-20260615073543-c7abb8, valset 0.6867). The current live program is candidate 18 (iteration 24, valset 0.6118 — major regression). The pattern of building on the best valset program (not the latest) has been confirmed correct: iterations 5, 8, 12, 18, and 21 all achieved new bests by branching from the prior best. CRITICAL: iterations 22, 23, and 24 were all accepted on subsample but regressed on valset (−11%, −3.5%, −11% respectively). Always branch from candidate 15 for all next experiments — NOT candidate 16, 17, or 18. Candidates 16, 17, and 18 must all be abandoned as bases.

  9. Gender-Filter Post-Processing for Phase 1 Entity Output — IMPLEMENTATION APPROACH FAILED (iterations 19-20), RISK ASSESSMENT REVISED by iteration 21: Phase 1 (EntityFinderSig) does not filter returned entities by gender even when the question specifies "uncle" (male), "aunt" (female), "male cousin", etc. In iteration 18, ex_id=35 ("Who is the second uncle of Rosina Robey?") — Phase 1 returned both gold uncles AND gold aunts, F1=0.667. Iteration 19 (GENDER QUALIFIER docstring: −7.7% valset) and iteration 20 (Python self.retrieve() gender filter: −3.9% subsample, systematic interference) both failed. CRITICAL CONSTRAINT (iteration 20): Any implementation that calls self.retrieve() inside forward() outside a DSPy ReAct module context causes systematic Phase 1 behavior changes — FORBIDDEN. NEW from iteration 21: Iteration 21 confirmed that strictly conditional, logically correct one-line additions to EntityFinderSig generalize positively (+5.2% valset). This revises the risk classification: a ONE-LINE conditional gender instruction in EntityFinderSig's entity block is now MEDIUM RISK (not HIGH RISK). Viable approaches: (1) Add one-line conditional gender rule to EntityFinderSig entity block ("For 'uncle/grandfather/male X' questions, include ONLY male entities; for 'aunt/grandmother/female X' questions, include ONLY female entities") — MEDIUM RISK, test from candidate 15; (2) Create a separate entity_finder_gender_aware module instance with a modified Signature for gender-specified entity questions only; (3) Accept current ~0.667 gender performance. Estimated valset impact if solved cleanly: +1-2%. Must branch from candidate 15.

  10. Phase 2 entity selection for count_pivot — Sequential cap increase FAILED (iter 23, −3.5%), Random sampling also FAILED (iter 24, −11.0%): All Phase 2 entity selection variants have now been tried from candidate 15:

  1. TRIED AND FAILED: Phase 1 max_iters=60 for entity questions (iteration 22): Increasing entity Phase 1 budget from 40→60 caused valset regression of −11.0% (0.6867→0.6112). The 40-iteration limit appears to be a BENEFICIAL constraint for entity questions — it prevents the model from exploring wrong paths too deeply. At 60 iterations, the model follows wrong branches more thoroughly, producing worse entity quality on questions that previously resolved correctly at the lower limit. Do NOT retry max_iters=60 for entity Phase 1. The hard entity failures (ex_id=93, ex_id=58, ex_id=90) are NOT solved by giving more budget — they are structural failures (ColBERT ceiling for deep chains, wrong traversal path choices) that require a different mechanism.

  2. Tautological Question Short-Circuit (STABLE): The tautological DOB short-circuit rule is stable. No action needed.

Other Important Notes