These are the instructions provided to CodeEvolver for each benchmark experiment.
HotpotQA Instructions
HotpotQA is designed to contain information from Wikipedia.
What's Allowed
- The program is allowed to create or remove modules
- The program is allowed to create or remove dynamic prompts
- The program is allowed to add rerankers
- There is no limit on the number of search results to retrieve per query (same cost)
- Available services: wikipedia colbert-server (Via dspy.Retrieve)
Constraints
- Do NOT search more than two times per question. This is a hard requirement.
- Do NOT use any external websearch services.
- Do NOT use the HotpotQA dataset as context.
- The program is limited to Wikipedia documents.
Available Services
Wikipedia colbert-server (Via dspy.Retrieve)
- Documentation: github.com/julianghadially/colbert-server
- Use cases: Retrieve information from Wikipedia abstracts. Useful for fact-checking and information retrieval.
- Python usage:
from dspy import Retrieve
Ideas for Optimization
- Consider increasing the number of k retrieved
- Consider different context retrieval pipelines, including query + rerankers (list rerankers, score-based reranking, sliding-window rerankers, etc.)
- Consider removing question summaries, and providing raw evidence instead (This performed well in a prior GEPA run)
- Consider replacing Generate Answer with Extract Answer — i.e., extracting the exact short factoid answer from the passages (This performed well in a prior GEPA run)
HoVer Instructions
HoVer is designed to retrieve information from 2017 Wikipedia Abstracts (5.9M) using multi-hop retrieval.
Task Type: Document Retrieval
This is a document retrieval task. The metric measures whether the correct supporting documents are retrieved — it does NOT measure claim verification or fact-checking accuracy. Do not add claim verification, fact-checking, or classification modules. Focus on improving retrieval recall and precision through better queries, re-ranking, or retrieval strategies.
What's Allowed
- The program is allowed to create or remove modules
- The program is allowed to create or remove dynamic prompts
- The program is allowed to add rerankers, provided the final document count remains the same — 21 final outputted documents
- There is no limit on the number of search results to retrieve per query (same cost)
- Available services: wikipedia colbert-server (Via dspy.Retrieve)
Constraints
- Do NOT search more than three queries per claim. This is a hard requirement.
- The final output of the system CANNOT return more than 21 documents. This is a hard requirement.
- Do NOT use any external websearch services.
Available Services
Wikipedia colbert-server (Via dspy.Retrieve)
- Documentation: github.com/julianghadially/colbert-server
- Use cases: Retrieve information from Wikipedia abstracts. Useful for fact-checking and information retrieval.
- Python usage:
from dspy import Retrieve
Ideas for Optimization
- Consider increasing the k retrieved per query, and then reranking the final results to 21 (final output limited to 21 documents)
- Consider different context retrieval pipelines, including different kinds of rerankers (list rerankers, score-based reranking, pairwise rerankers, etc.)
- Consider a gap analysis before generating queries
Additional note: If you increase k retrieved per query, do not increase it greater than 25 per query as it may overload the Colbert server.
PhantomWiki Instructions
Task Type: Document Retrieval + Question Answering
This is a document retrieval + Question Answering task. The metric measures F1 score — which includes the precision of predicted answers as well as the recall of all available answers. Some questions have multiple correct answers.
To score high, the system requires being able to reason over multiple retrieval steps. Additionally, many questions have multiple correct answers, so the system must be able to reason over multiple chains of logic simultaneously.
The core difficulty of PhantomWiki is the large amount of information that must be processed across multiple chains of logic. Additionally, if context accumulates across multiple chains of logic and the context is too long, the model can suffer from context rot.
What's Allowed
- The program is allowed to create or remove modules, dynamic prompts, tool calls, reasoning steps, etc.
- The program is allowed to change the module types (e.g., dspy.ReAct for tool calling, dspy.ChainOfThought, dspy.Predict, etc.)
- There is no limit on the number of search results to display per query or the number of searches to make
Constraints
- Colbertv2 retriever endpoint is used to retrieve documents from the wiki.
Ideas for Optimization
- Try to address the multiple chains of logic problem by breaking up investigation paths into multiple sub-agents.
- Taking inspiration from recursive language models, you could have recursive reasoning agents where the parent reasoning agent could spawn multiple sub-agents that each investigate a given path. However, limit the depth to one or two and the breadth per recursive spawn to four.
- Try adding reasoning steps and/or structured thinking and/or logic guidance via new modules in the workflow.
- Try creating a secondary workspace to jot down persistent reasoning logic that multiple agents can add to or remove as they interact with more documents.
- Try explicitly managing multiple chains of logic in the AI workflow architecture (e.g., modules or workspace file for work-in-progress reasoning logic).
- Try iterative search methods.
- Try increasing the maximum number of retrieval steps.
- Try modifying the number of documents returned per query.
Available Services
- PhantomWiki colbert-server (Via dspy.Retrieve)
Additional notes: Many answers are surfaced over multiple steps of reasoning and retrieving (i.e., pure parallel querying is unlikely to work without multiple retrieval steps). Different questions require different numbers of retrieval steps. RLMs underperformed in a previous trial and are not recommended.