A write-up argues prompt optimizers waste evaluations re-proposing edits users already rejected, like repeated 'be concise' suggestions that strip needed explanation. Reef Infra addresses this by passing recent rejected proposals—step, rejection reason, and mutations—to a custom proposer through a `rejected` argument. The proposer can then avoid repeating failures or return `None`, though the response strategy must be implemented by the developer.
A Reddit post describes Reef Infra, a tool for testing context-optimization strategies: a harness proposer proposes changes to a versioned context tree, and an evaluator scores completed task runs with the model fixed. The post argues that naive compression can silently drop rare-but-necessary evidence, so teams need workload-specific test cases and coverage-sensitive selection policies.
A Reddit post describes Reef Infra, a tool for testing context-optimization strategies: a harness proposer proposes changes to a versioned context tree, and an evaluator scores completed task runs with the model fixed. The post argues that naive compression can silently drop rare-but-necessary evidence, so teams need workload-specific test cases and coverage-sensitive selection policies.
A post describes how the GEPA prompt-optimization recipe implemented in Reef Infra avoids overfitting to failed examples. Optimizer revisions can be convincing yet merely tuned to the examples they fixed. The recipe layers distinct checks: minibatch adoption, full validation evaluation, and a separate held-out test.
A post describes how the GEPA prompt-optimization recipe implemented in Reef Infra avoids overfitting to failed examples. Optimizer revisions can be convincing yet merely tuned to the examples they fixed. The recipe layers distinct checks: minibatch adoption, full validation evaluation, and a separate held-out test.
A practitioner describes how patching one failure in a code-review prompt (missed hardcoded credentials) silently degraded style feedback on large pull requests. They attribute this to a fixed output budget under "be concise" where added emphasis on security crowds out other behaviors. They recommend rerunning a small fixed regression set of test inputs after every prompt change.
A practitioner describes how patching one failure in a code-review prompt (missed hardcoded credentials) silently degraded style feedback on large pull requests. They attribute this to a fixed output budget under "be concise" where added emphasis on security crowds out other behaviors. They recommend rerunning a small fixed regression set of test inputs after every prompt change.
Reef sits between a coding agent and its model endpoint, logging interactions and feedback. A proposer uses failure feedback to suggest harness changes (prompts, rules, skills, tools), and Reef runs candidate and current harnesses on the same tasks with unchanged grading logic. This lets operators attribute score changes to harness improvements rather than shifting evaluation criteria.
Reef sits between a coding agent and its model endpoint, logging interactions and feedback. A proposer uses failure feedback to suggest harness changes (prompts, rules, skills, tools), and Reef runs candidate and current harnesses on the same tasks with unchanged grading logic. This lets operators attribute score changes to harness improvements rather than shifting evaluation criteria.
A practitioner post argues that prompt optimization suffers a 'test-leak problem': when the candidate prompt, scoring rubric, and test cases can all change together, rising scores may reflect the optimizer gaming the test rather than genuine prompt improvement. The regression stays invisible during optimization because scores climb every iteration, so users detect it first.
A practitioner post argues that prompt optimization suffers a 'test-leak problem': when the candidate prompt, scoring rubric, and test cases can all change together, rising scores may reflect the optimizer gaming the test rather than genuine prompt improvement. The regression stays invisible during optimization because scores climb every iteration, so users detect it first.