Releases·Tool Release
Reef Infra tests context-optimization tradeoffs by running current and candidate harnesses on the same tasks
A Reddit post describes Reef Infra, a tool for testing context-optimization strategies: a harness proposer proposes changes to a versioned context tree, and an evaluator scores completed task runs with the model fixed. The post argues that naive compression can silently drop rare-but-necessary evidence, so teams need workload-specific test cases and coverage-sensitive selection policies.
6 engagement1 sourceFri, Sep 25, 2026, 04:20 AM
AI Overview
- Reef Infra runs current and candidate harnesses against the same fixed-model tasks, pairing context-reduction testing with task results.
- Its harness proposer returns changes to a versioned tree; the evaluator scores completed task runs.
- Default selection counts task wins against losses; a coverage-sensitive example accepts only when no task regresses.
- Reef Infra has no separate knowledge of which omitted evidence mattered if the evaluator rewards only brevity or token savings.
- A custom LangChain harness still needs an adapter to launch episodes and read traces; task fixtures and coverage definitions stay with the user.
Entities
Related
Products·launched
Reef: agent harness self-improvement loop with a fixed task set and grader
Reef sits between a coding agent and its model endpoint, logging interactions and feedback. A proposer uses failure feedback to suggest harness changes (prompts, rules, skills, tools), and Reef runs candidate and current harnesses on the same tasks with unchanged grading logic. This lets operators attribute score changes to harness improvements rather than shifting evaluation criteria.
Reef sits between a coding agent and its model endpoint, logging interactions and feedback. A proposer uses failure feedback to suggest harness changes (prompts, rules, skills, tools), and Reef runs candidate and current harnesses on the same tasks with unchanged grading logic. This lets operators attribute score changes to harness improvements rather than shifting evaluation criteria.
1 engagement1 sourceredditSat, Sep 12, 2026, 03:28 PM
Releases·Tool Release
Your prompt optimizer should remember the edits you rejected
A write-up argues prompt optimizers waste evaluations re-proposing edits users already rejected, like repeated 'be concise' suggestions that strip needed explanation. Reef Infra addresses this by passing recent rejected proposals—step, rejection reason, and mutations—to a custom proposer through a `rejected` argument. The proposer can then avoid repeating failures or return `None`, though the response strategy must be implemented by the developer.
A write-up argues prompt optimizers waste evaluations re-proposing edits users already rejected, like repeated 'be concise' suggestions that strip needed explanation. Reef Infra addresses this by passing recent rejected proposals—step, rejection reason, and mutations—to a custom proposer through a `rejected` argument. The proposer can then avoid repeating failures or return `None`, though the response strategy must be implemented by the developer.
1 engagement1 sourceredditThu, Oct 1, 2026, 12:36 PM
Developments·Paper
An Empirical Study of Harness Design for Coding Agents
An arXiv paper dissects how coding agent harness components—planning, action space, and context management—individually affect performance, using a fixed execution loop varied across 176 matched settings on four models. Findings give practitioners model- and budget-aware guidance: context management matters most under tight context budgets, and bash-capable models can drop predefined tools to cut costs. The work offers a modular framework for evaluating future harness components.
An arXiv paper dissects how coding agent harness components—planning, action space, and context management—individually affect performance, using a fixed execution loop varied across 176 matched settings on four models. Findings give practitioners model- and budget-aware guidance: context management matters most under tight context budgets, and bash-capable models can drop predefined tools to cut costs. The work offers a modular framework for evaluating future harness components.
0 engagement1 sourcearxivThu, Sep 17, 2026, 05:58 PM
Products·building
jev-gc: context garbage collector for LLM agent memory
jev-gc is a tool that manages agent context windows by hooking into OpenTelemetry spans and deciding in real time what stays in the next prompt, what gets compressed to a pointer, and what gets archived. It replaces re-sending everything each turn or periodic summarization with deterministic rules (pins, errors, recency, staleness) plus judgment calls on ambiguous cases. It targets developers building LLM agents whose contexts fill with stale tool results and dead-end investigations.
jev-gc is a tool that manages agent context windows by hooking into OpenTelemetry spans and deciding in real time what stays in the next prompt, what gets compressed to a pointer, and what gets archived. It replaces re-sending everything each turn or periodic summarization with deterministic rules (pins, errors, recency, staleness) plus judgment calls on ambiguous cases. It targets developers building LLM agents whose contexts fill with stale tool results and dead-end investigations.
10 engagement2 sourcesredditTue, Sep 22, 2026, 04:26 AM
Releases·Tool Release
We tested giving Claude Code a 450-line static prompt vs a self-learning harness (and measured a 38.4% drop in regression errors)
A Reddit post by a developer describes testing a 450-line static CLAUDE.md against a minimal 25-line root file with dynamic skill loading across 40 multi-file refactoring tasks on their backend repo. The dynamic setup cut regression errors 38.4%, dropped per-turn input tokens from ~18,500 to under 3,200, and reduced session token spend over 45%. The team open-sourced the tool as autoharness.
A Reddit post by a developer describes testing a 450-line static CLAUDE.md against a minimal 25-line root file with dynamic skill loading across 40 multi-file refactoring tasks on their backend repo. The dynamic setup cut regression errors 38.4%, dropped per-turn input tokens from ~18,500 to under 3,200, and reduced session token spend over 45%. The team open-sourced the tool as autoharness.
19 engagement1 sourceredditWed, Sep 23, 2026, 09:33 AM