inno-experiment-dev

Canonical Summary

Creates implementation plan, writes project code with judge feedback loop, and submits final experiment run. Use after code-survey in both Idea and Plan branches.

Trigger Rules

Use this skill when the user request matches its research workflow scope. Prefer the bundled resources instead of recreating templates or reference material. Keep outputs traceable to project files, citations, scripts, or upstream evidence.

Resource Use Rules

• Read from references/ only when the current task needs the extra detail.

Execution Contract

• Resolve every relative path from this skill directory first.
• Prefer inspection before mutation when invoking bundled scripts.
• If a required runtime, CLI, credential, or API is unavailable, explain the blocker and continue with the best manual fallback instead of silently skipping the step.
• Do not write generated artifacts back into the skill directory; save them inside the active project workspace.

Upstream Instructions

Inno Experiment Dev (Planning, Implementation, and Submission)

Merges the former inno-implementation-plan, inno-ml-dev-iteration, and the submit step of inno-experiment-submit-refine. Mirrors createimplementationplan (830-858), implementanditerate (861-920), and the submit portion of submitandrefineexperiments (922-945) in runinferidea_ours.py.

Inputs

Plan mode additionally uses ideas and survey-specific prompt variants (buildplanquerywithsurvey, builditerationqueryforplan, etc.).

Outputs

Cache Artifacts

Instructions

Phase 1: Create Implementation Plan

Mirrors createimplementation_plan.

1. Optional pre-step (Idea mode only): If refining the idea for implementation clarity, call the idea refinement agent to produce refinedfordownstream with tensor interfaces and forward-pass sketch.

1. Build plan query:

- Idea mode: planquery = buildplanquery(surveyres, references, updatedprepareres, codesurveyres, datasetdescription) (see prompts/buildplanquery.md) - Plan mode: Use buildplanquerywithsurvey(ideas, references, prepareres, codesurveyres, dataset_description)

1. Call Coding Plan Agent with messages = [{"role": "user", "content": plan_query}].

- The agent reviews codebases using tree / cat, then creates structured plans via plandataset, plantraining, plantesting. - Calls caseresolved to merge plans. - Set planres = planmessages[-1]["content"]. - See references/codingplanagent.md for agent details.

1. Verify the plan has clear sections: dataset, model, training, evaluation, file layout.

Phase 2: Implement and Iterate

Mirrors implementand_iterate.

1. Initial implementation: Build mldevquery = buildmldevquery(surveyres, prepareres, codesurveyres, planres, datasetdescription, corecode, codereferences) (see prompts/buildmldevquery.md). Use paths from instance.json: Experiment.corecode, Experiment.codereferences (absolute in Dr. Claw–created projects; use as-is or resolve with project path if relative). Call ML Agent with messages = [{"role": "user", "content": mldevquery}]. Set mldevres = ml_messages[-1]["content"].

- See references/mlagentinstructions.md for agent details.

1. Initial judge evaluation: Build judgequery = buildjudgequery(surveyres, prepareres, planres, mldevres) (see prompts/buildjudgequery.md). Call Judge Agent with inputmessages = [{"role": "user", "content": judgequery}]. Set judgeres = judgemessages[-1]["content"].

- See references/judgeagentinstructions.md for agent details.

1. Iteration loop (for i in 0..maxitertimes - 1):

a. Build iterationquery = builditerationquery(surveyres, prepareres, codesurveyres, planres, mldevres, judgeres, corecode, codereferences) (see prompts/builditerationquery.md). Use paths from instance.json (absolute in Dr. Claw–created projects; use as-is or resolve if relative). Plan mode uses builditerationqueryforplan. b. Append as user message to judgemessages. Call ML Agent with itertimes=i+1. Update mldevres. c. Build judgesimplequery = buildjudgesimplequery(surveyres, prepareres, planres, mldevres) (see prompts/buildjudgesimplequery.md). Plan mode uses buildjudgesimplequeryforplan. d. Append as user message to judgemessages. Call Judge Agent with itertimes=i+1. Update judgeres. e. If "fully_correct": true in last message, break early.

1. Preserve judge_messages for the submit step and for downstream inno-experiment-analysis.

Phase 3: Submit Experiment

Mirrors the submit portion of submitandrefineexperiments.

1. Build submit query: submitquery = buildsubmitquery(surveyres, mldevres, judgeres, corecode) (see prompts/buildsubmitquery.md). Resolve corecode from instance.Experiment.corecode. Plan mode uses buildsubmitqueryforplan.

1. Append to judgemessages as user message. Call ML Agent with itertimes="submit".

- The agent adjusts epochs (3-10), runs runtrainingtesting.py, ensures checkpoints are saved. - Set submitres = judgemessages[-1]["content"].

1. If the implementation is not runnable, ML Agent calls casenotresolved. Otherwise, case_resolved with statistical results and analysis.

Tool Mappings

All custom Python tools map to Claude Code built-in capabilities:

Checklist

• Optional idea refinement applied if desired (Idea mode).
• Correct buildplanquery variant used for Idea vs Plan mode.
• Coding Plan Agent called; plan_res has clear dataset/model/training/testing sections.
• ML Agent initial implementation completed; mldevres recorded.
• Judge Agent initial evaluation completed; judge_res recorded.
• Iteration loop runs with correct prompt variants; early exit on fully_correct.
• judge_messages preserved across all phases.
• Submit query appended to judge_messages; ML Agent submission run completed.
• Final model checkpoint saved to Experiment/corecode/checkpoints/modelfinal.pth.
• Cache artifacts saved to Experiment/corecode/logs/: codingplanagent.json, machinelearningagent.json, judgeagent.json, machinelearningagentitersubmit.json.

References

• runinferideaours.py: createimplementationplan (830-858), implementanditerate (861-920), submitandrefine_experiments submit step (922-945)
• prompttemplates.py: buildplanquery (203-233), buildmldevquery (236-381), buildjudgequery (384-417), builditerationquery (420-468), buildjudgesimplequery (471-494), buildsubmit_query (497-527)
• Agent definitions: planagent.py, mlagent.py, judgeagent.py in inno/agents/innoagent/