Qwen Training Data Miner (Prototype)

Metadata (YAML Frontmatter)

skillid: qwentrainingdataminerv1prototype name: qwentrainingdataminer description: Mine 012.txt for domain-specific training examples (MPS scoring, WSP patterns, decision rationale) version: 1.0prototype author: 0102design created: 2025-10-22 agents: [qwen] primaryagent: qwen intenttype: GENERATION promotionstate: prototype patternfidelitythreshold: 0.90 teststatus: needsvalidation

MCP Orchestration

mcporchestration: true breadcrumblogging: true owningdae: docdae executionphase: 1 nextskill: gemmadomaintrainerv1prototype

Input/Output Contract

inputs: - sourcefile: "O:/Foundups-Agent/012.txt (98,400 lines)" - domain: "Target knowledge domain (mpsscoring, wspapplication, roadmapanalysis, etc.)" - patterntype: "Type of pattern to extract (numericexamples, decisiontrees, rationalechains)" - minexamples: "Minimum number of examples to extract (default: 50)" outputs: - data/trainingdatasets/{domain}trainingdata.json: "Instruction-tuning dataset" - data/trainingdatasets/{domain}patternsummary.json: "Pattern analysis metadata" - executionid: "Unique execution identifier for breadcrumb tracking"

Dependencies

dependencies: datastores: - name: 012scrapbook type: text path: O:/Foundups-Agent/012.txt mcpendpoints: - endpointname: holoindex methods: [semanticsearch] throttles: [] requiredcontext: - domain: "Knowledge domain to mine" - patternregex: "Regex pattern for extraction"

Metrics Configuration

metrics: patternfidelityscoring: enabled: true frequency: everyexecution scoreragent: gemma writedestination: modules/infrastructure/wrecore/recursiveimprovement/metrics/qwentrainingdataminerfidelity.json promotioncriteria: minpatternfidelity: 0.90 minoutcomequality: 0.85 minexecutioncount: 100 requiredtestpass_rate: 0.95

Qwen Training Data Miner

Purpose: Mine 012.txt (0102's decision history) for domain-specific training examples to train Gemma models

Intent Type: GENERATION

Agent: qwen (1.5B, 32K context - can hold large sections of 012.txt)

Task

You are Qwen, a training data miner. Your job is to read 012.txt (98,400 lines of 0102's decision-making history) and extract high-quality training examples for specific knowledge domains. You create instruction-tuning datasets that Gemma can learn from.

Key Capability: Pattern recognition, example extraction, quality filtering

Domains You Can Mine:

1. mps_scoring - WSP 15 scoring examples with numeric calculations
2. wsp_application - How WSPs are applied to real problems
3. roadmap_analysis - Project planning, completion tracking
4. readme_patterns - Documentation structure, best practices
5. modlog_updates - Change documentation patterns
6. first_principles - Occam's Razor reasoning chains

Instructions (For Qwen Agent)

1. LOAD SOURCE FILE

Rule: Read 012.txt in chunks (32K token window)

Expected Pattern: source_loaded=True

Steps:

1. Open O:/Foundups-Agent/012.txt
2. Count total lines (should be ~98,400)
3. Calculate chunk size (fit within 32K context)
4. Load first chunk for analysis
5. Log: {"pattern": "sourceloaded", "value": true, "totallines": 98400, "chunk_size": 5000}

2. IDENTIFY DOMAIN PATTERNS

Rule: Search for domain-specific patterns using regex and semantic matching

Expected Pattern: domainpatternsidentified=True

Domain-Specific Patterns:

MPS Scoring Domain

patterns = [
    r"MPS.*Score:?\s*(\d+)",
    r"Complexity.*(\d)\s*,?\s*Importance.*(\d)\s*,?\s*Deferability.*(\d)\s*,?\s*Impact.*(\d)",
    r"Priority:?\s*(P[0-4])",
    r"MPS.*\(C:(\d),\s*I:(\d),\s*D:(\d),\s*P:(\d)\)"
]

WSP Application Domain

patterns = [
    r"WSP\s*(\d+).*compliance",
    r"WSP\s*(\d+).*violation",
    r"following\s+WSP\s*(\d+)",
    r"applied\s+WSP\s*(\d+)"
]

Roadmap Analysis Domain

patterns = [
    r"roadmap.*complete",
    r"roadmap.*incomplete",
    r"roadmap.*needs.*update",
    r"Phase\s*(\d+).*status",
    r"TODO.*implement"
]

Steps:

1. For target domain, compile regex patterns
2. Scan 012.txt for matching lines
3. Extract context (5 lines before/after match)
4. Store matches with line numbers
5. Log: {"pattern": "domainpatternsidentified", "value": true, "matches_found": N}

3. EXTRACT TRAINING EXAMPLES

Rule: Convert matched patterns into instruction-tuning format

Expected Pattern: examples_extracted=True

Instruction-Tuning Format:

{
  "instruction": "Apply WSP 15 MPS scoring to this task",
  "input": {
    "task_description": "Migrate agent_permissions to SQLite",
    "context": "Database consolidation, 21 tests passing, high priority"
  },
  "output": {
    "complexity": 3,
    "complexity_reason": "Moderate - requires schema design + migration",
    "importance": 5,
    "importance_reason": "Essential - blocks other migrations",
    "deferability": 5,
    "deferability_reason": "Cannot defer - P0 priority",
    "impact": 4,
    "impact_reason": "Major - enables autonomous permission system",
    "mps_total": 17,
    "priority": "P0"
  },
  "source": "012.txt:line_5234",
  "quality_score": 0.95
}

Steps:

1. For each match, extract:

- Instruction: What task is being performed? - Input: What context/data is provided? - Output: What is the correct answer/decision? - Source: Line number for verification

1. Quality filter:

- Complete examples only (has input + output) - Clear reasoning (not ambiguous) - Correct format (follows pattern)

1. Assign quality score (0.0-1.0)
2. Log: {"pattern": "examplesextracted", "value": true, "totalexamples": N, "high_quality": M}

4. QUALITY FILTERING

Rule: Only keep examples with quality_score >= 0.85

Expected Pattern: qualityfilteringapplied=True

Quality Criteria:

• ✅ Complete (has instruction + input + output)
• ✅ Clear reasoning (rationale provided)
• ✅ Correct format (matches instruction-tuning schema)
• ✅ Verifiable (can trace back to source line)
• ✅ Unambiguous (single correct interpretation)

Steps:

1. Review each extracted example
2. Score on 5 criteria (0.2 per criterion)
3. Filter: keep only if score >= 0.85 (4/5 criteria)
4. Remove duplicates (same input/output pattern)
5. Log: {"pattern": "qualityfilteringapplied", "value": true, "kept": N, "filtered": M}

5. GENERATE PATTERN SUMMARY

Rule: Analyze extracted examples for meta-patterns

Expected Pattern: patternsummarygenerated=True

Summary Metadata:

{
  "domain": "mps_scoring",
  "total_examples": 73,
  "high_quality_examples": 58,
  "quality_distribution": {
    "0.95-1.0": 23,
    "0.90-0.94": 20,
    "0.85-0.89": 15
  },
  "common_patterns": [
    "P0 tasks: MPS 16-20 (23 examples)",
    "P1 tasks: MPS 13-15 (19 examples)",
    "Complexity 3-4 most common (database migrations, refactoring)"
  ],
  "coverage_analysis": {
    "p0_examples": 23,
    "p1_examples": 19,
    "p2_examples": 12,
    "p3_examples": 3,
    "p4_examples": 1
  },
  "recommended_use": "Train Gemma on MPS scoring for cleanup tasks, project prioritization"
}

Steps:

1. Count examples by category/pattern
2. Identify common themes
3. Assess coverage (are all cases represented?)
4. Generate training recommendations
5. Log: {"pattern": "patternsummarygenerated", "value": true}

6. WRITE TRAINING DATASET

Rule: Output JSON file with instruction-tuning examples

Expected Pattern: trainingdatasetwritten=True

Output Format (EXECUTION-READY per First Principles):

{
  "dataset_id": "mps_scoring_training_v1",
  "created": "2025-10-22T02:30:00Z",
  "source": "012.txt (lines 1-98400)",
  "domain": "mps_scoring",
  "total_examples": 58,
  "quality_threshold": 0.85,

  "domain_priority_mps": {
    "complexity": 2,
    "complexity_reason": "Easy - pattern extraction from 012.txt",
    "importance": 4,
    "importance_reason": "Critical - enables autonomous MPS scoring",
    "deferability": 3,
    "deferability_reason": "Moderate - other wardrobes can be trained first",
    "impact": 5,
    "impact_reason": "Critical - foundation for cleanup automation",
    "total": 14,
    "priority": "P1",
    "training_order": 1
  },

  "examples": [
    {
      "example_id": "mps_001",
      "instruction": "...",
      "input": {...},
      "output": {...},
      "source": "012.txt:line_5234",
      "quality_score": 0.95
    },
    ...
  ],

  "metadata": {
    "pattern_summary": {...},
    "coverage_analysis": {...},
    "recommended_use": "..."
  },

  "recommended_wardrobe_config": {
    "wardrobe_id": "gemma_mps_scorer_v1",
    "lora_rank": 8,
    "learning_rate": 0.0002,
    "epochs": 3,
    "expected_accuracy": 0.87,
    "use_cases": [
      "Cleanup task prioritization",
      "Project scoring",
      "Issue triage"
    ]
  },

  "autonomous_execution": {
    "capable": true,
    "agent": "gemma_domain_trainer_v1",
    "confidence": 0.90,
    "estimated_tokens": 200,
    "estimated_time_seconds": 600,
    "requires_0102_approval": false,
    "execution_command": "python -m modules.infrastructure.wsp_orchestrator.src.wsp_orchestrator --skill gemma_domain_trainer --domain mps_scoring --dataset data/training_datasets/mps_scoring_training_data.json"
  },

  "verification": {
    "verify_command": "test -f data/training_datasets/mps_scoring_training_data.json && jq '.total_examples' data/training_datasets/mps_scoring_training_data.json",
    "success_criteria": "File exists + total_examples >= 50 + quality_threshold >= 0.85",
    "validation_script": "python -c \"import json; d=json.load(open('data/training_datasets/mps_scoring_training_data.json')); assert d['total_examples'] >= 50; assert d['quality_threshold'] >= 0.85; print('✓ Dataset validated')\""
  },

  "learning_feedback": {
    "pattern_extraction_stats": {
      "total_patterns_found": 73,
      "high_quality_kept": 58,
      "filter_rate": 0.79,
      "common_filter_reasons": [
        "Incomplete example (missing rationale) - 8 filtered",
        "Ambiguous input - 5 filtered",
        "Duplicate pattern - 2 filtered"
      ]
    },
    "domain_insights": [
      "P0 tasks: MPS 16-20 (23 examples) - database migrations, critical bugs",
      "P1 tasks: MPS 13-15 (19 examples) - feature requests, refactoring",
      "Complexity 3-4 most common - moderate difficulty tasks"
    ],
    "future_improvements": [
      "Add semantic deduplication (beyond exact match)",
      "Extract negative examples (what NOT to do)",
      "Mine multi-step reasoning chains for complex decisions"
    ],
    "store_to": "holo_index/adaptive_learning/training_data_mining_patterns.jsonl"
  }
}

Destination: data/trainingdatasets/{domain}training_data.json

Steps:

1. Create directory data/training_datasets/ if not exists
2. Calculate domainprioritymps (which domain should be trained first?)
3. Generate recommendedwardrobeconfig (LoRA hyperparameters)
4. Write training dataset JSON with all First Principles fields
5. Generate autonomous_execution command (can Gemma trainer auto-execute?)
6. Create verification script (validate dataset quality)
7. Extract learning_feedback (pattern extraction stats + future improvements)
8. Log: {"pattern": "trainingdatasetwritten", "value": true, "filesizekb": N, "autonomous_ready": true}

First Principles Additions:

• ✅ MPS Scoring: domainprioritymps determines training order (which wardrobe first?)
• ✅ Agent Mapping: autonomousexecution.agent = gemmadomaintrainerv1
• ✅ Executable Command: Can pipe to bash to start training automatically
• ✅ Verification: validation_script confirms dataset quality before training
• ✅ Learning Feedback: Stores pattern extraction stats for future mining improvements
• ✅ Recommended Config: Wardrobe hyperparameters (LoRA rank, learning rate, epochs)

Expected Patterns Summary

{
  "execution_id": "exec_qwen_miner_001",
  "skill_id": "qwen_training_data_miner_v1_prototype",
  "patterns": {
    "source_loaded": true,
    "domain_patterns_identified": true,
    "examples_extracted": true,
    "quality_filtering_applied": true,
    "pattern_summary_generated": true,
    "training_dataset_written": true
  },
  "total_examples_extracted": 73,
  "high_quality_examples": 58,
  "execution_time_ms": 3500
}

Fidelity Calculation: (patterns_executed / 6) - All 6 steps should run

Domain Catalog

1. MPS Scoring Domain

Purpose: Train Gemma to apply WSP 15 MPS scoring Patterns: Numeric scores, priority mapping, rationale Use Cases: Cleanup prioritization, project planning, issue triage

2. WSP Application Domain

Purpose: Train Gemma to recognize WSP violations and applications Patterns: WSP references, compliance checks, violation detection Use Cases: Code review, documentation validation, architecture audits

3. Roadmap Analysis Domain

Purpose: Train Gemma to analyze project roadmaps Patterns: Phase completion, TODO tracking, update detection Use Cases: Project status reports, roadmap audits, completion tracking

4. README Patterns Domain

Purpose: Train Gemma to validate README structure Patterns: Required sections, format consistency, completeness Use Cases: Documentation quality checks, README generation

5. ModLog Updates Domain

Purpose: Train Gemma to generate ModLog entries Patterns: Change descriptions, WSP references, rationale Use Cases: Automated ModLog updates, change tracking

6. First Principles Domain

Purpose: Train Gemma to apply Occam's Razor reasoning Patterns: Problem simplification, root cause analysis, decision trees Use Cases: Debugging, architecture design, problem-solving

Benchmark Test Cases

Test Set 1: MPS Scoring Extraction (10 cases)

1. Input: "MPS Score: 16" → Expected: Extract as P0 example
2. Input: "Complexity: 3, Importance: 5, Deferability: 2, Impact: 4" → Expected: Calculate MPS = 14
3. Input: "Priority: P1" → Expected: Map to MPS 13-15 range
4. Input: Incomplete example (missing rationale) → Expected: Quality score < 0.85, filtered
5. Input: Duplicate example → Expected: Deduplicated

Test Set 2: WSP Application Extraction (5 cases)

1. Input: "Following WSP 15 for scoring" → Expected: Extract WSP 15 application example
2. Input: "WSP 64 violation detected" → Expected: Extract violation example
3. Input: "WSP compliance: WSP 3, WSP 50" → Expected: Extract multi-WSP compliance
4. Input: Ambiguous WSP reference → Expected: Quality score < 0.85
5. Input: Clear WSP application with rationale → Expected: Quality score >= 0.90

Test Set 3: Quality Filtering (5 cases)

1. Input: Complete example with all fields → Expected: Quality score = 1.0
2. Input: Missing rationale → Expected: Quality score = 0.8 (filtered)
3. Input: Ambiguous input → Expected: Quality score = 0.6 (filtered)
4. Input: Clear but partial example → Expected: Quality score = 0.85 (kept)
5. Input: Excellent example with source → Expected: Quality score = 0.95

Total: 20 test cases across 3 categories

Success Criteria

• ✅ Pattern fidelity ≥ 90% (all 6 steps execute)
• ✅ Extract ≥ 50 high-quality examples per domain
• ✅ Quality threshold 0.85+ maintained
• ✅ Zero duplicate examples in output
• ✅ All examples have verifiable source (line number)
• ✅ Pattern summary provides actionable insights

Next Phase: Gemma Training

After extraction, examples feed into gemmadomaintrainer skill:

1. Load training dataset
2. Fine-tune Gemma 270M on domain examples
3. Validate accuracy on held-out test set
4. Deploy trained model for domain-specific tasks

Wardrobe Concept: Training as a Service

Different "training wardrobes" for different knowledge domains:

• qwenmpsscorer - Trained on MPS scoring examples
• qwenwspauditor - Trained on WSP compliance examples
• qwenroadmaptracker - Trained on roadmap analysis examples
• qwenreadmevalidator - Trained on README patterns

Each wardrobe:

• Mines 012.txt for domain-specific patterns
• Trains Gemma on extracted examples
• Deploys as reusable skill
• Evolves as more examples accumulate

Meta-skill: qwenwardrobegenerator - Automates creation of new training wardrobes for any domain!

Status: ✅ Ready for prototype testing - Mine 012.txt for MPS scoring examples first