Entry point: /faion-net — invoke this skill for automatic routing to the appropriate domain.

ML Ops Skill

Communication: User's language. Code: English.

Purpose

Handles ML model operations. Covers fine-tuning, evaluation, cost management, and observability.

Scope

Quick Start

Methodologies (15)

Fine-tuning (5):

• finetuning-basics: Fundamentals, when to fine-tune
• finetuning-datasets: Data preparation, quality
• fine-tuning-openai-basics: OpenAI API fine-tuning
• fine-tuning-openai-production: Production deployment
• lora-qlora: Efficient fine-tuning, parameter selection

Evaluation (3):

• evaluation-metrics: Accuracy, F1, perplexity, task metrics
• evaluation-framework: LLM-as-judge, human eval
• evaluation-benchmarks: MMLU, HumanEval, industry benchmarks

Cost Optimization (2):

• llm-cost-basics: Token counting, pricing models
• cost-reduction-strategies: Caching, compression, batching

Observability (5):

• llm-observability: Fundamentals, why monitor
• llm-observability-stack: Tools selection
• llm-observability-stack-2026: Latest tools (LangSmith, Langfuse)
• llm-management-observability: End-to-end management

Code Examples

OpenAI Fine-tuning

from openai import OpenAI

client = OpenAI()

# Upload training data
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={"n_epochs": 3}
)

# Monitor
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    if job.status == "succeeded":
        break

LoRA Fine-tuning

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8b")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    bias="none"
)

model = get_peft_model(model, lora_config)

Cost Tracking

import tiktoken

def count_tokens(text, model="gpt-4o"):
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

def estimate_cost(prompt, completion, model="gpt-4o"):
    prompt_tokens = count_tokens(prompt, model)
    completion_tokens = count_tokens(completion, model)

    # GPT-4o pricing
    prompt_cost = prompt_tokens * 0.000005
    completion_cost = completion_tokens * 0.000015

    return prompt_cost + completion_cost

LLM Observability with LangSmith

from langsmith import traceable

@traceable
def rag_pipeline(query: str) -> str:
    # Retrieval
    docs = retrieve(query)

    # Generation
    response = generate(query, docs)

    return response

Fine-tuning Decision Matrix

Cost Reduction Strategies

Evaluation Frameworks

Related Skills

ML Ops v1.0 | 15 methodologies