Train a sentence-transformers Model

This SKILL.md is a router, not a manual. It tells you which references and example scripts to load for your task. The actual content (recommended losses, evaluators, training-script structure, model selection, training-arg knobs, troubleshooting) lives in references/ and scripts/.

Do not synthesize a training script from this file alone. Open the per-type production template (scripts/train<type>example.py) and copy it as your starting point. The templates contain load-bearing scaffolding (autocast helper, model-card class, logger silencing list, force=True, seed, TF32, version-compatible imports, named-evaluator metric handling) that prior agent runs have repeatedly missed when rolling their own from a synthesized snippet.

1. Identify the model type

Tiebreakers when the request is ambiguous: "embedding model" / "vector search" / "similarity" → [SentenceTransformer]. "rerank" / "ranker" / "two-stage" → [CrossEncoder]. "SPLADE" / "sparse" / "inverted index" → [SparseEncoder]. "ColBERT" / "late interaction" / "multi-vector" / "MaxSim" / "ColPali" / "ColQwen" → [MultiVectorEncoder]. If still unclear, ask.

2. Required reading

Read these in full before writing any code. Do not triage by perceived relevance.

Per-type: always required

[SentenceTransformer]

• references/lossessentencetransformer.md: loss-to-data-shape mapping, BatchSamplers.NODUPLICATES requirement for MNRL-family, Cached* ↔ gradientcheckpointing incompatibility.
• references/evaluatorssentencetransformer.md: evaluator-to-task mapping, metricforbestmodel key construction (named vs unnamed), per-evaluator primarymetric values.
• references/model_architectures.md: encoder vs decoder vs static vs Router pipelines, pooling rules (mean / cls / lasttoken), auto-mean-pooling behavior for fresh-start MLM bases.
• scripts/trainsentencetransformer_example.py: production template. Copy this as your starting point.

[CrossEncoder]

• references/lossescrossencoder.md: pointwise / pairwise / listwise / distillation, posweight derivation, activationfn=Identity() mandatory for non-BCE losses (silent eval-rank collapse otherwise).
• references/evaluatorscrossencoder.md: CrossEncoderRerankingEvaluator recipe, named-evaluator key format eval{name}{primary_metric}.
• scripts/traincrossencoder_example.py: production template. Copy this as your starting point.

[SparseEncoder]

• references/lossessparseencoder.md: SpladeLoss wrapper requirement, FLOPS regularizer weights, smoke-test active-dim ramp behavior.
• references/evaluatorssparseencoder.md: SparseNanoBEIREvaluator (English-only) and the in-domain alternative, eval{name}{primary_metric} key format.
• scripts/trainsparseencoder_example.py: production template. Copy this as your starting point.

[MultiVectorEncoder]

• references/lossesmultivectorencoder.md: MaxSim scoring, scale choice per scoring mode (scale=1.0 for MaxSim, roughly the average query length for MeanMaxSim), MNRL / CachedMNRL / MarginMSE / DistillKLDiv, XTR-vs-ColBERT scoring, CachedMNRL ↔ gradientcheckpointing incompatibility.
• references/evaluatorsmultivectorencoder.md: MultiVectorNanoBEIREvaluator (English-only) and the in-domain alternative, evalNanoBEIRmeanmaxsim_ndcg@10 key format, distillation-eval spearman variant.
• scripts/trainmultivectorencoderexample.py: production template. Copy this as your starting point.

Cross-cutting: always required (regardless of task)

• references/trainingargs.md: TrainingArguments knobs, precision rules (load fp32 + autocast bf16/fp16, never torchdtype=bfloat16), warmupsteps (float) vs deprecated warmupratio, savesteps must be a multiple of evalsteps for loadbestmodelatend, schedulers, HPO, tracker, resume, hub-push variants.
• references/dataset_formats.md: column-matching rules (label name auto-detection, column-order-not-name), reshaping recipes, hard-negative mining options.
• references/basemodelselection.md: discovery commands, per-type model namespaces, ModernBERT-family maxseqlength=8192 trap, datasets >= 4 script-loader rejection, non-English starting-point shortcuts.
• references/troubleshooting.md: symptom-indexed failure recipes. Skim the section headings on every run, even a healthy one. The "Metrics don't improve" and "Hub push fails" entries cover bugs that bite frequently and are cheaper to recognize before they fire than to debug after.

Cross-cutting: load when applicable

• references/hardware_guide.md: VRAM sizing, multi-GPU, FSDP / DeepSpeed, HF Jobs flavors. Required for >24GB models, multi-GPU, or HF Jobs runs.
• references/hfjobsexecution.md: required when running on HF Jobs.
• references/promptsandinstructions.md: required when using prompt-tuned bases (E5, BGE, GTE, Qwen3-Embedding, Instructor, Nomic, etc.) or adding query: / passage: style prefixes.

Variant scripts (open when the task matches)

• [SentenceTransformer] scripts/trainsentencetransformer<matryoshka|multidataset|withlora|distillation|makemultilingual|staticembedding>example.py.
• [CrossEncoder] scripts/traincrossencoder<distillation|listwise>example.py.
• [SparseEncoder] scripts/trainsparseencoderdistillationexample.py.
• Hard-negative mining CLI: scripts/minehardnegatives.py.

3. Defaults

Override only if the user specifies otherwise:

• Local execution. Pitch HF Jobs only if local hardware can't fit the job.
• Single run. After it completes, propose experimentation if the user would benefit (weak/marginal verdict, "see how high you can push it" framing, etc.). Iteration rules in references/training_args.md (Experimentation section).
• Public Hub push at end-of-run, wrapped in try-except. On HF Jobs (ephemeral env) ALSO enable in-trainer push (pushtohub=True + hubstrategy="everysave"). Details in references/hfjobsexecution.md.

4. Constraints the produced script must satisfy

These are non-negotiable contracts. Implementation lives in the production templates and references. Do not reinvent.

• Capture the pre-training evaluator score as baseline_eval before trainer.train().
• Emit a single end-of-run line: VERDICT: WIN|MARGINAL|REGRESSION | score=... | baseline=... | delta=.... A monitor scrapes for this.
• Silence httpx, httpcore, huggingface_hub, urllib3, filelock, fsspec to WARNING (otherwise HF download URLs flood the agent's context).
• Tee logs to logs/{RUN_NAME}.log.
• End with model.pushtohub(...) wrapped in try/except.
• Smoke-test before any long run (maxsteps=1 + tiny dataset slice). The production templates show one common pattern (SMOKETEST env var).
• [CrossEncoder] Include EarlyStoppingCallback(patience>=3). CE rerankers often peak mid-training and regress.
• [SparseEncoder] Log queryactivedims / corpusactivedims on the verdict line. High nDCG with collapsed sparsity is not a win. The keys come back name-prefixed (e.g. ...queryactive_dims). Use suffix matching to pluck them. See the SPARSE production template for the exact pattern.
• [MultiVectorEncoder] Match scale to the scoring mode on any MNRL-family loss: near 1.0 for unnormalized MaxSim (do not copy scale=20.0 from bi-encoder MNRL), roughly the average query length with length-normalized MeanMaxSim, since each score is divided by its query's token count. XTRScores is a train-only similarity_fct: the evaluators reject it, so evaluation always scores with MaxSim, including for XTR-trained models.

5. Workflow

1. Identify the model type (§1). Ask if ambiguous.
2. Load the §2 required-reading files for that type.
3. Open scripts/train<type>example.py and copy it as your starting point.
4. Replace MODELNAME, DATASETNAME, RUNNAME, the loss, and the evaluator with the user's task. Cross-check loss/data-shape match against references/losses<type>.md. Cross-check the metricforbestmodel key against references/evaluators<type>.md (named evaluators format the key as eval{name}{primary_metric}).
5. Smoke-test (max_steps=1).
6. Run.
7. After the run, append to logs/experiments.md and propose iteration if the verdict is weak/marginal.

Prerequisites

pip install "sentence-transformers[train]>=5.0"        # add [train,image] / [audio] / [video] for [SentenceTransformer] multimodal
                                                       # [MultiVectorEncoder] requires >=6.0
pip install trackio                                    # optional tracker (or wandb / tensorboard / mlflow)
hf auth login                                          # or set HF_TOKEN with write scope (for Hub push)

GPU strongly recommended. CPU works only for demos and [SentenceTransformer] StaticEmbedding.