VoxFlow Transcribe / Dub / Translate Skill

Audio/video → text → other languages → re-voiced video. Five tightly-related commands:

Prerequisites

• npm install -g voxflow and voxflow login
• ffmpeg installed (brew install ffmpeg / sudo apt install ffmpeg) — required for dub --video, video-translate, audio extraction
• Optional: whisper.cpp for local engine (no quota cost). Install via brew install whisper-cpp or compile from source.
• Optional: sox / rec for --mic recording.

🎙 asr — speech recognition

Transcribe a local file, remote URL, or live mic input. Cloud (Tencent ASR) or local (Whisper).

Quick start

voxflow asr --input recording.mp3                                        # default cloud, zh
voxflow asr --input meeting.wav --speakers --speaker-number 3            # speaker diarization
voxflow asr --input video.mp4 --format srt --lang 16k_zh -o out.srt     # video → SRT
voxflow asr --input recording.mp3 --engine local --model small           # offline Whisper
voxflow asr --url https://example.com/audio.mp3                          # remote URL
voxflow asr --mic --lang 16k_en                                          # live mic

Engines

Azure path (durable jobs)

The azure engine is the right choice when a file would time out on Tencent's flash mode (anything over ~2 hours, or anything you want word-level timestamps on). The CLI:

1. Probes duration with ffprobe.
2. Extracts to 16 kHz mono WAV with ffmpeg to save uplink bandwidth (falls through to original file if ffmpeg is missing).
3. Uploads to Cloudflare R2 via backend-signed URLs (multipart for files >100 MiB).
4. Creates an asr_jobs row server-side, charges quota (per audio minute, see below), submits to Azure.
5. Polls the backend until terminal state.

Ctrl+C is safe — the server-side job keeps running and ~/.config/voxflow/jobs/asr-<jobId>.json lets you resume later via --job-id <uuid> or voxflow asr-jobs show <jobId>. Quota is automatically refunded on failure or cancel.

# 30-min Japanese meeting with speaker diarization → word-timed SRT
voxflow asr --input meeting-2h.mp4 --engine azure --lang ja-JP --diarize --speaker-number 4 --format srt

# Auto-detect language across the four Azure default candidates (en/zh/ja/ko)
voxflow asr --input multilang.mp3 --engine azure --lang auto

# Resume a job after closing the terminal
voxflow asr --engine azure --job-id 6f3c2798-87bf-4367-bb4c-08b872e12bef

Language codes: Azure uses BCP-47 (ja-JP, zh-CN, en-US, ko-KR). Tencent codes (16kzh, 16kja, …) are auto-mapped, so you can keep --lang 16k_zh working across engines.

Modes (cloud only)

Options

Common scenarios

Meeting → SRT with speakers

voxflow asr --input meeting.mp4 --speakers --speaker-number 4 --format srt -o meeting.srt

Quick transcript (no quota cost)

voxflow asr --input recording.mp3 --engine local --model small --format txt

Mixed Chinese + English audio

voxflow asr --input bilingual.mp3 --lang 16k_zh_en

📋 asr-jobs — browse Azure ASR job history

Azure batch jobs run server-side and survive across CLI sessions, so this command is the dashboard for them. All four subcommands talk to /api/asr/jobs/* directly.

# List the 20 most recent jobs (server-side, paginated)
voxflow asr-jobs list

# Show one job (with transcript preview if succeeded)
voxflow asr-jobs show 6f3c2798-87bf-4367-bb4c-08b872e12bef

# Re-emit the transcript locally without re-running ASR
voxflow asr-jobs download 6f3c2798-87bf-4367-bb4c-08b872e12bef --format srt -o meeting.srt

# Cancel a running job (refunds quota, deletes Azure-side transcription)
voxflow asr-jobs cancel 6f3c2798-87bf-4367-bb4c-08b872e12bef

# Machine-readable
voxflow asr-jobs list --json
voxflow asr-jobs show <jobId> --json

The download subcommand is idempotent — handy if you ran asr long ago and want to re-emit in a different format without paying again. The transcript is stored in result_json server-side for at least 30 days (RLS-scoped, only the owner can read it).

🌐 translate — SRT / text / file translation

LLM batch translation. Accepts SRT (preserves timing), inline text, or a .txt / .md file.

# Translate subtitles, preserve timing
voxflow translate --srt en.srt --to zh -o zh.srt

# Inline text
voxflow translate --text "你好世界" --to en

# File
voxflow translate --input article.md --to en -o article-en.md

# Smart re-timing for length-mismatched languages
voxflow translate --srt zh.srt --to en --realign

# Smaller batches for very long subtitle files
voxflow translate --srt long.srt --to ja --batch-size 5

Options

Use --realign when translating between languages with very different word density (CJK ↔ Latin) so subs don't crash into each other.

🎬 dub — SRT → timeline-aligned TTS

Re-voice a video using its SRT. Per-caption TTS placed at the exact timestamp. Optional speed compensation when speech overflows the slot.

# Basic — output WAV
voxflow dub --srt subtitles.srt -o dubbed.wav

# Merge into video (requires ffmpeg)
voxflow dub --srt subtitles.srt --video input.mp4 -o dubbed.mp4

# Multi-speaker (SRT lines tagged `[Speaker: Name]`)
voxflow dub --srt show.srt --voices speakers.json --speed-auto -o dubbed.wav

# With background music ducked under speech
voxflow dub --srt narration.srt --bgm music.mp3 --ducking 0.3 -o final.wav

# Patch mode — re-synthesize one caption without full re-run
voxflow dub --srt subtitles.srt --patch 5 -o dub-existing.wav

speakers.json format

{
  "Alice": "v-female-R2s4N9qJ",
  "Bob":   "v-male-s5NqE0rZ",
  "Narrator": "v-female-T8m4WxP7"
}

SRT captions tag speaker inline:

1
00:00:01,000 --> 00:00:03,500
[Speaker: Alice] Hello from Alice!

Unmatched speakers fall back to --voice.

--speed-auto (overflow protection)

When TTS audio is longer than the SRT time slot, dub computes alpha = Traw / Ttarget and re-synthesizes at speed × alpha. If alpha > 2.0, prints OVERFLOW_WARNING and tries at speed 2.0 (max).

Options

Pipeline

SRT → parseSrt() → captions[]
  ↓
per-caption TTS (with voice mapping + speed compensation)
  ↓
buildTimelineAudio(segments) → 24kHz / 16-bit / mono WAV
  ↓ optional
mixWithBgm() → mixed WAV
  ↓ optional
mergeAudioVideo() → MP4

🌍 video-translate — end-to-end pipeline

ASR → translate → dub → burn subtitles → auto-open. The "translate this video into another language" one-shot.

Pipeline (5 stages):

[1/5] FFmpeg check
[2/5] ASR (flash mode → Azure fallback) → fine-grained per-sentence SRT
[3/5] Translate → quality check → auto-retry untranslated captions
[4/5] TTS dub (speed-auto) → merge audio into video
[5/5] Burn subtitles into video → auto-open output

Output: <input>-<lang>.mp4 (with burned subtitles) + <input>-<lang>.srt (standalone subtitle file)

# Auto-detect source, dub to English
voxflow video-translate --input video.mp4 --to en

# Explicit source, with re-timing
voxflow video-translate --input video.mp4 --from zh --to en --realign

# Specific voice for dubbed track
voxflow video-translate --input video.mp4 --to ja --voice v-male-Bk7vD3xP

# All-local (no quota cost — uses Whisper for ASR; LLM still consumes quota for translation)
voxflow video-translate --input video.mp4 --to en --engine local

# Keep SRT + audio intermediates for inspection
voxflow video-translate --input video.mp4 --to en --keep-intermediates

Options

For multi-speaker dubbing, run asr separately first with --speakers, then edit the resulting SRT to tag [Speaker: Name] lines, then run dub with --voices — video-translate doesn't preserve speaker tags through translation by default.

🚀 publish — one-shot end-to-end (recommended for agents)

publish bundles ASR / translation / dubbing / merge into a single command and emits a structured JSON result via --json. Use this instead of stitching asr + translate + dub yourself when an agent or web orchestrator drives the run.

Three build modes (auto-selected from inputs):

Publish targets:

• --publish local (default) — copy artifact to ./published/<base>-<platform>-<ts>.mp4
• --publish webhook --publish-webhook <url> — POST artifact metadata to webhook, return URL
• --publish none — skip publish step, return final MP4 path only

Always pass --json when invoking from a Skill — stdout becomes a single JSON doc (artifact.path / artifact.durationSec / artifact.sizeBytes / publish.publishUrl / quotaUsed, etc.) instead of human-readable progress.

# End-to-end: source video → translated dubbed publish
voxflow publish --input talk.mp4 --to en --json

# Hand off to webhook publisher
voxflow publish --input talk.mp4 --to ja --publish webhook --publish-webhook https://hook.example.com/in --json

📋 summarize — content → slides (and optional video)

Long video/audio/text → ASR (if needed) → LLM summary → PPTX deck. Optional TTS narration and Remotion video render.

voxflow summarize --input lecture.mp4
voxflow summarize --input meeting.mp4 --lang zh --slides 10
voxflow summarize --input podcast.mp3 --engine local --tts
voxflow summarize --input lecture.mp4 --video --scheme aurora
voxflow summarize --text "长篇文章内容..." --slides 6 --lang zh

summarize --video overlaps with present. Use summarize when starting from a long video/audio source; use present when starting from a topic / pasted article.

Common workflows

Translate a YouTube video to Chinese (full pipeline)

# 1. Download (yt-dlp / etc.) → video.mp4
# 2. One-shot:
voxflow video-translate --input video.mp4 --to zh --realign --keep-intermediates
# Output: video-zh.mp4 + intermediate .srt files

Generate clean SRT from a recording

voxflow asr --input recording.mp3 --speakers --speaker-number 3 --format srt -o clean.srt

Long meeting / lecture (1 hr+) → word-timed SRT

# Kick off — Ctrl+C is safe, the server keeps working.
voxflow asr --input lecture-2h.mp4 --engine azure --lang ja-JP --diarize --speaker-number 4

# Days later, browse what ran:
voxflow asr-jobs list

# Re-download in TXT instead of SRT — no extra quota:
voxflow asr-jobs download <jobId> --format txt -o lecture.txt

Re-dub an existing SRT with different voices

voxflow dub --srt show.srt --video show.mp4 --voices new_cast.json --speed-auto -o new_cast.mp4

Patch a single bad caption without re-rendering

# Edit clean.srt's caption 12, then:
voxflow dub --srt clean.srt --patch 12 -o updated.wav

Lecture / podcast → summary deck

voxflow summarize --input lecture-2h.mp4 --slides 12 --lang zh --tts -o summary.pptx

Quota cost

Always voxflow status before long jobs.

Rules

1. Pick the right engine for the file:

- local — best when you have whisper.cpp installed and want zero quota. - cloud (Tencent) — best for short clips (≤2 hr), fast turnaround. - azure — best for 30-min+ recordings, word-level timestamps, multi-locale auto-detect, or anything you might need to resume after a disconnect. Server-side jobs survive Ctrl+C.

1. Translation batch size is critical:

- Default --batch-size 5 is the sweet spot. Larger batches (10+) cause LLM to silently drop captions. - If you see untranslated lines in output, reduce to --batch-size 3. - The CLI auto-retries failed batches up to 2 times with validation.

1. --realign when translating between very different language families (EN ↔ JA / ZH).
2. For multi-speaker dub, generate the SRT with --speakers first, manually verify and tag, then dub --voices.
3. Always test on a 1-minute clip before running video-translate on a long video — it consumes quota linearly.
4. After completion, auto-play: open output.mp4.
5. Keep --keep-intermediates on for the first run of any pipeline so you can inspect what failed.
6. For Azure jobs that are still running, don't re-submit if the user closes the terminal — voxflow asr-jobs list to find the jobId, then voxflow asr --engine azure --job-id <uuid> (or voxflow asr-jobs download <uuid>) resumes free of charge.
7. video-translate auto-enables --speed-auto — if TTS audio overflows the time slot, it auto-adjusts. No manual flag needed.
8. Quality self-check: After translation, the CLI checks if >30% captions are unchanged (untranslated). If so, it warns you to use a smaller batch size.

Troubleshooting