Entry point: /faion-net — invoke this skill for automatic routing to the appropriate domain.

Multimodal AI Skill

Communication: User's language. Code: English.

Purpose

Handles multimodal AI applications. Covers vision, image generation, video generation, speech, and voice synthesis.

Scope

Quick Start

Methodologies (12)

Vision (2):

• vision-basics: Image understanding, OCR, scene analysis
• vision-applications: Use cases, production patterns

Image Generation (2):

• img-gen-basics: Prompt engineering, models
• img-gen-tools: DALL-E 3, Midjourney, Stable Diffusion

Video Generation (2):

• video-gen-basics: Fundamentals, prompting
• video-gen-tools: Sora, Runway, Pika, Luma

Speech-to-Text (2):

• speech-to-text-basics: Whisper API, real-time
• speech-to-text-advanced: Diarization, timestamps

Text-to-Speech (2):

• tts-basics: Voice selection, SSML
• tts-implementation: Production patterns, streaming

Voice (2):

• voice-basics: Real-time voice, cloning
• voice-implementation: Integration patterns

Code Examples

GPT-4o Vision

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://..."}}
        ]
    }]
)

DALL-E 3 Image Generation

from openai import OpenAI

client = OpenAI()

response = client.images.generate(
    model="dall-e-3",
    prompt="A futuristic city with flying cars",
    size="1024x1024",
    quality="hd",
    n=1
)

image_url = response.data[0].url

Whisper Speech-to-Text

from openai import OpenAI

client = OpenAI()

audio_file = open("speech.mp3", "rb")
transcription = client.audio.transcriptions.create(
    model="whisper-1",
    file=audio_file,
    response_format="verbose_json",
    timestamp_granularities=["word"]
)

print(transcription.text)

OpenAI TTS

from openai import OpenAI
from pathlib import Path

client = OpenAI()

response = client.audio.speech.create(
    model="tts-1-hd",
    voice="alloy",
    input="Hello, this is a test of text to speech."
)

response.stream_to_file("speech.mp3")

Gemini Vision

import google.generativeai as genai

genai.configure(api_key="...")
model = genai.GenerativeModel('gemini-pro-vision')

image = PIL.Image.open("image.jpg")
response = model.generate_content([
    "Describe this image in detail",
    image
])

print(response.text)

Model Comparison

Vision Models

Image Generation

Speech-to-Text

Text-to-Speech

Use Cases

Related Skills

Multimodal AI v1.0 | 12 methodologies