Multimodal Embedding Serving — User

Run and call the embedding service. Run commands yourself and relay output. REST base URL: http://localhost:9777 (host port hardcoded by setup.sh; container 8000).

When to Use

• Deploy the embedding service (Docker Compose or standalone) and confirm health
• Embed text, images, or videos over REST on port 9777
• Choose or switch among the CLIP/SigLIP/MobileCLIP/CN-CLIP/Blip2/QwenText models
• Integrate embeddings in-process via the Python SDK wheel
• Diagnose 400/422 errors or model-capability mismatches

Example Prompts

Sample Problem-solving scenarios this skill handles end-to-end:

Docs & deploy files — with or without a clone

All paths below are relative to microservices/multimodal-embedding-serving/ in the edge-ai-libraries repo. No clone? Fetch any of them from GitHub raw:

https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving/<path>

Load these existing docs only when needed:

1. Context routing — repo clone or standalone? REST or SDK?

• In-process Python integration wanted (no separate server): the service

doubles as an SDK — build the wheel with poetry build (needs the repo) and use getmodelhandler(...) + EmbeddingModel; see docs/user-guide/sdk-usage.md. Rule of thumb: default to REST; pick the SDK when a Python process embeds heavily and an HTTP hop per item would dominate. Note: MobileCLIP/Blip2 extras exist only in the Docker image, and the wheel is not on PyPI.

• Otherwise detect a clone:

``bash [ -f setup.sh ] && grep -q 'name = "multimodal-embedding-serving"' pyproject.toml 2>/dev/null \ && echo REPO || echo STANDALONE ` REPO → Step 2 from the microservice root. STANDALONE → fetch the two deploy files, then the exact same Step 2: `bash RAW=https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/multimodal-embedding-serving mkdir -p embedding-serving/docker && cd embedding-serving curl -fsSL $RAW/setup.sh -o setup.sh curl -fsSL $RAW/docker/compose.yaml -o docker/compose.yaml ` Already running (curl -sf localhost:9777/health`) → Step 3.

2. Bring-up (identical in both contexts)

1. Pick a model — default CLIP/clip-vit-b-32 (512-dim, text+image+video).

Trade-offs: docs/user-guide/supported-models.md.

1. setup.sh must be sourced and hard-fails without

EMBEDDINGMODELNAME. REGISTRYURL=intel selects the prebuilt image; --no-build prevents a source build. Run in the background — first start downloads the model: ``bash bash -c 'export EMBEDDINGMODELNAME="CLIP/clip-vit-b-32" REGISTRYURL=intel TAG=latest \ && source setup.sh && docker compose -f docker/compose.yaml up -d --no-build' ` Intel GPU: also export EMBEDDING_DEVICE=GPU` (setup.sh then auto-enables OpenVINO + THROUGHPUT mode).

1. Wait for readiness:

``bash until curl -sf http://localhost:9777/health; do sleep 5; done ``

3. Capability check (mandatory before non-text inputs)

curl -s http://localhost:9777/model/capabilities

QwenText models are text-only — image/video requests return 400. GET /model/current shows the exact loaded model id to use in requests.

4. Embed

Text (single string or list of strings):

curl -s http://localhost:9777/embeddings -H 'Content-Type: application/json' -d '{
  "model": "CLIP/clip-vit-b-32",
  "input": {"type": "text", "text": "a red truck at a loading dock"},
  "encoding_format": "float"
}'

Response: {"embedding": [...]} — a flat vector for text/image; a list of per-frame vectors for video inputs.

• model must equal the loaded model (else 400).
• Image: {"type":"imageurl","imageurl":"https://…"} (plain string, not a

nested object) or imagebase64. Video: videourl/videobase64/ videoframes with segmentconfig (numframes default 64, extractionfps, frameindexes) — full shapes and examples: docs/user-guide/api-reference.md.

5. Stop / clean

• docker compose -f docker/compose.yaml down
• Volumes ov-models (model caches) and data-prep persist; removing them

forces re-downloads — confirm with the user first.

Troubleshooting