PromptLayer Skill

Product summary

PromptLayer is a platform for managing prompts, workflows, and evaluations outside your application code. It provides a Prompt Registry for versioning and releasing prompts with labels (e.g., "prod", "staging"), observability tools for logging and tracing LLM requests, Tables for running evaluations and batch jobs, and Workflows for building multi-step AI systems with conditional logic. Use the Python SDK (pip install promptlayer) or JavaScript SDK (npm install promptlayer) to fetch prompts and log requests. Authenticate with X-API-Key header or environment variable PROMPTLAYERAPIKEY. Primary docs: https://docs.promptlayer.com

When to use

• Prompt management: Create, version, and release prompts without code changes using release labels
• Observability: Log and trace LLM requests, associate them with prompt templates, add tags and metadata for filtering
• Evaluations: Build Tables with datasets, run prompts against test cases, score outputs with custom logic or LLM-as-judge
• Workflows: Build multi-step AI systems combining prompts, code, conditionals, loops, and external API calls
• Batch operations: Import request history or CSV data, run prompts across rows, compare model outputs, backtest changes
• Tool management: Define reusable tools in the Tool Registry, reference them in prompts, enable auto-execution
• Team collaboration: Share prompts for review, track version history, manage workspace access and roles

Quick reference

SDK initialization

from promptlayer import PromptLayer

client = PromptLayer(api_key="pl_your_key")

import { PromptLayer } from "promptlayer";

const client = new PromptLayer({ apiKey: "pl_your_key" });

Core methods

REST API endpoints

Template variables

Use f-string or Jinja2 syntax in prompts:

f-string: "Hello {{name}}, you have {{count}} messages"
Jinja2: "Hello {{ name }}, you have {{ count }} messages"

Release labels

Assign labels to prompt versions for deployment without code changes:

# Fetch with label
response = client.run(
    prompt_name="my-prompt",
    prompt_release_label="prod"  # or "staging", "beta_users"
)

Observability enrichment

Add context to requests:

response = client.run(
    prompt_name="my-prompt",
    input_variables={...},
    tags=["experiment-1", "user-test"],
    metadata={"user_id": "123", "session": "abc"}
)

Decision guidance

Workflow

1. Create and manage a prompt

1. Go to PromptLayer dashboard → New → Prompt
2. Name the prompt (e.g., my-classifier)
3. Write messages with template variables: {{variable_name}}
4. Set model and parameters
5. Click Save Template to create version 1
6. Test in Playground with sample inputs
7. Hover over version → Add Release Label → enter prod
8. Code now fetches with client.run(promptname="my-classifier", promptrelease_label="prod")

2. Log and observe requests

1. Initialize client with API key
2. Call client.run() or client.log_request() with tags and metadata
3. Go to Logs in dashboard, search by prompt name or tag
4. Click a log to inspect input, output, tokens, cost, latency
5. Use Search to filter by date, metadata, or content
6. Use Analytics to aggregate (e.g., avg latency by model)

3. Build an evaluation

1. Create a Table → Create from a prompt or Import request history
2. Add text columns for inputs, expected outputs, metadata
3. Add a computed column → Prompt Template → select your prompt
4. Add a scoring column → LLM-as-judge or Custom code
5. Configure scoring criteria (e.g., "Output contains required sections")
6. Click Run to execute prompts and score all rows
7. Review History to compare versions; use Analytics to track scores over time

4. Build a workflow

1. Go to New → Workflow
2. Drag nodes onto canvas: Prompt Template, Code, Conditional, For Loop, etc.
3. Connect nodes with edges; add conditionals for branching
4. Mark final node as Output Node
5. Click Save to create version 1
6. Add release label (e.g., prod)
7. Run with client.runworkflow(workflowidorname="...", input_variables={...})
8. View execution trace in Traces menu

5. Integrate with CI/CD

1. Create an evaluation pipeline (Table with scoring)
2. When saving a prompt version, select the evaluation pipeline
3. Each new version auto-runs the eval; scores appear in version history
4. Use webhooks (prompttemplateversioncreated, prompttemplatelabelmoved) to trigger external CI/CD

Common gotchas

• API key not set: Ensure PROMPTLAYERAPIKEY environment variable or pass api_key to client. Requests fail silently if missing.
• Provider API keys required: Set OPENAIAPIKEY, ANTHROPICAPIKEY, etc. as environment variables. PromptLayer never sends these to its servers; calls are made locally.
• Template variable mismatch: If prompt expects {{name}} but you pass inputvariables={"username": "..."}, rendering fails. Variable names must match exactly.
• Release label not found: If you reference a label that doesn't exist, the API returns the latest version. Always verify the label is attached to the intended version.
• Stale cells in Tables: After changing a computed column's source, cells marked "stale" won't recalculate automatically. Click Run or use the API to trigger recalculation.
• Scoring requires consistent types: All cells in a score column must be boolean or all numeric. Mixed types break scoring.
• Workflow node dependencies: If a node depends on another's output, ensure the dependency is listed. Circular dependencies are rejected.
• Trace closed: Once a trace is marked closed (via close_after=true in OTLP), no new spans can be added. Plan trace closure carefully.
• Custom logging format: log_request() requires input and output in Prompt Blueprint format (chat messages as arrays, not strings).
• Deprecated endpoints: Datasets and Evaluations APIs are deprecated. Use Tables API for new workflows.

Verification checklist

Before submitting work with PromptLayer:

• Prompt template saved and tested in Playground with sample inputs
• Release label (e.g., "prod") attached to the correct version
• API key set as environment variable or passed to client
• Provider API keys (OpenAI, Anthropic, etc.) set as environment variables
• Template variables in code match prompt definition exactly
• Tags and metadata added to client.run() for filtering and debugging
• Evaluation pipeline created and run; scores reviewed for regressions
• Workflow nodes connected correctly; output node marked
• Traces visible in dashboard; request logs show correct prompt association
• Search and analytics queries return expected results
• No deprecated Datasets/Evaluations endpoints used; Tables API used instead

Resources

• Comprehensive navigation: https://docs.promptlayer.com/llms.txt
• Quickstart: https://docs.promptlayer.com/quickstart
• Python SDK: https://docs.promptlayer.com/sdks/python
• REST API Reference: https://docs.promptlayer.com/reference/introduction

For additional documentation and navigation, see: https://docs.promptlayer.com/llms.txt