Nextflow Pipeline to Galaxy Workflow

When to Use

Use this skill when:

• Converting a complete Nextflow pipeline (e.g., nf-core pipeline)
• Creating a full Galaxy solution with multiple workflows
• Planning a large-scale conversion project

Don't use this skill if:

• Converting a single process (use nf-process-to-galaxy-tool instead)
• Converting a single subworkflow (use nf-subworkflow-to-galaxy-workflow instead)

Step-by-Step Process

Step 1: Clarify Workflow Scope with User

REQUIRED before starting conversion - ask the user:

1. What parts of the Nextflow pipeline to convert?

- Full pipeline end-to-end? - Specific subworkflow only? - Exclude certain optional steps? - Example: "Do you want preprocessing + alignment + variant calling + annotation, or just variant calling?"

1. Quality control and reporting requirements:

- Include QC steps (FastQC, etc.)? - Include aggregated reporting (MultiQC)? - Include intermediate QC checks?

1. Best-practice scientific / bioinformatics sanity check (REQUIRED):

- If the requested scope has conceptual issues (e.g., missing required preprocessing steps like sort/index, omitting essential QC, incompatible inputs/reference/annotation, or a too-small subset that would violate common analysis best practices), flag the issue. - Ask the user whether they want to: - Expand/adjust the scope to address the best-practice issue(s), or - Proceed with the user’s requested scope as-is (explicitly confirmed).

1. Workflow metadata:

- Workflow name (descriptive, user-facing) - Author/Creator name(s) - License (e.g., MIT, Apache-2.0, GPL-3.0) - Description/Annotation - Tags for categorization

NEVER use placeholder values - always get real information from the user. NEVER assume scope - explicitly confirm what to include/exclude.

Step 2: Analyze Pipeline Structure

Read the main Nextflow file and all imported workflows/subworkflows.

Document:

• All processes used
• Data flow between processes
• Conditionals and branches
• Input/output patterns

Step 3: Check Tool Availability and Requirements (CRITICAL)

For each process, verify a corresponding Galaxy tool exists:

• Search tools-iuc repository and verify tool XML exists
• Read the tool XML to confirm:

- Exact tool ID and current version - Actual input/output parameter names (for connections) - Conditional parameter structures - Whether tool requires pre-built databases/indices

• NEVER assume a tool exists without verification
• NEVER use placeholder or non-existent tools
• If a tool doesn't exist, inform user and discuss alternatives

CRITICAL: Verify tool owner/repository (COMMON ERROR):

• Tools can exist under multiple owners (e.g., devteam, iuc, bgruening)
• Check the actual ToolShed at https://toolshed.g2.bx.psu.edu/ to confirm correct owner
• Example: freebayes is under devteam, NOT iuc
• Example: samtools_sort is under iuc, NOT devteam
• Tool ID format: toolshed.g2.bx.psu.edu/repos/{owner}/{repo}/{tool}/{version}
• Wrong owner = "tool not available" error even if tool exists
• Always verify owner before using tool in workflow

Check for tool dependencies:

• Does the tool need a database built first? (e.g., SnpEff needs database, not just GTF)
• Does the tool need sorted/indexed input? (e.g., variant callers need sorted BAM + index)
• Are there helper tools needed? (e.g., samtools sort/index after alignment)

Use the check-tool-availability.md reference.

Prefer a splitter approach when the Nextflow workflow:

• Has multiple “modes” controlled by flags (--mode, --skip_*, etc.)
• Has large optional branches triggered by optional inputs
• Exposes a very large parameter surface (many knobs that overwhelm a Galaxy form)
• Produces materially different outputs depending on configuration

Splitter pattern:

• Create 2–N intent-focused Galaxy workflows with simpler inputs
• Keep an “advanced” variant only when needed
• Minimize workflow conditionals; prefer separate workflows when branches are substantial

Deliverable: your plan should explicitly state whether you are producing:

• One Galaxy workflow
• Multiple related Galaxy workflows (recommended for complex pipelines)
• Optional “meta-workflow” that chains subworkflows (only if it improves usability)

Example (CAPHEINE):

CAPHEINE Pipeline:
├── PIPELINE_INITIALISATION (setup)
├── CAPHEINE (main analysis)
│   ├── PROCESS_VIRAL_NONRECOMBINANT (preprocessing subworkflow)
│   ├── HYPHY_ANALYSES (analyses subworkflow)
│   └── DRHIP (aggregation)
└── PIPELINE_COMPLETION (reporting)

CAPHEINE splitter example (foreground branch): CAPHEINE has an optional branch controlled by providing a foreground regexp/list. In Galaxy, it may be clearer to publish two workflows:

• Workflow A: CAPHEINE (no foreground)

- Inputs: reference genes, unaligned sequences - Runs the baseline HyPhy analyses - Outputs: baseline result set

• Workflow B: CAPHEINE (with foreground)

- Inputs: reference genes, unaligned sequences, foreground regexp/list - Runs additional foreground-dependent HyPhy analyses - Outputs: baseline + additional foreground-dependent results

This keeps each workflow form smaller and avoids large conditional sections.

Step 2: Check Tool Availability for All Processes

Use: ../check-tool-availability.md and ../scripts/check_tool.sh

Create comprehensive tool inventory:

cd ../
# List all unique tools used in pipeline
for tool in tool1 tool2 tool3; do
    ./scripts/check_tool.sh $tool
done

Document all findings in a table:

| Process | Tool | Status | Action |
|---------|------|--------|--------|
| PROCESS_A | tool_a | ✅ Exists | Use existing |
| PROCESS_B | tool_b | ❌ Missing | Create tool |

Step 3: Create Conversion Plan

Plan must include:

1. Tool inventory (from Step 2)

- Existing tools and their locations - Missing tools that need creation

1. Tool creation strategy (for missing tools)

- Which should go in tools-iuc? - Which should be custom? - Ask user about tools-iuc access

1. Workflow structure

- Single workflow or multiple related workflows? - If multiple: how do they differ by intent/inputs/outputs? - Nested subworkflows? - Where do you intentionally split to reduce conditionals/knobs?

1. Implementation order

- Which tools to create first - Which workflows to build first - Testing strategy

Present plan to user and wait for approval before implementing.

See: ../tool-sources.md for tool placement decisions

Step 4: Create Missing Tools

For each missing tool, use the nf-process-to-galaxy-tool skill.

Wait for all tools to be created and validated before continuing.

Step 5: Build Workflows

For each subworkflow, use the nf-subworkflow-to-galaxy-workflow skill.

Caveats (important for most pipelines):

- UUIDs must be in proper UUID4 format: - Galaxy validates that all uuid fields are valid UUID4 strings (e.g., 550e8400-e29b-41d4-a716-446655440000). - UUIDs must be unique across the entire workflow JSON: - Galaxy rejects workflows if any uuid is duplicated (common failure: copying the workflow uuid into a step uuid, or reusing the same step UUID across multiple steps). - Check uniqueness for: - Workflow-level uuid - Every steps.<n>.uuid - Every steps.<n>.workflow_outputs[*].uuid - Validate before import (e.g., parse JSON and ensure the set of UUIDs has no duplicates).

- Interpret Galaxy warnings correctly: The user may provide a list of warnings produced by Galaxy for a workflow they try to import. - Benign (often OK): - Warnings for optional advanced parameters (e.g. “Disable grouping… Using default False”). - These usually just indicate the workflow JSON omitted optional parameters and Galaxy filled defaults. - Usually indicates a real workflow bug: - Warnings where the missing value is a dataset input (e.g. “BAM file … default ''”, “VCF Data … default ''”, “GFF dataset … default ''”). - This typically means: - inputconnections uses the wrong parameter name, or - a conditional selector in toolstate was not set, so Galaxy expects a different input branch. - Environment mismatch (action required): - “Tool is not installed” → the target Galaxy instance doesn’t have that exact tool_id. - “Using version X instead of version Y specified” → Galaxy substituted an installed version. - This is often OK, but you must re-check parameter names/outputs against the installed tool.

- These two mistakes are extremely common — preempt them: - Tool ID / owner / version mismatch: - The workflow may reference a real tool, but under the wrong owner or an uninstalled version. - Symptoms: “Tool is not installed”, “tool not available”, or silent version substitution. - Mitigation: verify ToolShed owner + inspect the tool XML for the exact toolid + version. - Conditional branch / input key mismatch: - The tool may have the correct upstream dataset available, but Galaxy still warns that the dataset input is empty. - Symptoms: “No value found for 'BAM file'… default ''” / “No value found for 'GFF dataset'… default ''”. - Mitigation: set the correct selector in toolstate and use the exact conditional parameter path in input_connections.

- Tool forms often require setting conditional selectors in toolstate: - Many tools expose inputs only after a selector is set (e.g. “use cached genome vs history dataset”, “single vs collection”, “region mode on/off”). - Even if you wire inputconnections correctly, Galaxy may still treat the input as missing unless the selector branch is chosen in tool_state.

- Validation checkpoint (recommended): - After generating the .ga, import into the target Galaxy and scan for: - Any “Tool is not installed” messages (fix tool IDs/owners/versions) - Any dataset-input warnings defaulting to empty (fix input_connections key names and/or conditional selectors) - Any version substitutions that might change parameter/output names - If any dataset-input warnings remain, treat the workflow as not runnable until resolved.

- Iterate using Galaxy’s import warnings report: - Do not assume the user will proactively provide these warnings. - If your first draft doesn’t import cleanly, ask the user to: - Import the .ga into their Galaxy instance - Copy/paste the resulting import warnings report - Use that report to quickly identify which failures are: - Missing tools (instance mismatch) - Wrong tool IDs/owners/versions - Wrong inputconnections keys - Missing conditional selectors in toolstate

- Tool existence and repository owner must be verified: - Tools can exist under multiple owners; wrong owner produces “tool not available” errors. - Confirm tool IDs via ToolShed URLs and/or tool XML in the authoritative repository.

• Tool existence must be verified (CRITICAL):

- NEVER reference a tool without verifying it exists in tools-iuc or target repository. - Check the actual tool XML file to confirm tool ID and versions. - NEVER use placeholder or assumed tool names. - If a tool doesn't exist, inform user and discuss alternatives

• Tool semantics must be validated (tool may exist but still be wrong) (CRITICAL):

- Do not stop at “the tool exists” — confirm it performs the intended transformation. - If semantics are unclear, ask the user what behavior is expected before drafting the step. - Example (CAPHEINE): seqkitsplit2 exists, but it splits into parts/chunks; for “one FASTA record → one dataset in a collection”, use an appropriate splitter tool (e.g. ToolShed rnateam/splitfasta / rbcsplitfasta).

• Tool input/output connections are REQUIRED (CRITICAL):

- Every tool step must have inputconnections that wire it to upstream steps (except workflow inputs). - Tools without connections will not receive data and will fail at runtime. - Each connection must specify: upstream step id + exact outputname from that step. - Read the actual tool XML to get exact input parameter names - do not guess. - Parameter names often use conditional paths (e.g., referencecond|referencehistory not reference). - Input names vary by tool (e.g., CAwlign uses fasta not query). - Output names must match tool definitions (e.g., labeled_tree not output). - Incorrect connections cause execution failures even if import succeeds. - Explicitly tell user which connections need verification.

• Dataset collections vs individual datasets:

- Paired FASTQ inputs should use datacollectioninput type with collection_type: "paired". - Tools that process collections need collection-aware parameter names (check tool XML). - Some tools iterate over collections, others process them as a unit. - Collection outputs have type: "input" to preserve collection structure.

• Tool dependencies and helper steps:

- Some tools require pre-built databases (e.g., SnpEff needs database built from GTF, not raw GTF). - Some tools require sorted/indexed input (e.g., variant callers need sorted BAM + BAI index). - Add helper tools as needed (e.g., samtools sort/index after alignment). - Check Nextflow workflow for these preprocessing steps.

• Tool versions in drafted .ga files are often placeholders:

- If you have access to the target Galaxy instance (UI or API), resolve each step's toolid/toolversion to what is actually installed (tool revisions and +galaxyN suffixes differ). - If you cannot check the instance, use the most recent version of each tool and treat any toolid/toolversion you emit as a placeholder and explicitly tell the user they must verify/adjust versions against their Galaxy.

Recommended structure:

• Break pipeline into logical workflow units
• Create subworkflows first
• Compose into main workflow(s)

Example (CAPHEINE):

Workflow 1: CAPHEINE_Preprocessing
  - Input: Raw sequences
  - Output: Alignment + Tree
  
Workflow 2: CAPHEINE_Analyses
  - Input: Alignment + Tree
  - Output: Analysis results + DRHIP report

Step 6: Integration Testing

Use the canonical testing docs: - Tool testing (Planemo): ../../tool-dev/references/testing.md - Workflow testing/validation (Galaxy instance): ../../galaxy-integration/galaxy-integration.md

../testing-and-validation.md is a short routing page that links to these.

At minimum:

1. Upload pipeline test data
2. Run workflows in sequence
3. Compare outputs to Nextflow results (structural/semantic match)
4. Document differences

Step 7: Documentation

Create documentation for users:

• Which workflows to run in what order
• Input data requirements
• Expected outputs
• Tool versions used
• Differences from Nextflow version (if any)

Quick Reference

Nextflow pipeline = Multiple Galaxy workflows + tools

Decomposition strategy:

1. Identify logical workflow boundaries
2. Check all tool availability
3. Create missing tools (use nf-process-to-galaxy-tool)
4. Build subworkflows (use nf-subworkflow-to-galaxy-workflow)
5. Compose and test

Resources

All detailed guides are in parent directory (../):

• check-tool-availability.md - Tool availability checking
• tool-sources.md - Where to create tools
• workflow-to-ga.md - Workflow creation guide
• nextflow-galaxy-terminology.md - Conceptual mappings
• testing-and-validation.md - Routing page to canonical testing docs

Related skills:

• nf-process-to-galaxy-tool - For creating missing tools
• nf-subworkflow-to-galaxy-workflow - For building workflow components

Complete Example: CAPHEINE Pipeline

See ../examples/capheine-mapping.md for full CAPHEINE conversion.

Key findings:

• ✅ 15/15 tools exist in tools-iuc (100% coverage)
• ✅ No tool creation needed
• ✅ Conversion is purely workflow assembly
• Recommended: 2 workflows (preprocessing + analyses)

This demonstrates the ideal case: Most established pipelines use common tools that already exist in Galaxy.