ae-dataops

CRITICAL - This skill is self-contained. Use the Global AE CLI Rules below; do not require a separate shared skill for DataOps-side tasks.

The AE Data Development and Operations domain provides capabilities for data warehouse management, flow orchestration, IDE SQL queries, data integration, operations and backfill management, including the following subcommands:

Global AE CLI Rules

AE CLI (ae-cli) is the command-line tool for the AE / TE / ThinkingEngine analysis platform. For AE analysis-side requests, prefer ae-cli and this skill's reference docs over model memory.

Global parameters:

Output and errors:

• Successful commands return machine-readable JSON by default. Envelope may include optional notice.hostcompat.
• Failed commands return { "ok": false, "error": { "type": "...", "message": "...", "hint": "..." } } and exit non-zero.
• CRITICAL — Host compat (do this first): After each ae-cli run, check stderr and notice.hostcompat. If either is present, open the user reply with a short ⚠️ version warning and quote the npm i -g / npx skills add (or update-cluster) lines verbatim, then present the business result. Soft tip; ok: true can still carry the notice.

Safety constraints:

• Read commands can execute directly after required IDs and references are verified.
• Ordinary write commands execute without --yes; use --yes only for a high-risk-write command after explicit user confirmation.
• Never invent command names, flags, JSON payloads, project_id, resource IDs, field names, event names, property names, metric definitions, or date formats. Read the matching command reference and discover real project metadata first.
• NEVER fabricate or guess resource names (reports, dashboards, events, properties, metrics, clusters, tags, alerts). Always use list commands to discover real resources first. If a resource is not found after fuzzy search and full list fallback, explicitly tell the user "resource not found" and stop - do not proceed with fabricated names.

Domains for DataOps: dataopsrepo, dataopsdatatable, dataopsflow, dataopsoperations, dataopside, dataopsintegration

Core Concepts and Rules

You must understand the following key concepts before use, otherwise errors are highly likely.

ID System

Environment and Defaults

Schema Naming Rules

• DEV environment: ws${spaceCode}dev
• PROD environment: ws${spaceCode}product

Responsibility Boundaries

Flow Lifecycle

Create DEV Flow → Create/Update DEV SQL, Integration, Workflow Instance Check, or Task Instance Check Tasks → Configure Dependencies/Schedule → Preview Release → Release to PROD → PROD Manual Execution / Operations Troubleshooting

Backfill lifecycle: Discover eligible PROD flow → Create or fully update DRAFT job → Run explicitly → Search / inspect plans → Stop or rerun the complete job; delete only after target inspection

CRON Format (6 fields)

second minute hour day month weekday — Note: one more "second" field than standard 5-field format.

• 0 0 2 ? — Daily at 2 AM
• 0 0 /4 * ? — Every 4 hours
• 0 30 8 1-5 — Weekdays at 8:30

Preset Repository vs Non-Preset Repository

• Preset Repository (teetl): datasourceId is teetl@TASKENGINETRINO, database field is empty, requires gatewayConfig
• Non-Preset Repository: datasourceId is specific datasource ID, database field is required

Scenario Routing

Choose the appropriate scenario skill based on user intent to get complete step-by-step workflow guidance.

1. Space Discovery

dataops_repo exposes only one read command. Use it to discover a valid spaceCode before calling DataOps commands that require one. It returns createTime, spaceCode, and spaceDisplayName.

• If the user already provided a trusted spaceCode, reuse it.
• If spaceCode is unknown, run +list_spaces first.
• If exactly one space is returned, use its spaceCode.
• If multiple spaces are returned and the user intent does not identify one, ask the user which space to use. Do not guess.

# List spaces accessible to the current user
ae-cli dataops_repo +list_spaces

2. Data Table and View Management

Detailed workflow, command flags, examples, and parameter notes live in [references/dataops-table.md](references/dataops-table.md).

Key constraints:

• Start with dataopsdatatable +dictsearch_tables for visible DataOps catalog discovery.
• Use dataopside +searchtables only for raw engine metadata, and dataopside +idelist_tables only for known catalog/schema browsing.
• Create tables/views with dataopsdatatable, not dataopside; creation is DEV-only and must be published with +publish_entity.
• DDL follows Trino syntax; current-space view DDL should keep the literal ${env} placeholder.

3. Flow Orchestration

Flow orchestration is divided into two scenario skills: creation and configuration and execution and monitoring.

Lifecycle: DEV configuration and preview → Release to PROD → PROD manual execution and operations troubleshooting

Detailed creation/configuration commands live in [references/dataops-flow-create.md](references/dataops-flow-create.md). Detailed execution, monitoring, operation instance, task log, and stop commands live in [references/dataops-flow-monitor.md](references/dataops-flow-monitor.md). Persistent multi-date backfill jobs live in [references/dataops-backfill.md](references/dataops-backfill.md).

Key constraints:

• Create and update tasks in DEV, preview/release before PROD execution.
• Treat +deletetask as high-risk: verify the target with +getflow_overview, preview with --dry-run, and use --yes only after explicit user confirmation. Deletion affects DEV; release the flow to apply it to PROD.
• +execute_flow always runs PROD; it returns executeId for early stop.
• Prefer flowInstanceId from operations search for stable inspection and troubleshooting.
• A backfill job is persistent and batches multiple base dates; do not emulate it by looping +execute_flow.
• Create and run backfill jobs as separate steps. +rerunbackfilljob reruns the complete job, not only failed plans.
• +updatebackfilljob replaces a DRAFT job's complete configuration; inspect the job first and do not treat it as a partial patch. Treat +deletebackfilljob as high-risk and preview it with --dry-run before confirmation.
• Reference workspace parameters in task SQL as ${paramKey}.

4. IDE SQL Queries

Detailed metadata browsing, SQL query, async download, and cancel workflows live in [references/dataops-query.md](references/dataops-query.md).

Key constraints:

• IDE is query-only; create/modify/delete tables with dataops_datatable.
• Prefer dataopsdatatable +dictsearch_tables for table discovery unless raw engine metadata or schema browsing is required.
• Submit exactly one read-only SQL query. It creates a platform-bounded download task; rows are not returned inline and the result is not an unlimited or full export.

5. Data Integration

Detailed datasource, metadata browsing, sync solution, execution, and monitoring workflows live in [references/dataops-integration.md](references/dataops-integration.md).

Key constraints:

• Generate sourceConfig, sinkConfig, channelConfig, and fieldsMapping from the reference templates; do not invent keys.
• MySQL Source read partitioning uses sourceConfig.splitColumn; fieldsMapping.shardingKey is column metadata and must not be used for it.
• +savesyncsolution is not a partial patch: call +getsyncdetail --withParams true first, then submit complete configs. syncName is accepted for compatibility but ignored.
• Preset repository sync uses teetl@TASKENGINE_TRINO and requires gateway configuration.
• Use +listsyncruns to get taskId before stopping a running sync.

Reference Documentation

For detailed command flags and usage, please refer to the command documentation in the [references/](references/) directory.