East Execution Engine (e3)

e3 is a durable dataflow execution engine for East programs with content-addressable caching.

Quick Start

// src/index.ts
import { East, StringType } from '@elaraai/east';
import e3 from '@elaraai/e3';

// Define an input
const name = e3.input('name', StringType, 'World');

// Define a task
const greet = e3.task(
  'greet',
  [name],
  East.function([StringType], StringType, ($, n) =>
    East.str`Hello, ${n}!`
  )
);

// Bundle and export
const pkg = e3.package('hello', '1.0.0', greet);
await e3.export(pkg, '/tmp/hello.zip');
export default pkg;

# Create repository
e3 repo create .

# Import and deploy
e3 package import . /tmp/hello.zip
e3 workspace create . dev
e3 workspace deploy . dev [email protected]

# Execute dataflow
e3 start . dev

# Get result
e3 get . dev.tasks.greet.output

SDK Reference (@elaraai/e3)

e3.input(name, type, defaultValue?)

Define an input dataset at .inputs.${name}.

const name = e3.input('name', StringType, 'default');
const count = e3.input('count', IntegerType);

e3.task(name, inputs, fn, config?)

Define a task that runs an East function.

const greet = e3.task(
  'greet',
  [name],  // dependencies (inputs or other task outputs)
  East.function([StringType], StringType, ($, n) =>
    East.str`Hello, ${n}!`
  )
);

// With custom runner
const pyTask = e3.task(
  'py_task',
  [input],
  East.function([IntegerType], IntegerType, ($, x) => x.multiply(2n)),
  { runner: ['uv', 'run', 'east-py', 'run', '-p', 'east-py-std'] }
);

// Chain tasks via .output
const shout = e3.task(
  'shout',
  [greet.output],
  East.function([StringType], StringType, ($, s) => s.toUpperCase())
);

e3.customTask(name, inputs, outputType, command)

Define a task that runs a shell command.

const process = e3.customTask(
  'process',
  [rawData],
  StringType,
  ($, input_paths, output_path) =>
    East.str`python script.py -i ${input_paths.get(0n)} -o ${output_path}`
);

e3.package(name, version, ...items)

Bundle into a package. Dependencies are collected automatically.

const pkg = e3.package('myapp', '1.0.0', finalTask);

e3.export(pkg, zipPath)

Export package to a .zip file.

await e3.export(pkg, '/tmp/myapp.zip');

CLI Reference

Repository Commands

e3 repo create <repo>             # Create a new repository
e3 repo status <repo>             # Show repository status
e3 repo remove <repo> [-r]        # Remove a repository (-r to remove workspaces first; remote servers require empty repos)
e3 repo gc <repo> [--dry-run]     # Garbage collect unreferenced objects

Package Commands

e3 package import <repo> <zipPath>       # Import from .zip
e3 package export <repo> <pkg> <zipPath> # Export to .zip
e3 package list <repo>                   # List packages
e3 package remove <repo> <pkg>           # Remove package

Workspace Commands

e3 workspace create <repo> <name>           # Create workspace
e3 workspace deploy <repo> <ws> <pkg[@ver]> # Deploy package
e3 workspace export <repo> <ws> <zipPath>   # Export workspace
e3 workspace import <repo> <ws> <zipPath>   # Import package zip into workspace
e3 workspace list <repo>                    # List workspaces
e3 workspace status <repo> <ws>             # Show workspace status
e3 workspace remove <repo> <ws>             # Remove workspace

Data Commands

e3 list <repo> [path]                       # List tree contents
e3 list <repo> <path> -r                    # List all dataset paths recursively
e3 list <repo> <path> -l                    # Immediate children with type/status/size
e3 list <repo> <path> -r -l                 # All datasets with type/status/size
e3 get <repo> <path> [-f east|json|beast2]  # Get dataset value
e3 set <repo> <path> <file> [--type <spec>] # Set dataset from file

Path format: workspace.path.to.dataset

e3 get . dev.inputs.name
e3 get . dev.tasks.greet.output
e3 set . dev.inputs.name data.east

Execution Commands

e3 start <repo> <ws> [--filter] [--concurrency <n>] [--force]
e3 run <repo> <pkg/task> [inputs...] -o <output>
e3 watch <repo> <ws> <source.ts> [--start] [--abort-on-change]
e3 logs <repo> <path> [--follow]

Utility Commands

e3 convert [input] [--from <fmt>] [--to <fmt>] [-o <output>]

Authentication (for remote servers)

e3 login <server>                 # Log in using OAuth2 Device Flow
e3 logout <server>                # Log out and clear credentials
e3 auth status                    # List all saved credentials
e3 auth token <server>            # Print access token (for curl/debugging)
e3 auth whoami [server]           # Show current identity

Remote URLs

All commands accept HTTP URLs instead of local paths:

# Start a server
e3-api-server --repos ./repos --port 3000

# Use remote repository
e3 repo create http://localhost:3000/repos/my-repo
e3 workspace list http://localhost:3000/repos/my-repo
e3 package import http://localhost:3000/repos/my-repo ./pkg.zip

Development Workflow

Watch Mode (recommended)

e3 watch . dev ./src/index.ts --start

Auto-compiles, deploys, and runs on file changes.

Manual Workflow

npm run build && npm run main
e3 package import . /tmp/pkg.zip
e3 workspace deploy . dev [email protected]
e3 start . dev

Packages

Project Structure

my-project/
├── package.json
├── tsconfig.json
├── pyproject.toml      # For Python runner
├── src/
│   └── index.ts        # Package definition
└── repo/               # Repository (created by e3 repo create)
    ├── objects/        # Content-addressable object store
    ├── packages/       # Package metadata
    └── workspaces/     # Workspace state

Caching

Tasks are cached by content hash. Re-runs only when:

• Task's East function IR changes
• Input values change

Use --force to bypass: e3 start . dev --force