Workflows
Deploy pipeline changes with confidence
Workflows and the Deployment Manager let you version pipeline graphs, preview diffs before you deploy them, and roll back safely—so teams can move fast without surprises.
Plan before you deploy
Preview the diff: what changes, and where it will run.
Scale selectively
Set placement and replicas per workflow node or group.
Rollback safely
Redeploy an older published version any time.
Overview
Workflows are versioned pipeline releases
A workflow is a publishable graph that describes what you want to run and how data moves between steps. The Deployment Manager turns a published version into a safe, planned change—with explicit placement, scaling, and rollback.
A workflow is authored (draft), then published (immutable).
Publishing creates a stable artifact you can deploy repeatedly and roll back to later.
A workflow is a control-plane concept: it is not a single long-running process. It is a description that the Deployment Manager turns into concrete jobs deployed onto workers.
Examples
What workflow graphs look like
Workflows are graphs. These are two common patterns teams ship and operate—with placement, scaling, and rollback handled by the Deployment Manager.
API harvest with fan-out
Refresh access, discover resources, ingest records, then fan out to related data and metrics before loading.
Control before ingest
Apply policy checks, masking, and routing rules before data ever lands in your systems.
Capabilities
Build, deploy, and operate—without surprises
Build
- Create workflows from scratch (drafts) and iterate quickly.
- Publish versions to produce immutable “known good” artifacts.
- Reuse building blocks (jobs and blueprints) instead of starting from scratch.
- Compose workflows from nested building blocks (blueprints) to create patterns like producer/consumer topologies.
Deploy
- Plan a deployment before making changes: preview what will be created/updated and where it will run.
- Deploy the change set: stage and roll out the resulting jobs onto workers.
- Choose placement: map workflow nodes to worker groups (execution pools) for isolation, cost control, and performance.
- Scale horizontally: set replicas per placement group and keep scaling changes targeted to the correct part of the workflow.
- Rollback by deploying an older published workflow version (no history rewriting; no editing “live” artifacts).
Operate
- See what’s running (workloads) and where it’s running (fleet/workers).
- Track deployment lifecycle (events, progress, warnings) and reconcile drift when manual changes occur.
- Keep changes explicit (“no surprises”): potentially disruptive placement changes are previewable, and recompute is an explicit action.
How it works
From draft to deployment
At a high level, workflows turn into running systems through a predictable lifecycle:
- 1
Draft design a versioned workflow graph using existing jobs and blueprints.
- 2
Publish an immutable workflow version—your “known good” release artifact.
- 3
Plan changes before you make them. Preview what will be created/updated and where it will run.
- 4
Deploy the change set onto your workers, with explicit placement and scaling.
- 5
Operate track events, inspect workloads, reconcile drift, and roll back by selecting an older version.
Under the hood
Built on Jobs and Workers
Workflows assemble runtime primitives—jobs (what runs) and workers (where it runs)—so you can place and scale parts of a pipeline independently.
Runtime primitives
- A job is the executable unit: it defines sources, transforms, and sinks (e.g., read from an API, normalize, write to a store).
- A worker is an execution environment that runs jobs. A system can have one worker (local) or many workers (horizontal scale).
- A worker group is an operator-defined set of workers (an execution pool) used for placement and scaling decisions.
Composability layers
Workflows add two composability layers:
- Templates / library jobs (catalog items) help you start from curated, reusable job definitions instead of starting blank.
- Blueprints are reusable sub-graphs (mini workflows). They package common patterns (connectors, fan-out pipelines, edge transports) into building blocks that can be imported into workflows and composed safely.
Inspectability
This layered model keeps the system understandable:
- You can inspect the workflow to understand the intent.
- You can inspect the planned jobs/channels to understand what will run.
- You can inspect workloads to see what is actually running and where.
Use cases
Patterns teams build with workflows
Isolate stages with placement
Run ingestion, normalization, enrichment, and serving as separate nodes, then place them onto different worker groups (execution pools) to isolate “hot” ingestion from “heavy” processing.
Fan-out pipelines with independent scaling
Fan out one producer to multiple consumers and scale the consumers independently for high-throughput ingestion and parallel processing.
Targeted scaling and blast-radius control
Map workflow nodes to worker groups and replica targets so you can scale only the part that needs it—without touching the rest of the pipeline.
Choose how data moves between stages
Use different delivery classes between nodes (local channels, landing zones, object stores, etc.) to match reliability and cost requirements, while the planner expands the right adapters underneath.
Repeatable releases and safe rollback
Published workflow versions are immutable, so you can redeploy the same artifact, roll back to a known good version, and avoid ad hoc copying or manual patching.
Detect drift and reconcile to desired state
When manual changes happen, re-plan and redeploy to converge back to desired state—with an audit trail of events for what changed and why.
Production-ready
Built for safe changes at scale
Workflows are designed as a production control plane: changes are planned and previewed before they’re deployed, rollbacks are explicit, and you get an audit trail from draft to deployment.
No surprises by default
Preview diffs, placement, and scaling intent before you deploy changes to production.
Stable references
Workflow node identities and blueprint aliases stay stable so placement and scaling remain portable across edits.
API-first automation
Everything in the UI is backed by APIs—so you can automate planning, deployments, reconciliation, and inspection as code.
FAQ