Governed routing. Observable execution.
Reduce ingest cost without losing fidelity
Filter noise, mask sensitive fields, and archive full-fidelity data—before it hits metered downstream tools.
Server → Workers → Jobs
Governed control plane between sources and tools.
Collect
Bring logs, metrics, traces, events, and streams into one workflow.
Transform
Filter noise, mask PII, parse structures, and enrich data upstream.
Route
Send governed subsets to SIEM, S3, observability, and analytics.
Observe
Trace every Job and Worker to see what moved where and why.
A governed pipeline that sits between your sources and your tools
LyftData replaces scattered agents and ad-hoc routing scripts with a clean, governed model built on three core components: Server, Workers, and Jobs.
Everything that enters LyftData is processed consistently — shaped, enriched, masked, and routed with full traceability.
Core components
The core components
Server (Control Plane)
The Server coordinates your entire pipeline: tracking Jobs, managing configuration, observing Worker health, and exposing metrics for every part of your data flow.
Learn more →Workers (Execution Engine)
Workers run your pipelines at scale. Add more Workers to increase throughput, handle bursts, or isolate workloads — all without rewriting pipelines.
Learn more →Jobs (Your Pipelines)
Jobs describe how data moves: where it comes from, the transformations applied through Actions, and where it goes. Jobs remain stable even as sources, Workers, or tools change.
Learn more →Architecture
A look at how LyftData is structured
The Server holds the source of truth, Workers run pipelines near your data, and Jobs describe the flow.
Next abstraction layer
Workflows & Deployment Manager
Workflows let you design versioned data pipelines as graphs. The Deployment Manager turns those graphs into running systems by safely planning and applying concrete job deployments onto workers, with explicit placement and horizontal scaling.
Learn about workflows →- A workflow is authored (draft), then published (immutable).
- Publishing creates a stable artifact you can deploy repeatedly and roll back to later.
- A workflow is a control-plane concept: it is not a single long-running process. It is a description that the Deployment Manager turns into concrete jobs deployed onto workers.
Example
Place stages onto worker groups
Keep “hot” ingest isolated from “heavy” processing by placing workflow nodes onto separate worker groups.
Building blocks
Workflow building blocks
LyftData stays simple: workflows describe intent, jobs do the work, workers run jobs, and blueprints + catalog items help you reuse patterns safely.
Workflow
A workflow is a versioned graph you draft, publish, and deploy. It describes how pipeline stages connect and how they should run.
Worker
A worker is an execution environment that runs jobs. A system can have one worker (local) or many workers (horizontal scale).
Worker group
A worker group is an operator-defined set of workers used for placement and scaling decisions.
Reusable building blocks:
Job
A job is the executable unit: it defines sources, transforms, and sinks (e.g., read from an API, normalize, write to a store).
Catalog items
Catalog items are curated starting points—templates and reusable job definitions you can import and adapt instead of starting from scratch.
Blueprints
Blueprints are reusable sub-graphs (mini workflows). They package common patterns (connectors, fan-out pipelines, edge transports) into building blocks that can be imported into workflows and composed safely.
This layered model keeps the system understandable:
Intent
Inspect the workflow to understand what you’re building.
Plan
Inspect planned jobs and channels to see what will run.
Reality
Inspect workloads to see what’s actually running and where.
Pain story
Because telemetry pipelines shouldn't be duct-taped together.
The Problem
Why it matters
Too many agents and forwarding rules
Per-tool agents and ad-hoc forwarding scripts break whenever sources or tools change.
Compliance requires proof
Audits demand evidence of how data flows, where it is masked, and who touched it.
Hard to govern and debug
Opaque routing logic makes it difficult to prove what moved where or to resolve incidents quickly.
Teams need stable pipelines
Tool sprawl shouldn't force rewrites — Jobs should stay stable as stacks change.
Costs spike before data is shaped
Skyrocketing ingest cost hits before teams can filter noise or mask sensitive fields.
Incidents demand clarity
Clear traceability across Server, Workers, and Jobs keeps investigations fast.
Product outcomes
What the product does
Unified ingestion, clean transformation, flexible routing — all upstream.
LyftData gives you a consistent pattern for shaping telemetry before it hits expensive or specialized tools.
Ingest from many sources
Bring logs, metrics, traces, events, and streams into one workflow — no per-tool agents required.
Shape data with Actions
Filter noise, mask PII, parse structures, normalize formats, and enrich events before they leave your environment.
Route intelligently
Send enriched or masked subsets to SIEM, full-fidelity logs to S3, and aggregated metrics to observability platforms — all from a single Job graph.
Scale horizontally
Workers scale linearly. Add more for volume spikes or high-throughput ingest with zero changes to Job logic.
Avoid vendor lock-in
Use the tools you want. Switch SIEMs, observability stacks, or storage systems without rewriting ingestion.
Trace your pipelines
Understand exactly what happened: which Job processed what, on which Worker, and how data moved through the system.
For Security, SRE, and Data teams
Why teams choose LyftData
Before LyftData
With LyftData
Skyrocketing ingest cost
Shape data before you pay for it.
Tool sprawl
Use one model across every tool.
Compliance requirements
Governance and traceability by default.
Opaque routing logic
Observable routing with stable Jobs.
Brittle per-tool agents
Workers replace per-tool agents.
Deployment model
Run LyftData where you already run your systems
LyftData is entirely self-managed: deploy Server and Workers wherever you already run your systems — VMs, containers, on-prem, VPC, or hybrid. Processing stays in your environment, and you control what leaves.
Deploy in your VPC
Deploy Server and Workers where you already run your systems — VMs or containers — and keep telemetry in your environment.
Deploy on-prem
Run LyftData alongside existing storage and analytics on-prem with the same governed pipeline and auditability.
Deploy in hybrid environments
Adopt incrementally and route between clouds, regions, or datacenters while Jobs stay stable.
Quickstart
Get started locally in minutes
Start locally with a built-in Worker, then deploy your first pipeline when you’re ready.
Automation
Automation via APIs
Everything you can do in the UI is backed by APIs, enabling automation and repeatable operations.
Explore docs →