Core Engine Architecture

Built on Experience

This architecture is not an academic exercise. It is the result of years of experience building mission-critical automation systems. We've seen every failure mode: API rate limits, database timeouts, worker crashes, and memory leaks.

We chose to build on top of proven Open Source technologies rather than inventing proprietary black boxes.

• Runtime: Node.js & Nitro for high-performance async I/O.
• Database: PostgreSQL for reliable, ACID-compliant state storage.
• Engine: Durable Execution Engine for resumable workflows.

By leveraging these battle-tested components, we focus our innovation on the workflow abstraction itself, not the plumbing.

Durable Execution

dataflows is built on the concept of Durable Execution. Instead of managing queues, retries, and state manually, you write standard TypeScript code that the engine automatically makes durable.

Reliability-as-Code

Move from hand-rolled queues and custom retries to durable, resumable code.

• No Queues: You don't define queues or workers. You declare a workflow and trigger it.
• No Timeouts: Workflows can run for seconds, days, or months.
• Resumable: If the process crashes, it resumes exactly where it left off.

Workflows and Steps

A workflow is defined with defineWorkflow. Inside its run function, every call wrapped in step.run(...) is an atomic, memoized step. The engine persists the result of each step in PostgreSQL — if the process crashes and restarts, it skips already executed steps and replays from the cache.

export const copyToClickUp = defineWorkflow({
  id: 'copy-to-clickup',
  trigger: azure.onWorkItemCreated(),

  async run({ event, step }) {
    // Each step runs once. On replay, the result is
    // fetched from the DB instead of hitting the API again.
    const item = await step.run('fetch-item', () =>
      azure.getWorkItem({ id: event.payload.id })
    )

    const task = await step.run('create-task', () =>
      clickup.createTask(item)
    )

    return task
  }
})

Workflows can also pause and wait for external signals or time:

// Wait for a human approval, then continue — even days later
const decision = await step.waitForEvent('approval', {
  timeout: '7d'
})

await step.sleep('cooldown', '24h')

Execution Flow

The engine manages the execution state in PostgreSQL. When a workflow sleeps or waits for an event, it offloads its state and releases compute resources.

sequenceDiagram
    participant Webhook
    participant API
    participant Engine
    participant DB

    Webhook->>API: POST /hooks/shopify
    API->>Engine: Start Workflow
    API-->>Webhook: 202 Accepted
    
    Engine->>DB: Create Execution
    Engine->>Engine: Run Step 1
    Engine->>DB: Save Result 1
    Engine->>Engine: Run Step 2
    Engine->>DB: Save Result 2
    
    Engine->>DB: Suspend (Sleep/Wait)
    Note right of Engine: Resources Released
    
    Engine->>DB: Resume (Timer/Event)
    Engine->>Engine: Replay Steps (Cached)
    Engine->>Engine: Run Step 3