Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
Tools
Tools
No tool list yet. Remote servers get one after the first health check.
FAQ
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents.
Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.
Does Duckle need an API key?
No required environment variable is listed on this record.
What transport does Duckle use?
Duckle runs locally over stdio.
When was Duckle last checked?
Last checked 2026-11-10. Pipelines you own. Author and deploy to your servers or cloud. Duckle is an open-source ETL platform for teams who want their pipelines running on their own infrastructure. Author on a canvas, in Python or in SQL, then ship the same file to your own server or cloud account: duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit tra
An open-source ETL platform you run on your own infrastructure. Drag sources, transforms, validators and sinks onto a canvas, wire them together, and press Run. Duckle compiles the graph to SQL and executes it on a real columnar engine, with live previews, the generated SQL visible on every node, and no hidden state.
You build a pipeline on a laptop and deploy that same file to a server, where it runs on a schedule under a web console with roles, alerts and an audit log. Nothing is rewritten in between, and nothing is metered.
In short: a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte - one pipeline for ingest, transform, and load that runs anywhere, and can also run dbt on DuckDB inside the same tool.
Three things set it apart:
An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs wherever Duckle does - no API key, no telemetry, no vendor round-trip. Point it at your own OpenAI-compatible endpoint instead if you would rather it did not run in-process.
400+ components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
A self-contained binary you can audit. 73 to 110 MB depending on your platform. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.
Why Duckle is different
Visual, never opaque
The canvas compiles to SQL you can read, and every node has a live preview tab. No black box.
An assistant with no API key
Runs in-process by default, or against your own OpenAI-compatible endpoint. Your prompts and your data stay inside your infrastructure either way.
Single-file binary, no bundled DB
73 to 110 MB depending on platform (it embeds the headless runner + MCP server). DuckDB downloads on first launch with a guided step. AI engine is opt-in.
Native speed
Execution runs through DuckDB: vectorized, columnar, local. A clean-and-export job that crawls in a spreadsheet finishes in milliseconds.
Git-friendly by design
Pipelines, connections, contexts, and routines persist as plain files in a folder you pick. Diff them, branch them, review them.
400+ components ready today
Files, databases, warehouses, lakehouses, object stores, SaaS APIs, NoSQL, streaming brokers, vector DBs, FTP, IMAP, SMTP. Each is covered by tests.
Honest about scope
Single-machine and embedded by design. Built to make local and small-team data work fast, not to replace a distributed warehouse.
Optionally click Install on Duckie AI Assistant (~1.1 GB, takes 5-10 min on average broadband).
Pick a workspace folder. Pipelines, connections, context variables, and routines live there as plain files.
Build a pipeline two ways:
Drag + wire: drag a CSV source in, point it at samples/orders.csv, hit Autodetect schema. Drag a Filter, wire it up. Drag a Parquet sink with an output path. Press Run, watch the nodes light up.
Ask Duckie: click the Sparkles icon (top-right of the toolbar), type "read orders.csv, filter where status = 'paid', write to paid.parquet". When Duckie streams back a pipeline, click Insert into canvas.
Inspect. Click any node for a live row sample in its Preview tab. The Plan tab, beside Canvas, shows the SQL generated for every stage.
That's a real, native ETL pipeline built and run in under a minute. CSV is just the easiest first node; swap in Parquet, JSON, S3, Snowflake, MongoDB, or Stripe the same way.
Double-click. Unsigned binary - Windows SmartScreen will warn the first time; click "More info" -> "Run anyway".
macOS (Apple Silicon)
Duckle-macos-arm64
chmod +x Duckle-macos-arm64 && ./Duckle-macos-arm64. Right-click -> Open the first time to bypass Gatekeeper.
Linux (x86_64)
Duckle-linux-x64
chmod +x Duckle-linux-x64 && ./Duckle-linux-x64. Requires WebKitGTK 4.1 (libwebkit2gtk-4.1-0 on Debian / Ubuntu) and a desktop session, X11 or Wayland with or without XWayland. On a server with no display, ./Duckle-linux-x64 serve runs the scheduler and web console without a window.
The single-file binary above is all you need for Build Pipeline too: the headless runner is embedded into the app at build time, and exporting a pipeline produces ONE self-contained executable (the engine, the DuckDB CLI, any needed extensions, and the resolved pipeline are all inside that one file). Copy that single file to your server and run or schedule it - no separate runner download required.
One command, nothing installed: it scaffolds sample data and a pipeline, compiles it to SQL, runs it on DuckDB, and shows you the rows.
uvx duckle quickstart
Let an agent do it
Paste this into Claude Code, Cursor, or Codex:
Run uvx duckle quickstart to build my first pipeline and run it
Nothing to install first. The agent fetches Duckle and the DuckDB engine on demand, runs a real pipeline, and shows you the rows.
CLI only (CI, cron, containers)
If you do not want the desktop studio, install just the headless runner. It is about 27 MB rather than 100 MB or more, has no GUI dependency, and is what a build step actually needs.
pip install duckle
That is the whole install. It brings the DuckDB CLI with it (via the duckdb-cli package published by the DuckDB Foundation), so there is nothing else to fetch and it works offline. Wheels ship for Linux, macOS and Windows on x86-64 and arm64.
It also gives you a Python API, where pipelines are built as code and executed by DuckDB rather than by Python:
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())
Python expressions compile to vectorized SQL at plan time, so no rows pass through the interpreter. See the PyPI page for the full API.
The same package provides the duckle command-line runner for CI, cron, and containers - it bundles the headless runner and the MCP server per platform:
pip install duckle # or run ad hoc, no install: uvx duckle --help
Pipelines execute as SQL on the DuckDB CLI, so the runner needs a duckdb on PATH or DUCKLE_DUCKDB_BIN set (pip install duckdb-cli is the quickest route). Validation does not:
duckle validate # compile-check every pipeline under ./pipelines
duckle validate --json # machine-readable, for a CI step
duckle --pipeline my.json # run one
validate opens no source and writes no sink, so it needs no engine, no credentials and no network. Exit codes are stable: 0 clean, 1 a real finding (a pipeline failed or did not compile), 2 the runner could not start (bad usage, unreadable file, missing engine).
The binary is 73 to 110 MB depending on platform (it embeds the headless runner and the bundled MCP server). On first launch you'll be guided through downloading two engines into your app-data directory:
Engine
Size
Required?
What it powers
DuckDB CLI
~30 MB + extensions
Yes - cannot run pipelines without it
Every source / transform / sink that runs as SQL
Duckie AI Assistant
~1.1 GB (llama-server + Qwen 2.5 Coder 1.5B GGUF)
Optional
The chat sidebar that generates pipelines from natural language
Press Run in the toolbar. Nodes light up in execution order; row counts appear under each.
Open the Output tab (bottom panel) to see per-stage timing.
Click any node for sampled rows in its Preview tab; the Plan tab beside Canvas shows the generated SQL for every stage.
5. Iterate
Add a Group By before the sink to aggregate. Re-run. Sub-second on small data.
Cancel mid-run with the Stop button - the DuckDB process is killed cleanly.
Save your work: Cmd/Ctrl-S writes a JSON pipeline file to your workspace folder.
Or start from a query you already have
New pipeline -> From SQL, then paste a SELECT. Each CTE becomes a SQL step
named after it, the final SELECT becomes the last one, and each table the
query reads becomes a source node to point at your data. The steps still read
each other by name, so the pipeline computes exactly what the query did (a test
runs both and compares the rows). Files named in the query (FROM 'orders.csv') stay in the SQL. A WITH RECURSIVE query is kept as one step,
and the node says why.
Where Duckle runs
You build a pipeline on your laptop. The server runs that same file. Nothing is rewritten, exported or converted in between.
flowchart LR
D["Duckle Desktop<br/>your machine"] -->|deploy, needs admin| W
B["Console in a browser<br/>your machine"] -->|turn it on, needs operator| W
W["Workspace on your server<br/>a new schedule lands OFF"] --> C["Scheduler<br/>every 15s, takes what is due"]
C --> R["It runs<br/>on that box, unattended"]
R --> O["Run history, logs, metrics,<br/>alerts, and an audit log"]
O -->|you watch it here| B
How
What you get
Server
duckle-runner serve --workspace /srv/pipelines
Headless web console, cron scheduler, roles, audit log, alerts
Docker
Dockerfile.web
The same console in a container, behind your own ingress
CI
duckle-runner --pipeline p.json
Any runner. Exit codes and NDJSON logs, nothing to install
Standalone
Build Pipeline
One self-contained executable. Drop it on a box, run it from cron or systemd
Desktop
The app
Author, debug and inspect. Optional, and never required to run anything
Nothing here depends on a person's machine being switched on:
Pipelines are plain files in git. Review them in a pull request, roll them back, and let them outlive whoever wrote them. There is no proprietary repository and no exported binary artifact.
The console has roles and an audit log, so more than one person can operate it and you can see who did what.
Secrets are not in the pipeline file. They resolve from the environment or an encrypted per-workspace store at run time.
Working recipes for AWS (EC2, ECS, EKS), Azure (VM, Container Apps, AKS) and Google Cloud (Compute Engine, GKE), with manifests and the mistakes worth avoiding, are at duckle.org/deploy. Three things worth knowing before you start:
On a non-loopback bind with no credential the console starts unclaimed, and for 15 minutes it can be claimed by someone holding the setup code it prints to its own output when it starts. Reading that output - the terminal, or docker logs - is what distinguishes the operator from anyone who can reach the port; the code is 128 bits, new on every restart, and never written to disk. A refused claim is recorded in the audit log, so guessing is visible rather than silent. Pass --token, set DUCKLE_CONSOLE_TOKEN, or create accounts with duckle-runner console add-user to skip the window entirely. An empty value is refused rather than treated as absent, so an unresolved secret fails loudly instead of opening that window. Who can do what, and how one request is decided, is set out under Sign-in and roles.
The scheduler runs in serve, not in the editor. Start the editor with schedules armed and it now says so rather than leaving you to wonder why nothing fired.
GET /healthz needs no credential and answers ok, so a Kubernetes probe or a load balancer can check liveness without holding a token. Every other route is authenticated, so pointing a probe anywhere else reports the pod unhealthy forever.
Deploying a pipeline to a running server
POST /api/deploy lands a pipeline on a server from wherever it was authored, with the schedule it should eventually run on:
Two things are deliberate. The schedule arrives disabled, so a cadence someone set while testing on a laptop cannot start firing the moment it reaches production; enabling it is a separate call. And deploying needs admin while enabling needs operator, because a deployed pipeline runs shell and SQL on that host: shipping the code and starting it are two acts, and the audit log records both with the name of whoever did them.
API keys, for machines
A person signs in and gets a session. A machine has no browser and nobody to rotate a password, so it gets a key of its own:
duckle-runner console key-add ci-deployer --role admin --expires-days 90
duckle-runner console key-list # role, state, and when each was last used
duckle-runner console key-revoke ci-deployer
A key carries its own role, so a deploy runner can be admin while a metrics scraper is viewer. It is printed once and stored only as a hash, so a lost key is replaced rather than recovered. key-list shows when each was last used, which is the question actually worth answering before revoking one, and revoking takes effect immediately on a console that is already running rather than at the next restart. Revoked keys are marked rather than deleted, so a key that turns up in an old log can still be named.
Accounts, sessions and keys live in .duckle/console.db. An existing console-users.json is carried into it on first start and renamed to .migrated, so an upgrade neither locks anyone out nor destroys the only copy of a credential store. For the roles, and a diagram of how one request is decided, see Sign-in and roles.
Scaling it
duckle-runner serve is an ordinary service. Run it on EC2, EKS, a VM or a container next to everything else you operate, and scale it the way you scale any service:
More cores. The engine is parallel and uses every core on the box by default, so a bigger instance is a faster pipeline with no change to the pipeline. Bound it with DUCKLE_THREADS when you would rather it did not take the whole machine.
Object storage reads. On DuckDB 2.0, Parquet, DuckDB files and uncompressed CSV over S3 (and other object storage) are read on DuckDB's own pool of I/O threads, 4 per core by default, which reads ahead within the memory limit. JSON and compressed CSV are still read synchronously. DUCKLE_THREADS does not bound that pool: DUCKLE_ASYNC_THREADS does, and DUCKLE_READ_AHEAD_DEPTH sets how far ahead it reads (0 turns read-ahead off, -1 is DuckDB's automatic default). Both need DuckDB 2.0, and a run records them in its manifest. The automatic depth grows with DUCKLE_THREADS, about four requests in flight per thread, so a small worker on high-latency storage gains most from raising it. Measured through Duckle on 2 threads, reading 104 MB of a 24M-row Parquet file from an S3 stand-in that answers each request after 50 ms: 6.4 s on DuckDB 1.5.5, 2.1 s on 2.0 as it ships, and 1.1 s with DUCKLE_ASYNC_THREADS=32 and DUCKLE_READ_AHEAD_DEPTH=32. A fixed depth is outside DuckDB's memory budget, so depth times the row-group size has to fit in memory.
More RAM. Set memoryLimitMb per stage, or a workspace default, and spill to disk past it.
More pipelines at once.DUCKLE_MAX_CONCURRENT_RUNS raises how many run together; it ships at 1 so an unattended server stays predictable until you decide otherwise.
More machines.duckle-runner work drains a queued batch from as many workers as you start, on as many hosts as you like, each claiming its items under a lock so nothing runs twice. The hosts need shared storage whose locks work: a worker tests that when it starts and refuses to run where they do not, and work --check runs just that test.
One serve per workspace. Each serve decides for itself when a schedule is due, so a second one on the same workspace fires every schedule again: measured on one machine, every cron occurrence ran twice. Add capacity with workers rather than consoles. docs/design/horizontal-scaling.md has the measurements and the proposed fix.
Bigger than any box. Turn on pushdown and the query runs verbatim inside Postgres, Oracle, SQL Server or Snowflake. The warehouse does the scan; Duckle keeps the scheduling, lineage, data quality and alerting.
Measured, rather than asserted:
96,000,000 rows out of live Postgres to Parquet in 39.9s (details)
Oracle extract at 65.0s, against 68.6s for python-oracledb with pyarrow on the same machine (details)
The one thing Duckle does not do is split a single query across a cluster the way a distributed warehouse does. When you need that, push the work down into the system that has it and let Duckle orchestrate around it.
Server deployment (Build Pipeline)
Want the studio to publish straight to a running server instead? That is the other
route: connect a server once, then Deploy to a server from the editor. Step by step
in docs/current/server-deployment.md.
Promoting from CI instead, or driving Duckle from Airflow, Dagster or Temporal? See
docs/current/ci-and-orchestration.md, with
copyable GitHub Actions and GitLab CI templates in docs/ci/.
Want to know exactly what crosses the wire, and where every credential is stored?
docs/current/client-server-architecture.md
is the diagrammed answer, sharp edges included.
The in-app scheduler runs only while Duckle is open. To run a pipeline on a server with no desktop app, Build Pipeline turns it into ONE self-contained executable - the equivalent of a standalone "Job".
Right-click a pipeline (in the project tree or on the canvas) and choose Build Pipeline. The output is a single file named after the pipeline (orders_etl.exe on Windows, orders_etl on macOS / Linux) that embeds everything it needs:
the headless execution engine,
the DuckDB CLI,
only the DuckDB extensions that pipeline's components actually use,
the resolved pipeline (context variables substituted, routines inlined),
its secrets (see below).
On first run it self-extracts to a temp cache and uses its own embedded DuckDB, so the server needs nothing installed - no Duckle, no DuckDB. There is no folder to copy, no run.sh, and no separate runner download. A CSV-to-CSV pipeline builds to about 28 MB; only the extensions a pipeline uses are bundled, so the file stays lean.
./orders_etl # or orders_etl.exe on Windows
The process exits 0 on success and non-zero on failure, and writes the same NDJSON run logs under logs/ (Splunk / Dynatrace friendly).
Build options
Option
What it does
Target OS
Pick Windows, Linux, or macOS in the build dialog. The native OS always builds; a Linux server file can be cross-built from any host (the Linux engine is bundled for you), while a macOS file can only be produced on a Mac. Appending the payload makes the file unsigned, so do not codesign / Authenticode-sign it.
Context
Pick a context at build time; its non-secret variables are baked into the pipeline.
Secrets: Environment
Each secret becomes a ${ENV:KEY} placeholder, so nothing sensitive is written into the file. The runner resolves real environment variables first, then a secrets.env (KEY=VALUE lines) placed next to the file.
Secrets: Passphrase
Secrets are encrypted inside the file with AES-256-GCM, decrypted at run time from the DUCKLE_BUNDLE_PASSPHRASE environment variable.
Schedule it with whatever the server already has - point the OS scheduler straight at the file:
# Linux cron - run every day at 02:00
0 2 * * * /opt/duckle/orders_etl >> /var/log/orders_etl.log 2>&1
On Windows use Task Scheduler; on macOS a launchd plist; on Linux a systemd timer. Full examples in docs/current/scheduler.md.
Run against an existing workspace - the same embedded headless runner can also execute a pipeline JSON directly, resolving context the way the app does:
A scheduled pipeline already consumes a stream without gaps: a source that
tracks its position (src.kafka with trackOffset, xf.incremental) resumes
where the last successful run stopped. What a schedule cannot give is
latency - the scheduler wakes every 15 seconds, and every run pays process
start, DuckDB resolution and document parsing again.
follow keeps the same execution model and removes that per-batch overhead.
The document is read and resolved once, the engine is built once, and the
pipeline then runs in a loop. Each pass is one micro-batch:
wait N ms after a pass whose sinks wrote nothing (default 1000)
--max-batches N
stop after N passes (default: until stopped)
--on-error stop|continue
stop on a failed batch (default), or keep going
A failed batch never advances the source position. The position is queued
during the run and written only when the run reaches ok, which is after every
sink has written - so a failure anywhere, transform, quality gate or sink,
leaves the position where it was and the next pass re-reads exactly the records
that did not land. Killing the process is safe for the same reason; Ctrl-C
finishes the batch in hand first, which only saves you a truncated output file.
That ordering is the difference between a correct micro-batch loop and a lossy
one, so it is covered by a regression test that fails if the position ever
advances past a batch that did not land.
Masking what you look at, not what you write
A pipeline's sinks can be perfectly governed and its data still be read off a
screen. Previews, profiles, reject rows, error bodies, API responses and the
rows an MCP tool hands an agent are all places production person-data appears
without anyone's permissions being wrong.
a short stable digest, so rows stay distinguishable without the value appearing
mask:null / mask:last4 / mask:redact / mask:hash
picked explicitly
It never changes what the pipeline writes. The sink still receives the real
value, because the sink is governed by policy and the screen is not. Changing
written data is what qa.mask is for.
Masking happens as the previews are assembled, so the desktop panel, the CLI,
the console API and MCP are consistent by construction rather than by four
callers remembering. Both execution paths are covered, and there is a test for
each - a masking point wired into only one of them would leak on the other.
Nothing is inferred from a column name. A heuristic that masked company_name
because it contains "name" would teach people to distrust the masking, and one
that quietly failed to mask something would be worse.
Schedules in a named time zone
A cron expression is civil time: 0 3 * * * means three in the morning as a
person reads a clock. With no zone set that is the machine's clock, which is
what every existing schedule already means. Set one and it stops depending on
where the runner is deployed:
timezone: Europe/Brussels
A Brussels registry pipeline stays at 03:00 Brussels whether the container runs
on UTC or the operator is watching from another continent. An unknown zone is
refused when you save it, not at fire time, so Europe/Brussel is a typo
you see rather than a job quietly running on UTC for a quarter.
Daylight saving is decided, not discovered. Twice a year a civil time is not
a single instant, so both cases are pinned and tested:
case
what happens
the clock skips it (spring)
the occurrence is skipped, and the skip is reported. A job asked to run at 02:30 on a day with no 02:30 has not been missed by the scheduler; the day was short. It is not nudged to 03:30.
the clock repeats it (autumn)
it fires once, at the earlier of the two instants
Intervals are untouched.every 24 hours is an elapsed duration, not "the
same clock time tomorrow", and a zone must not quietly turn one into the other.
Both schedulers - the desktop one and the web console's - now evaluate through
the same code. They disagreed once before, and the way it showed up was one
expression firing at two different times depending on which surface owned it.
Days a schedule must not fire on are a small calendar rather than a holiday
provider:
exclude:
weekdays: [sunday]
dates: [2026-12-25]
The dates are civil dates in the schedule's own zone, which is why this
belongs with time zones rather than beside them: a schedule at 00:30 Brussels on
the 25th is 23:30 UTC on the 24th, so a UTC-based check would exclude the wrong
day. A skipped occurrence is reported rather than merely not happening, and a
misspelled weekday or date is refused when you save it - "sundy" excludes
nothing, which looks exactly like no exclusion at all until the day arrives.
Real holiday calendars vary by country, region and year; a first version that
tried to know them would be wrong somewhere and confidently so. A date list is
something an operator can check by reading it.
Typed pipeline parameters
A pipeline can declare what it takes, and the contract is checked once for every
surface rather than separately by each:
They go through the same check as every other surface, so --param jurisdiction=FR stops before anything runs (jurisdiction is "FR", which is not one of the accepted values). A --param without =, or the same name twice, is
a usage error rather than a guess at which was meant, and the run record gives
their source as --param.
In the editor, desktop or web, Run opens a form with a control for each
declared parameter: a list for an enum, true / false for a boolean, a date
or date-time picker, a number field that knows its minimum and maximum, and a
masked field for a secret, each with its description. A blank field sends
nothing, so the declared default applies and a required one is refused
(run_date is required). The values go to the engine rather than being
substituted in the browser, so a run started from the editor is held to the
contract like any other, and the pipeline's maxRunSeconds and resourcePool
apply to it too. A value a context gives a declared parameter is filled in for
you to keep or change.
A schedule can bind values too: the Schedules dialog shows the same controls
under Parameters, and every run the schedule starts is given them, checked
against the contract like any other value, on the desktop scheduler and under
duckle-runner serve alike. A blank field uses the pipeline's default. Through
the console's API a schedule takes them as "params": { "region": "us" }, and a
save that does not mention params keeps the ones the schedule has. A schedule
that runs a plan binds none itself: each pipeline in a plan has its own contract,
so its values live on the plan's steps (see Plans).
Where a value came from is kept. When two surfaces bind the same parameter -
a schedule and the run that starts, say - the later one wins, which is a
documented rule and not a clever one. What is not thrown away is that something
was displaced:
Only a differing value counts as an override. Two places binding a parameter
to the same value is a duplicate and harmless; recording both alike would bury
the case that matters in the noise of the one that does not. The record lands on
the run receipt, so "was this deliberately overridden, or bound twice by
accident?" is answerable after the fact rather than only while it happens. A
secret is *** here for exactly the reason it is elsewhere - a provenance record
must not become the one place a credential is written down.
Validated at one boundary. Every surface - desktop, console, CLI, HTTP API,
MCP, scheduler, Plans - reaches substitution through the same function, so the
contract is enforced there. Validating per surface is how the desktop ends up
accepting a value the scheduler refuses, and the bug is then in neither of them.
Every problem at once, with a stable code (param:unknown, param:missing,
param:type, param:enum, param:range, param:pattern) and what was wanted.
A form filling in one mistake per round trip is not a contract, it is a
guessing game. An undeclared name is refused rather than ignored, with the near
names suggested - a typo is far more likely than a new parameter, and a silently
ignored one means the run used the default and nobody noticed.
secret is a declared type, not a guess from the name. A secret value never
reaches run history, and a constraint failure on one never echoes the value into
an error message. In history it is replaced rather than dropped, because a
missing key reads as "never supplied" and "was this run given a token?" is worth
being able to answer.
A pipeline that declares nothing behaves exactly as before: any unresolved
${name} is simply prompted for.
skip src src.csv has no SQL to check
skip pg src.postgres sends its SQL to the remote system, so DuckDB cannot
validate it. Checking it here would say nothing true about that dialect.
hint pg more than one statement: a source sends ONE query, wrapped as
`SELECT * FROM (...)`, so anything after the first semicolon is a
syntax error rather than a second step
FAIL q 2:12: Referenced column "amountt" not found in FROM clause! (did you mean amount?)
skip out node "out" is a sink stage, which produces no relation to describe - and
running it to find out would perform its writes
4 node(s) checked, 2 problem(s)
Every SQL-bearing node is bound against the columns its upstreams actually
produce, before anything runs, and the output schema each node infers is
carried forward to the next. Nothing with effects is executed: a sink is refused
rather than run to see what it returns, and only a plain derived view is bound.
The diagnostics were always there and were being thrown away. DuckDB already
says which column, where, and what it thinks you meant; all of it was collapsed
into one error string, so the editor could report only that a node did not
resolve. The candidate list is the most useful part and was the first thing lost.
A wrong position is worse than none. The position DuckDB reports is an offset
into the SQL Duckle compiled, not the SQL you wrote - and DuckDB truncates the
line it echoes for any wide statement, which Duckle's always is. So the position
is recovered by finding the named token in your own SQL, and only when it occurs
exactly once. Twice, and there is no position at all rather than a confident
guess at the first one.
A source's query is not checked, and says so. It runs on Postgres or
BigQuery; binding it against DuckDB would either reject valid SQL or accept
invalid SQL, and either way the answer would be about the wrong engine. "Not
checked" and "checked and clean" never read the same.
But some things are wrong in every dialect. An unclosed quote, an unbalanced
parenthesis, a second statement where exactly one query is sent, a DELETE in a
position that reads rows, a ${placeholder} nothing substituted. Those are
reported as hint on a node that still says it was not validated - the round
trip to the remote system tells you the same thing, slower, in a message about a
token far from the mistake. Nothing here guesses at a dialect: ARRAY_AGG,
QUALIFY, LISTAGG, toDateTime and GENERATE_UUID() are left alone, because
checking Postgres SQL with anything other than Postgres tells you about the
checker rather than about your query. A test asserts exactly that.
The editor shows them. Selecting a node runs the same bind and puts what
DuckDB said above the form - the position in your own SQL, and the column it
suggests instead:
2:12 Referenced column "amountt" not found in FROM clause! did you mean amount?
Cleared as soon as the node binds again, because a stale error under a line the
author has already fixed is worse than no error at all.
And what could come next.complete_node_sql suggests upstream columns with
their types, the relations the node can read, the pipeline's declared parameters,
DuckDB's functions and keywords - ranked for the position:
SELECT reg -> column region (String) ${region_filter} regexp_escape(
SELECT * FROM -> input src (relations only)
WHERE x = ${re -> ${region_filter} (nothing else can be meant)
A column beats a function where a column belongs, a prefix beats a substring, and
the list is stable between identical edits - a list that reshuffles between
keystrokes is one nobody builds muscle memory against.
It never runs your SQL. The only thing read from DuckDB is its own function
list, cached for the process; the columns come from the caller. That is what lets
it answer on every keystroke when the bind cannot.
The editor uses it. A SQL field suggests as you type, with arrow keys, Enter
or Tab to accept, Escape to dismiss and Ctrl-Space to ask at a fresh position. It
is still a plain textarea: swapping it for a code editor to add completion would
change everything about typing in order to change one thing. Requests are
debounced, and a reply for an earlier keystroke is discarded rather than shown -
it describes text that is no longer there.
Same analysis from duckle-runner sql check, the MCP tools check_node_sql and
complete_node_sql, and both editors - one function, so they cannot come to
disagree. SARIF carries a real
region, so a code-scanning viewer jumps to the token.
OpenLineage export
Drop an openlineage.json in the workspace and every run emits START and a
terminal event. No file means no events, no local writes and no network.
Emitted from retry::begin and retry::finish, which every execution
surface already goes through - the desktop, the console, the CLI, MCP, the
scheduler and Plans. A feed covering six of eight surfaces is one nobody can
reason about, because the missing runs look like runs that never happened.
The run id is derived, not random. OpenLineage requires a UUID and Duckle's
ids are readable strings, so they are mapped by name: START and COMPLETE are
emitted by different calls and must agree, or a collector shows two unrelated
runs and no completed one. The original id travels in a facet.
interrupted is ABORT, not FAIL - the run stopped being observed, it did
not fail, and a consumer that treats those the same re-runs work that may have
finished.
A run-time reference is marked, not asserted. A dataset name still holding a
${...} does not address a single dataset, and it is emitted with
unresolved: true rather than silently joining to the wrong thing in someone's
graph. Datasets come from the catalog joined to the receipt by node id, so a
node the run never reached is not reported as touched: absent is not zero. A run
that finds the catalog missing or out of date builds it, so a new workspace needs
no catalog build before its events name anything.
Telemetry cannot fail a run. Events are appended to
logs/openlineage.ndjson first and only then POSTed, so a collector that is
down costs one short timeout and the events are already durable. Catalog asset
ids are credential-free by construction; query strings are stripped on top of
that, so a signed URL never carries its signature off the machine.
hashDatasetNames replaces names with a digest and keeps the namespace, for an
organisation that wants the shape of its graph in a shared tool without the
table names.
Alerting without scraping the UI (/metrics, /readyz)
curl -H "Authorization: Bearer $TOKEN" http://console:8080/metrics
curl http://console:8080/readyz # no credential; so does /healthz
The run-history half is rendered by the engine, the same function that writes
logs/duckle_metrics.prom for a node_exporter textfile collector - so the
endpoint and the file cannot come to disagree about what a series means. What
the endpoint adds is what a file cannot carry: what this process is doing right
now. duckle_run_permits_free at zero for any length of time is runs queueing,
and duckle_scheduler_seconds_since_tick growing past a few tick intervals is a
scheduler that has stopped: no schedule is firing.
Liveness and readiness are separate, because they fail differently and an
orchestrator acts differently on each: a process that is alive but not ready
should stop receiving traffic, not be restarted. /readyz writes and deletes a
probe file under .duckle/, so it catches a read-only mount or a full disk - the
states that stop runs being recorded while every read still succeeds. It checks
nothing external: a source being down is not this server being unready. On
serve it also checks the scheduler: a scheduler thread that has died or hung
leaves the console answering every request while no schedule fires, which from
outside looks exactly like a quiet night. Five missed ticks, and never under a
minute, answer 503 naming it. The web editor schedules nothing and is not held
to it.
Both probes are unauthenticated; /metrics is not. A probe says the process
is up and tells an anonymous caller nothing else. Pipeline names are the shape of
someone's business, so a scraper sends the same bearer token any other API client
does.
Labels are bounded, and when the budget bites it says so:
duckle_metrics_pipelines_omitted above zero means those pipelines are not being
monitored. Nothing prunes runs/, so a workspace that has ever run thousands of
pipelines would otherwise emit thousands of label values forever.
External components
An iXBRL parser, an OCR adapter, a country-specific registry reader - Duckle
should not contain all of those, and they should not have to be escape hatches
either. Drop one in <workspace>/components/<name>/:
if req["format"] == "arrow":
with open(req["inputs"]["main"], "rb") as f:
table = ipc.open_stream(f).read_all() # the STREAM format, .arrows
Arrow IPC needs DuckDB's arrow extension, which is a community one -
INSTALL arrow from the core repository 404s - so it may be unavailable on a
machine with no network. The host probes once and falls back to Parquet rather
than failing a run over an interchange preference. A component that never
mentions interchange gets Parquet, exactly as before.
What DuckDB writes is the Arrow IPC stream format, so a component reaches for
open_stream, not open_file; the files are named .arrows so the name does
not promise the other one. Control messages stay JSON because they are small and
structured.
Sources and sinks too, not only transforms. The ports a component declares
decide what it is: no inputs is a source, no outputs is a sink.
g ok (5 rows) - ext.gen: 5 row(s) -> g
e ok - ext.emit: 5 row(s) delivered
A sink has delivered its rows somewhere Duckle does not model - an API, a queue,
a file of its own - and has no relation to hand back, so it is not asked for one.
Requiring one failed a sink that had already done its job.
Rows it cannot handle go to the reject port, on the same __reject contract
every built-in uses, so a downstream edge reads them identically whoever wrote
the component:
con.execute(f"COPY (SELECT * FROM read_parquet(?) WHERE name IS NULL) "
f"TO '{req['reject']}' (FORMAT PARQUET)", [src])
A component that writes no rejects still gets an empty reject relation with the
right columns. Without it, wiring the port to a component that happens never to
reject fails the run with Table with name v__reject does not exist - and
"this component rejects nothing" is an ordinary thing for a component to be.
An external id must start with ext., so a component can never shadow a
built-in one. A component called xf.filter that quietly replaced the real one
would be the worst failure this could have.
Policy gates them like anything else.components.deny: ["ext.*"] refuses
the family; the existing denylist covers external components by construction
rather than by remembering to.
Declared, not discovered by running. Ports, properties and version come from
the manifest, so "what components exist here" never means executing third-party
code. duckle-runner components external lists them with their manifest and lock
hashes.
They appear in the palette. Opening a workspace loads its external
components into an External category - their own, rather than mixed into
Sources and Transforms, because a component Duckle did not write should be
visibly not one Duckle wrote. The property form comes from the component's own
manifest, so a tile you can drop is a tile you can configure. kind is derived
from the declared ports: no inputs is a source, no outputs is a sink.
Both editors get the same list from the same endpoint, and MCP's
list_components includes them when given a workspace - so an agent asking
what it can build with sees what the workspace installed, not only what was
compiled in.
A component that hangs is killed at its declared timeout, and a manifest that
does not parse is reported rather than silently missing from the list.
Does your component behave?
duckle-runner components conform ext.upper --workspace .
pass schema validation id, version and runtime.command are declared; 1 input, 1 output
pass initialize answered without doing the work
pass interchange declares arrow, parquet; this host would use arrow
pass empty typed input zero rows in, zero rows out, with a readable schema
pass large batch 200000 rows in, 200000 out, in 0.9s
pass crash cleanup reported the failure: IO Error: No files found ...
pass secret redaction the request carries property values and paths, no credentials
pass cancellation killed after 60s; the host enforces this bound
pass reject output wrote 2 rejected row(s) the host can read
pass artifact lineage 1 artifact(s) exist and hash as declared (29 bytes)
Real invocations through the same code the engine uses, so "conforming" means
what the engine will actually do. Against a deliberately broken component it
reports what is wrong and exits 1:
FAIL empty typed input reported success but wrote no output; zero rows is still a table
FAIL crash cleanup reported success on an input that does not exist
A case for something the host cannot do says unsupported, not pass. A
green tick for a feature nobody built is the most misleading result a
conformance kit can produce - and it is not a failure either, or every component
would look broken because the host is incomplete.
A lifecycle, not just a call. A component may answer an initialize - a
configuration check that must not do the work - and may report progress while it
runs:
if req.get("phase") == "initialize":
emit({"type": "result", "ok": True}); raise SystemExit(0)
emit({"type": "progress", "rows": n, "fraction": 0.5, "message": "stage 2 of 4"})
w ext.slow: 1 row(s), 50%, stage 2 of 4
w ext.slow: 2 row(s), 75%, stage 3 of 4
A component written before progress existed emits one {"ok": true} and keeps
working unchanged - a bare object carrying ok is still a result. A stray log
line is ignored rather than treated as a malformed one.
Cancelling stops the process. A portable in-band cancel would require every
component to read stdin as a live stream while working, which the simple case -
json.load(sys.stdin) - cannot do; so the host terminates it and cleans up its
own files. A component that outlives its declared timeout is killed the same way:
error: ext.slow: did not finish within 1s
Files are referenced, not streamed. A document, a model or a report is a
file: the component writes it into the directory the host provides and names it
back, and the run's provenance records where it is and what it hashed to.
A declared hash is verified, never trusted - a hash a component asserts about
its own output is worth nothing if nobody checks it, and being able to tell later
that the file changed is the whole reason to record one. A component that
declares a file it did not write, or one whose hash does not match, fails the
run. Declare no hash and the host computes it.
Every run records what it ran against. The receipt carries each external
component the pipeline names with its manifest and lock hashes:
The hash is the point, not the version: a component edited in place keeps its
version, which is exactly the case worth being able to detect. A component the
pipeline names and the workspace does not have is recorded as missing rather
than omitted - an absent entry is indistinguishable from a run that used no
external components at all.
Chunked, resumable extraction
A single query over a billion-row table holds a snapshot for hours, fails near
the end and restarts from zero. Declare how it should be split:
strategy range on company_id
chunks 5 concurrency 4
snapshot best effort - each chunk reads when it runs
fallback one query, as today, if chunking is removed
note chunks are separate queries: a row written while the extract runs is in
one chunk or none. This source cannot pin a snapshot across them.
1..1000000 company_id >= 1 AND company_id < 1000001
4000001..4200000 company_id >= 4000001 AND company_id <= 4200000
Range, time and hash strategies. Refusing is the feature: a connector that
cannot give stable semantics is told so rather than emulated, a key with NULLs is
refused with the row count it would silently lose, and a column name that is not
a plain identifier is refused rather than escaped, because a pipeline file is not
a trusted source of SQL fragments. Hash bucketing is spelled out per family -
hashtext, ORA_HASH, CHECKSUM, CRC32 - because getting it wrong does not
error, it silently produces overlapping or empty buckets.
The extent of the key is asked of the source, through the same engine the
extract will use, so the probe reaches the source the way the extract does
rather than by a second path that could succeed where the extract then fails.
--min / --max / --nulls override it, and are what to reach for when this
machine cannot see the database. Numbers typed in once are right once: a table
grows, and nothing notices.
A chunk is a slice, so it is the same ledger.source extract writes one
entry per chunk into the ledger a partitioned backfill uses, and everything
around it comes from there rather than from a second executor: claiming,
bounded concurrency, resource-pool admission, reuse of an identical occurrence,
restart reconciliation, run ids and receipts. backfill status <id> shows a
chunked extract, and backfill retry <id> resumes one - neither of them knows
it is an extract, which is the point. Only the generator differs: a partition
binds a time window, a chunk binds a predicate.
"The query finished" is not "the chunk succeeded."
query completed -> part fsynced, hashed, renamed into place -> slice succeeds
A process that dies between the read and the commit would otherwise leave a
chunk marked done whose part is not there, and the retry that exists to fix
exactly that would skip it - silently, after an hour of database time. So the
part is committed before the ledger moves, and the executor refuses to record a
success without one. On a restart, a chunk whose part is gone or the wrong size
goes back to requested; --verify re-hashes every part, which is the only
thing that catches one edited in place and costs a full read to do.
Assembly is free: the extract IS the parts read together, so a completed extract
prints its read_parquet([...]) and nothing is merged or copied. A partial one
refuses to print a read at all, because a short extract that looks whole is the
failure the design exists to prevent.
The predicate goes into the read, not after it. A filter applied on Duckle's
side would make every chunk fetch the whole table. With pushdown on and your
own SQL, the predicate is placed inside the statement that runs on the server;
otherwise the node's read is wrapped and pushdown is turned off, because the
rewritten SQL names the local attach alias and a remote server has never heard
of it.
2020-01-01 succeeded run-backfill-accounts-...
2020-01-03 failed run-backfill-accounts-... IO Error: No files found ...
bf-accounts-...: 1 failed, 4 succeeded
# the missing file arrives
retrying 1 partition(s) -> 5 succeeded, and one new run, not five
Addressable over the server and from MCP, not only from a CLI:
GET /api/backfills # every plan, or ?id= for one
POST /api/backfills # {"action":"create"|"retry"|"cancel", ...}
Create and retry are accepted and run on a thread, returning the plan id at once
rather than holding a connection open for hours. dryRun lists the partitions
and queues nothing - "what would this queue" must not be a question that queues
anything. Reading needs a viewer; creating, retrying or cancelling needs an
operator.
The MCP tool backfill takes the same five actions, through the same engine
functions, so an agent and an operator cannot get different behaviour.
A slice knows what it is, so it is not done twice. Its identity is
pipeline + partition + release + the schedule occurrence that caused it, hashed
deterministically - so a restart, or the same schedule firing again, finds the
work already done:
first firing 2020-01-01 ok
same occurrence 2020-01-01 already done by bf-accounts-1788359538501
receipts: 3, not 6
The release is part of the identity because the same date against different code
is different work. --force runs them anyway, and a retry is always explicit.
A backfill's own bound is an additional ceiling, not a way around the
machine's. Each slice still acquires the pool its pipeline asks for, so
--max-concurrent 4 over a pipeline in a pool of one runs one at a time:
Each slice is an ordinary durable run - its own receipt, run id, release and
log lines - with the backfill named as its parent, so "which slice produced this
output" is answerable from the receipt alone.
Boundaries are computed in the partition's own zone. A Brussels day is 23
hours in March and 25 in October; generating UTC days would silently process an
hour twice and skip another. Each window ends exactly where the next begins.
The plan is written before anything runs and updated after every slice, so a
kill halfway through leaves something resumable. Slices still marked running on
the next start become interrupted, the same reconciliation a run receipt gets.
Retrying touches only the failures. A thousand days failing on four costs
four runs to finish, not a thousand.
Parameters go through the same boundary as everything else, with partition
as the source - so a value that came from the slice is distinguishable from one
you passed.
Named execution pools
One heavy join should not have to serialise eight cheap HTTP jobs. Name the kinds
of work in <workspace>/.duckle/pools.json:
Admission only. A pool answers may this run start now. What a run may then
use - threads, memory, temp disk - is the existing resources block and is
untouched; otherwise a pipeline could widen its own memory limit by choosing a
different pool.
A pipeline may choose a pool, never widen one. Point DUCKLE_POOLS_FILE at a
server-authoritative file and a workspace can select among those pools and ask
for less, never more; a name the server does not define falls back to default
rather than becoming a new unbounded pool.
One definition, two gates. The runner gates with a condvar and the scheduler
with a tokio semaphore because one is sync and the other async - but both read
the same numbers, because two limiters each parsing their own config is how the
two schedulers came to disagree about time zones.
A run waiting for capacity already exists. It is written as queued with a
queueReason before the wait, so an API or MCP caller gets a durable id
immediately instead of holding its request open, and can inspect or cancel it
while it waits - nothing has started, so there is nothing to undo. When the
permit arrives it becomes running with startedAt and queueMs. A queued run
whose process died is reconciled to interrupted on the next start, so a restart
does not leave stale capacity behind.
A Plan takes no workload slot. It is a supervisor that spends its time
waiting for children, each of which acquires the pool its own pipeline asks for.
Holding a slot while waiting for a child that needs the same pool is a deadlock
the size of the pool.
Every run records resourcePool and, where it actually waited, queueMs - a
pool that is never saturated and one that queues for ten minutes are otherwise
indistinguishable. /metrics carries duckle_pool_permits_free{pool="..."},
because a network pool at 8/8 and an idle heavy pool sum to something that looks
half busy.
A separate duckle-runner invocation is a separate process and an in-process
semaphore cannot bound it. That path records the pool it belongs to and no queue
time, because it never queued.
A run still going after that many seconds is stopped and reported as a
failure - the run exceeded its time limit of 3600s - so failure alerts
fire and its schedule is free for the next occurrence, instead of a hung query
holding it for as long as it hangs. Stopping kills the running DuckDB process,
so it works mid-query, and a Wait node now sleeps in short slices, so a limit
or a person pressing Cancel stops a wait too rather than queueing behind it.
The limit is on the run that was started, on every surface - desktop, CLI,
scheduler, API, MCP - because the engine enforces it where they all meet. A
child pipeline it calls runs under the caller's limit and does not arm its
own.
Which tested version is running? (release)
duckle-runner release build # record the control plane
duckle-runner release diff <from> [<to>] # what changed
duckle-runner release activate <id> --environment production
duckle-runner release rollback --environment production
release c2cc337f3dd609fb... 2 pipeline(s), format v1
production is now running release 15963a9b750d6492...
changed load
production rolled back to release c2cc337f3dd609fb...
A release holds the content, not only its hash. Every pipeline, plan and
schedule is stored immutably, content-addressed and shared between releases, and
activatematerialises it into the workspace. That is what makes
activate A / activate B / rollback actually execute A, B, A - and what makes
releaseId: A on a run mean the run executed A, rather than that A happened to
be the pointer when it started.
Two releases differing in one pipeline cost one extra object, not a second copy
of the workspace. Rebuilding an unchanged workspace produces the same id, so
"has anything changed?" stays a comparison rather than an investigation.
Activation refuses before it mutates. Every check runs and every problem is
reported: the release's stored content must be intact and still compile, every
connection a pipeline names must exist, and policy must load. An operator fixing
a production activation should not discover the second problem after fixing the
first. It does not require the workspace to already match the release - that
requirement is what would make rollback impossible, since rolling back to A is
exactly the case where the workspace holds B.
Uncommitted work is named, not discarded. Activating overwrites the
control-plane files, so if the workspace differs it lists what would be
overwritten or removed and refuses without --force.
The pointer swap is one rename. Never remove-then-rename: std::fs::rename
replaces the destination even while a reader holds it open, so unlinking first
buys nothing and costs exactly the guarantee - a window where the environment
points at nothing.
Rollback is deliberately not gated on those checks. It is what an operator
reaches for when the current release is broken, and one that refuses because the
workspace is in a bad state is one that never works when it is needed.
Every run records the release it ran under, read when the run starts - so a
run already in flight when someone activates keeps naming the release it began
with, because it did not silently change code halfway through.
The hashes are the ones already in use. A pipeline's hash is
retry::pipeline_hash, the same one its run receipt records; a second hash would
eventually disagree with the first. Only hashes and declarations are stored -
connection references so activation can check them, never their values.
Sign in through an identity provider (OIDC)
Optional, and off unless <workspace>/.duckle/oidc.json exists:
Authorization-code flow with PKCE (S256), state, nonce, and RS256 ID-token
verification against the provider's JWKS. It adds a protocol and nothing else -
the flow ends by minting the same session a password login mints, and every
request after that is authorised by the same code as before.
A subject no rule matches is refused, unless defaultRole says otherwise.
An identity provider saying who someone is does not say what they may do here,
and absent means deny because that is the answer that cannot surprise anyone.
First matching rule wins, in file order, so the result is a property of the
config rather than of iteration order. A group name matches whole: contains: "data-admins" does not match a group called not-data-admins-really.
The callback is bound to the browser that started the login. The redirect
sets a short-lived HttpOnly cookie and the callback requires it back, compared in
constant time. Single-use and a five-minute TTL stop a state being replayed; they
do not stop an attacker starting a login, taking the callback URL for their own
identity and getting a victim to visit it - which would sign the victim in as the
attacker.
Audit names the provider's subject. A display name is self-service at most
providers, so a session labelled with one lets a user choose their own actor
string - including the label the break-glass admin runs under - and every action
they take afterwards is recorded against it. The actor is sub (display name),
and the part that identifies is the subject.
Break-glass is untouched. The --token / DUCKLE_CONSOLE_TOKEN admin lives
only in the process and never in the store, so it still works when the provider
does not. Scoped API keys are unaffected.
No reverse-proxy identity headers, by not implementing them: in any
deployment where the proxy can be bypassed, an X-Forwarded-User header is an
admin login. No provider tokens are stored - the ID token is verified and
dropped, and only the subject, a display name and the mapped role reach the
session. Audit records the provider's stable sub, not the display name, because
a name or an email can be reassigned to a different person.
Reading GeoParquet
The Geospatial source reads GeoParquet as well as GeoJSON, Shapefile,
GeoPackage, KML, GPX and GML.
ST_Read is GDAL-backed and the spatial extension DuckDB ships does not carry
GDAL's Parquet driver, so a .geoparquet path failed with Could not open GDAL dataset - the file was perfectly readable, just not by that function. Parquet
paths now go through read_parquet, which returns a real GEOMETRY with its CRS
intact; everything else still goes through ST_Read.
Spatial sort on Parquet export
Name a GEOMETRY column on a Parquet sink and rows are sorted along a Hilbert
curve before writing, so geometries close on the ground land in the same row
group and a spatial filter can skip more of the file.
Measured on 2,000 scattered points written in 40 row groups of 50:
written
mean row-group bbox area
as they arrive
954,575
Hilbert
19,549
The curve is scaled to this dataset's own extent, which costs one extra pass
to find it - that is the trade the option exists to make. Leave the field empty
and nothing changes: the emitted SQL is byte-for-byte what it was.
One field rather than a checkbox and a column, because a checkbox ticked with no
column chosen is a state the engine would have to guess at, and guessing which
column holds the geometry is how the wrong one gets sorted on.
A watcher is not a run
follow polls continuously, and most polls find nothing. A source checked every
ten seconds is unchanged thousands of times between real arrivals, so there are
two identities rather than one:
session follow-orders-1788270909966 the watcher: is it up, when did it last look?
run run-follow-orders-1788270910157 one execution: what did it do, can I retry it?
A poll that finds nothing updates the session - lastPollAt, pollCount,
lastError - and nothing else. A poll that actually moves rows, or fails, gets a
normal run id from the same primitive every other surface uses, names the session
as its parent, and lands in run history: retryable, comparable and addressable
exactly like a scheduled or manual run.
pollCount - runCount is how much of the watching was quiet. lastPollAt and
lastEventAt are separate because "healthy and idle" and "healthy and ingesting"
are different states and one field cannot say which.
A killed watcher is interrupted, not running forever. The session is
written before the loop starts and reconciled on the next start, the same way a
run receipt
