I Built the Flink SQL Connector DuckDB Didn’t Have

A new open-source Apache Flink SQL connector for DuckDB: write or upsert streaming results straight into a DuckDB file with a CREATE TABLE and an INSERT INTO, no custom DataStream job and no hand-rolled JDBC plumbing — plus the one correctness trap that had nothing to do with SQL at all.

DuckDB is one of the most widely used embedded analytical databases, and as of this writing there is no official Flink SQL connector for DuckDB, and no well-known community one either. A widely-used embedded database, a widely-used streaming engine, and until now nothing connecting them in SQL.

The shape

CREATE TABLE sink_table (id BIGINT, name STRING, score DOUBLE)
WITH ('connector' = 'duckdb', 'database-path' = '/data/my.db', 'table-name' = 't',
      'key-fields' = 'id');

INSERT INTO sink_table SELECT id, name, score FROM source_stream;

Plain insert by default, upsert-by-key (ON CONFLICT) when key-fields is set. A plain dynamic table factory, same shape as any other Flink sink.

The challenges

Challenge one: a correct error that looked like a connector bug.

DuckDB’s locking model is simple to state and easy to get wrong in practice: it locks its database file for exclusive access from one OS process. Flink, by default, runs multiple subtasks sharing a TaskManager process — same JVM, separate threads and connections. Naively set sink.parallelism above 1 against a local DuckDB file and you don’t get a clean error — you get a job that silently hangs or races, because from DuckDB’s perspective it’s getting concurrent writers from inside one process, each with a connection that looks legitimate until it isn’t.

Fix: don’t just document the limitation — enforce it. The connector validates sink.parallelism at table-creation time and rejects anything above 1 against a local file target, failing fast at SQL-planning time instead of leaving a job stuck waiting on task slots it will never use productively.

CREATE TABLE sink_table (id BIGINT, name STRING)
WITH ('connector' = 'duckdb', 'database-path' = '/data/my.db',
      'table-name' = 't', 'sink.parallelism' = '4');
-- rejected at planning time: local DuckDB files only support parallelism = 1

Verified directly: setting sink.parallelism = 4 against a local file is rejected at planning time with a clear error, instead of the job quietly hanging. The only way to raise parallelism is pointing at a concurrency-safe remote target, such as MotherDuck, instead of a local file.

Challenge two: how much durability are you actually promising.

Get Ashfaq’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Each flush commits independently of Flink’s checkpoint barriers — there’s no two-phase commit here. That makes plain inserts at-least-once (a restart-from-checkpoint can replay rows written after the last successful checkpoint and duplicate them), while upserts with key-fields set are effectively exactly-once, because ON CONFLICT-based upsert-by-key makes replays idempotent. The fix wasn’t a bigger engineering effort to bolt on exactly-once everywhere — it was being explicit about which guarantee applies when, so the caller decides with real information instead of assuming. Set key-fields if you care about replay safety; the connector won’t silently give you more than it actually guarantees.

What’s honestly scoped out

Sink-only, v1 — no changelog/retract-stream upsert, only ON CONFLICT-on-duplicate, which covers the common case but not an aggregation or join upstream of the sink that needs real retraction semantics. The table has to already exist — this connector doesn’t create one for you.

Testing: DuckDB is embedded, so all 14 tests open a real temporary .db file directly — no Docker, no testcontainers — and they’re still testing against the real thing, not mocks: inserts, upserts, DECIMAL/DATE/TIMESTAMP/null round-tripping, and batch-size-triggered flushing.

Recap

• No official or well-known community Flink SQL connector for DuckDB existed — this closes that gap with a real dynamic table factory, not a one-off DataStream job.
• The write logic worked first try; the trap was DuckDB’s process-level file lock silently breaking under Flink’s default parallelism.
• Fixed by rejecting unsafe parallelism at planning time, not letting a job hang.
• At-least-once for plain inserts, effectively exactly-once with key-fields set — stated plainly rather than assumed.

The lesson worth keeping: the SQL layer was never the hard part. The hard part was understanding exactly how the target database’s own concurrency model collides with Flink’s default execution model — and refusing to let that collision happen silently.

Try it

git clone https://github.com/Ashfaqbs/flink-connector-duckdb.git
cd flink-connector-duckdb
mvn clean install

Repo: https://github.com/Ashfaqbs/flink-connector-duckdb

If you’re searching for a Flink SQL connector for DuckDB or a way to stream data from Apache Flink into an embedded analytical database, this is that connector.