As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :)
What's the story for version control, debugging, testing, releasing? It'd be cool to have everything together for data locality and simplifying the stack, but it feels you'd lose a lot of useful knowledge about how to do stuff "properly".
One would be able to trigger maintenance jobs via simple lambda functions whose duration is capped.
For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th...
Also for example, you dont get ultra-wide high dimensionality vectors.
It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?
Why would I want to store my control flow in the database and not in code? It feels strange.
Not trying to dismiss the project, I'm just not getting it yet I think.
df.wait_for_schedule()
How does this call work? Is it idempotent if I call it from an application? If I run it 2x with the same parameters, does it double tick? Am I invoking this manually from a query console to only do this one time? Am I running this as part of a migration script?For this[0]:
-- Wait for human signal (5 minute timeout)
~> (df.wait_for_signal('approval', 300) |=> 'sig')
~> df.if(
$$SELECT NOT ($sig::jsonb->>'timed_out')::boolean
AND ($sig::jsonb->'data'->>'approved')::boolean$$,
Is the `timed_out` a fixed constant that is returned on timeout?Also not immediately clear: how to handle errors/exceptions?
[0] https://github.com/microsoft/pg_durable/blob/main/examples/i...
https://github.com/microsoft/duroxide - also OSS, the durable execution framework pg_durable is built on itself supports function versions. We can leverage that to get similar support in pg_durable.
That said, we did hand-build a simple job queue (just lock, poll, reserve on a column, poll and update reservation to mark job done) on top of postgres at my previous startup. Something like pgque would have made that much more polished.
I mean, we used to keep our SQL code in git too for projects where we had DB triggers. I think some were even shoved in there via Django migrations just to let someone setup locally and have the triggers available in their local database.
How is this project at all comparable to something like Temporal? Am I misunderstanding the limitation implied by this particular recommendation?
i have always had maintenance packages for this type of stuff. if i could deploy them alongside the database itself that could be kind of cool.
but yeah i agree with you that i do prefer having this in the code layer.
Is the proposal to be able to export pg_dump formatted data on some schedule or trigger, entirely hosted in PostgreSQL and with timeouts? There are already extension that can export to blob/file storage and can be combined with pg_durable or pg_cron, so I assume the challenge is pg_dump compatible data export from SQL running in the database?
This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to cross-reference the workflow log with the codebase and trace through it line by line to figure out what the state is. Plus I assume it's less overhead and latency, and operationally one less thing to spin up.
[1] https://learn.microsoft.com/en-us/azure/durable-task/common/...
The provider is an extensibility point. We just shipped the simplest version of it. Happy to take contribs if someone sends a pgmq based provider!
We use Postgres for that on https://transport.data.gouv.fr (Elixir app which does a fair bit of processing), and it helps.
Not familiar yet with pg_durable though, but I have used or implemented similar solutions and can relate.
Also if all the "state" is in one database, then you have better chance of getting consistent backups.
It's an interesting technical achievement I guess, but it's very bizarre to try and read this
SELECT df.start(
@> (
($$SELECT ... FROM demo.invoices WHERE status = 'pending'$$ |=> 'inv')
~> df.if_rows('inv',
$$UPDATE ... SET status = 'processing'$$
~> (df.http(...) |=> 'resp')
~> df.if($$SELECT $r.ok$$,
-- classify, branch, wait for signal ...
),
df.sleep(5)
)
),
'invoice-approval-pipeline'
);Don't need to synchronize the backups with anything else that is part of the same data store, good for ETL pipelines and other state machine type jobs.
If your ETL is mostly SQL anyway, then having the actual job being run on the same server helps as well.
Long-running, fault-tolerant SQL functions for teams that already keep their state in Postgres and want to stop stitching together cron jobs, workers, queues, and status tables to make background work reliable. Define the workflow in SQL, let pg_durable checkpoint each step, and resume after crashes, restarts, or failed steps.
Durable execution is now a standard industry pattern, and pg_durable brings it inside Postgres with no extra service infrastructure required. Part of our mission to bring compute close to data.
A pg_durable function is a graph of SQL steps that PostgreSQL executes and checkpoints as it goes. If the database crashes, restarts, or a step fails, execution resumes from the last durable checkpoint instead of making you reconstruct state by hand.
pgvector.pg_cron plus a jobs table, status columns, retry counters, and a polling worker.plpgsql procedure that works until a crash or long-running transaction forces you to start over.df.start(...).df.instances, using the same auth and backup model as your data.INSERT ... SELECT or one ordinary SQL statement.~> and |=>.df.start() and get back an instance ID.The model is intentionally SQL-shaped. If a step needs arbitrary code, a non-HTTP SDK, or rich in-memory control flow, you may need to wrap that logic in a SQL function, expose it behind an HTTP endpoint for df.http(), or use a general-purpose orchestrator for that part of the system.
-- A durable function that processes data in steps
SELECT df.start(
'SELECT id FROM documents WHERE processed = false LIMIT 100' |=> 'batch'
~> 'UPDATE documents SET processed = true WHERE id = ANY($batch)'
);
Tagged releases publish Debian packages for PostgreSQL 17 and 18 on amd64 from the GitHub release assets. Packages are named pg-durable-postgresql-<PG major>_<pg_durable version>-1_<arch>.deb and install the extension library, control file, and SQL upgrade files into the matching PostgreSQL installation directories.
After installing a package, add pg_durable to shared_preload_libraries, restart PostgreSQL, and create the extension in the configured pg_durable database:
CREATE EXTENSION pg_durable;
The default pg_durable database is postgres; see User Guide for background worker configuration and privilege setup.
Release assets also include source archives for building from source.
The main branch prebuild installs PostgreSQL 17, builds pg_durable, and prepares a local cluster under ~/.pgrx with the extension ready. PostgreSQL is not left running, so start it when you begin working.
# Start PostgreSQL
./scripts/pg-start.sh
# Connect
~/.pgrx/17.*/pgrx-install/bin/psql -h localhost -p 28817 -d postgres
On a branch without a ready prebuild, run pg-start.sh β it will build and install the extension on first run (expect a few minutes):
./scripts/pg-start.sh
A VS Code Dev Container (.devcontainer/) provides Rust, cargo-pgrx, and PostgreSQL 17 pre-installed. For a bare local machine, install the toolchain first by following the steps in .devcontainer/onCreateCommand.sh.
# Build, initialize PostgreSQL, and install the extension
# This takes a while - go do something else
./scripts/pg-start.sh
# Connect to the local pgrx PostgreSQL instance
~/.pgrx/17.*/pgrx-install/bin/psql -h localhost -p 28817 -d postgres
pg-start.sh bootstraps new local data directories with a postgres superuser and also creates a matching superuser role for the current OS user, so default local psql usage continues to work. Use -U postgres if you want to force the canonical bootstrap role explicitly.
# Build and test
./scripts/test-e2e-docker.sh --rebuild
# Optional: Deploy to ACR (for custom PG17 image with pg_durable baked-in)
./scripts/deploy-acr.sh
CREATE EXTENSION pg_durable does not grant any privileges to PUBLIC. After installing the extension, the admin must explicitly grant access to application roles. Row-level security (RLS) ensures each user can only see and manage their own durable function instances and nodes.
Grant privileges to an application role:
-- Grant to specific roles after CREATE EXTENSION
SELECT df.grant_usage('app_role');
Alternatively, create an indirection role and grant membership to application roles:
-- Create a shared role for pg_durable access
CREATE ROLE pg_durable_user NOLOGIN;
SELECT df.grant_usage('pg_durable_user');
-- Grant membership to application roles
GRANT pg_durable_user TO app_backend, etl_service;
See the User Guide β Privilege Grants section for the full list of individual grants, revoking access, and hardening upgraded installs.
Note:
GRANT EXECUTE ON ALL FUNCTIONSonly applies to functions that exist when the grant runs. After upgrading pg_durable withALTER EXTENSION pg_durable UPDATE, re-rundf.grant_usage('role')(or re-issue the manual grants) so new functions are accessible.
Key points:
pg_durable.worker_role GUC, default: azuresu) must be a superuser β it bypasses RLS to manage all users' instancesSELECT + INSERT on df.instances / df.nodes, column-level UPDATE (status, updated_at) on instances for df.cancel()submitted_by) cannot be modified by usersdf.vars uses per-user scoping β each user has their own variable namespace via an owner column and RLS. Superusers bypass RLS but DSL functions still scope to the calling user via explicit filters. Avoid storing secrets in plain textAll pull requests must pass the following checks before merging:
cargo fmt --checkcargo clippy, unit tests (cargo pgrx test pg17), pg_regress tests, and E2E testsThe CI workflow is defined in .github/workflows/ci.yml. It uses pgrx to download and manage PostgreSQL.
pg_durable has two test suites:
Fast, deterministic tests for core DSL functionality using PostgreSQL's standard testing framework.
Test SQL lives in sql/, expected output in expected/, and PGXS is configured in the root Makefile.
make test-regress # full reset + run
make installcheck # run only (PostgreSQL must already be running)
Complex local integration tests with pgrx PostgreSQL:
./scripts/test-e2e-local.sh # All local SQL E2E tests, including special restart/config phases
./scripts/test-e2e-local.sh 04_parallel # Specific test
./scripts/test-e2e-local.sh --default-build-phases # Only the default-build phase group
See tests/e2e/ for details.
pg_durable is a PostgreSQL extension (built with pgrx) β everything runs inside the PostgreSQL server, no external services. The extension exposes a SQL DSL for building function graphs and registers a background worker that executes them durably on top of two lower-level Rust libraries:
duroxide.* schema owned by the extension.ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PostgreSQL β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β pg_durable extension (pgrx) β β
β β β β
β β SQL DSL 'sql' |=> 'name' ~> 'sql2' β β
β β df.if() | df.join() | df.loop() β β
β β β β
β β Background worker (hosts the duroxide runtime in-process) β β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β
β β β duroxide (orchestration runtime) β β β
β β β ββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β
β β β β duroxide-pg (PostgreSQL state provider) β β β β
β β β ββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β
β β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Schemas β
β df.* DSL graphs (nodes, instances, vars) β
β duroxide.* runtime state (owned by duroxide-pg) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
If you'd rather author durable functions in Rust, Python, or Node while still persisting state in PostgreSQL, you can use duroxide and duroxide-pg directly from your host language β pg_durable is what you'd build on top of that pair when you'd prefer authoring in SQL.
Preview - This project is currently in preview.
Use GitHub Issues for bug reports and feature requests. Do not report security vulnerabilities through public GitHub issues; follow the instructions in SECURITY.md instead.
This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with questions or comments.
Microsoft takes the security of our software products and services seriously. Please do not report security vulnerabilities through public GitHub issues. See SECURITY.md for security reporting instructions.
pg_durable does not send telemetry to Microsoft.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos is subject to those third-party policies.
PostgreSQL License
Indeed Durable tasks is an exceptional project and was a unique innovation at the time.
pg_durable brings the same reliability and durablity semantics to long running operations within the database.
We have tons of interesting scenarios on the roadmap. Stay tuned! :)
The steps are:
1. Get all the pending invoices
2. Set their state to "processing"
3. Call out to an external service/process to do the actual processing, wait for a response.
4. If the response is OK, do something
5. Wait 5 seconds and then start again.
Not sure I love the syntax and the way SQL is embedded between the $$
But it is in the database, can be updated and modified in the same way as all the other stored procedures/functions, allows job control, I assume other control structures for parallel steps etc.
Gonna go read the doco now.
Not trying to dismiss the project - it looks like a lot of hard work has gone in and somebody has a use for it. I just come from an airflow style external orchestrator frame of mind that manages durability state in postgres but keeps the control flow out. Sorry if I came off as a bit snarky