Logical decoding server for PostGraphile. (Can also act as a library with no server.)
Connects to a database and streams logical decoding events to interested parties.
NOTICE: If you're just getting started, refer to
Crowd-funded open-source software
To help us develop this software sustainably under the MIT license, we ask all individuals and businesses that use it to help support its ongoing maintenance and development via sponsorship.
And please give some love to our featured sponsors 🤩:
We currently only support PG10+; if you need support for 9.x please get in touch.
wal2json; this is available on:
- Amazon RDS
- (please send a PR adding more compatible providers)
If you don't already have it, you can install it. On your Unix-based OS (assuming
pg_config is in your path, and points to the correct PostgreSQL installation) you can add it in a few seconds with:
git clone https://github.com/eulerto/wal2json.gitcd wal2jsonUSE_PGXS=1 makeUSE_PGXS=1 make install
(No need to restart PostgreSQL)
postgresql.conf you need to ensure that the following settings are set:
wal_level = logical max_wal_senders = 10 max_replication_slots = 10
(You can set max_wal_senders and max_replication_slots to a number at least 1.)
You can determine if your PostgreSQL instance is configured correctly with this:
DO $$BEGINif current_setting('wal_level') is distinct from 'logical' thenraise exception 'wal_level must be set to ''logical'', your database has it set to ''%''. Please edit your `%` file and restart PostgreSQL.', current_setting('wal_level'), current_setting('config_file');end if;if (current_setting('max_replication_slots')::int >= 1) is not true thenraise exception 'Your max_replication_slots setting is too low, it must be greater than 1. Please edit your `%` file and restart PostgreSQL.', current_setting('config_file');end if;if (current_setting('max_wal_senders')::int >= 1) is not true thenraise exception 'Your max_wal_senders setting is too low, it must be greater than 1. Please edit your `%` file and restart PostgreSQL.', current_setting('config_file');end if;perform pg_create_logical_replication_slot('compatibility_test', 'wal2json');perform pg_drop_replication_slot('compatibility_test');raise notice 'Everything seems to be in order.';end;$$ LANGUAGE plpgsql;
If you see the following message then all should be good:
NOTICE: 00000: Everything seems to be in order.
I've not put any argument parsing into this yet, so everything's done with envvars:
LD_DATABASE_URL- the database URL to connect to for logical decoding; must have permission to create the logical replication slot
LD_TABLE_PATTERN- optional setting, allows us to ignore changes in tables you don't care about; default: '.', recommended: 'app_public.*' (assuming your PostGraphile schema is called 'app_public').
PORT- the port number to run this server on, defaults to
LD_HOST- the host name to listen on, defaults to
0.0.0.0to listen on all interfaces, e.g. if running inside of Docker)
LD_SLOT_NAME- optional name of the logical decoding slot to use, we use
postgraphileby default. Be sure to drop the old slot (see below) if you change this.
LD_MAX_CLIENTS- set to the maximum number of clients to allow to connect to the server. Defaults to
50. (Each PostGraphile instance counts as one client.)
LD_WAIT- duration in milliseconds to pause between logical decoding polls; defaults to
200, reduce for lower latency but higher CPU usage.
This package installs the
graphile-lds command. If you installed it globally you should be able to execute it directly, otherwise we advise you run
npx to run it:
LD_DATABASE_URL="postgres:///my_db" npx graphile-lds
It's essential that you drop the logical replication slot when you no longer need it, otherwise your disk will fill up. PostgreSQL does this for you with slots marked as "temporary", but for normal slots you must do this manually:
SELECT pg_drop_replication_slot('postgraphile'); -- or whatever slot name you were using.
(It's okay to keep the slot active whilst you're running the LDS because we'll keep consuming the data and it'll be cleared automatically. It's only when LDS isn't running that data will build up.)