Supabase Postgres 14.1.0 (#98)

* chore: bump postgres and pgbouncer
- #67 #68

* chore: #65 sha-scram-256 encryption

* chore: Bump to Postgres 14.0

* chore: bump up existing extensions

* Use PG 14 version of postgresql.conf

* move up running of SQL files
- to a part where the DB is still up and running

* new extensions:
- rum
- pg_hashids

* Cleanup build dependencies immediately after installation

* Remove hardcoded values for Postgres major

* Update variables

* Update wal-g installation process

* add build dependencies for supautils

* changes to pgbouncer config
- use scram-sha-256 encryption
- list out additional build dependencies
- assign postgres as stats user

* Revert "Cleanup build dependencies immediately after installation"

This reverts commit 25c93de.

* Minimise packages cleaned up
- some extensions were dependent to them

* link pg binaries only at the end

* remove postgres as pgbouncer stats_users

* bump to Postgres 14.1

* rename filename for pgsodium

* update regex for session_preload_libraries

* Update README

* update download URLs for postgrest

* Bump versions

* feat: build AMIs using Github Actions (#101)

* feat: build AMIs using Github Actions

* eliminate two-step build

Co-authored-by: dragarcia <[email protected]>

* Transfer ownership of pgbouncer process to pgbouncer user

* increased timeout for get_url

* further hardening of pgbouncer
- no access to public schema
- cannot connect to port 5432 by pgbouncer user other than localhost

* bump common.vars.json

* Enhancements to docker images (#108)

* GA for dockerhub

* Shift removal of unused packages at the bottom
- For some reason, this disrupts the ansible process
- Err: the connection plugin 'local' was not found

* No longer use init.sh for docker

* Use Dockerfile when building docker images

* cleanup /tmp via the Dockerfile

* more changes to docker build
- move more cleanup to the Dockerfile
- run everything under one command in the Dockerfile

* Commenting these out for now
- Need to configure runners first
- Will uncomment in another PR once done

* ignore caretion of symbolic link for plv8 in docker arm build

* Temporarily remove CI files for Dockerhub

Co-authored-by: Div Arora <[email protected]>

Author: dragarcia

.github/workflows/ci.yml

@@ -0,0 +1,19 @@
+name: Run CI checks
+
+on:
+  push:
+    branches:
+      - develop
+
+jobs:
+  build:
+    runs-on: [self-hosted, linux]
+    timeout-minutes: 150
+
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v2
+
+      - name: Build AMI
+        run: |
+          packer build -timestamp-ui -color=false -on-error=abort -var-file common.vars.json -var-file development-arm.vars.json amazon.json

Dockerfile

@@ -0,0 +1,21 @@
+ARG PLATFORM
+ARG VERSION
+
+FROM --platform=$PLATFORM postgres:$VERSION
+
+COPY ansible/ /tmp/ansible/
+
+RUN apt update && \
+    apt install -y ansible && \
+    cd /tmp/ansible && \
+    ansible-playbook playbook-docker.yml && \
+    apt -y update && \
+    apt -y upgrade && \
+    apt -y autoremove && \
+    apt -y autoclean && \
+    apt install -y default-jdk-headless && \
+    rm -rf /tmp/* /var/lib/apt/lists/* /var/tmp/* 
+
+ENV LANGUAGE=en_US.UTF-8
+ENV LANG=en_US.UTF-8
+ENV LC_ALL=en_US.UTF-8

README.md

@@ -3,32 +3,35 @@
 Unmodified Postgres with some useful plugins. Our goal with this repo is not to modify Postgres, but to provide some of the most common extensions with a one-click install.
 
 ## Primary Features
-- ✅ Postgres [13](https://www.postgresql.org/about/news/postgresql-13-released-2077/).
+- ✅ Postgres [14](https://www.postgresql.org/about/news/postgresql-14-released-2318/).
 - ✅ Ubuntu 20.04 (Focal Fossa).
 - ✅ [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) = logical and [max_replication_slots](https://www.postgresql.org/docs/current/runtime-config-replication.html) = 5. Ready for replication.
 - ✅ [Large Systems Extensions](https://github.com/aws/aws-graviton-getting-started#building-for-graviton-and-graviton2). Enabled for ARM images.
 
 ## Extensions 
-| Extension | Description |
-| ------------- | ------------- |
-| [Postgres contrib modules](https://www.postgresql.org/docs/current/contrib.html) | Because everyone should enable `pg_stat_statements`. |
-| [PostGIS](https://postgis.net/) | Postgres' most popular extension - support for geographic objects. |
-| [pgRouting](https://pgrouting.org/) | Extension of PostGIS - provides geospatial routing functionalities. |
-| [pgTAP](https://pgtap.org/) | Unit Testing for Postgres. |
-| [pg_cron](https://github.com/citusdata/pg_cron) | Run CRON jobs inside Postgres. |
-| [pgAudit](https://www.pgaudit.org/) | Generate highly compliant audit logs. |
-| [pgjwt](https://github.com/michelp/pgjwt) | Generate JSON Web Tokens (JWT) in Postgres. |
-| [pgsql-http](https://github.com/pramsey/pgsql-http) | HTTP client for Postgres. |
-| [plpgsql_check](https://github.com/okbob/plpgsql_check) | Linter tool for PL/pgSQL. |
-| [pg-safeupdate](https://github.com/eradman/pg-safeupdate) | Protect your data from accidental updates or deletes. |
-| [wal2json](https://github.com/eulerto/wal2json) | JSON output plugin for logical replication decoding. |
-| [PL/Java](https://github.com/tada/pljava) | Write in Java functions in Postgres. |
-| [plv8](https://github.com/plv8/plv8) | Write in Javascript functions in Postgres. |
-| [pg_plan_filter](https://github.com/pgexperts/pg_plan_filter) | Only allow statements that fulfill set criteria to be executed. |
-| [pg_net](https://github.com/supabase/pg_net) | Expose the SQL interface for async networking. |
-| [pg_sodium](https://github.com/michelp/pgsodium) | Modern encryption API using libsodium. |
-
-Can't find your favorite extension? Suggest for it to be added into future versions [here](https://github.com/supabase/supabase/discussions/679)!
+| Extension | Version | Description |
+| ------------- | :-------------: | ------------- |
+| [Postgres contrib modules](https://www.postgresql.org/docs/current/contrib.html) | - | Because everyone should enable `pg_stat_statements`. |
+| [PostGIS](https://postgis.net/) | [3.1.4](https://git.osgeo.org/gitea/postgis/postgis/raw/tag/3.1.4/NEWS) | Postgres' most popular extension - support for geographic objects. |
+| [pgRouting](https://pgrouting.org/) | [v3.3.0](https://github.com/pgRouting/pgrouting/releases/tag/v3.3.0) | Extension of PostGIS - provides geospatial routing functionalities. |
+| [pgTAP](https://pgtap.org/) | [v1.1.0](https://github.com/theory/pgtap/releases/tag/v1.1.0) | Unit Testing for Postgres. |
+| [pg_cron](https://github.com/citusdata/pg_cron) | [v1.4.1](https://github.com/citusdata/pg_cron/releases/tag/v1.4.1) | Run CRON jobs inside Postgres. |
+| [pgAudit](https://www.pgaudit.org/) | [1.6.1](https://github.com/pgaudit/pgaudit/releases/tag/1.6.1) | Generate highly compliant audit logs. |
+| [pgjwt](https://github.com/michelp/pgjwt) | [commit](https://github.com/michelp/pgjwt/commit/9742dab1b2f297ad3811120db7b21451bca2d3c9) | Generate JSON Web Tokens (JWT) in Postgres. |
+| [pgsql-http](https://github.com/pramsey/pgsql-http) | [1.3.1](https://github.com/pramsey/pgsql-http/releases/tag/v1.3.1) | HTTP client for Postgres. |
+| [plpgsql_check](https://github.com/okbob/plpgsql_check) | [2.0.6](https://github.com/okbob/plpgsql_check/releases/tag/v2.0.6) | Linter tool for PL/pgSQL. |
+| [pg-safeupdate](https://github.com/eradman/pg-safeupdate) | [1.4](https://github.com/eradman/pg-safeupdate/releases/tag/1.4) | Protect your data from accidental updates or deletes. |
+| [wal2json](https://github.com/eulerto/wal2json) | [2.4](https://github.com/eulerto/wal2json/releases/tag/wal2json_2_4) | JSON output plugin for logical replication decoding. |
+| [PL/Java](https://github.com/tada/pljava) | [1.6.3](https://github.com/tada/pljava/releases/tag/V1_6_3) | Write in Java functions in Postgres. |
+| [plv8](https://github.com/plv8/plv8) | [commit](https://github.com/plv8/plv8/commit/3656177d384e3e02b74faa8e2931600f3690ab59) | Write in Javascript functions in Postgres. |
+| [pg_plan_filter](https://github.com/pgexperts/pg_plan_filter) | [commit](https://github.com/pgexperts/pg_plan_filter/commit/5081a7b5cb890876e67d8e7486b6a64c38c9a492) | Only allow statements that fulfill set criteria to be executed. |
+| [pg_net](https://github.com/supabase/pg_net) | [v0.3](https://github.com/supabase/pg_net/releases/tag/v0.3) | Expose the SQL interface for async networking. |
+| [rum](https://github.com/postgrespro/rum) | [1.3.9](https://github.com/postgrespro/rum/releases/tag/1.3.9) | An alternative to the GIN index. |
+| [pg_hashids](https://github.com/iCyberon/pg_hashids) | [commit](https://github.com/iCyberon/pg_hashids/commit/83398bcbb616aac2970f5e77d93a3200f0f28e74) | Generate unique identifiers from numbers. |
+| [pg_sodium](https://github.com/michelp/pgsodium) | [v1.3.0](https://github.com/michelp/pgsodium/releases/tag/v1.3.0) | Modern encryption API using libsodium. |
+
+
+Can't find your favorite extension? Suggest for it to be added into future releases [here](https://github.com/supabase/supabase/discussions/679)!
 
 ## Enhanced Security
 *This is only available for our AWS EC2/ DO Droplet images*
@@ -42,11 +45,11 @@ Aside from having [ufw](https://help.ubuntu.com/community/UFW),[fail2ban](https:
 ## Additional Goodies
 *This is only available for our AWS EC2/ DO Droplet images*
 
-| Goodie | Description |
-| ------------- | ------------- |
-| [PgBouncer](https://www.pgbouncer.org/) | Set up Connection Pooling. |
-| [PostgREST](https://postgrest.org/en/stable/) | Instantly transform your database into an RESTful API. |
-| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | Tool for physical database backup and recovery. |
+| Goodie | Version | Description |
+| ------------- | :-------------: | ------------- |
+| [PgBouncer](https://www.pgbouncer.org/) | [1.16.1](http://www.pgbouncer.org/changelog.html#pgbouncer-116x) | Set up Connection Pooling. |
+| [PostgREST](https://postgrest.org/en/stable/) | [v8.0.0](https://github.com/PostgREST/postgrest/releases/tag/v8.0.0) | Instantly transform your database into an RESTful API. |
+| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | [v1.1](https://github.com/wal-g/wal-g/releases/tag/v1.1) | Tool for physical database backup and recovery. |
 
 ## Install
 

amazon.json

@@ -1,34 +1,36 @@
 {
   "variables": {
-    "aws_access_key": "",
-    "aws_secret_key": "",
-    "region": "af-south-1",
-    "ami_regions": "af-south-1",
-    "ami": "ami-08a4b40f2fe1e4b35",
-    "ami_name": "supabase-postgres-13.3.0.4",
-    "environment": "prod",
-    "ansible_arguments": "--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
+    "profile": "{{env `AWS_PROFILE`}}"
   },
   "builders": [
     {
       "type": "amazon-ebs",
-      "access_key": "{{user `aws_access_key`}}",
-      "secret_key": "{{user `aws_secret_key`}}",
+      "profile": "{{user `profile`}}",
       "region": "{{user `region`}}",
       "ami_regions": "{{user `ami_regions`}}",
-      "source_ami": "{{user `ami`}}",
-      "instance_type": "m5.2xlarge",
+      "source_ami": "{{user `ubuntu-2004`}}",
+      "instance_type": "{{user `instance-type`}}",
       "ssh_username": "ubuntu",
-      "ami_name": "{{user `ami_name`}}",
+      "ami_name": "supabase-postgres-{{user `postgres-version`}}",
       "tags": {
         "environment": "{{user `environment`}}",
-        "appType": "postgres"
+        "appType": "postgres",
+        "creator": "packer"
+      },
+      "run_tags": {
+        "creator": "packer"
+      },
+      "snapshot_tags": {
+        "creator": "packer"
+      },
+      "run_volume_tags": {
+        "creator": "packer"
       },
       "launch_block_device_mappings": [
         {
           "device_name": "/dev/sda1",
           "volume_size": 16,
-          "volume_type": "gp2",
+          "volume_type": "gp3",
           "delete_on_termination": true
         }
       ]
@@ -45,7 +47,7 @@
       "type": "ansible",
       "user": "ubuntu",
       "playbook_file": "ansible/playbook.yml",
-      "extra_arguments": "{{user `ansible_arguments`}}"
+      "extra_arguments": "--skip-tags,install-postgrest"
     },
     {
       "execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",

ansible/files/docker_mnt/init.sh

@@ -113,7 +113,7 @@ unix_socket_dir = /tmp
 ;;;
 
 ;; any, trust, plain, md5, cert, hba, pam
-auth_type = md5
+auth_type = scram-sha-256
 auth_file = /etc/pgbouncer/userlist.txt
 
 ;; Path to HBA-style auth config

ansible/files/pgbouncer_config/pgbouncer.ini.j2

@@ -30,7 +30,7 @@ After=network.target
 
 [Service]
 Type=notify
-User=postgres
+User=pgbouncer
 ExecStart=/usr/local/bin/pgbouncer /etc/pgbouncer/pgbouncer.ini
 ExecReload=/bin/kill -HUP $MAINPID
 KillSignal=SIGINT

ansible/files/pgbouncer_config/pgbouncer.service.j2

@@ -1,5 +1,7 @@
 CREATE USER pgbouncer;
 
+REVOKE ALL PRIVILEGES ON SCHEMA public FROM pgbouncer;
+
 CREATE SCHEMA pgbouncer AUTHORIZATION pgbouncer;
 
 CREATE OR REPLACE FUNCTION pgbouncer.get_auth(p_usename TEXT)

ansible/files/pgbouncer_config/pgbouncer_auth_schema.sql

@@ -1,2 +1,2 @@
 # Directory for PostgreSQL sockets, lockfiles and stats tempfiles
-d /run/pgbouncer 2775 postgres postgres - -
+d /run/pgbouncer 2775 pgbouncer postgres - -

ansible/files/pgbouncer_config/tmpfiles.d-pgbouncer.conf.j2

@@ -84,11 +84,11 @@ local  all  all    peer
 # IPv4 local connections:
 host  all  all  127.0.0.1/32  trust
 # IPv6 local connections:
-host  all  all  ::1/128  md5
+host  all  all  ::1/128  scram-sha-256
 # Local root Unix user, passwordless access
 local  all  postgres    peer map=root_as_postgres
 # IPv4 external connections
-host  all  all  0.0.0.0/0  md5
+host  all  all  0.0.0.0/0  scram-sha-256
 
 # MD5 hashed password hosts
 

ansible/files/postgresql_config/pg_hba.conf.j2

@@ -24,7 +24,8 @@
 # "postgres -c log_connections=on".  Some parameters can be changed at run time
 # with the "SET" SQL command.
 #
-# Memory units:  kB = kilobytes        Time units:  ms  = milliseconds
+# Memory units:  B  = bytes            Time units:  us  = microseconds
+#                kB = kilobytes                     ms  = milliseconds
 #                MB = megabytes                     s   = seconds
 #                GB = gigabytes                     min = minutes
 #                TB = terabytes                     h   = hours
@@ -92,7 +93,7 @@ listen_addresses = '*'		# what IP address(es) to listen on;
 # - Authentication -
 
 authentication_timeout = 1min		# 1s-600s
-password_encryption = md5	# scram-sha-256 or md5
+password_encryption = scram-sha-256	# scram-sha-256 or md5
 db_user_namespace = off
 
 # GSSAPI using Kerberos
@@ -105,6 +106,7 @@ ssl = off
 ssl_ca_file = ''
 ssl_cert_file = ''
 ssl_crl_file = ''
+ssl_crl_dir = ''
 ssl_key_file = ''
 ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
 ssl_prefer_server_ciphers = on
@@ -124,28 +126,28 @@ ssl_passphrase_command_supports_reload = off
 
 shared_buffers = 128MB			# min 128kB
 					# (change requires restart)
-# huge_pages = try			# on, off, or try
+#huge_pages = try			# on, off, or try
 					# (change requires restart)
-# huge_page_size = 0			# zero for system default
+#huge_page_size = 0			# zero for system default
 					# (change requires restart)
-# temp_buffers = 8MB			# min 800kB
-# max_prepared_transactions = 0		# zero disables the feature
+#temp_buffers = 8MB			# min 800kB
+#max_prepared_transactions = 0		# zero disables the feature
 					# (change requires restart)
 # Caution: it is not advisable to set max_prepared_transactions nonzero unless
 # you actively intend to use prepared transactions.
-# work_mem = 4MB				# min 64kB
-# hash_mem_multiplier = 1.0		# 1-1000.0 multiplier on hash table work_mem
-# maintenance_work_mem = 64MB		# min 1MB
-# autovacuum_work_mem = -1		# min 1MB, or -1 to use maintenance_work_mem
-# logical_decoding_work_mem = 64MB	# min 64kB
-# max_stack_depth = 2MB			# min 100kB
-# shared_memory_type = mmap		# the default is the first option
+#work_mem = 4MB				# min 64kB
+#hash_mem_multiplier = 1.0		# 1-1000.0 multiplier on hash table work_mem
+#maintenance_work_mem = 64MB		# min 1MB
+#autovacuum_work_mem = -1		# min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB	# min 64kB
+#max_stack_depth = 2MB			# min 100kB
+#shared_memory_type = mmap		# the default is the first option
 					# supported by the operating system:
 					#   mmap
 					#   sysv
 					#   windows
 					# (change requires restart)
-# dynamic_shared_memory_type = posix	# the default is the first option
+#dynamic_shared_memory_type = posix	# the default is the first option
 					# supported by the operating system:
 					#   posix
 					#   sysv
@@ -765,8 +767,6 @@ jit_provider = 'llvmjit'		# JIT library to use
 
 #exit_on_error = off			# terminate session on any error?
 #restart_after_crash = on		# reinitialize after backend crash?
-#remove_temp_files_after_crash = on	# remove temporary files after
-					# backend crash?
 #data_sync_retry = off			# retry or panic on failure to fsync
 					# data?
 					# (change requires restart)

ansible/files/postgresql_config/postgresql.conf.j2

@@ -1,2 +1,2 @@
 [general]
-postgres_version=13
+postgres_version=14

ansible/files/supabase_facts.ini

@@ -1,26 +1,5 @@
-- name: Preparing Docker container
-  hosts: localhost
-  tasks:
-  - name: Pull Postgres Image
-    docker_container:
-      name: "supabase-postgres-build"
-      image: "postgres:13.3"
-      env:
-        LANGUAGE: "en_US.UTF-8"
-        LANG: "en_US.UTF-8"
-        LC_ALL: "en_US.UTF-8"
-      state: started
-      memory: 4G
-      memory_swap: 6G
-      command: tail -f /dev/null
-  - name: Add Postgres Image to Ansible Hosts
-    add_host:
-      name: "supabase-postgres-build"
-      ansible_connection: docker
-      ansible_ssh_user: root
-
 - name: Build Supabase Postgres
-  hosts: "supabase-postgres-build"
+  hosts: localhost
   gather_facts: false
 
   vars_files:
@@ -34,19 +13,4 @@
       import_tasks: tasks/setup-extensions.yml
 
     - name: Cleanup container
-      import_tasks: tasks/docker/cleanup.yml
-
-- name: Create supabase/postgres docker image
-  hosts: localhost
-  tasks:
-    - name: Commit Docker image
-      command: docker commit --change='CMD ["postgres"]' "supabase-postgres-build" "supabase/postgres"
-
-- name: Clean Up Postgres Image
-  hosts: localhost
-  tasks:
-    - name: Remove Running Base Image
-      docker_container:
-        name: supabase-postgres-build
-        state: absent
-        force_kill: yes
+      import_tasks: tasks/docker/cleanup.yml