[WIP] Major Upgrading of Supabase Postgres

README.md

@@ -2,23 +2,44 @@
 
 Unmodified Postgres with some useful plugins. Our goal with this repo is not to modify Postgres, but to provide some of the most common extensions with a one-click install.
 
-## Features
-
-- ✅ Postgres [12](https://www.postgresql.org/about/news/1976/). Includes [generated columns](https://www.postgresql.org/docs/12/ddl-generated-columns.html) and [JSON path](https://www.postgresql.org/docs/12/functions-json.html#FUNCTIONS-SQLJSON-PATH) support.
-- ✅ Ubuntu 18.04 (Bionic).
-- ✅ [pg-contrib-12](https://www.postgresql.org/docs/12/contrib.html). Because everyone should enable `pg_stat_statements`.
+## Primary Features
+- ✅ Postgres [13](https://www.postgresql.org/about/news/postgresql-13-released-2077/).
+- ✅ Ubuntu 20.04 (Focal Fossa).
 - ✅ [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) = logical and [max_replication_slots](https://www.postgresql.org/docs/current/runtime-config-replication.html) = 5. Ready for replication.
-- ✅ [PostGIS](https://postgis.net/). Postgres' most popular extension - support for geographic objects.
-- ✅ [pgTAP](https://pgtap.org/). Unit Testing for Postgres.
-- ✅ [pgAudit](https://www.pgaudit.org/). Generate highly compliant audit logs.
-- ✅ [pgjwt](https://github.com/michelp/pgjwt). Generate JSON Web Tokens (JWT) in Postgres.
-- ✅ [pgsql-http](https://github.com/pramsey/pgsql-http). HTTP client for Postgres.
-- ✅ [plpgsql_check](https://github.com/okbob/plpgsql_check). Linter tool for PL/pgSQL.
-- ✅ [plv8](https://github.com/plv8/plv8). Write in Javascript functions in Postgres.
-- ✅ [PL/Java](https://github.com/tada/pljava). Write in Java functions in Postgres.
-- ✅ [pg_cron](https://github.com/citusdata/pg_cron). Run CRON jobs inside Postgres.
-- ✅ [pg-safeupdate](https://github.com/eradman/pg-safeupdate). Protect your data from accidental updates or deletes.
-- ✅ [wal2json](https://github.com/eulerto/wal2json). JSON output plugin for logical replication decoding
+- ✅ [Large Systems Extensions](https://github.com/aws/aws-graviton-getting-started#building-for-graviton-and-graviton2). Enabled for ARM images.
+
+## Extensions 
+| Extension | Description |
+| ------------- | ------------- |
+| [Postgres contrib modules](https://www.postgresql.org/docs/current/contrib.html) | Because everyone should enable `pg_stat_statements`. |
+| [PostGIS](https://postgis.net/) | Postgres' most popular extension - support for geographic objects. |
+| [pgRouting](https://pgrouting.org/) | Extension of PostGIS - provides geospatial routing functionalities. |
+| [pgTAP](https://pgtap.org/) | Unit Testing for Postgres. |
+| [pg_cron](https://github.com/citusdata/pg_cron) | Run CRON jobs inside Postgres. |
+| [pgAudit](https://www.pgaudit.org/) | Generate highly compliant audit logs. |
+| [pgjwt](https://github.com/michelp/pgjwt) | Generate JSON Web Tokens (JWT) in Postgres. |
+| [pgsql-http](https://github.com/pramsey/pgsql-http) | HTTP client for Postgres. |
+| [plpgsql_check](https://github.com/okbob/plpgsql_check) | Linter tool for PL/pgSQL. |
+| [pg-safeupdate](https://github.com/eradman/pg-safeupdate) | Protect your data from accidental updates or deletes. |
+| [wal2json](https://github.com/eulerto/wal2json) | JSON output plugin for logical replication decoding. |
+| [PL/Java](https://github.com/tada/pljava) | Write in Java functions in Postgres. |
+| [plv8](https://github.com/plv8/plv8) | Write in Javascript functions in Postgres. |
+
+Can't find your favorite extension? Suggest for it to be added into future versions [here](https://github.com/supabase/supabase/discussions/679)!
+
+## Enhanced Security
+Aside from having [ufw](https://help.ubuntu.com/community/UFW),[fail2ban](https://www.fail2ban.org/wiki/index.php/Main_Page), and [unattended-upgrades](https://wiki.debian.org/UnattendedUpgrades) installed, we also have the following enhancements in place: 
+| Enhancement | Description |
+| ------------- | ------------- |
+| fail2ban filter for PostgreSQL access | Monitors for brute force attempts over at port `5432`. |
+| fail2ban filter for PgBouncer access | Monitors for brute force attempts over at port `6543`. |
+
+## Additional Goodies
+| Goodie | Description |
+| ------------- | ------------- |
+| [PgBouncer](https://postgis.net/) | Set up Connection Pooling. |
+| [PostgREST](https://postgrest.org/en/stable/) | Instantly transform your database into an RESTful API. |
+| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | Tool for physical database backup and recovery. |
 
 ## Install
 
@@ -28,19 +49,29 @@ See all installation instructions in the [repo wiki](https://github.com/supabase
 [![Digital Ocean](https://github.com/supabase/postgres/blob/master/docs/img/digital-ocean.png)](https://github.com/supabase/postgres/wiki/Digital-Ocean)
 [![AWS](https://github.com/supabase/postgres/blob/master/docs/img/aws.png)](https://github.com/supabase/postgres/wiki/AWS-EC2)
 
-### Building
-
-Install the anxs.postgresql role
+### Marketplace Images
+|   | Postgres & Extensions | PgBouncer | PostgREST | WAL-G |
+|---|:---:|:---:|:---:|:---:|
+| Supabase Postgres |  ✔️   | ❌    | ❌   |  ✔️   |
+| Supabase Postgres: PgBouncer Bundle  |  ✔️   |  ✔️  | ❌    |   ✔️ |
+| Supabase Postgres: PostgREST Bundle |  ✔️   |  ❌  |  ✔️   |   ✔️ |
+| Supabase Postgres: Complete Bundle |  ✔️  |  ✔️   | ✔️   | ✔️   |
 
-```bash
-ansible-galaxy install anxs.postgresql -r ansible/install_roles.yml --force -vvv
-```
+#### Availability
+|   | AWS ARM | AWS x86 | Digital Ocean x86 |
+|---|:---:|:---:|:---:|
+| Supabase Postgres | Coming Soon | Coming Soon | Coming Soon |
+| Supabase Postgres: PgBouncer Bundle  | Coming Soon | Coming Soon | Coming Soon |
+| Supabase Postgres: PostgREST Bundle | Coming Soon | Coming Soon | Coming Soon |
+| Supabase Postgres: Complete Bundle | Coming Soon | Coming Soon | Coming Soon |
 
-Set the `supabase_internal` flag to `false` to avoid baking in components that are specific to Supabase's hosted offering.
+### Quick Build
 
 ```bash
 $ time packer build -timestamp-ui \
-  -var "ansible_arguments=--skip-tags,update-only,-v,-e,supabase_internal='false'" \
+  --var "aws_access_key=<insert aws access key>" \
+  --var "aws_secret_key=<insert aws secret key>" \
+  --var "ami_regions=<insert desired regions>" \
   amazon-arm.json
 ```
 

amazon-arm.json

@@ -4,10 +4,10 @@
     "aws_secret_key": "",
     "region": "ap-northeast-1",
     "ami_regions": "eu-central-1,eu-west-1,eu-west-2,ap-south-1,ap-southeast-1,ap-southeast-2,us-west-1,us-east-1,ca-central-1,sa-east-1,ap-northeast-1",
-    "ami": "ami-0d24b4f369844fc54",
-    "ami_name": "supabase-postgres-0.16.2",
+    "ami": "ami-076d8ebdd0e1ec091",
+    "ami_name": "supabase-postgres-13.3.0",
     "environment": "prod",
-    "ansible_arguments": "--skip-tags,update-only -v"
+    "ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
   },
   "builders": [
     {
@@ -17,7 +17,7 @@
       "region": "{{user `region`}}",
       "ami_regions": "{{user `ami_regions`}}",
       "source_ami": "{{user `ami`}}",
-      "instance_type": "c6g.2xlarge",
+      "instance_type": "r6g.2xlarge",
       "ssh_username": "ubuntu",
       "ami_name": "{{user `ami_name`}}",
       "tags": {
@@ -51,7 +51,6 @@
       "execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
       "type": "shell",
       "scripts": [
-        "scripts/01-test",
         "scripts/02-credentials_cleanup.sh",
         "scripts/90-cleanup.sh",
         "scripts/91-log_cleanup.sh"

amazon.json

@@ -4,10 +4,10 @@
     "aws_secret_key": "",
     "region": "af-south-1",
     "ami_regions": "af-south-1",
-    "ami": "ami-07d30d5bf2d29a582",
-    "ami_name": "supabase-postgres-0.16.2",
+    "ami": "ami-08a4b40f2fe1e4b35",
+    "ami_name": "supabase-postgres-13.3.0.4",
     "environment": "prod",
-    "ansible_arguments": "--skip-tags,update-only"
+    "ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
   },
   "builders": [
     {

ansible/files/docker_mnt/init.sh

@@ -0,0 +1,3 @@
+cat /etc/postgresql/postgresql.conf > $PGDATA/postgresql.conf
+echo "host replication $POSTGRES_USER 0.0.0.0/0 trust" >> $PGDATA/pg_hba.conf
+echo "host  all  all  127.0.0.1/32  trust" >> $PGDATA/pg_hba.conf

ansible/files/fail2ban_config/filter-pgbouncer.conf.j2

@@ -0,0 +1,2 @@
+[Definition]
+failregex = ^.+@<HOST>:.+error: password authentication failed$

ansible/files/fail2ban_config/filter-postgresql.conf.j2

@@ -0,0 +1,3 @@
+[Definition]
+failregex = ^.*,.*,.*,.*,"<HOST>:.*password authentication failed for user.*$
+ignoreregex = ^.*,.*,.*,.*,"127\.0\.0\.1.*password authentication failed for user.*$

ansible/files/fail2ban_config/jail-pgbouncer.conf.j2

@@ -0,0 +1,7 @@
+[pgbouncer]
+enabled = true
+port    = 6543
+protocol = tcp
+filter = pgbouncer
+logpath = /var/log/pgbouncer.log
+maxretry = 3

ansible/files/fail2ban_config/jail-postgresql.conf.j2

@@ -0,0 +1,7 @@
+[postgresql]
+enabled = true
+port    = 5432
+protocol = tcp
+filter = postgresql
+logpath = /var/lib/postgresql/data/pg_log/postgresql.csv
+maxretry = 3

ansible/files/logrotate-postgres

@@ -1,10 +1,9 @@
-/var/lib/postgresql/12/main/pg_log/postgresql.csv {
-    daily
-    size 50M 
+/var/lib/postgresql/data/pg_log/postgresql.log {
+    size 50M
     rotate 3  
     copytruncate
     delaycompress
     compress
     notifempty
     missingok
-}
+}

ansible/files/pgbouncer.ini → ansible/files/pgbouncer_config/pgbouncer.ini.j2

@@ -43,8 +43,8 @@
 ;;; Administrative settings
 ;;;
 
-logfile = /var/log/postgresql/pgbouncer.log
-pidfile = /var/run/postgresql/pgbouncer.pid
+logfile = /var/log/pgbouncer.log
+pidfile = /var/run/pgbouncer/pgbouncer.pid
 
 ;;;
 ;;; Where to wait for clients
@@ -56,10 +56,9 @@ listen_port = 6543
 
 ;; Unix socket is also used for -R.
 ;; On Debian it should be /var/run/postgresql
-;unix_socket_dir = /tmp
+unix_socket_dir = /tmp
 ;unix_socket_mode = 0777
 ;unix_socket_group =
-unix_socket_dir = /var/run/postgresql
 
 ;;;
 ;;; TLS settings for accepting clients

ansible/files/pgbouncer_config/pgbouncer.service.j2

@@ -0,0 +1,40 @@
+# Example systemd service unit for PgBouncer
+#
+# - Adjust the paths in ExecStart for your installation.
+#
+# - The User setting requires careful consideration.  PgBouncer needs
+#   to be able to place a Unix-domain socket file where PostgreSQL
+#   clients will look for it.  In the olden days, this was in /tmp,
+#   but systems using systemd now prefer something like
+#   /var/run/postgresql/.  But then some systems also lock down that
+#   directory so that only the postgres user can write to it.  That
+#   means you need to either
+#
+#   - run PgBouncer as the postgres user, or
+#
+#   - create a separate user and add it to the postgres group and
+#     make /var/run/postgresql/ group-writable, or
+#
+#   - use systemd to create the sockets; see pgbouncer.socket nearby.
+#
+#   For packagers and deployment systems, this requires some
+#   coordination between the PgBouncer and the PostgreSQL
+#   packages/components.
+#
+[Unit]
+Description=connection pooler for PostgreSQL
+Documentation=man:pgbouncer(1)
+Documentation=https://www.pgbouncer.org/
+After=network.target
+#Requires=pgbouncer.socket
+
+[Service]
+Type=notify
+User=postgres
+ExecStart=/usr/local/bin/pgbouncer /etc/pgbouncer/pgbouncer.ini
+ExecReload=/bin/kill -HUP $MAINPID
+KillSignal=SIGINT
+#LimitNOFILE=1024
+
+[Install]
+WantedBy=multi-user.target

ansible/files/pgbouncer_config/tmpfiles.d-pgbouncer.conf.j2

@@ -0,0 +1,2 @@
+# Directory for PostgreSQL sockets, lockfiles and stats tempfiles
+d /run/pgbouncer 2775 postgres postgres - -