Skip to content

Commit 3d5ec92

Browse files
dragarciadragarcia
committed
Merge branch 'develop' of github.com:supabase/postgres
2 parents 49f2e7e + a6545cd commit 3d5ec92

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

71 files changed

+3542
-711
lines changed

‎.DS_Store

-6 KB
Binary file not shown.

‎.github/workflows/provision.yml

-59
This file was deleted.

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.DS_Store
2+
.python-version
3+
venv/

‎README.md

+71-16
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,44 @@
22

33
Unmodified Postgres with some useful plugins. Our goal with this repo is not to modify Postgres, but to provide some of the most common extensions with a one-click install.
44

5-
## Features
6-
7-
- ✅ Postgres [12](https://www.postgresql.org/about/news/1976/). Includes [generated columns](https://www.postgresql.org/docs/12/ddl-generated-columns.html) and [JSON path](https://www.postgresql.org/docs/12/functions-json.html#FUNCTIONS-SQLJSON-PATH) support.
8-
- ✅ Ubuntu 18.04 (Bionic).
9-
- ✅ [pg-contrib-12](https://www.postgresql.org/docs/12/contrib.html). Because everyone should enable `pg_stat_statements`.
5+
## Primary Features
6+
- ✅ Postgres [13](https://www.postgresql.org/about/news/postgresql-13-released-2077/).
7+
- ✅ Ubuntu 20.04 (Focal Fossa).
108
- ✅ [wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html) = logical and [max_replication_slots](https://www.postgresql.org/docs/current/runtime-config-replication.html) = 5. Ready for replication.
11-
- ✅ [PostGIS](https://postgis.net/). Postgres' most popular extension - support for geographic objects.
12-
- ✅ [pgTAP](https://pgtap.org/). Unit Testing for Postgres.
13-
- ✅ [pgAudit](https://www.pgaudit.org/). Generate highly compliant audit logs.
14-
- ✅ [pgjwt](https://github.com/michelp/pgjwt). Generate JSON Web Tokens (JWT) in Postgres.
15-
- ✅ [pgsql-http](https://github.com/pramsey/pgsql-http). HTTP client for Postgres.
16-
- ✅ [plpgsql_check](https://github.com/okbob/plpgsql_check). Linter tool for PL/pgSQL.
17-
- ✅ [plv8](https://github.com/plv8/plv8). Write in Javascript functions in Postgres.
18-
- ✅ [plpython3u](https://www.postgresql.org/docs/current/plpython-python23.html). Python3 enabled by default. Write in Python functions in Postgres.
19-
- ✅ [PL/Java](https://github.com/tada/pljaval). Write in Java functions in Postgres.
9+
- ✅ [Large Systems Extensions](https://github.com/aws/aws-graviton-getting-started#building-for-graviton-and-graviton2). Enabled for ARM images.
10+
11+
## Extensions
12+
| Extension | Description |
13+
| ------------- | ------------- |
14+
| [Postgres contrib modules](https://www.postgresql.org/docs/current/contrib.html) | Because everyone should enable `pg_stat_statements`. |
15+
| [PostGIS](https://postgis.net/) | Postgres' most popular extension - support for geographic objects. |
16+
| [pgRouting](https://pgrouting.org/) | Extension of PostGIS - provides geospatial routing functionalities. |
17+
| [pgTAP](https://pgtap.org/) | Unit Testing for Postgres. |
18+
| [pg_cron](https://github.com/citusdata/pg_cron) | Run CRON jobs inside Postgres. |
19+
| [pgAudit](https://www.pgaudit.org/) | Generate highly compliant audit logs. |
20+
| [pgjwt](https://github.com/michelp/pgjwt) | Generate JSON Web Tokens (JWT) in Postgres. |
21+
| [pgsql-http](https://github.com/pramsey/pgsql-http) | HTTP client for Postgres. |
22+
| [plpgsql_check](https://github.com/okbob/plpgsql_check) | Linter tool for PL/pgSQL. |
23+
| [pg-safeupdate](https://github.com/eradman/pg-safeupdate) | Protect your data from accidental updates or deletes. |
24+
| [wal2json](https://github.com/eulerto/wal2json) | JSON output plugin for logical replication decoding. |
25+
| [PL/Java](https://github.com/tada/pljava) | Write in Java functions in Postgres. |
26+
| [plv8](https://github.com/plv8/plv8) | Write in Javascript functions in Postgres. |
27+
28+
Can't find your favorite extension? Suggest for it to be added into future versions [here](https://github.com/supabase/supabase/discussions/679)!
29+
30+
## Enhanced Security
31+
Aside from having [ufw](https://help.ubuntu.com/community/UFW),[fail2ban](https://www.fail2ban.org/wiki/index.php/Main_Page), and [unattended-upgrades](https://wiki.debian.org/UnattendedUpgrades) installed, we also have the following enhancements in place:
32+
| Enhancement | Description |
33+
| ------------- | ------------- |
34+
| [fail2ban filter](https://github.com/supabase/postgres/blob/develop/ansible/files/fail2ban_config/filter-postgresql.conf.j2) for PostgreSQL access | Monitors for brute force attempts over at port `5432`. |
35+
| [fail2ban filter](https://github.com/supabase/postgres/blob/develop/ansible/files/fail2ban_config/filter-pgbouncer.conf.j2) for PgBouncer access | Monitors for brute force attempts over at port `6543`. |
36+
37+
## Additional Goodies
38+
| Goodie | Description |
39+
| ------------- | ------------- |
40+
| [PgBouncer](https://postgis.net/) | Set up Connection Pooling. |
41+
| [PostgREST](https://postgrest.org/en/stable/) | Instantly transform your database into an RESTful API. |
42+
| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | Tool for physical database backup and recovery. |
2043

2144
## Install
2245

@@ -26,13 +49,39 @@ See all installation instructions in the [repo wiki](https://github.com/supabase
2649
[![Digital Ocean](https://github.com/supabase/postgres/blob/master/docs/img/digital-ocean.png)](https://github.com/supabase/postgres/wiki/Digital-Ocean)
2750
[![AWS](https://github.com/supabase/postgres/blob/master/docs/img/aws.png)](https://github.com/supabase/postgres/wiki/AWS-EC2)
2851

52+
### Marketplace Images
53+
| | Postgres & Extensions | PgBouncer | PostgREST | WAL-G |
54+
|---|:---:|:---:|:---:|:---:|
55+
| Supabase Postgres | ✔️ | ❌ | ❌ | ✔️ |
56+
| Supabase Postgres: PgBouncer Bundle | ✔️ | ✔️ | ❌ | ✔️ |
57+
| Supabase Postgres: PostgREST Bundle | ✔️ | ❌ | ✔️ | ✔️ |
58+
| Supabase Postgres: Complete Bundle | ✔️ | ✔️ | ✔️ | ✔️ |
59+
60+
#### Availability
61+
| | AWS ARM | AWS x86 | Digital Ocean x86 |
62+
|---|:---:|:---:|:---:|
63+
| Supabase Postgres | Coming Soon | Coming Soon | Coming Soon |
64+
| Supabase Postgres: PgBouncer Bundle | Coming Soon | Coming Soon | Coming Soon |
65+
| Supabase Postgres: PostgREST Bundle | Coming Soon | Coming Soon | Coming Soon |
66+
| Supabase Postgres: Complete Bundle | Coming Soon | Coming Soon | Coming Soon |
67+
68+
### Quick Build
69+
70+
```bash
71+
$ time packer build -timestamp-ui \
72+
--var "aws_access_key=<insert aws access key>" \
73+
--var "aws_secret_key=<insert aws secret key>" \
74+
--var "ami_regions=<insert desired regions>" \
75+
amazon-arm.json
76+
```
77+
2978
## Motivation
3079

31-
After talking to a lot of techies, we've found that most believe Postgres is the best (operational) database but they *still* choose other databases. This is overwhelmingly because "the other one was quicker/easier". Our goal is to make it fast and simple to get started with Postgres, so that we never hear that excuse again.
80+
After talking to a lot of techies, we've found that most believe Postgres is the best (operational) database but they _still_ choose other databases. This is overwhelmingly because "the other one was quicker/easier". Our goal is to make it fast and simple to get started with Postgres, so that we never hear that excuse again.
3281

3382
Our secondary goal is to show off a few of Postgres' most exciting features. This is to convince new developers to choose it over other database (a decision we hope they'll appreciate once they start scaling).
3483

35-
Finally, this is the same build we offer at [Supabase](https://supabase.io), and everything we do is opensource. This repo makes it easy to *install* Postgres, Supabase makes it easy to *use* Postgres.
84+
Finally, this is the same build we offer at [Supabase](https://supabase.io), and everything we do is opensource. This repo makes it easy to _install_ Postgres, Supabase makes it easy to _use_ Postgres.
3685

3786
## Roadmap
3887

@@ -43,3 +92,9 @@ Finally, this is the same build we offer at [Supabase](https://supabase.io), and
4392
## License
4493

4594
[The PostgreSQL License](https://opensource.org/licenses/postgresql). We realize that licensing is tricky since we are bundling all the various plugins. If we have infringed on any license, let us know and we will make the necessary changes (or remove that extension from this repo).
95+
96+
## Sponsors
97+
98+
We are building the features of Firebase using enterprise-grade, open source products. We support existing communities wherever possible, and if the products don’t exist we build them and open source them ourselves.
99+
100+
[![New Sponsor](https://user-images.githubusercontent.com/10214025/90518111-e74bbb00-e198-11ea-8f88-c9e3c1aa4b5b.png)](https://github.com/sponsors/supabase)

‎amazon-arm.json

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
{
2+
"variables": {
3+
"aws_access_key": "",
4+
"aws_secret_key": "",
5+
"region": "ap-northeast-1",
6+
"ami_regions": "eu-central-1,eu-west-1,eu-west-2,ap-south-1,ap-southeast-1,ap-southeast-2,us-west-1,us-east-1,ca-central-1,sa-east-1,ap-northeast-1",
7+
"ami": "ami-076d8ebdd0e1ec091",
8+
"ami_name": "supabase-postgres-13.3.0",
9+
"environment": "prod",
10+
"ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
11+
},
12+
"builders": [
13+
{
14+
"type": "amazon-ebs",
15+
"access_key": "{{user `aws_access_key`}}",
16+
"secret_key": "{{user `aws_secret_key`}}",
17+
"region": "{{user `region`}}",
18+
"ami_regions": "{{user `ami_regions`}}",
19+
"source_ami": "{{user `ami`}}",
20+
"instance_type": "r6g.2xlarge",
21+
"ssh_username": "ubuntu",
22+
"ami_name": "{{user `ami_name`}}",
23+
"tags": {
24+
"environment": "{{user `environment`}}",
25+
"appType": "postgres"
26+
},
27+
"launch_block_device_mappings": [
28+
{
29+
"device_name": "/dev/sda1",
30+
"volume_size": 16,
31+
"volume_type": "gp2",
32+
"delete_on_termination": true
33+
}
34+
]
35+
}
36+
],
37+
"provisioners": [
38+
{
39+
"type": "shell",
40+
"inline": [
41+
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
42+
]
43+
},
44+
{
45+
"type": "ansible",
46+
"user": "ubuntu",
47+
"playbook_file": "ansible/playbook.yml",
48+
"extra_arguments": "{{user `ansible_arguments`}}"
49+
},
50+
{
51+
"execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
52+
"type": "shell",
53+
"scripts": [
54+
"scripts/02-credentials_cleanup.sh",
55+
"scripts/90-cleanup.sh",
56+
"scripts/91-log_cleanup.sh"
57+
]
58+
}
59+
]
60+
}

‎amazon.json

+38-26
Original file line numberDiff line numberDiff line change
@@ -2,31 +2,45 @@
22
"variables": {
33
"aws_access_key": "",
44
"aws_secret_key": "",
5-
"region": "ap-southeast-1",
6-
"ami_regions": "ap-southeast-1",
7-
"ami": "ami-0f7719e8b7ba25c61",
8-
"ansible_arguments": "--skip-tags,update-only"
5+
"region": "af-south-1",
6+
"ami_regions": "af-south-1",
7+
"ami": "ami-08a4b40f2fe1e4b35",
8+
"ami_name": "supabase-postgres-13.3.0.4",
9+
"environment": "prod",
10+
"ansible_arguments": "--skip-tags,update-only,--skip-tags,install-postgrest,--skip-tags,install-pgbouncer,--skip-tags,install-supabase-internal"
911
},
10-
"builders": [{
11-
"type": "amazon-ebs",
12-
"access_key": "{{user `aws_access_key`}}",
13-
"secret_key": "{{user `aws_secret_key`}}",
14-
"region": "{{user `region`}}",
15-
"ami_regions": "{{user `ami_regions`}}",
16-
"source_ami": "{{user `ami`}}",
17-
"instance_type": "m5.2xlarge",
18-
"ssh_username": "ubuntu",
19-
"ami_name": "supabase-postgres-0.13.0",
20-
"launch_block_device_mappings": [
21-
{
22-
"device_name": "/dev/sda1",
23-
"volume_size": 16,
24-
"volume_type": "gp2",
25-
"delete_on_termination": true
26-
}
27-
]
28-
}],
12+
"builders": [
13+
{
14+
"type": "amazon-ebs",
15+
"access_key": "{{user `aws_access_key`}}",
16+
"secret_key": "{{user `aws_secret_key`}}",
17+
"region": "{{user `region`}}",
18+
"ami_regions": "{{user `ami_regions`}}",
19+
"source_ami": "{{user `ami`}}",
20+
"instance_type": "m5.2xlarge",
21+
"ssh_username": "ubuntu",
22+
"ami_name": "{{user `ami_name`}}",
23+
"tags": {
24+
"environment": "{{user `environment`}}",
25+
"appType": "postgres"
26+
},
27+
"launch_block_device_mappings": [
28+
{
29+
"device_name": "/dev/sda1",
30+
"volume_size": 16,
31+
"volume_type": "gp2",
32+
"delete_on_termination": true
33+
}
34+
]
35+
}
36+
],
2937
"provisioners": [
38+
{
39+
"type": "shell",
40+
"inline": [
41+
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
42+
]
43+
},
3044
{
3145
"type": "ansible",
3246
"user": "ubuntu",
@@ -37,11 +51,9 @@
3751
"execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
3852
"type": "shell",
3953
"scripts": [
40-
"scripts/01-test",
4154
"scripts/02-credentials_cleanup.sh",
4255
"scripts/90-cleanup.sh",
43-
"scripts/91-log_cleanup.sh",
44-
"scripts/99-img_check.sh"
56+
"scripts/91-log_cleanup.sh"
4557
]
4658
}
4759
]

‎ansible/files/docker_mnt/init.sh

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
cat /etc/postgresql/postgresql.conf > $PGDATA/postgresql.conf
2+
echo "host replication $POSTGRES_USER 0.0.0.0/0 trust" >> $PGDATA/pg_hba.conf
3+
echo "host all all 127.0.0.1/32 trust" >> $PGDATA/pg_hba.conf

‎ansible/files/fail2ban_config/filter-pgbouncer.conf.j2

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[Definition]
2+
failregex = ^.+@<HOST>:.+error: password authentication failed$

‎ansible/files/fail2ban_config/filter-postgresql.conf.j2

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[Definition]
2+
failregex = ^.*,.*,.*,.*,"<HOST>:.*password authentication failed for user.*$
3+
ignoreregex = ^.*,.*,.*,.*,"127\.0\.0\.1.*password authentication failed for user.*$

‎ansible/files/fail2ban_config/jail-pgbouncer.conf.j2

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[pgbouncer]
2+
enabled = true
3+
port = 6543
4+
protocol = tcp
5+
filter = pgbouncer
6+
logpath = /var/log/pgbouncer.log
7+
maxretry = 3

‎ansible/files/fail2ban_config/jail-postgresql.conf.j2

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[postgresql]
2+
enabled = true
3+
port = 5432
4+
protocol = tcp
5+
filter = postgresql
6+
logpath = /var/lib/postgresql/data/pg_log/postgresql.csv
7+
maxretry = 3

‎ansible/files/logrotate-postgres

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
/var/lib/postgresql/data/pg_log/postgresql.log {
2+
size 50M
3+
rotate 3
4+
copytruncate
5+
delaycompress
6+
compress
7+
notifempty
8+
missingok
9+
}

‎ansible/files/node_exporter.service.j2

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
[Unit]
2+
Description=Node Exporter
3+
After=network-online.target
4+
5+
[Service]
6+
Type=simple
7+
ExecStart=/opt/node_exporter/node_exporter --web.disable-exporter-metrics --collector.disable-defaults {% for collector in collectors %} --collector.{{ collector }} {% endfor %}
8+
9+
User=root
10+
StandardOutput=file:/var/log/node_exporter.stdout
11+
StandardError=file:/var/log/node_exporter.error
12+
Restart=on-failure
13+
RestartSec=3
14+
15+
[Install]
16+
WantedBy=multi-user.target

0 commit comments

Comments
 (0)