Web standard APIs (cross-platform support)

[gregnr]

This is a somewhat major refactor that changes pg-gateway to use only web standard APIs.

Why?

Cross-platform support. Building on web streams and web crypto means that pg-gateway can work on other platforms beyond Node.js like Deno, custom runtimes, and even the browser.

What changed?

There are 5 main changes in the PR:

1. Web streams
2. Async iterables
3. Web crypto
4. Uint8Array
5. Tests 🎉

Web streams

PostgresConnection now accepts a ReadableStream and WritableStream instead of a Node.js Socket. Specifically it accepts a DuplexStream, which is simply an object containing the ReadableStream and WritableStream:

interface DuplexStream<T = unknown> {
  readable: ReadableStream<T>;
  writable: WritableStream<T>;
}

const connection = new PostgresConnection(duplex, {
  ...
});

The design for this duplex interface was not arbitrary - it is a pattern encouraged in the web streams spec, and is used by most duplex-like implementations, like Deno's Conn (TCP socket), TransformStream, etc.

A helper function exists for each platform to simplify converting between their native socket and a DuplexStream.

eg. Node.js Socket:

import { fromNodeSocket } from 'pg-gateway/node';

const connection = await fromNodeSocket(socket, {
  onMessage(data) {
    console.log(data);
  },
});

or Deno's Conn:

import { fromDenoConn } from 'pg-gateway/deno';

const connection = await fromDenoConn(conn, {
  onMessage(data) {
    console.log(data);
  },
});

Note that these helper functions are imported from separate entry points (package exports) so that their native dependencies don't interfere with bundlers if you don't use them.

Worth considering: The above helper functions were actually really simple to make and might not be needed. For example, to convert from a Node.js Socket to DuplexStream is actually as simple as calling:

import { Duplex } from 'node:stream';

const duplex: DuplexStream<Uint8Array> = Duplex.toWeb(socket);

and Deno's Conn actually already implements DuplexStream<Uint8Array>, so no conversion was needed (just wrapped).

We do a bit of extra work though in fromNodeSocket() to support TLS upgrades (more on this below), and other future implementations might not be so simple, so it may be worth keeping these helper functions for consistency.

TLS upgrades

Abstracting Socket into a standard DuplexStream is great, but it makes TLS upgrades tricky since this logic is platform specific. Right now this is solved by moving upgradeTls() to the consumer who is now responsible for TLS upgrades by implementing this method. The method should return a new DuplexStream that operates on top of the new encrypted channel (like a Node.js TLSSocket).

To make this easy for Node.js users, fromNodeSocket() implements this method for you using our previous TLS upgrade logic. Unfortunately Deno does not yet offer APIs to upgrade a TCP connection to TLS from the server side, so TLS is not yet supported there (but likely in the future).

Async iterables

Async iterables are a modern way to work with streams. For example Deno uses them to handle connections on TCP servers and listening for new data within each stream.

It turns out that ReadableStream implements AsyncIterable, so we can now easily use this within our own implementation. Instead of listening for a socket.on('data') event, we now iterate over the readable:

for await (const data of this.duplex.readable) {
  // work with data chunk
}

A nice side effect from this is that reading data is now pull-based instead of push-based. This means we no longer need to worry about pausing and resuming the socket like we did before since new data won't arrive until we ask for it (will get buffered under the hood).

In addition to the above, this PR modifies all downstream logic to use async generators. For example, handleClientMessage() is an async generator that calls yield whenever it wants to write data back to the client (instead of this.sendData()):

async *handleClientMessage(message: Uint8Array) {
  yield createMyResponse();
}

Nested methods can also yield data and will be forwarded by parent functions:

async *handleClientMessage(message: Uint8Array) {
  // yield* is syntactic sugar to forward all yields from the downstream generator
  yield* this.completeAuthentication(); 
}

Web crypto

Another area that needed to change was auth, since it heavily relied on Node's crypto APIs. Most functions we used had a direct 1-to-1 mapping to Web Crypto, with the exception of MD5 which is not implemented (probably for security reasons). To support MD5, we use Deno's @std/crypto library which implements missing algorithms like MD5 using Web Assembly.

Uint8Array

Up until now we've used Node's Buffer to represent binary data, but it's not a web standard. This PR modifies every occurrence of Buffer to use Uint8Array instead. Any missing methods (like concat(), toString('base64')) are implemented using Deno's @std/bytes and @std/encoding standard libraries.

Tests

We finally have some tests 🎉 Given all the refactors, testing was the only way to be sure things don't break. We were also long overdue for creating tests. This PR introduces baseline tests for Node and Deno, with the goal to increase test coverage in subsequent PRs.

[gregnr]

Tests are now set up and automated via GitHub actions (all tests are passing). We use vitest as the testing framework.

[gregnr]

This PR is now fully web API compatible, including the browser. We have browser tests for chromium, firefox, and webkit (via vitest + playwright) that are also automated in CI and passing.

Browser tests use a createDuplexPair() utility that simulates a socket connection via in-memory buffer:

const [clientDuplex, serverDuplex] = createDuplexPair<Uint8Array>();

const db = new PGlite();

new PostgresConnection(serverDuplex, {
  async onStartup() {
    await db.waitReady;
  },
  async onMessage(data, { isAuthenticated }) {
    if (!isAuthenticated) {
      return;
    }
    return await db.execProtocolRaw(data);
  },
});

// read/write `clientDuplex`

In the tests we actually use an in-browser version of pg (node-postgres) and knex to communicate through pg-gateway to the PGlite instance, all in browser:

import pg from '@nodeweb/pg';

const { Client } = pg;
const [clientDuplex, serverDuplex] = createDuplexPair<Uint8Array>();

const db = new PGlite();

new PostgresConnection(serverDuplex, {
  async onStartup() {
    await db.waitReady;
  },
  async onMessage(data, { isAuthenticated }) {
    if (!isAuthenticated) {
      return;
    }
    return await db.execProtocolRaw(data);
  },
});

const client = new Client({
  user: 'postgres',
  stream: socketFromDuplexStream(clientDuplex),
});

await client.connect();

const res = await client.query("select 'Hello world!' as message");
const [{ message }] = res.rows;
expect(message).toBe('Hello world!');
await client.end();

socketFromDuplexStream() is another utility method that simulates a Node Socket by wrapping a DuplexStream. pg allows you to pass a stream option that it will use instead of creating it's own outbound TCP connection.

[gregnr]

TLS logic is confirmed working via ./packages/pg-gateway/test/node/tls.test.ts for Node.js. TLS is not yet supported on Deno or browser. We test:

• TLS upgrades work to spec using server cert/key + self-signed CA
• SNI sends correctly
• Client certs send and validate correctly
• TLS upgrades work over an in-memory duplex pair instead of a TCP socket (but still using Node's APIs for upgrade logic)

[gregnr]

PGlite extended query tests are now passing via fixes in @electric-sql/[email protected]. We can now test & create examples for popular ORMs like Prisma.

[jgoux]

Stellar work @gregnr 👏

Just a couple of questions and little suggestions, but nothing blocking for merging! 🫡