add: use SO_REUSEPORT on platform supporting it

Author: mkleczek

CHANGELOG.md

@@ -15,6 +15,7 @@ All notable changes to this project will be documented in this file. From versio
 - Add config `db-timezone-enabled` for optional querying of timezones by @taimoorzaeem in #4751
 - Log schema cache queries timings on `log-level=debug` by @steve-chavez in #4805
 - Add GHC runtime metrics to the metrics endpoint by @mkleczek in #4862
+- Enable starting multiple PostgREST instances using the same ports on platforms supporting it by @mkleczek in #4703 #4694
 
 ### Fixed
 

docs/how-tos/zero-downtime-upgrades.rst

@@ -0,0 +1,164 @@
+.. _zero_downtime_upgrades:
+
+Zero-Downtime Upgrades
+======================
+
+When :ref:`server-reuseport` is enabled on an operating system that supports
+``SO_REUSEPORT``, PostgREST can start more than one process on the same
+:ref:`server-host` and :ref:`server-port`. This allows a new PostgREST process
+to start and become ready before the old process is stopped.
+
+While both processes are running, the operating system distributes new
+connections between them. After the old process exits, the new process receives
+all new connections.
+
+This is useful for upgrades and restarts:
+
+1. Keep the old PostgREST process serving requests.
+2. Start the new PostgREST process on the same host and port.
+3. Wait for the new process to report ``/ready``.
+4. Stop the old process.
+
+Configuration
+-------------
+
+Both processes should use the same public host and port:
+
+.. code-block:: ini
+
+  # /etc/postgrest/postgrest.conf
+  server-host = "127.0.0.1"
+  server-port = 3000
+  server-reuseport = true
+
+  admin-server-host = "127.0.0.1"
+  admin-server-port = 3001
+
+The second process can use the same configuration file and override only the
+admin server port:
+
+.. code-block:: bash
+
+  PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf
+
+.. important::
+
+  Use a different :ref:`admin-server-port` for each PostgREST process during
+  the handover. Admin ports are not shared between processes. This keeps
+  readiness checks unambiguous: ``/ready`` on the new admin port can only be
+  answered by the new process.
+
+Before using this in production, keep these details in mind:
+
+- This works for host and port based servers. It does not apply when
+  :ref:`server-unix-socket` is used.
+- If :ref:`server-reuseport` is disabled, the new process will fail to start
+  with an address-in-use error and the old process will keep serving requests.
+- If :ref:`server-reuseport` is enabled on an operating system that does not
+  support ``SO_REUSEPORT``, PostgREST will fail to start because the
+  configuration is not supported on that platform.
+- If the new process uses the same :ref:`admin-server-port` as the old process,
+  it will fail to start because that admin port is already in use.
+- Each PostgREST process has its own :ref:`db-pool`. During the handover, the
+  total possible database connections can temporarily double.
+- The old and new processes may both serve requests for a short time. Database
+  migrations should be compatible with both versions while they overlap.
+
+Manual Handover
+---------------
+
+Assuming the old process is already serving on ``127.0.0.1:3000`` and its PID
+is stored in ``OLD_PID``:
+
+.. code-block:: bash
+
+  PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf &
+  NEW_PID=$!
+
+  curl --fail http://127.0.0.1:3002/ready
+
+  kill -TERM "$OLD_PID"
+
+The ``curl`` request checks the new process through its own admin server port.
+If the new process cannot load its configuration, connect to the database, or
+load the schema cache, ``/ready`` will not return a successful response and the
+old process can keep serving traffic.
+
+Example Script
+--------------
+
+The following script shows the full sequence for a setup that stores the old
+process PID in a PID file. Adapt the start and stop commands to your process
+manager.
+
+.. code-block:: bash
+
+  #!/usr/bin/env bash
+  set -euo pipefail
+
+  POSTGREST=${POSTGREST:-postgrest}
+  CONFIG=${CONFIG:-/etc/postgrest/postgrest.conf}
+  PID_FILE=${PID_FILE:-/run/postgrest.pid}
+
+  ADMIN_HOST=${ADMIN_HOST:-127.0.0.1}
+  NEW_ADMIN_PORT=${NEW_ADMIN_PORT:-3002}
+  READY_TIMEOUT=${READY_TIMEOUT:-30}
+  STOP_TIMEOUT=${STOP_TIMEOUT:-30}
+
+  if [[ ! -s "$PID_FILE" ]]; then
+    echo "PID file not found or empty: $PID_FILE" >&2
+    exit 1
+  fi
+
+  OLD_PID=$(<"$PID_FILE")
+
+  if ! kill -0 "$OLD_PID" 2>/dev/null; then
+    echo "Old PostgREST process is not running: $OLD_PID" >&2
+    exit 1
+  fi
+
+  PGRST_ADMIN_SERVER_HOST="$ADMIN_HOST" \
+  PGRST_ADMIN_SERVER_PORT="$NEW_ADMIN_PORT" \
+    "$POSTGREST" "$CONFIG" &
+  NEW_PID=$!
+
+  cleanup_new_process() {
+    kill "$NEW_PID" 2>/dev/null || true
+  }
+  trap cleanup_new_process EXIT INT TERM
+
+  READY_URL="http://$ADMIN_HOST:$NEW_ADMIN_PORT/ready"
+  READY_DEADLINE=$((SECONDS + READY_TIMEOUT))
+
+  until curl --fail --silent --show-error --output /dev/null "$READY_URL"; do
+    if ! kill -0 "$NEW_PID" 2>/dev/null; then
+      echo "New PostgREST process exited before it became ready" >&2
+      exit 1
+    fi
+
+    if (( SECONDS >= READY_DEADLINE )); then
+      echo "New PostgREST process did not become ready at $READY_URL" >&2
+      exit 1
+    fi
+
+    sleep 1
+  done
+
+  printf '%s\n' "$NEW_PID" > "$PID_FILE"
+
+  kill -TERM "$OLD_PID" 2>/dev/null || true
+
+  STOP_DEADLINE=$((SECONDS + STOP_TIMEOUT))
+
+  while kill -0 "$OLD_PID" 2>/dev/null; do
+    if (( SECONDS >= STOP_DEADLINE )); then
+      echo "Old PostgREST process did not stop after SIGTERM; sending SIGKILL" >&2
+      kill -KILL "$OLD_PID"
+      break
+    fi
+
+    sleep 1
+  done
+
+  trap - EXIT INT TERM
+  echo "PostgREST handover complete: $OLD_PID -> $NEW_PID"

docs/postgrest.dict

@@ -34,6 +34,7 @@ DSL
 DevOps
 Dramatiq
 dockerize
+downtime
 enum
 Enums
 Entra
@@ -59,6 +60,7 @@ HMAC
 htmx
 Htmx
 Homebrew
+handover
 hstore
 HTTP
 HTTPS
@@ -113,6 +115,7 @@ ov
 parametrized
 passphrase
 PBKDF
+PID
 PgBouncer
 pgcrypto
 pgjwt
@@ -144,6 +147,7 @@ Redux
 refactor
 reloadable
 Reloadable
+reuseport
 requester's
 RESTful
 RLS

docs/references/admin_server.rst

@@ -16,9 +16,15 @@ Two endpoints ``live`` and ``ready`` will then be available. Both these endpoint
 
 .. important::
 
-  If you have a machine with multiple network interfaces and multiple PostgREST instances in the same port, you need to specify a unique :ref:`hostname <server-host>`
-  in the configuration of each PostgREST instance for the health check to work correctly. Don't use the special values(``!4``, ``*``, etc) in this case because the health check
-  could report a false positive.
+  Multiple PostgREST instances can share the same public API host and port when
+  :ref:`server-reuseport` is enabled on operating systems that support
+  ``SO_REUSEPORT``. Admin ports are not shared: give each instance a different
+  :ref:`admin-server-port`, otherwise the new instance will fail to start.
+
+  If the machine has multiple network interfaces, configure concrete
+  :ref:`server-host` and :ref:`admin-server-host` values when you need health
+  checks to target a specific process. Avoid special values (``!4``, ``*``, etc)
+  in this case because the health check could report a false positive.
 
 Live
 ----

docs/references/configuration.rst

@@ -176,6 +176,11 @@ admin-server-port
 
   Specifies the port for the :ref:`admin_server`. Cannot be equal to :ref:`server-port`.
 
+  When running multiple PostgREST instances on the same :ref:`server-port`, use
+  a different ``admin-server-port`` for each instance. Admin ports are not shared
+  between instances, so readiness checks always target one specific PostgREST
+  instance. See :ref:`zero_downtime_upgrades`.
+
 .. _app.settings.*:
 
 app.settings.*
@@ -899,6 +904,50 @@ server-port
 
   The TCP port to bind the web server. Use ``0`` to automatically assign a port.
 
+  When :ref:`server-reuseport` is enabled on an operating system that supports
+  ``SO_REUSEPORT``, you can start multiple PostgREST instances on the same
+  :ref:`server-host` and ``server-port``. For example, two PostgREST processes
+  can use the same configuration:
+
+  .. code:: ini
+
+    server-host = "127.0.0.1"
+    server-port = 3000
+    server-reuseport = true
+
+  New connections are then distributed by the operating system between the
+  running PostgREST processes. This can be used to start a replacement process
+  before stopping the old one, or to run several PostgREST processes behind one
+  port.
+
+  If ``server-reuseport`` is disabled, starting another PostgREST process on
+  the same host and port will fail with the usual address-in-use error.
+
+  For a step-by-step example, see :ref:`zero_downtime_upgrades`.
+
+.. _server-reuseport:
+
+server-reuseport
+----------------
+
+  =============== =================================
+  **Type**        Bool
+  **Default**     false
+  **Reloadable**  N
+  **Environment** PGRST_SERVER_REUSEPORT
+  **In-Database** `n/a`
+  =============== =================================
+
+  Enables ``SO_REUSEPORT`` on the TCP server socket. This allows multiple
+  PostgREST processes to bind to the same :ref:`server-host` and
+  :ref:`server-port` when the operating system supports it.
+
+  Enabling this setting on an operating system that does not support
+  ``SO_REUSEPORT`` is a configuration error. PostgREST will fail to start
+  instead of falling back to a normal TCP socket.
+
+  This setting does not apply when :ref:`server-unix-socket` is used.
+
 .. _server-trace-header:
 
 server-trace-header

src/PostgREST/App.hs

@@ -65,7 +65,9 @@ import PostgREST.Version              (docsVersion, prettyVersion)
 
 import qualified Data.ByteString.Char8     as BS
 import qualified Data.List                 as L
-import           Data.Streaming.Network    (bindPortTCP)
+import           Data.Streaming.Network    (HostPreference,
+                                            bindPortGenEx,
+                                            bindPortTCP)
 import qualified Data.Text                 as T
 import qualified Network.HTTP.Types        as HTTP
 import qualified Network.HTTP.Types.Header as HTTP
@@ -265,11 +267,18 @@ initServerSocket AppConfig{..} = case configServerUnixSocket of
   -- I'm not using `streaming-commons`' bindPath function here because it's not defined for Windows,
   -- but we need to have runtime error if we try to use it in Windows, not compile time error
   Just path -> createAndBindDomainSocket path configServerUnixSocketMode
-  Nothing -> bindPortTCP configServerPort (fromString $ T.unpack configServerHost)
+  Nothing
+    | configServerReusePort -> bindPortTCPWithReusePort configServerPort (fromString $ T.unpack configServerHost)
+    | otherwise             -> bindPortTCP configServerPort (fromString $ T.unpack configServerHost)
 
 initAdminServerSocket :: AppConfig -> IO (Maybe NS.Socket)
 initAdminServerSocket AppConfig{..} =
   traverse (`bindPortTCP` adminHost) configAdminServerPort
   where
     adminHost = fromString $ T.unpack configAdminServerHost
 
+bindPortTCPWithReusePort :: Int -> HostPreference -> IO NS.Socket
+bindPortTCPWithReusePort port hostPreference =
+  bindPortGenEx [(NS.ReusePort, 1)] NS.Stream port hostPreference >>= listenSocket
+  where
+    listenSocket sock = NS.listen sock (max 2048 NS.maxListenQueue) $> sock

src/PostgREST/Config.hs

@@ -118,6 +118,7 @@ data AppConfig = AppConfig
   , configServerCorsAllowedOrigins :: Maybe [Text]
   , configServerHost               :: Text
   , configServerPort               :: Int
+  , configServerReusePort          :: Bool
   , configServerTraceHeader        :: Maybe (CI.CI BS.ByteString)
   , configServerTimingEnabled      :: Bool
   , configServerUnixSocket         :: Maybe FilePath
@@ -203,6 +204,7 @@ toText conf =
       ,("server-cors-allowed-origins",      q . maybe "" (T.intercalate ",") . configServerCorsAllowedOrigins)
       ,("server-host",               q . configServerHost)
       ,("server-port",                   show . configServerPort)
+      ,("server-reuseport",              T.toLower . show . configServerReusePort)
       ,("server-trace-header",       q . T.decodeUtf8 . maybe mempty CI.original . configServerTraceHeader)
       ,("server-timing-enabled",         T.toLower . show . configServerTimingEnabled)
       ,("server-unix-socket",        q . maybe mempty T.pack . configServerUnixSocket)
@@ -318,6 +320,7 @@ parser optPath env dbSettings roleSettings roleIsolationLvl =
     <*> parseCORSAllowedOrigins "server-cors-allowed-origins"
     <*> (defaultServerHost <$> optString "server-host")
     <*> parseServerPort "server-port"
+    <*> (fromMaybe False <$> optBool "server-reuseport")
     <*> (fmap (CI.mk . encodeUtf8) <$> optString "server-trace-header")
     <*> (fromMaybe False <$> optBool "server-timing-enabled")
     <*> (fmap T.unpack <$> optString "server-unix-socket")
@@ -779,6 +782,7 @@ exampleConfigFile = S.unlines
   , ""
   , "server-host = \"!4\""
   , "server-port = 3000"
+  , "server-reuseport = false"
   , ""
   , "## Allow getting the request-response timing information through the `Server-Timing` header"
   , "server-timing-enabled = false"

test/io/configs/expected/aliases.config

@@ -34,6 +34,7 @@ openapi-server-proxy-uri = ""
 server-cors-allowed-origins = ""
 server-host = "!4"
 server-port = 3000
+server-reuseport = false
 server-trace-header = ""
 server-timing-enabled = false
 server-unix-socket = ""

test/io/configs/expected/boolean-numeric.config

@@ -34,6 +34,7 @@ openapi-server-proxy-uri = ""
 server-cors-allowed-origins = ""
 server-host = "!4"
 server-port = 3000
+server-reuseport = false
 server-trace-header = ""
 server-timing-enabled = false
 server-unix-socket = ""

test/io/configs/expected/boolean-string.config

@@ -34,6 +34,7 @@ openapi-server-proxy-uri = ""
 server-cors-allowed-origins = ""
 server-host = "!4"
 server-port = 3000
+server-reuseport = false
 server-trace-header = ""
 server-timing-enabled = false
 server-unix-socket = ""