Skip to content

Commit 033bd60

Browse files
mkleczekmkleczek
committed
add: use SO_REUSEPORT on platform supporting it
1 parent ea08a4d commit 033bd60

10 files changed

Lines changed: 267 additions & 86 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ All notable changes to this project will be documented in this file. From versio
1414
- Add `Vary` header to responses by @develop7 in #4609
1515
- Add config `db-timezone-enabled` for optional querying of timezones by @taimoorzaeem in #4751
1616
- Log schema cache queries timings on `log-level=debug` by @steve-chavez in #4805
17+
- Enable starting multiple PostgREST instances using the same ports on platforms supporting it by @mkleczek in #4703 #4694
1718

1819
### Fixed
1920

‎docs/how-tos/zero-downtime-upgrades.rst‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
.. _zero_downtime_upgrades:
2+
3+
Zero-Downtime Upgrades
4+
======================
5+
6+
On operating systems that support ``SO_REUSEPORT``, PostgREST can start more
7+
than one process on the same :ref:`server-host` and :ref:`server-port`. This
8+
allows a new PostgREST process to start and become ready before the old process
9+
is stopped.
10+
11+
While both processes are running, the operating system distributes new
12+
connections between them. After the old process exits, the new process receives
13+
all new connections.
14+
15+
This is useful for upgrades and restarts:
16+
17+
1. Keep the old PostgREST process serving requests.
18+
2. Start the new PostgREST process on the same host and port.
19+
3. Wait for the new process to report ``/ready``.
20+
4. Stop the old process.
21+
22+
Configuration
23+
-------------
24+
25+
Both processes should use the same public host and port:
26+
27+
.. code-block:: ini
28+
29+
# /etc/postgrest/postgrest.conf
30+
server-host = "127.0.0.1"
31+
server-port = 3000
32+
33+
admin-server-host = "127.0.0.1"
34+
admin-server-port = 3001
35+
36+
The second process can use the same configuration file and override only the
37+
admin server port:
38+
39+
.. code-block:: bash
40+
41+
PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf
42+
43+
.. important::
44+
45+
Use a different :ref:`admin-server-port` for each PostgREST process during
46+
the handover. Admin ports are not shared between processes. This keeps
47+
readiness checks unambiguous: ``/ready`` on the new admin port can only be
48+
answered by the new process.
49+
50+
Before using this in production, keep these details in mind:
51+
52+
- This works for host and port based servers. It does not apply when
53+
:ref:`server-unix-socket` is used.
54+
- If the operating system does not support ``SO_REUSEPORT``, the new process
55+
will fail to start with an address-in-use error and the old process will keep
56+
serving requests.
57+
- If the new process uses the same :ref:`admin-server-port` as the old process,
58+
it will fail to start because that admin port is already in use.
59+
- Each PostgREST process has its own :ref:`db-pool`. During the handover, the
60+
total possible database connections can temporarily double.
61+
- The old and new processes may both serve requests for a short time. Database
62+
migrations should be compatible with both versions while they overlap.
63+
64+
Manual Handover
65+
---------------
66+
67+
Assuming the old process is already serving on ``127.0.0.1:3000`` and its PID
68+
is stored in ``OLD_PID``:
69+
70+
.. code-block:: bash
71+
72+
PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf &
73+
NEW_PID=$!
74+
75+
curl --fail http://127.0.0.1:3002/ready
76+
77+
kill -TERM "$OLD_PID"
78+
79+
The ``curl`` request checks the new process through its own admin server port.
80+
If the new process cannot load its configuration, connect to the database, or
81+
load the schema cache, ``/ready`` will not return a successful response and the
82+
old process can keep serving traffic.
83+
84+
Example Script
85+
--------------
86+
87+
The following script shows the full sequence for a setup that stores the old
88+
process PID in a PID file. Adapt the start and stop commands to your process
89+
manager.
90+
91+
.. code-block:: bash
92+
93+
#!/usr/bin/env bash
94+
set -euo pipefail
95+
96+
POSTGREST=${POSTGREST:-postgrest}
97+
CONFIG=${CONFIG:-/etc/postgrest/postgrest.conf}
98+
PID_FILE=${PID_FILE:-/run/postgrest.pid}
99+
100+
ADMIN_HOST=${ADMIN_HOST:-127.0.0.1}
101+
NEW_ADMIN_PORT=${NEW_ADMIN_PORT:-3002}
102+
READY_TIMEOUT=${READY_TIMEOUT:-30}
103+
STOP_TIMEOUT=${STOP_TIMEOUT:-30}
104+
105+
if [[ ! -s "$PID_FILE" ]]; then
106+
echo "PID file not found or empty: $PID_FILE" >&2
107+
exit 1
108+
fi
109+
110+
OLD_PID=$(<"$PID_FILE")
111+
112+
if ! kill -0 "$OLD_PID" 2>/dev/null; then
113+
echo "Old PostgREST process is not running: $OLD_PID" >&2
114+
exit 1
115+
fi
116+
117+
PGRST_ADMIN_SERVER_HOST="$ADMIN_HOST" \
118+
PGRST_ADMIN_SERVER_PORT="$NEW_ADMIN_PORT" \
119+
"$POSTGREST" "$CONFIG" &
120+
NEW_PID=$!
121+
122+
cleanup_new_process() {
123+
kill "$NEW_PID" 2>/dev/null || true
124+
}
125+
trap cleanup_new_process EXIT INT TERM
126+
127+
READY_URL="http://$ADMIN_HOST:$NEW_ADMIN_PORT/ready"
128+
READY_DEADLINE=$((SECONDS + READY_TIMEOUT))
129+
130+
until curl --fail --silent --show-error --output /dev/null "$READY_URL"; do
131+
if ! kill -0 "$NEW_PID" 2>/dev/null; then
132+
echo "New PostgREST process exited before it became ready" >&2
133+
exit 1
134+
fi
135+
136+
if (( SECONDS >= READY_DEADLINE )); then
137+
echo "New PostgREST process did not become ready at $READY_URL" >&2
138+
exit 1
139+
fi
140+
141+
sleep 1
142+
done
143+
144+
printf '%s\n' "$NEW_PID" > "$PID_FILE"
145+
146+
kill -TERM "$OLD_PID" 2>/dev/null || true
147+
148+
STOP_DEADLINE=$((SECONDS + STOP_TIMEOUT))
149+
150+
while kill -0 "$OLD_PID" 2>/dev/null; do
151+
if (( SECONDS >= STOP_DEADLINE )); then
152+
echo "Old PostgREST process did not stop after SIGTERM; sending SIGKILL" >&2
153+
kill -KILL "$OLD_PID"
154+
break
155+
fi
156+
157+
sleep 1
158+
done
159+
160+
trap - EXIT INT TERM
161+
echo "PostgREST handover complete: $OLD_PID -> $NEW_PID"

‎docs/postgrest.dict‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ DSL
3434
DevOps
3535
Dramatiq
3636
dockerize
37+
downtime
3738
enum
3839
Enums
3940
Entra
@@ -57,6 +58,7 @@ HMAC
5758
htmx
5859
Htmx
5960
Homebrew
61+
handover
6062
hstore
6163
HTTP
6264
HTTPS
@@ -111,6 +113,7 @@ ov
111113
parametrized
112114
passphrase
113115
PBKDF
116+
PID
114117
PgBouncer
115118
pgcrypto
116119
pgjwt
@@ -200,4 +203,4 @@ webuser
200203
wfts
201204
www
202205
debouncing
203-
deduplicates
206+
deduplicates

‎docs/references/admin_server.rst‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,15 @@ Two endpoints ``live`` and ``ready`` will then be available. Both these endpoint
1616

1717
.. important::
1818

19-
If you have a machine with multiple network interfaces and multiple PostgREST instances in the same port, you need to specify a unique :ref:`hostname <server-host>`
20-
in the configuration of each PostgREST instance for the health check to work correctly. Don't use the special values(``!4``, ``*``, etc) in this case because the health check
21-
could report a false positive.
19+
Multiple PostgREST instances can share the same public API host and port on
20+
operating systems that support ``SO_REUSEPORT``. Admin ports are not shared:
21+
give each instance a different :ref:`admin-server-port`, otherwise the new
22+
instance will fail to start.
23+
24+
If the machine has multiple network interfaces, configure concrete
25+
:ref:`server-host` and :ref:`admin-server-host` values when you need health
26+
checks to target a specific process. Avoid special values (``!4``, ``*``, etc)
27+
in this case because the health check could report a false positive.
2228

2329
Live
2430
----

‎docs/references/configuration.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,11 @@ admin-server-port
176176

177177
Specifies the port for the :ref:`admin_server`. Cannot be equal to :ref:`server-port`.
178178

179+
When running multiple PostgREST instances on the same :ref:`server-port`, use
180+
a different ``admin-server-port`` for each instance. Admin ports are not shared
181+
between instances, so readiness checks always target one specific PostgREST
182+
instance. See :ref:`zero_downtime_upgrades`.
183+
179184
.. _app.settings.*:
180185

181186
app.settings.*
@@ -899,6 +904,26 @@ server-port
899904

900905
The TCP port to bind the web server. Use ``0`` to automatically assign a port.
901906

907+
On operating systems that support ``SO_REUSEPORT``, you can start multiple
908+
PostgREST instances on the same :ref:`server-host` and ``server-port``. For
909+
example, two PostgREST processes can use the same configuration:
910+
911+
.. code:: ini
912+
913+
server-host = "127.0.0.1"
914+
server-port = 3000
915+
916+
New connections are then distributed by the operating system between the
917+
running PostgREST processes. This can be used to start a replacement process
918+
before stopping the old one, or to run several PostgREST processes behind one
919+
port.
920+
921+
No additional PostgREST setting is required. If the operating system does not
922+
support this behavior, starting another PostgREST process on the same host and
923+
port will fail with the usual address-in-use error.
924+
925+
For a step-by-step example, see :ref:`zero_downtime_upgrades`.
926+
902927
.. _server-trace-header:
903928

904929
server-trace-header

‎src/PostgREST/Admin.hs‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,20 +22,20 @@ import qualified PostgREST.AppState as AppState
2222
import qualified Network.Socket as NS
2323
import Protolude
2424

25-
runAdmin :: AppState -> Maybe NS.Socket -> NS.Socket -> Warp.Settings -> IO ()
26-
runAdmin appState maybeAdminSocket socketREST settings = do
25+
runAdmin :: AppState -> Maybe NS.Socket -> IO (Maybe NS.Socket) -> Warp.Settings -> IO ()
26+
runAdmin appState maybeAdminSocket getSocketREST settings = do
2727
whenJust maybeAdminSocket $ \adminSocket -> do
2828
address <- resolveSocketToAddress adminSocket
2929
observer $ AdminStartObs address
3030
void . forkIO $ Warp.runSettingsSocket settings adminSocket adminApp
3131
where
32-
adminApp = admin appState socketREST
32+
adminApp = admin appState getSocketREST
3333
observer = AppState.getObserver appState
3434

3535
-- | PostgREST admin application
36-
admin :: AppState.AppState -> NS.Socket -> Wai.Application
37-
admin appState socketREST req respond = do
38-
isMainAppReachable <- isRight <$> reachMainApp socketREST
36+
admin :: AppState.AppState -> IO (Maybe NS.Socket) -> Wai.Application
37+
admin appState getSocketREST req respond = do
38+
isMainAppReachable <- getSocketREST >>= maybe (pure False) (fmap isRight . reachMainApp)
3939
isLoaded <- AppState.isLoaded appState
4040
isPending <- AppState.isPending appState
4141

@@ -44,8 +44,8 @@ admin appState socketREST req respond = do
4444
respond $ Wai.responseLBS (if isMainAppReachable then HTTP.status200 else HTTP.status500) [] mempty
4545
["ready"] ->
4646
let
47-
status | not isMainAppReachable = HTTP.status500
48-
| isPending = HTTP.status503
47+
status | isPending = HTTP.status503
48+
| not isMainAppReachable = HTTP.status500
4949
| isLoaded = HTTP.status200
5050
| otherwise = HTTP.status500
5151
in

0 commit comments

Comments
 (0)