Skip to content

Commit 2b8643e

Browse files
mkleczekmkleczek
committed
add: use SO_REUSEPORT on platform supporting it
1 parent ea08a4d commit 2b8643e

9 files changed

Lines changed: 255 additions & 83 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ All notable changes to this project will be documented in this file. From versio
1414
- Add `Vary` header to responses by @develop7 in #4609
1515
- Add config `db-timezone-enabled` for optional querying of timezones by @taimoorzaeem in #4751
1616
- Log schema cache queries timings on `log-level=debug` by @steve-chavez in #4805
17+
- Enable starting multiple PostgREST instances using the same ports on platforms supporting it by @mkleczek in #4703 #4694
1718

1819
### Fixed
1920

‎docs/how-tos/zero-downtime-upgrades.rst‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
.. _zero_downtime_upgrades:
2+
3+
Zero-Downtime Upgrades
4+
======================
5+
6+
On operating systems that support ``SO_REUSEPORT``, PostgREST can start more
7+
than one process on the same :ref:`server-host` and :ref:`server-port`. This
8+
allows a new PostgREST process to start and become ready before the old process
9+
is stopped.
10+
11+
While both processes are running, the operating system distributes new
12+
connections between them. After the old process exits, the new process receives
13+
all new connections.
14+
15+
This is useful for upgrades and restarts:
16+
17+
1. Keep the old PostgREST process serving requests.
18+
2. Start the new PostgREST process on the same host and port.
19+
3. Wait for the new process to report ``/ready``.
20+
4. Stop the old process.
21+
22+
Configuration
23+
-------------
24+
25+
Both processes should use the same public host and port:
26+
27+
.. code-block:: ini
28+
29+
# /etc/postgrest/postgrest.conf
30+
server-host = "127.0.0.1"
31+
server-port = 3000
32+
33+
admin-server-host = "127.0.0.1"
34+
admin-server-port = 3001
35+
36+
The second process can use the same configuration file and override only the
37+
admin server port:
38+
39+
.. code-block:: bash
40+
41+
PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf
42+
43+
.. important::
44+
45+
Use a different :ref:`admin-server-port` for each process during the
46+
handover. Although the admin server can also share a port on systems that
47+
support ``SO_REUSEPORT``, a shared admin port makes readiness checks
48+
ambiguous: ``/ready`` could be answered by the old process instead of the new
49+
one.
50+
51+
Before using this in production, keep these details in mind:
52+
53+
- This works for host and port based servers. It does not apply when
54+
:ref:`server-unix-socket` is used.
55+
- If the operating system does not support ``SO_REUSEPORT``, the new process
56+
will fail to start with an address-in-use error and the old process will keep
57+
serving requests.
58+
- Each PostgREST process has its own :ref:`db-pool`. During the handover, the
59+
total possible database connections can temporarily double.
60+
- The old and new processes may both serve requests for a short time. Database
61+
migrations should be compatible with both versions while they overlap.
62+
63+
Manual Handover
64+
---------------
65+
66+
Assuming the old process is already serving on ``127.0.0.1:3000`` and its PID
67+
is stored in ``OLD_PID``:
68+
69+
.. code-block:: bash
70+
71+
PGRST_ADMIN_SERVER_PORT=3002 postgrest /etc/postgrest/postgrest.conf &
72+
NEW_PID=$!
73+
74+
curl --fail http://127.0.0.1:3002/ready
75+
76+
kill -TERM "$OLD_PID"
77+
78+
The ``curl`` request checks the new process through its own admin server port.
79+
If the new process cannot load its configuration, connect to the database, or
80+
load the schema cache, ``/ready`` will not return a successful response and the
81+
old process can keep serving traffic.
82+
83+
Example Script
84+
--------------
85+
86+
The following script shows the full sequence for a setup that stores the old
87+
process PID in a PID file. Adapt the start and stop commands to your process
88+
manager.
89+
90+
.. code-block:: bash
91+
92+
#!/usr/bin/env bash
93+
set -euo pipefail
94+
95+
POSTGREST=${POSTGREST:-postgrest}
96+
CONFIG=${CONFIG:-/etc/postgrest/postgrest.conf}
97+
PID_FILE=${PID_FILE:-/run/postgrest.pid}
98+
99+
ADMIN_HOST=${ADMIN_HOST:-127.0.0.1}
100+
NEW_ADMIN_PORT=${NEW_ADMIN_PORT:-3002}
101+
READY_TIMEOUT=${READY_TIMEOUT:-30}
102+
STOP_TIMEOUT=${STOP_TIMEOUT:-30}
103+
104+
if [[ ! -s "$PID_FILE" ]]; then
105+
echo "PID file not found or empty: $PID_FILE" >&2
106+
exit 1
107+
fi
108+
109+
OLD_PID=$(<"$PID_FILE")
110+
111+
if ! kill -0 "$OLD_PID" 2>/dev/null; then
112+
echo "Old PostgREST process is not running: $OLD_PID" >&2
113+
exit 1
114+
fi
115+
116+
PGRST_ADMIN_SERVER_HOST="$ADMIN_HOST" \
117+
PGRST_ADMIN_SERVER_PORT="$NEW_ADMIN_PORT" \
118+
"$POSTGREST" "$CONFIG" &
119+
NEW_PID=$!
120+
121+
cleanup_new_process() {
122+
kill "$NEW_PID" 2>/dev/null || true
123+
}
124+
trap cleanup_new_process EXIT INT TERM
125+
126+
READY_URL="http://$ADMIN_HOST:$NEW_ADMIN_PORT/ready"
127+
READY_DEADLINE=$((SECONDS + READY_TIMEOUT))
128+
129+
until curl --fail --silent --show-error --output /dev/null "$READY_URL"; do
130+
if ! kill -0 "$NEW_PID" 2>/dev/null; then
131+
echo "New PostgREST process exited before it became ready" >&2
132+
exit 1
133+
fi
134+
135+
if (( SECONDS >= READY_DEADLINE )); then
136+
echo "New PostgREST process did not become ready at $READY_URL" >&2
137+
exit 1
138+
fi
139+
140+
sleep 1
141+
done
142+
143+
printf '%s\n' "$NEW_PID" > "$PID_FILE"
144+
145+
kill -TERM "$OLD_PID" 2>/dev/null || true
146+
147+
STOP_DEADLINE=$((SECONDS + STOP_TIMEOUT))
148+
149+
while kill -0 "$OLD_PID" 2>/dev/null; do
150+
if (( SECONDS >= STOP_DEADLINE )); then
151+
echo "Old PostgREST process did not stop after SIGTERM; sending SIGKILL" >&2
152+
kill -KILL "$OLD_PID"
153+
break
154+
fi
155+
156+
sleep 1
157+
done
158+
159+
trap - EXIT INT TERM
160+
echo "PostgREST handover complete: $OLD_PID -> $NEW_PID"

‎docs/postgrest.dict‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ DSL
3434
DevOps
3535
Dramatiq
3636
dockerize
37+
downtime
3738
enum
3839
Enums
3940
Entra
@@ -57,6 +58,7 @@ HMAC
5758
htmx
5859
Htmx
5960
Homebrew
61+
handover
6062
hstore
6163
HTTP
6264
HTTPS
@@ -111,6 +113,7 @@ ov
111113
parametrized
112114
passphrase
113115
PBKDF
116+
PID
114117
PgBouncer
115118
pgcrypto
116119
pgjwt
@@ -200,4 +203,4 @@ webuser
200203
wfts
201204
www
202205
debouncing
203-
deduplicates
206+
deduplicates

‎docs/references/configuration.rst‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,10 @@ admin-server-port
176176

177177
Specifies the port for the :ref:`admin_server`. Cannot be equal to :ref:`server-port`.
178178

179+
On operating systems that support ``SO_REUSEPORT``, the admin server can also
180+
share the same host and port across multiple PostgREST instances. See
181+
:ref:`zero_downtime_upgrades`.
182+
179183
.. _app.settings.*:
180184

181185
app.settings.*
@@ -899,6 +903,26 @@ server-port
899903

900904
The TCP port to bind the web server. Use ``0`` to automatically assign a port.
901905

906+
On operating systems that support ``SO_REUSEPORT``, you can start multiple
907+
PostgREST instances on the same :ref:`server-host` and ``server-port``. For
908+
example, two PostgREST processes can use the same configuration:
909+
910+
.. code:: ini
911+
912+
server-host = "127.0.0.1"
913+
server-port = 3000
914+
915+
New connections are then distributed by the operating system between the
916+
running PostgREST processes. This can be used to start a replacement process
917+
before stopping the old one, or to run several PostgREST processes behind one
918+
port.
919+
920+
No additional PostgREST setting is required. If the operating system does not
921+
support this behavior, starting another PostgREST process on the same host and
922+
port will fail with the usual address-in-use error.
923+
924+
For a step-by-step example, see :ref:`zero_downtime_upgrades`.
925+
902926
.. _server-trace-header:
903927

904928
server-trace-header

‎src/PostgREST/Admin.hs‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,20 +22,20 @@ import qualified PostgREST.AppState as AppState
2222
import qualified Network.Socket as NS
2323
import Protolude
2424

25-
runAdmin :: AppState -> Maybe NS.Socket -> NS.Socket -> Warp.Settings -> IO ()
26-
runAdmin appState maybeAdminSocket socketREST settings = do
25+
runAdmin :: AppState -> Maybe NS.Socket -> IO (Maybe NS.Socket) -> Warp.Settings -> IO ()
26+
runAdmin appState maybeAdminSocket getSocketREST settings = do
2727
whenJust maybeAdminSocket $ \adminSocket -> do
2828
address <- resolveSocketToAddress adminSocket
2929
observer $ AdminStartObs address
3030
void . forkIO $ Warp.runSettingsSocket settings adminSocket adminApp
3131
where
32-
adminApp = admin appState socketREST
32+
adminApp = admin appState getSocketREST
3333
observer = AppState.getObserver appState
3434

3535
-- | PostgREST admin application
36-
admin :: AppState.AppState -> NS.Socket -> Wai.Application
37-
admin appState socketREST req respond = do
38-
isMainAppReachable <- isRight <$> reachMainApp socketREST
36+
admin :: AppState.AppState -> IO (Maybe NS.Socket) -> Wai.Application
37+
admin appState getSocketREST req respond = do
38+
isMainAppReachable <- getSocketREST >>= maybe (pure False) (fmap isRight . reachMainApp)
3939
isLoaded <- AppState.isLoaded appState
4040
isPending <- AppState.isPending appState
4141

@@ -44,8 +44,8 @@ admin appState socketREST req respond = do
4444
respond $ Wai.responseLBS (if isMainAppReachable then HTTP.status200 else HTTP.status500) [] mempty
4545
["ready"] ->
4646
let
47-
status | not isMainAppReachable = HTTP.status500
48-
| isPending = HTTP.status503
47+
status | isPending = HTTP.status503
48+
| not isMainAppReachable = HTTP.status500
4949
| isLoaded = HTTP.status200
5050
| otherwise = HTTP.status500
5151
in

0 commit comments

Comments
 (0)