feat: rotate the internal PostgreSQL logs

The internal vector-db PostgreSQL redirected all output to a single
"$DATA_DIR/logfile" (pg_ctl -l) that grew without bound and was never
rotated, eventually filling the persistent storage on long-running
instances.

Drop the -l redirect and enable PostgreSQL's built-in logging collector
with the documented weekday scheme: one file per day named
postgresql-Mon.log through postgresql-Sun.log under "$DATA_DIR/log".
Size-based rotation stays off (log_rotation_size=0): with weekday
filenames PostgreSQL reopens the same file on a size rotation without
truncating, so a size cap cannot bound a day's file anyway.

The collector overwrites a weekday file only when it rotates while
running across a day boundary, not on startup, so a container that
restarts frequently would keep appending to the same-named weekday file
week after week. To keep the logs bounded to about a week regardless of
restart frequency, remove the legacy single logfile and any weekday log
at least six days old before starting the server, so last week's file is
always gone before its name is reused. This also reclaims the old
unbounded logfile on upgraded installs.

The options are passed on the pg_ctl command line, so they also take
effect for already-initialised data directories on the next restart and
need no edits to persisted configuration. Document the new log location
and retention in the README.

Fixes #303

Signed-off-by: Cesar <275373127+sanzakicesarr@users.noreply.github.com>

Author: sanzakicesarr

README.md

@@ -104,6 +104,7 @@ volumes:
 Logs are stored in the `logs/` directory in the persistent directory. In a docker container, it should be at `/nc_app_context_chat_backend/logs/`. The log file is named `ccb.log` and is set to otate at 20 MB with 10 backups. These logs are in JSONL format, i.e. each line is a valid JSON object.
 Now only warning and above logs are printed to the console. All the debug logs are written to the log file if `debug` is set to `true` in the config file.
 The logs of the embedding server are written to `logs/embedding_server_[date].log` in the persistent directory, it rotates with date change and is not in JSONL format, just raw stdout and stderr from the embedding server's process.
+The internal vector database (PostgreSQL) writes its logs to `vector_db_data/pgsql/log/` in the persistent directory, one file per weekday (e.g. `postgresql-Mon.log`); each is overwritten when that weekday comes round again, so the logs are kept for about a week.
 
 ## Configuration
 Configuration resides inside the persistent storage as `config.yaml`. The location is `$APP_PERSISTENT_STORAGE`. By default it would be at `/nc_app_context_chat_backend_data/config.yaml` inside the container.

changelog.md

@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - add gh workflows for docker builds and do separate cpu, cuda and rocm (vulkan) images (#295) @kyteinsky
 
 ### Changed
+- rotate the internal PostgreSQL logs into one file per weekday instead of a single ever-growing logfile (#303) @sanzakicesarr
 - update readme according to the latest changes (#300) @kyteinsky
 - bump llama_cpp_python to 0.3.23 (#301) @kyteinsky
 

dockerfile_scripts/pgsql/setup.sh

@@ -11,7 +11,7 @@ source "$(dirname $(realpath $0))/env"
 
 if [ x"$1" == "xstop" ]; then
     echo -n Stopping PostgreSQL...
-    sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -l "${DATA_DIR}/logfile" -o "-p ${PG_PORT}" stop
+    sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -o "-p ${PG_PORT}" stop
     exit 0
 fi
 
@@ -47,8 +47,19 @@ if [ ! -d "$DATA_DIR/base" ]; then
     sudo -u postgres ${PG_BIN}/initdb -D "$DATA_DIR" -E UTF8
 fi
 
+# Rotate the internal PostgreSQL logs with the built-in logging collector: one
+# file per weekday (postgresql-Mon.log ... postgresql-Sun.log) under
+# "$DATA_DIR/log". The collector overwrites a weekday file only on time-based
+# rotation, not on startup, so a frequently-restarted container would otherwise
+# append to last week's same-named file. Before starting, drop the pre-rotation
+# single logfile and any weekday log at least six days old (find -mtime +5), so
+# last week's file is always gone before its name is reused. See #303.
+rm -f "${DATA_DIR}/logfile"
+find "${DATA_DIR}/log" -maxdepth 1 -type f -name 'postgresql-*.log' -mtime +5 -delete 2>/dev/null || true
+PG_LOG_OPTS="-c logging_collector=on -c log_directory=log -c log_filename=postgresql-%a.log -c log_rotation_age=1d -c log_rotation_size=0 -c log_truncate_on_rotation=on"
+
 echo "Starting PostgreSQL..."
-sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -l "${DATA_DIR}/logfile" -o "-p ${PG_PORT}" start
+sudo -u postgres ${PG_BIN}/pg_ctl -D "$DATA_DIR" -o "-p ${PG_PORT} ${PG_LOG_OPTS}" start
 
 echo "Waiting for PostgreSQL to start..."
 until sudo -u postgres ${PG_SQL} -c "SELECT 1" > /dev/null 2>&1; do