Create function, endpoint, frontend implementation for log persistence

Update CHANGELOG.md

Lint ctrl.py

Add persist_logs bool and integrate throughout the config flow
Add many logger.info chunks for testing

Add more logic to persist_logs toggle endpoint

Change how default value is handled

Change default permissions of journald.conf

Fix spelling errors, add logger on frontend

Move toggle_log_persistance away from FastAPI and towards the Api object

Linting

Call reinit at end of log toggle, reword models.py description of persist_logs

Remove all uses of persist_logs from amplipi core, move to updater side

Remove lingering changes from app.py, ctrl.py

pylint

pylint (2)

Finish porting new admin settings tab to updater

Change name of Updates tab on settings page

Fix spelling in comment in app.py

Robustify setPersist function against edge cases, add documentation

pylint

Add delete logs endpoint, cronjob that calls it at midnight

Updates/Admin

Change comment on delete_persisted_logs_crontab

Update CHANGELOG.md

Remove deletion endpoint + supporting files

remove optional "value" bool from post persist_logs

Fix wording in index.html

Change configure.py to only set logs to volatile if the /var/log/journal directory doesn't exist (i.e., only set volatility on first deploy)

Reduce verbocity in configure.py

Add auto_off_delay feature to admin settings

Pylint

Pylint

Rename attempt_persist_logs_auto_off.py to increment_auto_off.py
Change logic paths of POST /settings/persist_logs to avoid interacting with the files more often than necessary

PyCodeStyle

Update crontab to have a user associated with it

remove unused variables from increment_auto_off.py

Swap from using subprocess.run and configparser to just responses in increment_auto_off.py

Copy files properly in configure.py

Cover some oversights in asgi.py

Add hoverover tooltip for Auto Off Delay label

Make sure to pip install requests in configure.py

pylint
segregate standard imports from package import in increment_auto_off.py

Update styling

Move password setting functionality to Admin Settings page, add loading spinner to save log state button

update IDs, classnames to use JavaScript naming instead of Python naming ( - instead of _ )

Add new success and failure notification for persist save button

Fix formatting for smaller screens on the updater

Update CHANGELOG.md

Update CHANGELOG.md

Fix styling of admin panel

Correct error handling, fix styling issue

pylint

Extend safety net

remove pip install requests from configure.py, that should be handled by requirements.txt

Update styling, tooltips

Gate the delay of increment_auto_off.py behind an if statement

Revamp increment_auto_off.py to be more efficient

Update verbiage around the updater

Update increment_auto_off.py to not run or loop if persistance is disabled

Edit github workflows to include more files
Add more, better file validation to /persist_logs endpoints
Remove looping, add checking to increment_auto_off.py

fix typo in python-app.yaml

Add docstring to increment_auto_off.py, add same file to github pylinting

Reconfigure file validation functions
Simplify persistence toggle function
Rework increment_auto_off.py to be able to use the backdoor if the frontdoor doesn't work out

linting

linting (2)

Linting (3)

Simplify file verification functions
Include minor verification in the non-api route of increment_auto_off.py

pycodestyle

Further robustify auto_off.py and validator functions against direct user interventions

pypcodestyle

Update password set function to give feedback in the same way as updating persistence and delay

Author: SteveMicroNova

.github/workflows/python-app.yml

@@ -41,10 +41,12 @@ jobs:
         pylint --exit-zero amplipi --generated-members "signal.Signals,GPIO.*"
         pylint -E amplipi --generated-members "signal.Signals,GPIO.*"
         pylint -E streams
+        pylint -E scripts/increment_auto_off.py
     - name: Lint with mypy, static type checker
       run: |
         pip install mypy
         mypy amplipi/ --ignore-missing-imports
+        mypy scripts/increment_auto_off.py --ignore-missing-imports
     - name: Test mock using pytest # rpi cannot be tested directly due to hardware...
       run: |
         pip install pytest pytest-cov

CHANGELOG.md

@@ -6,6 +6,10 @@
   * Add error handling on browser page for instances where the selected stream isn't browsable
   * Add scrollbars to tall modals
   * Change how events are handled with Modals to reduce accidental closures
+  * Rename Updater to Admin Panel
+  * Add "Admin Settings" tab to the Updater/Admin Panel
+    * Add toggleable option to persist system logs to Admin Settings
+    * Move "Set password" page to be accessible via Admin Settings tab
 * System
   * Make update process properly report errors
 

amplipi/app.py

@@ -226,7 +226,7 @@ def shutdown():
   """
   # preemptively save the state (just in case the shutdown procedure doesn't invoke a save)
   get_ctrl().save()
-  # start the shutdown process and returning immediately (hopeully before the shutdown process begins)
+  # start the shutdown process and returning immediately (hopefully before the shutdown process begins)
   Popen('sleep 1 && sudo systemctl poweroff', shell=True)
 
 

amplipi/ctrl.py

@@ -222,7 +222,7 @@ def reinit(self, settings: models.AppSettings = models.AppSettings(), change_not
     except Exception as exc:
       logger.exception("Error setting is_streamer flag: {exc}")
 
-    # determine if we're in LMS mode, based on a file
+    # determine if we're in LMS mode, based on a file existing
     lms_mode_path = Path(defaults.USER_CONFIG_DIR, 'lms_mode')
     if lms_mode_path.exists():
       logger.info("lms mode")

amplipi/updater/asgi.py

@@ -36,6 +36,8 @@
 import shutil
 import asyncio
 
+import configparser
+
 # web framework
 import requests
 from fastapi import FastAPI, Request, File, UploadFile, Depends, APIRouter, Response
@@ -48,7 +50,8 @@
 # pylint: disable=no-name-in-module
 from pydantic import BaseModel
 
-from ..auth import CookieOrParamAPIKey, router as auth_router, set_password_hash, unset_password_hash, NotAuthenticatedException, not_authenticated_exception_handler, create_access_key
+from ..auth import CookieOrParamAPIKey, router as auth_router, set_password_hash, unset_password_hash,\
+  NotAuthenticatedException, not_authenticated_exception_handler, create_access_key
 
 app = FastAPI()
 router = APIRouter(dependencies=[Depends(CookieOrParamAPIKey)])
@@ -62,6 +65,72 @@
 sse_messages: queue.Queue = queue.Queue()
 
 
+def validate_logging_ini():
+  """Fallback in case the ini file or any individual header  doesn't exist, set to default settings. Only really comes up during tests."""
+  tmp = '/tmp/logging.ini.tmp'
+  ini = '/var/log/logging.ini'
+  conf = configparser.ConfigParser(strict=False, allow_no_value=True)
+
+  with open(tmp, "+w", encoding="utf-8") as file:
+    if os.path.exists(ini):
+      conf.read(ini)
+    else:
+      conf.read(file)
+
+    if not conf.has_section("logging"):
+      conf.add_section("logging")
+
+    if not conf.has_option("logging", "auto_off_delay"):
+      conf.set("logging", "auto_off_delay", "14")
+    auto_off_delay = conf.get("logging", "auto_off_delay", fallback="14")
+    if not auto_off_delay.isdigit() and bool(re.fullmatch(r'\d*\.\d+', auto_off_delay)):
+      # regex to check decimal state, this would lead to "123.45" and ".45" being true but not "123."
+      # Exclude anything that isdigit() as to not overwrite valid user settings
+      rounded = round(float(auto_off_delay)) if round(float(auto_off_delay)) > 0 else 1  # Avoid instances where it could be zero as to not set the "do not deactivate" setting
+      conf.set("logging", "auto_off_delay", str(rounded))
+    elif not auto_off_delay.isdigit():  # Cannot be merged with the first check in an OR case as valid regex catches would be intercepted by that
+      conf.set("logging", "auto_off_delay", "14")
+    conf.write(file)
+
+  subprocess.run(['sudo', 'mv', tmp, ini], check=True)
+
+
+def validate_journald_conf():
+  """Fallback in case the config file or any individual header doesn't exist, set to default settings"""
+  tmp = '/tmp/journald.conf.tmp'
+  conf = '/etc/systemd/journald.conf'
+  confparse = configparser.ConfigParser(strict=False, allow_no_value=True)
+
+  with open(tmp, "+w", encoding="utf-8") as file:
+    if os.path.exists(conf):
+      confparse.read(conf)
+    else:
+      confparse.read(file)
+
+    if not confparse.has_section("Journal"):
+      confparse.add_section("Journal")
+
+    if not confparse.has_option("Journal", "Storage") or confparse.get("Journal", "Storage") not in ("volatile", "persistent"):
+      # While volatile is the system's default, having that value either not exist or be invalid points to there being a system error that caused it to get that way
+      # Set to persistent just in case
+      confparse.set("Journal", "Storage", "persistent")
+
+    # Set everything else to default while preserving user settings
+    if not confparse.has_option("Journal", "SyncIntervalSec"):
+      confparse.set('Journal', 'SyncIntervalSec', '30s')
+    if not confparse.has_option("Journal", "SystemMaxUse"):
+      confparse.set('Journal', 'SystemMaxUse', '64M')
+    if not confparse.has_option("Journal", "RuntimeMaxUse"):
+      confparse.set('Journal', 'RuntimeMaxUse', '64M')
+    if not confparse.has_option("Journal", "ForwardToConsole"):
+      confparse.set('Journal', 'ForwardToConsole', 'no')
+    if not confparse.has_option("Journal", "ForwardToWall"):
+      confparse.set('Journal', 'ForwardToWall', 'no')
+
+    confparse.write(file)
+  subprocess.run(['sudo', 'mv', tmp, conf], check=True)
+
+
 class ReleaseInfo(BaseModel):
   """ Software Release Information """
   url: str
@@ -95,6 +164,81 @@ class ReleaseInfo(BaseModel):
   logger.exception(f'Error loading identity file: {e}')
 
 
+class Persist_Logs(BaseModel):
+  """Basemodel that consists of a bool and int, used to change different config files around the system via POST /settings/persist_logs"""
+  persist_logs: bool
+  auto_off_delay: int
+
+
+@router.get("/settings/persist_logs")
+def get_log_persist_state():
+  """
+  Checks /etc/systemd/journald.conf to find if the current storage setting is persistent and returns a bool
+  Note that returning false doesn't necessarily mean that logs are set to volatile, and could just mean that the config file is missing the line being read
+  """
+  validate_journald_conf()
+  journalconf = configparser.ConfigParser(strict=False, allow_no_value=True)
+  journalconf.read('/etc/systemd/journald.conf')
+
+  validate_logging_ini()
+  logconf = configparser.ConfigParser(strict=False, allow_no_value=True)
+  logconf.read('/var/log/logging.ini')
+
+  # Fallback set is the default value of the Storage variable under the Journal header of the conf file
+  # Used when the variable cannot be read but the file itself can (implying that the variable is missing, and should be set to a default)
+  ret = Persist_Logs(persist_logs=journalconf.get("Journal", "Storage", fallback="volatile") == "persistent", auto_off_delay=logconf.get("logging", "auto_off_delay", fallback="14"),)
+  return ret
+
+
+@router.post("/settings/persist_logs")
+def toggle_persist_logs(data: Persist_Logs):
+  """Toggles the option within journald to save logs to memory or storage, and sets the length of time before that setting is reset to volatile"""
+  try:
+    # Just in case
+    validate_logging_ini()
+    validate_journald_conf()
+
+    state = get_log_persist_state()
+
+    if state.persist_logs != data.persist_logs:
+      journalconf = '/etc/systemd/journald.conf'
+      journaltmp = '/tmp/journald.conf.tmp'
+      journal = configparser.ConfigParser(strict=False, allow_no_value=True)
+      journal.read(journalconf)
+
+      if not journal.has_section("Journal"):
+        journal.add_section('Journal')
+
+      # goal_value is true if you wish to turn persistent logging on and false if you wish to turn it off
+      journal.set('Journal', 'Storage', 'persistent' if data.persist_logs else 'volatile')
+
+      with open(journaltmp, 'w', encoding="utf-8") as conf_file:
+        journal.write(conf_file)
+
+      subprocess.run(['sudo', 'mv', journaltmp, journalconf], check=True)
+      subprocess.run(['sudo', 'systemctl', 'restart', 'systemd-journald'], check=True)
+      logger.info(f"persist_logs set to {data.persist_logs}")
+    else:
+      logger.info("persist_logs unchanged")
+
+    if state.auto_off_delay != data.auto_off_delay:
+      logconf = '/var/log/logging.ini'
+      logtmp = '/tmp/logging.ini.tmp'
+      log = configparser.ConfigParser(strict=False, allow_no_value=True)
+      log.read(logconf)
+      log.set('logging', 'auto_off_delay', f"{data.auto_off_delay}")  # Accept auto_off_delay as an int for type checking, parse to str for configParser validity
+      with open(logtmp, 'w', encoding='utf-8') as file:
+        log.write(file)
+      subprocess.run(['sudo', 'mv', logtmp, logconf], check=True)
+      logger.info(f"auto_off_delay set to {data.auto_off_delay}")
+
+    else:
+      logger.info("auto_off_delay unchanged")
+  except Exception as exc:
+    logger.exception(str(exc))
+    return 500
+
+
 @router.get('/update')
 def get_index():
   """ Get the update website """

amplipi/updater/static/css/styles.css

@@ -1,6 +1,20 @@
 /*
   A couple styles to make the demo page look good
  */
+
+ .column-container {
+  display: flex;
+  flex-direction: column;
+  gap: 64px;
+}
+
+.column-item {
+  display: flex;
+  flex-direction: column;
+  justify-content: space-around;
+  gap: 16px;
+}
+
 body {
   padding-bottom: 2rem;
   padding-top: 1rem;
@@ -79,6 +93,15 @@ hr {
   background: #4a4a4a;
 }
 
+.margin{
+  margin-left: 0px;
+}
+@media (min-width: 768px) {
+  .margin{
+    margin-left: 10px;
+  }
+}
+
 .fa-circle-notch {
   animation: fa-spin 1s infinite linear;
 }
@@ -88,3 +111,41 @@ hr {
   background-color: #5a6268;
   border-color: #545b62;
 }
+
+.container {
+  padding-left: 1rem;
+  padding-right: 1rem;
+}
+
+
+.set-password-dialog {
+  color: #aaaaaa !important;
+}
+
+/* lifted from https://www.w3schools.com/css/css_tooltip.asp */
+
+.hover-label {
+  position: relative;
+  display: inline-block;
+  border-bottom: 1px dotted black;
+}
+
+.hover-label .hover-tooltip{
+  top: 100%;
+  left: 50%;
+  visibility: hidden;
+  width: 750px;
+  max-width: 70vw;
+  background-color: rgba(100, 100, 100, 1);
+  color: #ffffff;
+  text-align: center;
+  padding: 5px 0;
+  border-radius: 6px;
+
+  position: absolute;
+  z-index: 1;
+}
+
+.hover-label:hover .hover-tooltip{
+  visibility: visible;
+}