chore: refactor with black and flake8 (#156)

This pull request focuses on improving logging practices, code
documentation, and minor code quality enhancements across both the
`admin-api-lib` and `extractor-api-lib` libraries. The main changes
include replacing generic error logging with `logger.exception()` for
better stack trace visibility, adding or improving docstrings for
classes and methods, and updating project dependencies. Additionally,
some minor code cleanups and consistency improvements were made.

**Logging improvements:**

* Replaced `logger.error` (often with explicit `traceback.format_exc()`)
with `logger.exception()` in multiple files to ensure stack traces are
logged automatically and consistently. This affects error handling in
file and source uploaders, document retrievers, and extractors.
[[1]](diffhunk://#diff-5cf6e4adf466c99d1afc322897549f6cda78687be53f7d1d2f43ca7551e32aa4L57-R58)
[[2]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL112-R113)
[[3]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL127-R137)
[[4]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL199-R199)
[[5]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL232-R231)
[[6]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L114-R121)
[[7]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L153-R155)
[[8]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L200-R204)
[[9]](diffhunk://#diff-6d8a274cc6bd15025ebdd55cdb7525c2b01da207871c44440632c9ebea0ce4deL79-R77)

**Documentation enhancements:**

* Added or improved docstrings for classes and methods, including
module-level docstrings in several files (e.g.,
`default_file_uploader.py`, `default_source_uploader.py`,
`file_extractor.py`, `source_extractor.py`).
[[1]](diffhunk://#diff-7dddc401b91c2ff4593c718a6211b203b1f5e15eb78cb33ec9e6da5756a0d173R11)
[[2]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455R33)
[[3]](diffhunk://#diff-6788b6d236b1f7d9e8ddcd74ed1f7c265ba82fa4b193f2eaa9d2ff9790a35c67R1-R2)
[[4]](diffhunk://#diff-73a4b2010a5a6312fd5a5496546dca10ff9fc480e4c4be037c96372647366c7eR1-R2)
[[5]](diffhunk://#diff-bcba3b03484e8a08bc3addb98d2b8f50cdaa5cf8e8d9605270bc69a6231599b7R25)
[[6]](diffhunk://#diff-8980643c87d06c313959eb7f400a57787367a482168872181e875355cb55d64bL16-R17)
[[7]](diffhunk://#diff-2b5524f0cb01b11e336def1a99356a243662de61a73be6dd5da1be89227cf112R39)
[[8]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeR1-L3)
[[9]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455R1-R2)

**Dependency and configuration updates:**

* Updated the version of both `admin-api-lib` and `extractor-api-lib` to
`v3.3.0` in their respective `pyproject.toml` files.
[[1]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL7-R7)
[[2]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L7-R7)
* Enabled `flake8-logging-format` and `flake8-docstrings` in both
libraries for improved code linting and documentation enforcement.
[[1]](diffhunk://#diff-9c5aeb0db77c2eec077d07ddc3b3810ae1a4a1e50ee7061fba37a46706c513fbL102-R103)
[[2]](diffhunk://#diff-dede389bcfb615c4b45cd1da7ac14cbe9535305f41f19cce09e321c91a8bb323L141-R142)

**Code quality and consistency:**

* Improved method and parameter documentation for several upload and
extraction methods, and made minor adjustments to method docstrings for
clarity and consistency.
[[1]](diffhunk://#diff-7dddc401b91c2ff4593c718a6211b203b1f5e15eb78cb33ec9e6da5756a0d173L19-R20)
[[2]](diffhunk://#diff-07e3a38bb2ae6f51f1ce64c6e8e1898be13dc441c1c43071c8c2ecc5f352531eL24-R24)
[[3]](diffhunk://#diff-8065840d5a5800fba5b7e366084521208dd6acdb106779806da0a04443d18667L152-R152)
[[4]](diffhunk://#diff-8065840d5a5800fba5b7e366084521208dd6acdb106779806da0a04443d18667L184-R184)
[[5]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL81-R82)
[[6]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L81-R84)
[[7]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L98)
[[8]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeL127-R137)
[[9]](diffhunk://#diff-3c2003f1db64f5b9610d691e451bb2ea5f65a92f1a81b5b210588931a4ecb455L114-R121)
* Minor code cleanups, such as removing unnecessary imports
(`traceback`), and improving warning message punctuation.
[[1]](diffhunk://#diff-5cf6e4adf466c99d1afc322897549f6cda78687be53f7d1d2f43ca7551e32aa4L5)
[[2]](diffhunk://#diff-2f472a7af7eccd952a8fc1daa5e849c9cd0649fae490380ff2c421bb57e16feeR1-L3)
[[3]](diffhunk://#diff-6d8a274cc6bd15025ebdd55cdb7525c2b01da207871c44440632c9ebea0ce4deL6-L7)
[[4]](diffhunk://#diff-7c45d126e034917dfbd2dbd921a73cd39c2b82a91752a1b6d3d60f9a01a04287L336-R336)

These changes collectively improve maintainability, error visibility,
and code documentation across the codebase.

---------

Co-authored-by: Andreas Klos <andreas.klos@stackit.cloud>

Author: robodev-r2d2

.gitignore

@@ -1,3 +1,4 @@
+**/.nx/*
 *.tgz
 */Chart.lock
 helm-chart/charts/admin-backend-0.0.1.tgz

libs/admin-api-lib/poetry.lock

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "admin-api-lib"
-version = "v3.2.1"
+version = "v3.3.0"
 description = "The admin backend is responsible for the document management. This includes deletion, upload and returning the source document."
 authors = [
     "STACKIT GmbH & Co. KG <data-ai@stackit.cloud>",
@@ -99,8 +99,8 @@ flake8-wot = "^0.2.0"
 flake8-function-order = "^0.0.5"
 flake8-tidy-imports = "^4.10.0"
 black = "^25.1.0"
-# flake8-logging-format = "^2024.24.12"
-# flake8-docstrings = "^1.7.0"
+flake8-logging-format = "^2024.24.12"
+flake8-docstrings = "^1.7.0"
 pytest-asyncio = "^1.0.0"
 
 [tool.poetry.dependencies]

libs/admin-api-lib/pyproject.toml

@@ -8,6 +8,7 @@
 
 
 class FileUploader(UploaderBase):
+    """File uploader endpoint of the admin API."""
 
     @abstractmethod
     async def upload_file(
@@ -16,7 +17,7 @@ async def upload_file(
         file: UploadFile,
     ) -> None:
         """
-        Uploads a source file for content extraction.
+        Upload a source file for content extraction.
 
         Parameters
         ----------

libs/admin-api-lib/src/admin_api_lib/api_endpoints/file_uploader.py

@@ -21,7 +21,7 @@ async def upload_source(
         timeout: Optional[float],
     ) -> None:
         """
-        Uploads the parameters for source content extraction.
+        Upload the parameters for source content extraction.
 
         Parameters
         ----------

libs/admin-api-lib/src/admin_api_lib/api_endpoints/source_uploader.py

@@ -7,9 +7,7 @@ class UploaderBase:
     """Base class for uploader API endpoints."""
 
     def __init__(self):
-        """
-        Initialize the UploaderBase.
-        """
+        """Initialize the UploaderBase."""
         self._background_threads = []
 
     def _prune_background_threads(self) -> list[Thread]:

libs/admin-api-lib/src/admin_api_lib/api_endpoints/uploader_base.py

@@ -149,7 +149,7 @@ async def upload_file(
     request: Request,
 ) -> None:
     """
-    Uploads user selected sources.
+    Upload user selected sources.
 
     Parameters
     ----------
@@ -181,7 +181,7 @@ async def upload_source(
     key_value_pair: List[KeyValuePair] = Body(None, description="The key-value pairs for the source"),
 ) -> None:
     """
-    Uploads user selected sources.
+    Upload user selected sources.
 
     Parameters
     ----------

libs/admin-api-lib/src/admin_api_lib/apis/admin_api.py

@@ -2,7 +2,6 @@
 
 import io
 import logging
-import traceback
 
 from fastapi import HTTPException, Response, status
 
@@ -54,10 +53,9 @@ async def adocument_reference_id_get(self, identification: str) -> Response:
                 self._file_service.download_file(identification, document_buffer)
                 logger.debug("DONE retrieving document with id: %s", identification)
                 document_data = document_buffer.getvalue()
-            except Exception as e:
-                logger.error(
-                    "Error retrieving document with id: %s. Error: %s %s", identification, e, traceback.format_exc()
-                )
+            except Exception:
+                # Log full stack trace without embedding the exception object in the message (G200)
+                logger.exception("Error retrieving document with id: %s.", identification)
                 raise ValueError(f"Document with id '{identification}' not found.")
             finally:
                 document_buffer.close()

libs/admin-api-lib/src/admin_api_lib/impl/api_endpoints/default_document_reference_retriever.py

@@ -1,6 +1,7 @@
+"""Module for the default file uploader implementation."""
+
 import logging
 from pathlib import Path
-import traceback
 import urllib
 import tempfile
 import asyncio
@@ -78,7 +79,7 @@ async def upload_file(
         file: UploadFile,
     ) -> None:
         """
-        Uploads a source file for content extraction.
+        Upload a source file for content extraction.
 
         Parameters
         ----------
@@ -109,7 +110,7 @@ async def upload_file(
             raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(e))
         except Exception as e:
             self._key_value_store.upsert(source_name, Status.ERROR)
-            logger.error("Error while uploading %s = %s", source_name, str(e))
+            logger.exception("Error while uploading %s", source_name)
             raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(e))
 
     def _log_task_exception(self, task: asyncio.Task) -> None:
@@ -124,19 +125,16 @@ def _log_task_exception(self, task: asyncio.Task) -> None:
         if task.done() and not task.cancelled():
             try:
                 task.result()  # This will raise the exception if one occurred
-            except Exception as e:
-                logger.error("Background task failed with exception: %s", str(e))
-                logger.debug("Background task exception traceback: %s", traceback.format_exc())
+            except Exception:
+                logger.exception("Background task failed with exception.")
 
     def _prune_background_tasks(self) -> None:
-        """
-        Remove completed background tasks from the list.
-        """
+        """Remove completed background tasks from the list."""
         self._background_tasks = [task for task in self._background_tasks if not task.done()]
 
     def _check_if_already_in_processing(self, source_name: str) -> None:
         """
-        Checks if the source is already in processing state.
+        Check if the source is already in processing state.
 
         Parameters
         ----------
@@ -196,9 +194,9 @@ async def _handle_source_upload(
             await asyncio.to_thread(self._rag_api.upload_information_piece, rag_information_pieces)
             self._key_value_store.upsert(source_name, Status.READY)
             logger.info("Source uploaded successfully: %s", source_name)
-        except Exception as e:
+        except Exception:
             self._key_value_store.upsert(source_name, Status.ERROR)
-            logger.error("Error while uploading %s = %s", source_name, str(e))
+            logger.exception("Error while uploading %s", source_name)
 
     def _add_file_url(self, file_name: str, base_url: str, chunked_documents: list[Document]):
         document_url = f"{base_url.rstrip('/')}/document_reference/{urllib.parse.quote_plus(file_name)}"
@@ -229,6 +227,6 @@ async def _asave_new_document(
 
                 self._file_service.upload_file(Path(temp_file_path), filename)
                 return filename
-        except Exception as e:
-            logger.error("Error during document saving: %s %s", e, traceback.format_exc())
+        except Exception:
+            logger.exception("Error during document saving")
             self._key_value_store.upsert(source_name, Status.ERROR)

libs/admin-api-lib/src/admin_api_lib/impl/api_endpoints/default_file_uploader.py

@@ -1,3 +1,5 @@
+"""Module for the default source uploader implementation."""
+
 import logging
 import asyncio
 from threading import Thread
@@ -28,6 +30,7 @@
 
 
 class DefaultSourceUploader(SourceUploader):
+    """The DefaultSourceUploader is responsible for uploading source files for content extraction."""
 
     def __init__(
         self,
@@ -78,7 +81,7 @@ async def upload_source(
         kwargs: list[KeyValuePair],
     ) -> None:
         """
-        Uploads the parameters for source content extraction.
+        Upload the parameters for source content extraction.
 
         Parameters
         ----------
@@ -95,7 +98,6 @@ async def upload_source(
         -------
         None
         """
-
         self._prune_background_threads()
 
         source_name = f"{source_type}:{sanitize_document_name(name)}"
@@ -111,12 +113,12 @@ async def upload_source(
             raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(e))
         except Exception as e:
             self._key_value_store.upsert(source_name, Status.ERROR)
-            logger.error("Error while uploading %s = %s", source_name, str(e))
+            logger.exception("Error while uploading %s", source_name)
             raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(e))
 
     def _check_if_already_in_processing(self, source_name: str) -> None:
         """
-        Checks if the source is already in processing state.
+        Check if the source is already in processing state.
 
         Parameters
         ----------
@@ -150,7 +152,7 @@ def _thread_worker(self, source_name, source_type, kwargs, timeout):
             logger.error("Upload of %s timed out after %s seconds", source_name, timeout)
             self._key_value_store.upsert(source_name, Status.ERROR)
         except Exception:
-            logger.error("Error while uploading %s", source_name)
+            logger.exception("Error while uploading %s", source_name)
             self._key_value_store.upsert(source_name, Status.ERROR)
         finally:
             loop.close()
@@ -197,6 +199,6 @@ async def _handle_source_upload(
             await asyncio.to_thread(self._rag_api.upload_information_piece, rag_information_pieces)
             self._key_value_store.upsert(source_name, Status.READY)
             logger.info("Source uploaded successfully: %s", source_name)
-        except Exception as e:
+        except Exception:
             self._key_value_store.upsert(source_name, Status.ERROR)
-            logger.error("Error while uploading %s = %s", source_name, str(e))
+            logger.exception("Error while uploading %s", source_name)