Additions to Media de-duplication efforts (#210)

A little bit of clean up. Abstracting how the local path to the file is
resolved. Ensure consistency when removing media, that if there are
still references to the media that only the metadata is removed until it
is the last reference.

Resolving concerns about moving existing media by leaving it in place
and only new media is de-duplicated.

Stage incoming remote media into it's legacy file path until the sha256
can be discovered, and only after it is written to the database metadata
is it moved into it's new place. This specific set of steps works with
the file path resolving, so if the moving process is interrupted it will
still be usable.

Always provide `StorageProvider` classes with a consistent relative
"path". For a media id of `1234567890`, it would break down to
`12/34/567890`. Do not use the sha256 for this path. `StorageProvider`s
are provided the `FileInfo` class as well, if they want to do media
de-dupe they can get it from there. Should we provide an enhancement to
the bundled `StorageProvider` to allow for this capability?

Depends on #155

Author: jason-famedly

.github/workflows/famedly-tests.yml

@@ -25,7 +25,7 @@ jobs:
       - uses: Swatinem/rust-cache@68b3cb7503c78e67dae8373749990a220eb65352
       - uses: matrix-org/setup-python-poetry@v2
         with:
-          python-version: "3.x"
+          python-version: "3.13"
           poetry-version: "2.1.1"
           extras: "all"
       - run: poetry run scripts-dev/generate_sample_config.sh --check
@@ -49,7 +49,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.x"
+          python-version: "3.13"
       - run: .ci/scripts/check_lockfile.py
 
   lint:
@@ -63,6 +63,7 @@ jobs:
         uses: matrix-org/setup-python-poetry@v2
         with:
           poetry-version: "2.1.1"
+          python-version: "3.13"
           install-project: "false"
 
       - name: Run ruff check
@@ -91,6 +92,7 @@ jobs:
           # https://github.com/matrix-org/synapse/pull/15376#issuecomment-1498983775
           # To make CI green, err towards caution and install the project.
           install-project: "true"
+          python-version: "3.13"
           poetry-version: "2.1.1"
 
       # Cribbed from
@@ -124,6 +126,7 @@ jobs:
       - uses: matrix-org/setup-python-poetry@v2
         with:
           poetry-version: "2.1.1"
+          python-version: "3.13"
           extras: "all"
       - run: poetry run scripts-dev/check_pydantic_models.py
 
@@ -161,7 +164,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.x"
+          python-version: "3.13"
       - run: "pip install rstcheck"
       - run: "rstcheck --report-level=WARNING README.rst"
 

rust/src/http_client.rs

@@ -137,7 +137,7 @@ fn get_runtime<'a>(reactor: &Bound<'a, PyAny>) -> PyResult<PyRef<'a, PyTokioRunt
 static DEFER: OnceCell<PyObject> = OnceCell::new();
 
 /// Access to the `twisted.internet.defer` module.
-fn defer(py: Python<'_>) -> PyResult<&Bound<PyAny>> {
+fn defer(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>> {
     Ok(DEFER
         .get_or_try_init(|| py.import("twisted.internet.defer").map(Into::into))?
         .bind(py))