fix: address various review comments for file sharing (#3978)

Document

Remove dead code

Document more things in the schema

Add another doc string

rename column requested_at ->  upload_requested_at

...

Update kubernetes/loculus/values.schema.json

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
Update preprocessing/specification.md

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
Update backend/src/main/kotlin/org/loculus/backend/model/SubmitModel.kt

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
Rename function

publish -> release

truncate new table

fix test

Refactor huge 'useEffect' in FolderUploadComponent

Update schema documentation based on migration changes

Replace * with concrete imports

Replace * with concrete imports

change 600 to 300

Fix formatting issue

rename fileField -> fileCategory in the backend

Update backend/src/main/kotlin/org/loculus/backend/controller/FilesController.kt

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
Update kubernetes/loculus/templates/_s3-endpoint.tpl

Co-authored-by: Cornelius Roemer <cornelius.roemer@gmail.com>
remove multipleFiles

...

rename in frontend
some changes

format

Add more docs to the endpoints

undo *

try catch mapping parsing

Update error

clarify naming of properties

Add admin docs

Allow region to be null

Test if we can remove the region and everything is still fine

format

test fixes

Add us-east-1 back

Add some arch docs

improve docs

Document required bucket policy

format

Add more docs

API docs formatting fix

escape

Add help text

Remove '.'

Add 'underline' to file links

update sequence diagram with better description

improve docs

Update backend/docs/file_sharing.md

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>
Update backend/src/main/kotlin/org/loculus/backend/config/BackendSpringConfig.kt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
eager initialize clients for thread safety

Update backend/src/main/kotlin/org/loculus/backend/api/SubmissionTypes.kt

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>
Adress Annas comments

file field -> file category

tail

rename

Update preprocessing/specification.md

Co-authored-by: Anna (Anya) Parker <50943381+anna-parker@users.noreply.github.com>
rename getFileName to getFileIdPath

Update schema documentation based on migration changes

remove 'silo' property from values.yaml

try out a container lifeycle hook

...

Add checking of 'enabled' in backend

test fix

configure resources for minio

Author: fhennig

.github/workflows/e2e-k3d.yml

@@ -34,7 +34,7 @@ jobs:
     env:
       ALL_BROWSERS: ${{ github.ref == 'refs/heads/main' || github.event.inputs.all_browsers && 'true' || 'false' }}
       sha: ${{ github.event.pull_request.head.sha || github.sha }}
-      wait_timeout: ${{ github.ref == 'refs/heads/main' && 900 || 600 }}
+      wait_timeout: ${{ github.ref == 'refs/heads/main' && 900 || 300 }}
     steps:
       - name: Shorten sha
         run: echo "sha=${sha::7}" >> $GITHUB_ENV

backend/docs/db/schema.sql

@@ -2,8 +2,8 @@
 -- PostgreSQL database dump
 --
 
--- Dumped from database version 15.12 (Debian 15.12-1.pgdg120+1)
--- Dumped by pg_dump version 16.8 (Debian 16.8-1.pgdg120+1)
+-- Dumped from database version 15.13 (Debian 15.13-1.pgdg120+1)
+-- Dumped by pg_dump version 16.9 (Debian 16.9-1.pgdg120+1)
 
 SET statement_timeout = 0;
 SET lock_timeout = 0;
@@ -272,10 +272,10 @@ ALTER VIEW public.external_metadata_view OWNER TO postgres;
 
 CREATE TABLE public.files (
     id uuid NOT NULL,
-    requested_at timestamp without time zone NOT NULL,
+    upload_requested_at timestamp without time zone NOT NULL,
     uploader text NOT NULL,
     group_id integer NOT NULL,
-    published_at timestamp without time zone
+    released_at timestamp without time zone
 );
 
 

backend/docs/file_sharing.md

@@ -0,0 +1,44 @@
+# File sharing
+
+Loculus supports a "file sharing" feature, where arbitrary files can be submitted alongside regular sequence entry data.
+The feature uses S3 blob storage to store the actual raw files, and a table in the backend to keep track of the files.
+
+Just like the rest of sequence data files are not publicly accessible until they are released.
+
+## Submission
+
+![submission](./plantuml/sequenceFileSharingSubmission.svg)
+
+## Preprocessing
+
+![preprocessing](./plantuml/sequenceFileSharingPrepro.svg)
+
+## Releasing
+
+When a sequence entry is released, the associated files are made public as well.
+This is done by setting a `released_at` timestamp in the `files` table, and making the file object in S3 public.
+The bucket is configured to allow public access for objects which are tagged with `public=true`,
+so to make a file public, the backend sets this tag on the object in S3.
+
+## The files table
+
+The database has a table to keep track of the files:
+
+```
+CREATE TABLE public.files (
+    id uuid NOT NULL,
+    upload_requested_at timestamp without time zone NOT NULL,
+    uploader text NOT NULL,
+    group_id integer NOT NULL,
+    released_at timestamp without time zone
+);
+```
+
+Entries in this table are created by the backend when the `request-uploads` endpoint is called.
+The request records who made the request, for which group and when.
+Note that the file does not yet exist in S3, the user still needs to upload it.
+
+On release, the release time is also marked in the database.
+
+This table can also be used to find "orphaned" files, i.e. files that have been requested and uploaded,
+but haven't been referenced in any sequence submission.

backend/docs/plantuml/sequenceFileSharingPrepro.puml

@@ -0,0 +1,26 @@
+@startuml
+    participant "Processing Pipeline" as pipeline #LightGreen
+    participant "Backend" as backend #Orange
+    participant "S3 / Object Storage" as s3 #LightGray
+    database "Database" as DB
+
+    pipeline -> backend: request entries needing processing
+    backend -> DB: fetch entries (including file IDs and file names)
+    backend -> pipeline: return entry data + pre-signed read URLs
+
+    loop for each file
+        pipeline -> s3: download file using pre-signed URL
+    end loop
+
+    pipeline -> backend: request new pre-signed URLs for processed files
+    backend -> DB: store new file IDs
+    backend -> pipeline: return new pre-signed write URLs + file IDs
+
+    loop for each processed file
+        pipeline -> s3: upload file using pre-signed write URL
+    end loop
+
+    pipeline -> backend: submit processed file IDs + other data
+    backend -> DB: store processed file data
+
+@enduml

backend/docs/plantuml/sequenceFileSharingPrepro.svg

@@ -0,0 +1,16 @@
+@startuml
+    participant "Frontend / User" as frontend #LightCyan
+    participant "Backend" as backend #Orange
+    participant "S3 / Object Storage" as s3 #LightGray
+    database "Database" as DB
+
+    frontend -> backend: request pre-signed URLs and file IDs
+    backend -> DB: insert file records with file IDs and requesting user
+    backend -> frontend: return pre-signed URLs and file IDs
+
+    loop for each file
+        frontend -> s3: upload file using pre-signed URL
+    end loop
+
+    frontend -> backend: submit metadata, sequence data and file IDs per submission ID
+@enduml