Skip to content

Commit 0141ef1

Browse files
committed
feat: add cloud storage mode with jellyfin guardrails
1 parent d3d94de commit 0141ef1

14 files changed

Lines changed: 1039 additions & 9 deletions

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,15 @@ JELLYSTAT_DB_PASSWORD=
4747
# --- OPTIONAL: MUSIC (Lidarr + Tidarr) ---
4848
# Uncomment if using music services (bash scripts/setup-music.sh)
4949
# UN_LIDARR_0_API_KEY=your_lidarr_api_key_here
50+
51+
# --- CLOUD STORAGE (rclone + mergerfs) ---
52+
# Uncomment if using cloud storage (bash scripts/setup-cloud-storage.sh)
53+
# Note: cloud-backed playback on macOS requires MEDIA_SERVER=jellyfin and TDARR_MODE=docker.
54+
# CLOUD_STORAGE_ENABLED=true
55+
# RCLONE_REMOTE=myremote
56+
# RCLONE_REMOTE_PATH=
57+
# RCLONE_VFS_CACHE_MODE=full
58+
# RCLONE_VFS_CACHE_MAX_SIZE=50G
59+
# RCLONE_VFS_CACHE_MAX_AGE=72h
60+
# RCLONE_VFS_READ_CHUNK_SIZE=128M
61+
# CLOUD_UPLOAD_MIN_AGE_HOURS=24

.github/workflows/validate.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ jobs:
3535
scripts/install-vpn-failover.sh \
3636
scripts/log-prune.sh \
3737
scripts/setup-music.sh \
38+
scripts/setup-cloud-storage.sh \
39+
scripts/cloud-upload.sh \
3840
scripts/vpn-failover-watch.sh
3941
4042
- name: Validate compose config
@@ -43,4 +45,16 @@ jobs:
4345
docker compose --profile autoupdate --profile music config >/dev/null
4446
docker compose --profile jellyfin config >/dev/null
4547
docker compose --profile autoupdate --profile music --profile jellyfin config >/dev/null
48+
49+
# Cloud storage compose override
50+
sed -i 's/^# CLOUD_STORAGE_ENABLED=true/CLOUD_STORAGE_ENABLED=true/' .env
51+
sed -i 's/^# RCLONE_REMOTE=myremote/RCLONE_REMOTE=testremote/' .env
52+
sed -i 's/^# RCLONE_REMOTE_PATH=/RCLONE_REMOTE_PATH=/' .env
53+
sed -i 's/^# RCLONE_VFS_CACHE_MODE=full/RCLONE_VFS_CACHE_MODE=full/' .env
54+
sed -i 's/^# RCLONE_VFS_CACHE_MAX_SIZE=50G/RCLONE_VFS_CACHE_MAX_SIZE=50G/' .env
55+
sed -i 's/^# RCLONE_VFS_CACHE_MAX_AGE=72h/RCLONE_VFS_CACHE_MAX_AGE=72h/' .env
56+
sed -i 's/^# RCLONE_VFS_READ_CHUNK_SIZE=128M/RCLONE_VFS_READ_CHUNK_SIZE=128M/' .env
57+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage config >/dev/null
58+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile jellyfin config >/dev/null
59+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile tdarr-docker config >/dev/null
4660
rm .env

README.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
<img src="https://img.shields.io/badge/Tidarr-1DB954?style=flat-square&logoColor=white" />
2222
<img src="https://img.shields.io/badge/Recyclarr-FF6B35?style=flat-square&logoColor=white" />
2323
<img src="https://img.shields.io/badge/Kometa-FF4081?style=flat-square&logoColor=white" />
24+
<img src="https://img.shields.io/badge/rclone-2596be?style=flat-square&logo=rclone&logoColor=white" />
2425
<img src="https://img.shields.io/badge/macOS-000000?style=flat-square&logo=apple&logoColor=white" />
2526
<br><br>
2627
<img src="https://img.shields.io/github/stars/liamvibecodes/mac-media-stack-advanced?style=flat-square&color=yellow" />
@@ -55,6 +56,7 @@ New to self-hosted media? Start with the [basic version](https://github.com/liam
5556
| **Intro Skipper** | Jellyfin plugin: auto-detects intros and adds a "Skip Intro" button on TV shows |
5657
| **TMDb Box Sets** | Jellyfin plugin: auto-creates franchise collections from TMDb data (Jellyfin's Kometa) |
5758
| **Jellystat** | Jellyfin analytics dashboard (like Tautulli for Plex: watch history, user stats, library insights) |
59+
| **Cloud Storage** | rclone + mergerfs: transparent cloud/local merged library (Google Drive, S3, B2, Dropbox) |
5860

5961
## Choosing Your Media Server
6062

@@ -94,6 +96,62 @@ After the stack is running:
9496

9597
Jellystat will start tracking watch history, user activity, and library statistics automatically.
9698

99+
## Optional: Cloud Storage (rclone + mergerfs)
100+
101+
Transparent cloud/local merged library. rclone FUSE-mounts your cloud storage inside Docker (where FUSE works natively), mergerfs overlays it with local storage, and all existing services see a single unified path. Local-first writes keep downloads fast; a periodic upload script moves stable media to the cloud.
102+
103+
Supports Google Drive, S3, Backblaze B2, Dropbox, and [40+ other providers](https://rclone.org/overview/).
104+
105+
### Compatibility Requirements (macOS)
106+
107+
- Set `MEDIA_SERVER=jellyfin` for cloud-backed playback.
108+
- Set `TDARR_MODE=docker` if you use Tdarr with cloud storage.
109+
- Native macOS apps (like Plex app and native Tdarr) cannot read Docker FUSE merged mounts directly.
110+
111+
### Quick Start
112+
113+
```bash
114+
bash scripts/setup-cloud-storage.sh
115+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile jellyfin --profile tdarr-docker up -d
116+
```
117+
118+
If you also use optional profiles, append them to the same command (example: `--profile jellyfin`, `--profile music`, `--profile tdarr-docker`).
119+
120+
Or include it in bootstrap:
121+
122+
```bash
123+
bash bootstrap.sh --cloud-storage
124+
```
125+
126+
### How It Works
127+
128+
```
129+
rclone-mount --> FUSE mounts cloud provider to /cloud
130+
mergerfs --> overlays /local + /cloud into /merged
131+
Radarr/Sonarr --> read/write /merged (local-first writes)
132+
cloud-upload --> every 6h, moves files >24h old to cloud
133+
```
134+
135+
### Library paths when cloud storage is enabled
136+
137+
- **Jellyfin (Docker):** keep `/data/movies` and `/data/tvshows` (override maps those to merged paths).
138+
- **Tdarr Docker mode:** keep `/movies` and `/tv` (override maps those to merged paths).
139+
140+
### Configuration
141+
142+
| Variable | Default | Description |
143+
|----------|---------|-------------|
144+
| `CLOUD_STORAGE_ENABLED` | `false` | Set to `true` to enable |
145+
| `RCLONE_REMOTE` | (required) | Remote name from rclone.conf |
146+
| `RCLONE_REMOTE_PATH` | (empty) | Subfolder on remote |
147+
| `RCLONE_VFS_CACHE_MODE` | `full` | VFS cache mode (full recommended) |
148+
| `RCLONE_VFS_CACHE_MAX_SIZE` | `50G` | Max local cache size |
149+
| `RCLONE_VFS_CACHE_MAX_AGE` | `72h` | Max cache age |
150+
| `RCLONE_VFS_READ_CHUNK_SIZE` | `128M` | Read chunk size |
151+
| `CLOUD_UPLOAD_MIN_AGE_HOURS` | `24` | Only upload files older than this |
152+
153+
**Cloud mode note:** Use `TDARR_MODE=docker` when cloud storage is enabled. Native Tdarr cannot access merged Docker FUSE mounts on macOS.
154+
97155
## Optional: Music (Lidarr + Tidarr)
98156

99157
| Service | What It Does |
@@ -108,6 +166,12 @@ bash scripts/setup-music.sh
108166
docker compose --profile music up -d
109167
```
110168

169+
If cloud storage is enabled, use:
170+
171+
```bash
172+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile music up -d
173+
```
174+
111175
Then open Tidarr at `http://localhost:8484` to authenticate with your Tidal account, and Lidarr at `http://localhost:8686` to configure your music library. See the [Music Setup](#music-setup) section below for details.
112176

113177
## Automation
@@ -121,6 +185,7 @@ Then open Tidarr at `http://localhost:8484` to authenticate with your Tidal acco
121185
| Log prune | Daily | Removes log files older than 30 days |
122186
| Franchise sort | Optional/manual | Sorts franchise collection movies by release date in Plex (Plex only) |
123187
| VPN failover | Every 2 min (optional) | Auto-switches between ProtonVPN and NordVPN on sustained failure |
188+
| Cloud upload | Every 6 hours (optional) | Moves local media older than 24h to cloud storage |
124189
| Watchtower | Daily at 04:00 (optional) | Auto-pulls latest container images and recreates updated services |
125190

126191
Franchise sorting is kept manual by default because it requires your Plex token:
@@ -290,6 +355,8 @@ bash scripts/vpn-mode.sh nord
290355
| `scripts/vpn-mode.sh` | Manual VPN provider switcher |
291356
| `scripts/vpn-failover-watch.sh` | Automatic VPN failover daemon |
292357
| `scripts/run-kometa.sh` | Trigger Kometa metadata run |
358+
| `scripts/setup-cloud-storage.sh` | Sets up rclone + mergerfs cloud storage integration |
359+
| `scripts/cloud-upload.sh` | Periodic upload of local media to cloud storage |
293360
| `scripts/setup-music.sh` | Creates music directories and Tidarr config (optional) |
294361
| `scripts/log-prune.sh` | Prunes old log files (30-day default retention) |
295362
| `scripts/franchise-sort.py` | Auto-sorts franchise collections in Plex by release date (Plex only) |
@@ -340,6 +407,12 @@ docker compose --profile music up -d
340407
# Host: gluetun, Port: 8080
341408
```
342409

410+
If cloud storage is enabled, use this start command in step 2 instead:
411+
412+
```bash
413+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile music up -d
414+
```
415+
343416
### Tidarr Download Config
344417

345418
The setup script creates a default `tiddl` config at `<MEDIA_DIR>/config/tidarr/.tiddl/config.toml` (default `<MEDIA_DIR>` is `~/Media`). Key settings:
@@ -388,6 +461,12 @@ docker compose --profile music stop lidarr tidarr
388461
export COMPOSE_PROFILES=music
389462
```
390463

464+
If cloud storage is enabled, include both compose files and profiles when starting music services:
465+
466+
```bash
467+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile music up -d
468+
```
469+
391470
## Media Archiving
392471

393472
If you're running out of space on your primary drive, the archive script moves old or watched media to an external drive. Dry-run by default so you can preview what would be moved before committing.

SETUP.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,7 @@ Replace `YOUR_PLEX_TOKEN` and `YOUR_TMDB_API_KEY`, then save.
268268
- Movies source: `/movies`
269269
- TV source: `/tv`
270270
- Temp/cache: `/temp`
271+
- If `CLOUD_STORAGE_ENABLED=true`, use `TDARR_MODE=docker` so Tdarr can read merged cloud paths.
271272
3. Assign the preloaded flow: `Quality-First HEVC (Resolution Preserving)`
272273
4. Keep the defaults in that flow for quality-first behavior:
273274
- No resolution downscale
@@ -334,6 +335,65 @@ Everything else is fully automated.
334335

335336
---
336337

338+
## Optional: Cloud Storage (rclone + mergerfs)
339+
340+
If your local disk is too small for your full library, cloud storage lets you extend it transparently. rclone mounts your cloud provider (Google Drive, S3, B2, Dropbox, etc.) inside Docker, and mergerfs overlays it with your local storage so all services see one unified library.
341+
342+
### Compatibility (macOS)
343+
344+
- Cloud-backed playback requires `MEDIA_SERVER=jellyfin`.
345+
- If using Tdarr with cloud storage, use `TDARR_MODE=docker`.
346+
- Native macOS apps (Plex app, native Tdarr) cannot directly read Docker FUSE merged mounts.
347+
348+
### Setup
349+
350+
```bash
351+
bash scripts/setup-cloud-storage.sh
352+
```
353+
354+
The setup wizard will:
355+
1. Create the required directories
356+
2. Help you configure an rclone remote (interactive Docker wizard, copy existing conf, or skip)
357+
3. Write `CLOUD_STORAGE_ENABLED=true` and `RCLONE_REMOTE` to your `.env`
358+
4. Create `Movies` and `TV Shows` folders on your remote
359+
360+
### Start with cloud storage
361+
362+
```bash
363+
docker compose -f docker-compose.yml -f docker-compose.cloud-storage.yml --profile cloud-storage --profile jellyfin --profile tdarr-docker up -d
364+
```
365+
366+
Or pass `--cloud-storage` to `bootstrap.sh` for a full install.
367+
368+
### How it works
369+
370+
- **Write path:** Downloads land on local disk (fast). Every 6 hours, `cloud-upload.sh` moves files older than 24h to the cloud via `rclone move`.
371+
- **Read path:** Docker services (Jellyfin, Arr apps, Tdarr Docker) read from the mergerfs overlay, which transparently serves files from local or cloud storage.
372+
- **VFS cache:** rclone caches recently accessed cloud files locally (default 50GB, 72h max age). Adjust `RCLONE_VFS_CACHE_MAX_SIZE` in `.env` based on your available disk space.
373+
374+
### Library paths when cloud storage is enabled
375+
376+
- **Jellyfin (Docker):** keep default container paths `/data/movies` and `/data/tvshows` (mapped to merged paths by compose override).
377+
- **Tdarr Docker mode:** keep `/movies` and `/tv` (mapped to merged paths by compose override).
378+
379+
### Troubleshooting cloud storage
380+
381+
```bash
382+
# Check rclone and mergerfs health
383+
bash scripts/health-check.sh
384+
385+
# View cloud upload logs
386+
tail -50 ~/Media/logs/cloud-upload.log
387+
388+
# Test rclone connectivity
389+
docker exec rclone-mount rclone ls myremote: --max-depth 1
390+
391+
# Restart cloud storage (order matters: rclone first, then mergerfs)
392+
docker restart rclone-mount && sleep 15 && docker restart mergerfs
393+
```
394+
395+
---
396+
337397
## Troubleshooting
338398

339399
**Check overall health:**

0 commit comments

Comments
 (0)