Skip to content

Commit b243142

Browse files
authored
feat(nextcloud): trigger endpoint for bulk migrations (#4728)
## Summary Adds `POST /remote/nextcloud/migration` so the Settings UI can start a Nextcloud-to-Cozy bulk migration end-to-end. The endpoint probes the supplied credentials, upserts an `io.cozy.accounts` document with the resolved WebDAV user ID, creates an `io.cozy.nextcloud.migrations` tracking document in `pending` state, and publishes a `nextcloud.migration.requested` command to the `migration` RabbitMQ exchange. A separate service consumes the command and drives the existing `/remote/nextcloud/:account/*` routes, writing progress back to the tracking document so the UI can render a real-time progress bar. Only one migration per instance can be in flight at a time; failed migrations do not block retries. ## API `POST /remote/nextcloud/migration` (requires `POST io.cozy.nextcloud.migrations`) ```json { "nextcloud_url": "https://nextcloud.example.com", "nextcloud_login": "alice", "nextcloud_app_password": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx", "source_path": "/", "target_dir": "/Nextcloud" } ``` `source_path` and `target_dir` are optional. `source_path` defaults to `/`. `target_dir` defaults to `/Nextcloud` and must be a clean absolute path (validated with `path.Clean`, so `..`, double slashes, and relative segments are rejected). `nextcloud_app_password` should be a Nextcloud app password, not the account password. ### Error responses | Code | When | | ---- | ---- | | 400 Bad Request | Required fields missing or `target_dir` is not a clean absolute path | | 401 Unauthorized | The Nextcloud server rejected the supplied credentials | | 409 Conflict | A `pending` or `running` migration already exists for this instance | | 500 Internal Server Error | Account upsert or tracking document creation failed | | 502 Bad Gateway | The Nextcloud instance is unreachable | | 503 Service Unavailable | RabbitMQ publish failed; the tracking document is marked `failed` before returning | ## Companion route: recursive size `GET /remote/nextcloud/:account/size/*path` returns the recursive byte total of a Nextcloud folder via a single PROPFIND on `oc:size`. The migration consumer uses it as a pre-flight quota check so it knows the true source size before starting; the previous shallow-sum estimate could be off by several orders of magnitude on deep trees. ## RabbitMQ contract Exchange `migration`, routing key `nextcloud.migration.requested`. The consumer is responsible for declaring its queue and binding. ```json { "migrationId": "d4e5f6a7b8c94d0ea1b2c3d4e5f6a7b8", "workplaceFqdn": "alice.cozy.example.com", "accountId": "a1b2c3d4e5f6", "sourcePath": "/", "timestamp": 1712563200 } ``` Credentials are never in the payload; they live in the `io.cozy.accounts` document referenced by `accountId`. `target_dir` is read by the consumer from the tracking document so legacy messages and new ones both flow through the same path. `MessageID` is set to the migration ID for cross-system tracing. ## Credentials probe The OCS probe itself is not new. It already existed as a lazy fallback behind `(nc *NextCloud).fetchUserID()`, called from `fillUserID` only when an existing account document was missing its cached `webdav_user_id`. In practice that path was almost never exercised, so a latent bug sat there undetected: the probe targeted `apps/user_status`, and any non-200 (including a 404 from a managed Nextcloud host that strips the optional `user_status` app) was turned into `webdav.ErrInvalidAuth`. What this PR adds is the synchronous, user-facing path: the migration trigger endpoint calls `FetchUserIDWithCredentials` before persisting anything, so the probe is now on the critical path of every trigger. That exposure turned the latent classification bug into a user-visible 401 for anyone pointing the feature at a managed Nextcloud like `thegood.cloud`. A follow-up commit on this branch fixes it by switching the probe to OCS Core (`/ocs/v2.php/cloud/user`, which cannot be disabled by an admin) and narrowing the auth-failure classification to 401/403 only, so other non-2xx statuses bubble up with their real cause. Both code paths (the old lazy fallback and the new synchronous call) benefit from the fix. Validated end-to-end against a real Nextcloud instance: the probe, the encrypted-at-rest account persistence, the tracking document, the `/size/*path` pre-flight, the RabbitMQ publish with no credentials in the payload, and the consumer-driven transfer all behave as documented.
2 parents 84a60d7 + cb91de2 commit b243142

18 files changed

Lines changed: 1969 additions & 30 deletions

docs/nextcloud.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -579,3 +579,138 @@ Authorization: Bearer eyJhbG...
579579
```http
580580
HTTP/1.1 204 No Content
581581
```
582+
583+
## GET /remote/nextcloud/:account/size/*path
584+
585+
This route returns the recursive size in bytes of a folder (or a single
586+
file) on the NextCloud account. It is backed by a single Depth:0 PROPFIND
587+
asking for the `oc:size` property that NextCloud maintains in its metadata
588+
table, so it runs in constant time regardless of how many files are in the
589+
tree. Pass the account root by using an empty `*path` (just `/size/`).
590+
591+
The `:account` parameter is the identifier of the NextCloud `io.cozy.account`.
592+
593+
**Note:** a permission on `GET io.cozy.files` is required to use this route.
594+
595+
### Request (sub-folder)
596+
597+
```http
598+
GET /remote/nextcloud/4ab2155707bb6613a8b9463daf00381b/size/Photos HTTP/1.1
599+
Host: cozy.example.net
600+
Authorization: Bearer eyJhbG...
601+
```
602+
603+
### Response
604+
605+
```http
606+
HTTP/1.1 200 OK
607+
Content-Type: application/json
608+
```
609+
610+
```json
611+
{
612+
"size": 5656463
613+
}
614+
```
615+
616+
### Request (account root)
617+
618+
```http
619+
GET /remote/nextcloud/4ab2155707bb6613a8b9463daf00381b/size/ HTTP/1.1
620+
Host: cozy.example.net
621+
Authorization: Bearer eyJhbG...
622+
```
623+
624+
#### Status codes
625+
626+
- 200 OK, with the recursive byte total of the target path
627+
- 401 Unauthorized, when the NextCloud credentials are rejected
628+
- 404 Not Found, when the account or the target path does not exist
629+
630+
## POST /remote/nextcloud/migration
631+
632+
This route triggers a one-shot bulk migration of a user's Nextcloud files into
633+
their Cozy. The Stack validates the credentials, persists an
634+
`io.cozy.accounts` document, creates an `io.cozy.nextcloud.migrations`
635+
tracking document in `pending` state, and publishes a
636+
`nextcloud.migration.requested` command to the `migration` RabbitMQ exchange.
637+
The actual transfer is performed by an external migration service that
638+
consumes the command and drives the existing `/remote/nextcloud/:account/*`
639+
routes, updating the tracking document as it progresses.
640+
641+
Before persisting anything, the Stack probes the supplied credentials against
642+
the Nextcloud instance via the OCS `user_status` endpoint, so wrong passwords
643+
and unreachable hosts surface synchronously instead of being deferred to the
644+
migration service. The probe also resolves the WebDAV user ID, which is
645+
cached on the account document so the migration service does not need to
646+
re-fetch it.
647+
648+
When an existing `io.cozy.accounts` document for the same `account_type:
649+
"nextcloud"` + `auth.url` + `auth.login` triplet is found, it is reused with
650+
its stored password and `webdav_user_id` refreshed from the request. Only one
651+
migration can be in flight per instance at a time: if a `pending` or `running`
652+
tracking document already exists, the Stack returns `409 Conflict`. Failed
653+
migrations do not block new attempts.
654+
655+
**Note:** a permission on `POST io.cozy.nextcloud.migrations` is required to
656+
use this route.
657+
658+
### Request
659+
660+
```http
661+
POST /remote/nextcloud/migration HTTP/1.1
662+
Host: cozy.example.net
663+
Authorization: Bearer eyJhbG...
664+
Content-Type: application/json
665+
```
666+
667+
```json
668+
{
669+
"nextcloud_url": "https://nextcloud.example.com",
670+
"nextcloud_login": "alice",
671+
"nextcloud_app_password": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx",
672+
"source_path": "/"
673+
}
674+
```
675+
676+
`source_path` is optional and defaults to `/`. The `nextcloud_app_password`
677+
should be a Nextcloud app password, not the user's main account password.
678+
679+
### Response
680+
681+
```http
682+
HTTP/1.1 201 Created
683+
Content-Type: application/vnd.api+json
684+
```
685+
686+
```json
687+
{
688+
"data": {
689+
"id": "d4e5f6a7b8c94d0ea1b2c3d4e5f6a7b8",
690+
"type": "io.cozy.nextcloud.migrations",
691+
"attributes": {
692+
"status": "pending",
693+
"target_dir": "/Nextcloud",
694+
"progress": {
695+
"files_imported": 0,
696+
"files_total": 0,
697+
"bytes_imported": 0,
698+
"bytes_total": 0
699+
},
700+
"errors": [],
701+
"skipped": [],
702+
"started_at": null,
703+
"finished_at": null
704+
}
705+
}
706+
}
707+
```
708+
709+
#### Status codes
710+
711+
- 201 Created, when the migration has been queued and the tracking document is returned
712+
- 401 Unauthorized, when the Nextcloud credentials are rejected by the remote host
713+
- 409 Conflict, when a `pending` or `running` migration already exists
714+
- 500 Internal Server Error, when the conflict check, account upsert, or tracking document creation fails
715+
- 502 Bad Gateway, when the Nextcloud instance is unreachable
716+
- 503 Service Unavailable, when the migration command cannot be published to RabbitMQ. The tracking document is marked `failed` before returning

docs/rabbitmq.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,46 @@ rabbitmq:
134134
- If queue-level `dlx_name`/`dlq_name` are not specified, exchange-level defaults are used.
135135
- Messages that exceed the `delivery_limit` or are rejected will be sent to the DLX and routed to the DLQ.
136136

137+
### Publishers
138+
139+
The Stack also publishes messages to RabbitMQ. Publishers do not declare any
140+
queue or exchange on the Stack side: the exchange must already exist on the
141+
broker, and a queue must be bound by the consuming service. Publishes use the
142+
AMQP `mandatory` flag, so a publish to an exchange with no matching binding
143+
fails with `PublishReturnedError` and the caller is expected to surface the
144+
failure to the user.
145+
146+
#### `auth` exchange
147+
148+
Routing key: `user.deletion.requested`. Published from
149+
`POST /settings/instance/deletion/force` when a user requests permanent
150+
deletion of their account. The payload is the `UserDeletionRequestedMessage`
151+
struct in `pkg/rabbitmq/contracts.go`.
152+
153+
#### `migration` exchange
154+
155+
Routing key: `nextcloud.migration.requested`. Published from
156+
`POST /remote/nextcloud/migration` when a user starts a Nextcloud-to-Cozy
157+
bulk migration. The payload is the `NextcloudMigrationRequestedMessage`
158+
struct in `pkg/rabbitmq/contracts.go`:
159+
160+
```json
161+
{
162+
"migrationId": "d4e5f6a7b8c94d0ea1b2c3d4e5f6a7b8",
163+
"workplaceFqdn": "alice.cozy.example.com",
164+
"accountId": "a1b2c3d4e5f6",
165+
"sourcePath": "/",
166+
"timestamp": 1712563200
167+
}
168+
```
169+
170+
Credentials are never in the payload: they live in the `io.cozy.accounts`
171+
document referenced by `accountId`. The Stack populates `MessageID` with the
172+
migration ID for cross-system tracing. The consuming service is responsible
173+
for declaring its queue, binding it to this exchange, and processing the
174+
messages; if no queue is bound when the Stack publishes, the user receives
175+
`503` and the tracking document is marked `failed`.
176+
137177
### Handlers
138178

139179
Handlers implement a simple interface:

0 commit comments

Comments
 (0)