rcat interface

[cbertelli]

I don't know if it is possible with this protocol, but we usually stream the dump from our databases towards the destination like this:

mariadb-dump | xz -6 - | rclone --s3-chunk-size 15M rcat dbdump:db20260629_dmp.xz

in this case the final destination is an s3 bucket, but I would like to use Internxt as a backend. Would it be possible to plan an rcat even for internxt?

[cbertelli]

Maybe the solution is just using the webdav interface, but this is less easy for unattended backup.

[CrispStrobe]

should work now

[cbertelli]

Yes, it does!

$ PGPASSWORD="aPasswordHere" pg_dump -Fc -Z 9 -U my_dba mydb | python3.9 cli.py rcat mydb_20260630.dmp
🔄 Refreshing authentication token...
🔄 DEBUG: Refreshing tokens for one.user@email.com
🔐 DEBUG: Computing Bridge Auth for UserID: $2b$08$etzgU7wFsUBJooA9AK38eeVE6Fadm49jmCEvgPJaYLrgaAHI8SZnq
🎯 Target: '/mydb_20260630.dmp' (parent '/')
✅ Parent folder ready (UUID: 2e2a6828...)
📥 Buffering stdin to a temporary file...
✅ Buffered 1.8 MB from stdin
-> Preparing upload: 'ixt-rcat-_texzdqa.tmp' (1.8 MB) to '/mydb_20260630.dmp'
-> Target does not exist, proceeding with upload
📤 Uploading 'dide_20260630.dmp' (1.8 MB)...
⏱️ Upload timeout: 300s (~5 minutes)
✅ Network upload complete (file id: 6a435cb33ea704d9836f03a5)
✅ File entry created (UUID: 8ae6b169-dbd4-9054-cb72-4e6101afabc3)

 🎉 Upload complete!
    Total time: 0.9s (0.0 minutes)
    Average speed: 2.1 MB/s

-> ✅ Successfully uploaded: mydb_20260630.dmp
🎉 Streamed 1.8 MB to '/mydb_20260630.dmp'

If a password is needed, it doesn't wait for it, it may become a problem if the user is not aware it's needed, even if it's more convenient to pass it in a variable. For this usage it would be better to handle Internxt credentials in this way as well.

[cbertelli]

I see, I didn't notice before, that login lets me use -e and -p for credentials. For security reasons, using env vars could be better.

[CrispStrobe]

indeed. we also support keychain and env params now.

[cbertelli]

Thank you, even for the --password-stdin option in login, it's a very welcome addition for security.
It still needs a separate login action, it would be better to allow an implicit login when piping.
Still it does not show the password request from the database. You don't need to change this behaviour, it is sufficient to state It into rcat help.

[cbertelli]

Even keeping login and rcat as two distinct actions, it works well with "&&" between the phases:

python3.9 cli.py login -e one.user@email.com -p aSecretOne\! && \
PGPASSWORD="aPasswordHere" \
pg_dump -Fc -Z 9 -U my_dba mydb | python3.9 cli.py rcat mydb_20260630a.dmp && \
python3.9 cli.py logout

obviously I need to trap at least the errors for login and rcat.

[CrispStrobe]

great. btw also sped up crypto...

[CrispStrobe]

Shipped a batch of follow-ups based on this thread — all on main (CI green on Python 3.10/3.11/3.12):

Implicit auto-login when piping. You no longer need a separate login step. When there's no valid stored session, commands auto-login from INTERNXT_EMAIL / INTERNXT_PASSWORD (+ INTERNXT_TFA_SECRET for 2FA). It also kicks in transparently when a stored token has expired, so an unattended run won't get stranded mid-pipeline:

INTERNXT_EMAIL=you@example.com INTERNXT_PASSWORD="$PW" \
  PGPASSWORD="$DBPW" pg_dump -Fc mydb | python cli.py rcat /backups/mydb.dmp

Your explicit login && … | rcat … && logout flow still works unchanged.

Quieter / safer logs. The internal auth TRACE/DEBUG lines — which echoed the account email and bridge-auth material on every authenticated command (you saw these in your rcat output) — are now off by default and only emitted to stderr under INTERNXT_DEBUG=1. They no longer pollute stdout or backup logs.

logout now exits non-zero on failure, so your login && rcat && logout chain can properly trap a failed logout.

Docs. rcat --help and the README now state the upstream-password caveat you raised: while piping into rcat the dump tool has no terminal, so it can't prompt for its own DB password — supply it non-interactively via PGPASSWORD / ~/.pgpass (Postgres) or MYSQL_PWD / ~/.my.cnf (MySQL/MariaDB). The auto-login behavior is documented there too.

Tests added for all of the above (env auto-login, expired-token fallback, TOTP-secret 2FA, logout exit code); mypy + bandit clean.

One thing I left out by design: true unknown-size streaming (skipping the temp spool). It'd remove the temp-disk requirement for big dumps, but a pipe can't be re-read, so per-part upload retry on a network blip is lost — spooling stays the reliable default. Happy to add it as an opt-in --size N flag with a warning if that tradeoff is worth it for your setup.

[cbertelli]

Great indeed!
Thank you for everything, for the powerful enhancements and the reasonable steering.