MongoDB Atlas support

[alwynpan]

Description of the feature
Currently docker-db-backup does not support MongoDB Atlas. It would be great to support mongodb+srv:// type of hosts. The simplest way is to use --uri instead of --host with mongodump.

Benefits of feature
By replacing --host with --uri and adding DB_HOST_TYPE (default to mongodb and can be set to mongodb+srv), it makes the mongodump more flexible.

[tiredofit]

Hi there this has been added to version 3.4.0 . Changelog - Please checkout the README to understand how to change it to atlas mode and also the URI override that has been built in. Let me know if you encounter any issues.

[Dinip]

Hi,

Having some issues using this new version.
When specifying MONGO_HOST_TYPE=atlas and setting DB_HOST=clustername.id.mongodb.net, the code from line 379 is preventing the backup from running, since you are trying to resolve a SRV record with nc and it is failing:
Mongo Host 'clustername.id.mongodb.net' is not accessible, retrying.. (5 seconds so far)

[tiredofit]

Thanks for the report. 3.4.2 should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.

[Dinip]

Thanks for the report. 3.4.2 should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.

Thanks! I'll try it :)

[Dinip]

Thanks for the report. 3.4.2 should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.

It wasn't working, so I checked the code again and found: --uri="${MONGO_URI_PREFIX}${DB_HOST}:${DB_PORT}.
When using an SRV record, the port shouldn't be specified, just tried it in a local mongodump to be sure:

In my opinion, it would be nice to have a way, through ENV vars, to specify the full uri just like mongodb+srv://username:password@cluster.id.mongodb.net instead of building it from multiple vars like host, user, password, etc. Maybe check if MONGO_CUSTOM_URI is specified and use it instead of building this: --uri="${MONGO_URI_PREFIX}${DB_HOST}:${DB_PORT}" ${MONGO_USER_STR}${MONGO_PASS_STR}${MONGO_AUTH_STR}${MONGO_DB_STR}.

[tiredofit]

I agree with your opinion. I can do that.
Help me understand, if you send a username and password in the URI - that would negate having to use the Mongo User and Pass variables I have following the URI? A few real world examples would help and I can certainly build something in.

[Dinip]

if you send a username and password in the URI - that would negate having to use the Mongo User and Pass variables I have following the URI?

Yes. The uri can contain the username and password in the format that I mentioned before mongodb+srv://username:password@cluster.id.mongodb.net/defaultDb?options=extraStuff
Here is an example of an uri from mongo compass:

Basically if you accept an URI from the env vars (don't modify it) and pass it through to mongodump, it would allow the user to access any mongodb instance (self-hosted or atlas) and even specify extra options.
For most of us would be the best way to handle mongodb connections, since we use atlas with SRV records and the URI is automatically handled to use by atlas.

Connection manual from mongo

[tiredofit]

Perfect, thanks for the input here. Let me give this a think over the next 24 hours and come up with a way forward.

[alwynpan]

@tiredofit @Dinip, my understanding is we should be able to use mongodb+srv://cluster.id.mongodb.net with the --uri parameter and specify the username and password etc in separate parameters.

I am sorry, I created this issue and planned to implement this myself, but I was dragged into some urgent work and didn't have a chance to do that. Once I finished the urgent job (hopefully by tomorrow), I can help to do testing and debugging.

[tiredofit]

In a377f57 I've added support for a MONGO_CUSTOM_URI environment variable. I had to do some parsing of the URI string for filename generation, but also left the parsed variables in place for other needs if ever down the road.

If you aren't happy with the DB_NAME or DB_HOST of the generated file you can override it.
Instead of pushing a bad release - I've switched over to tiredofit/db-backup:develop if you were willing to pull and test.

[Dinip]

Something must be wrong with this validation if [ "${MONGO_HOST_TYPE,,}" != "atlas" ] || [ -n "${MONGO_CUSTOM_URI}" ]; then
My env vars:

environment:
      - DB_TYPE=mongo
      - MONGO_CUSTOM_URI=mongodb+srv://user_redacted:pass_redacted@cluster.redacted.mongodb.net

[tiredofit]

Thanks for feedback. Pushing new develop with proper variable check.

[Dinip]

Pulled the new docker image and seems to be doing the same 🤔

[tiredofit]

That's bizarre, I had switched it to this:
if [ "${MONGO_HOST_TYPE,,}" != "atlas" ] || [ -z "${MONGO_CUSTOM_URI}" ]

For time being I am going to strip any host checking just to get us to the good stuff.

[Dinip]

The backup now works but is having some errors at the end and it does not upload to the s3 bucket.

Timeline:
At the beginning

[tiredofit/db-backup:develop 19:40:24 /] $ backup-now
** Performing Manual Backup
/assets/functions/10-db-backup: line 462: syntax error near unexpected token `fi'
/assets/functions/10-db-backup: line 462: `    fi'
/etc/services.available/10-db-backup/run: line 56: check_availability: command not found

Reads the data from the mongo instance and writes it to the archive.gz file

At the end

/assets/functions/10-db-backup: line 187: check_exit_code: command not found
/assets/functions/10-db-backup: line 188: generate_checksum: command not found
/assets/functions/10-db-backup: line 189: move_dbbackup: command not found
/assets/functions/10-db-backup: line 190: post_dbbackup: command not found
2022-09-20.19:41:46 [INFO] ** [db-backup] Backup routines finish time: 2022-09-20 19:41:46 GMT with overall exit code 0
2022-09-20.19:41:46 [NOTICE] ** [db-backup] Backup routines time taken: Hours: 0 Minutes: 01 Seconds: 19
/etc/services.available/10-db-backup/run: line 79: cleanup_old_data: command not found

Doesn't upload to S3

[tiredofit]

Thanks, looking into and will have a fix momentarily.

[tiredofit]

Right - Next :develop will fix and work :)

[Dinip]

Yep, now seems to work!
Thanks for the update 😀
If you need help testing after you re-enable the host verification, let me know!

[tiredofit]

Awesome. I'm going to do that one right now - If you wanted to check the new :develop and we pass we should be good to move this to master and have a new release.

[Dinip]

something is weird with that verification 🤔

[tiredofit]

Interesting. Can I get some of the environment variables you are using? I can't seem to recreate this whatsoever.

[Dinip]

Not much of a bash expert but just done some research and seems like -z "$variable" checks if the variable is unset or is set to an empty string. In this case, $MONGO_CUSTOM_URI is defined, so it would return false.

For example, in line 68

 if [ -z "${MONGO_CUSTOM_URI}" ] ; then
    [[ ( -n "${DB_USER}" ) ]] && MONGO_USER_STR=" --username ${DB_USER}"
    [[ ( -n "${DB_PASS}" ) ]] && MONGO_PASS_STR=" --password ${DB_PASS}"
    [[ ( -n "${DB_NAME}" ) ]] && MONGO_DB_STR=" --db ${DB_NAME}"
    [[ ( -n "${DB_AUTH}" ) ]] && MONGO_AUTH_STR=" --authenticationDatabase ${DB_AUTH}"
else

your are checking if custom uri is empty to set the values to the other variables.

With this check

if [ "${MONGO_HOST_TYPE,,}" = "atlas" ] || [ -z "${MONGO_CUSTOM_URI}" ]; then
    print_debug "Skipping Connectivity Check"
else

you are checking if either the host type is atlas or the custom uri is empty to skip the check, but the check for custom uri will return false since it is set.

Made some tests:

But going back to the check that you had before [ -n "${MONGO_CUSTOM_URI}" ] seems to work

[tiredofit]

New :develop is ready - I stripped the || OR Statement as I'm clearly not capable this week of building those :)

[Dinip]

Well...
Here if [ -n "${MONGO_CUSTOM_URI}" ]; then MONGO_DB_TYPE="atlas" ; fi you are setting MONGO_DB_TYPE but it should be MONGO_HOST_TYPE :)

[tiredofit]

Death by environment variables - New :develop being built!