Sync to custom S3 apis

[Juliaria08]

Similar to the Azure sync that would be implemented in #369, the GCP sync implemented in GothenburgBitFactory/taskwarrior#3185, and the AWS sync implemented in #368, we should be able to sync replicas to a S3-compatible object storages like MinIO, seaweedfs, Garage, Versity's S3 Gateway, or even more-enterprise solutions like Tigris.

I'm not sure how easy it'd be to allow the use of a custom endpoint, nor if it's already implemented by using the aws cli client option, but I tried setting the AWS_ENDPOINT_URL environment-variable to https://play.min.io:9000, having the aws client configured properly, and it didn't work IIRC.

[djmitche]

This is a good idea! If you can figure out how to do it with the normal AWS shell command (profile config??), it should be pretty easy to replicate that in the TW config options.

[Juliaria08]

This is a good idea! If you can figure out how to do it with the normal AWS shell command (profile config??), it should be pretty easy to replicate that in the TW config options.

The docs say to use aws --endpoint-url http://server.url.here.tld.invalid s3 ls

Suppposedly, there's also a way to do it by setting environment variables, or editing the config (manually?): https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/

[djmitche]

It looks like this can be included in a profile: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-config-endpoint_url

If that works, then that might be the best option -- we can document how to set up the profile, but avoid adding purpose-specific config to TaskChampion and Taskwarrior.

If that doesn't work, could you check if a patch like this works?

diff --git src/server/cloud/aws.rs src/server/cloud/aws.rs
index 264b6824b..3c64460c3 100644
--- src/server/cloud/aws.rs
+++ src/server/cloud/aws.rs
@@ -67,16 +67,17 @@ impl AwsService {
         bucket: String,
         creds: AwsCredentials,
     ) -> Result<Self> {
         let rt = Runtime::new()?;
 
         let config =
             rt.block_on(async {
                 let mut config_provider = aws_config::defaults(BehaviorVersion::v2024_03_28());
+                config_provider = config_provider.endpoint_url("https://foo.com");
                 match creds {
                     AwsCredentials::AccessKey {
                         access_key_id,
                         secret_access_key,
                     } => {
                         config_provider = config_provider.credentials_provider(
                             Credentials::from_keys(access_key_id, secret_access_key, None),

[Juliaria08]

It looks like this can be included in a profile: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-config-endpoint_url

If that works, then that might be the best option -- we can document how to set up the profile, but avoid adding purpose-specific config to TaskChampion and Taskwarrior.

It doesn't seem to work. As I get a "InvalidAccessKeyId" message, and I don't see any requests in my server's log.

If that doesn't work, could you check if a patch like this works?

I checked, and with the patch it behaves as if I were to have set the AWS_ENDPOINT_URL, which doesn't seem to work because of dns, but dns works because the official client seems to work?

$ ./task sync   
unhandled error: dispatch failure: io error: error trying to connect: dns error: failed to lookup address information: No address associated with hostname: dns error: failed to lookup address information: No address associated with hostname: failed to lookup address information: No address associated with hostname
$ AWS_ENDPOINT_URL=https://s3.tilde.green task sync
unhandled error: dispatch failure: io error: error trying to connect: dns error: failed to lookup address information: No address associated with hostname: dns error: failed to lookup address information: No address associated with hostname: failed to lookup address information: No address associated with hostname

I've noticed that aws's endpoint-url in the cli behaves differently than when used with the aws crate like taskwarrior is doing, because somehow I'm getting a dns query for bucket.s3.tilde.green, when I have the endpoint as s3.tilde.green.

Ok, there's two ways to handle a endpoint, one is path style and the other is subdomain style, and by default minio does path style, whilst things like the aws crate do subdomain style, but the aws cli tool seemed to do path style, so maybe there needs to be another knob for that.

See: awslabs/aws-sdk-rust#390

[Juliaria08]

So, after configuring my minio instance, and setting it up to have subdomain style, it works, it seems to have made 5 objects, taking up 2MiB.

[djmitche]

OK, it sounds like the earlier tests were with an incorrect configuration of minio, then? Could you verify whether the profile approach works?

[Juliaria08]

OK, it sounds like the earlier tests were with an incorrect configuration of minio, then? Could you verify whether the profile approach works?

Sorry, I wasn't clear enough. The profile approach doesn't work, it only works when I either recompile taskwarrior with the modified taskchampion applying your patch, OR I set the AWS_ENDPOINT_URL environment variable.

In both of the second cases where it works, it uses subdomain style, not subpath style but I guess implementing support for subpath style is not needed, but if wanted look at awslabs/aws-sdk-rust#390 (comment), in concrete https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/tests/endpoints.rs#L36-L43 is the test)

[djmitche]

OK, sounds like we'll need to add an endpoint URL config option, as requiring that an env variable be set when running task sync (much less requiring it when using taskchampion algorithmically) is pretty bad UX. I'm not sure what will be the best way to accomplish that in a non-breaking fashion. Perhaps adding a new AwsEndpoint variant to ServerConfig?

[Juliaria08]

OK, sounds like we'll need to add an endpoint URL config option, as requiring that an env variable be set when running task sync (much less requiring it when using taskchampion algorithmically) is pretty bad UX. I'm not sure what will be the best way to accomplish that in a non-breaking fashion. Perhaps adding a new AwsEndpoint variant to ServerConfig?

Yes, I guess? I'm not too well versed into this, so I'm not sure, but I guess your idea LGTM