Support updating Account info in PutMetadata API by welansari · Pull Request #5762 · scality/cloudserver

welansari · 2025-03-18T11:25:47Z

Update Account info when the account id is provided. In S3C, Backbeat is the one that calling this API and replacing the info in the MD before sending them, but in Zenko we have to delegate this task to the replication destination's Cloudserver as Vault Admin APIs are not accessible externally.
Passed an additional flag to the MongoClient's PutMetadata to update the master to the latest version each time we put an object.
Re-enabled BackbeatRoutes CRR tests and added additional tests to cover the changes made.
Generate coverage in functional tests

bert-e · 2025-03-18T11:25:51Z

Hello kerkesni,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2025-03-18T11:26:45Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

codecov · 2025-03-18T13:09:26Z

Codecov Report

Attention: Patch coverage is 91.17647% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.69%. Comparing base (32a1ae8) to head (089611f).
Report is 11 commits behind head on development/9.0.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/routes/routeBackbeat.js	90.90%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/auth/streamingV4/V4Transform.js	`89.60% <100.00%> (+11.19%)`	⬆️
lib/routes/routeBackbeat.js	`44.33% <90.90%> (+10.99%)`	⬆️

... and 59 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.0    #5762      +/-   ##
===================================================
+ Coverage            65.44%   73.69%   +8.24%     
===================================================
  Files                  186      186              
  Lines                11699    11718      +19     
===================================================
+ Hits                  7656     8635     +979     
+ Misses                4043     3083     -960

Flag	Coverage Δ
ceph-backend-test	`44.23% <85.29%> (?)`
file-ft-tests	`47.50% <29.41%> (?)`
kmip-ft-tests	`26.44% <0.00%> (?)`
mongo-v0-ft-tests	`47.83% <29.41%> (?)`
mongo-v1-ft-tests	`47.84% <29.41%> (?)`
multiple-backend	`31.97% <82.35%> (?)`
quota-tests	`32.09% <0.00%> (?)`
quota-tests-inflights	`34.06% <0.00%> (?)`
unit	`65.51% <73.52%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

francoisferrand · 2025-03-19T18:10:48Z

Update Account info when the account id is provided. In S3C, Backbeat is the one that calling this API and replacing the info in the MD before sending them, but in Zenko we have to delegate this task to the replication destination's Cloudserver as Vault Admin APIs are not accessible externally.

is there a reason to keep the existing S3C behavior (i.e. mapping done in backbeat, on the source), or should we move S3C to this new approach as well?

francoisferrand · 2025-03-19T18:24:59Z

should this target 9.0, or a (to be created before merging) 9.1 ?

bert-e · 2025-03-21T09:20:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

welansari · 2025-03-21T09:49:02Z

is there a reason to keep the existing S3C behavior (i.e. mapping done in backbeat, on the source), or should we move S3C to this new approach as well?

There's nothing keeping us from doing it aside from updating both Cloudserver and Backbeat in S3C.

should this target 9.0, or a (to be created before merging) 9.1 ?

No need to do as no breaking change was introduced. PutMetadata is only used to update existing objects in Zenko so the repairMaster flag will never be used.

ghost · 2025-03-21T10:15:16Z

+                            // The new data location is set to null when archiving to a Cold site.
+                            // In that case "removing old data location key" is handled by the lifecycle
+                            // transition processor.
+                            omVal.location && Array.isArray(omVal.location) &&


Can be simplified?

Suggested change

omVal.location && Array.isArray(omVal.location) &&

Array.isArray(omVal.location) &&

ghost · 2025-03-21T10:17:35Z

+                            async.eachLimit(objMd.location, 5,
+                            (loc, nextEach) => dataWrapper.data.delete(loc, log, err => {


Not related to your changes but IIRC there is the data.batchDelete method from the data wrapper, that might help simplifying your function (not blocking, just to share)

This is needed for Zenko's CRR as vault admin APIs are not exposed externally. Issue: CLDSRV-618

When using the MongoDB backend, we have to pass repairMaster to make it update the master with the latest version, otherwise the master is only updated when the version being put has the same versionId as the master. Issue: CLDSRV-618

- Updated tests to perform cross-account CRR - Made bucket names unique per test Issue: CLDSRV-618

Issue: CLDSRV-618

Issue will be investigated in CLDSRV-626 This was triggered by using the MongoDB metadata backend instead of the memory backend. Issue: CLDSRV-618

Issue: CLDSRV-618

welansari · 2025-03-21T12:44:56Z

/approve

bert-e · 2025-03-21T12:54:53Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.0

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8

Please check the status of the associated issue CLDSRV-618.

Goodbye kerkesni.

The following options are set: approve

scality deleted a comment from bert-e Mar 18, 2025

welansari force-pushed the improvement/CLDSRV-618 branch from 1a13e70 to 29bffcf Compare March 18, 2025 13:05

welansari force-pushed the improvement/CLDSRV-618 branch 5 times, most recently from 444ed9f to 2bc936b Compare March 18, 2025 16:18

scality deleted a comment from bert-e Mar 18, 2025

welansari force-pushed the improvement/CLDSRV-618 branch 9 times, most recently from e99b461 to e9a23ed Compare March 19, 2025 17:12

welansari changed the title ~~Support CRR In Zenko~~ Support updating Account info in PutMetadata API Mar 19, 2025

welansari mentioned this pull request Mar 19, 2025

support getCanonicalIdsByAccountIds in vault helper class scality/Arsenal#2318

Merged

francoisferrand reviewed Mar 19, 2025

View reviewed changes

Comment thread lib/routes/routeBackbeat.js Outdated

francoisferrand reviewed Mar 19, 2025

View reviewed changes

Comment thread lib/routes/routeBackbeat.js Outdated

francoisferrand reviewed Mar 19, 2025

View reviewed changes

Comment thread tests/multipleBackend/routes/routeBackbeat.js

ghost reviewed Mar 20, 2025

View reviewed changes

Comment thread lib/routes/routeBackbeat.js Outdated

Comment thread lib/routes/routeBackbeat.js

Comment thread .github/workflows/tests.yaml Outdated

welansari force-pushed the improvement/CLDSRV-618 branch 2 times, most recently from aa4b6e1 to 44318dc Compare March 20, 2025 15:20

welansari force-pushed the improvement/CLDSRV-618 branch 5 times, most recently from f678f27 to 66fe8e4 Compare March 21, 2025 09:20

scality deleted a comment from bert-e Mar 21, 2025

ghost approved these changes Mar 21, 2025

View reviewed changes

BourgoisMickael approved these changes Mar 21, 2025

View reviewed changes

francoisferrand reviewed Mar 21, 2025

View reviewed changes

Comment thread .github/docker/docker-compose.yaml Outdated

francoisferrand reviewed Mar 21, 2025

View reviewed changes

Comment thread .github/workflows/tests.yaml Outdated

francoisferrand reviewed Mar 21, 2025

View reviewed changes

Comment thread .github/workflows/tests.yaml Outdated

francoisferrand approved these changes Mar 21, 2025

View reviewed changes

Kerkesni added 10 commits March 21, 2025 12:49

support replacing account info in PutMetadata

114bd7b

This is needed for Zenko's CRR as vault admin APIs are not exposed externally. Issue: CLDSRV-618

update master when putting metadata for CRR

3f47d2b

When using the MongoDB backend, we have to pass repairMaster to make it update the master with the latest version, otherwise the master is only updated when the version being put has the same versionId as the master. Issue: CLDSRV-618

bump arsenal

5d5c660

enable BackbeatRoutes replication tests

ed3f6ca

- Updated tests to perform cross-account CRR - Made bucket names unique per test Issue: CLDSRV-618

add additional BackbeatRoutes replication tests

23eccf1

Issue: CLDSRV-618

skip null version replication tests

27efae5

Issue will be investigated in CLDSRV-626 This was triggered by using the MongoDB metadata backend instead of the memory backend. Issue: CLDSRV-618

remove duplicate commands from dockerfile

b24f758

Issue: CLDSRV-618

add unit tests

401a990

Issue: CLDSRV-618

add coverage to functional tests

cde97fa

Issue: CLDSRV-618

pass missing argument to authenticateV4Request

80ef863

Issue: CLDSRV-618

welansari force-pushed the improvement/CLDSRV-618 branch from c916742 to 80ef863 Compare March 21, 2025 11:49

bump version to 9.0.6

089611f

Issue: CLDSRV-618

bert-e merged commit 089611f into development/9.0 Mar 21, 2025
19 checks passed

bert-e deleted the improvement/CLDSRV-618 branch March 21, 2025 12:54

	omVal.location && Array.isArray(omVal.location) &&
	Array.isArray(omVal.location) &&

		async.eachLimit(objMd.location, 5,
		(loc, nextEach) => dataWrapper.data.delete(loc, log, err => {

Conversation

welansari commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Mar 18, 2025

Hello kerkesni,

Uh oh!

bert-e commented Mar 18, 2025

Waiting for approval

Uh oh!

codecov bot commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

francoisferrand commented Mar 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

francoisferrand commented Mar 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 21, 2025

Waiting for approval

Uh oh!

welansari commented Mar 21, 2025

Uh oh!

Uh oh!

ghost Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

ghost Mar 21, 2025 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

welansari commented Mar 21, 2025

Uh oh!

bert-e commented Mar 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

welansari commented Mar 18, 2025 •

edited

Loading

codecov bot commented Mar 18, 2025 •

edited

Loading

ghost Mar 21, 2025 •

edited by ghost

Loading