Inserted child record silently dropped when batched with a conflicting parent

[lukaskubanek]

Description

After reporting #274, I’ve encountered another incorrect CloudKit sync behavior. When a device sends a batch containing (a) an update to a parent record and (b) an inserted child record referencing that parent, and the parent has a newer version on the server, the child record is silently dropped and never synced again.

I wasn’t able to reproduce this in any built-in sample apps, so I slightly tweaked the Reminders example. The modification lives on this branch and adds a toolbar button for triggering the database operations. See also this diff.

Here is a step-by-step description of the scenario:

1. Devices A and B both start with an empty reminders list titled “List” with the ID 80ef064a-3dd8-4406-9a7b-548bedd18c65.

2. Both devices go offline.

3. Device B updates the title of the list to “List changed on B”.

UPDATE "remindersLists" 
SET "title" = 'List changed on B'
WHERE ("remindersLists"."id") IN (('80ef064a-3dd8-4406-9a7b-548bedd18c65'))

4. Device A updates the title to “List changed on A” and inserts a new reminder titled “Reminder added on A” with ID 48107186-4283-41a7-ad75-23bbf018438b.

UPDATE "remindersLists"
SET "title" = 'List changed on A'
WHERE ("remindersLists"."id") IN (('80ef064a-3dd8-4406-9a7b-548bedd18c65'))

INSERT INTO "reminders" ("id", "dueDate", "isFlagged", "notes", "position", "priority", "remindersListID", "status", "title")
VALUES (NULL, NULL, 0, '', 0, NULL, '80ef064a-3dd8-4406-9a7b-548bedd18c65', 0, 'Reminder added on A')

5. Both devices go online.

Warning

At this point, a race begins between the two devices. The issue occurs only if device B sends its changes first, and device A attempts to send its changes before fetching B’s updates from the server. Due to the nondeterminism in CKSyncEngine, I haven’t found a reliable way to force this exact ordering, so it may take a few attempts. The following steps assume this sequence.

6. Device B sends its changes to the server.

SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action   ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └──────────┴────────────────┴─────────────────────────────────────────────────────┘

7. Device A then attempts to send its changes in a single batch. Because a newer parent record now exists on the server, the update to the reminders list is rejected with a serverRecordChanged error. However, the save of the newly inserted child reminder is also rejected, this time with a batchRequestFailed error.

SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ reminders      │ 48107186-4283-41a7-ad75-23bbf018438b:reminders      │
  │ ➡️ Sending │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action         ┃ recordType     ┃ recordName                                          ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ 🛑 Save failed │ reminders      │ 48107186-4283-41a7-ad75-23bbf018438b:reminders      │ batchRequestFailed (22)  │
  │ 🛑 Save failed │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │ serverRecordChanged (14) │
  └────────────────┴────────────────┴─────────────────────────────────────────────────────┴──────────────────────────┘

I’m not entirely sure about this behavior, but it appears that CloudKit refuses to save a child record when its parent record is considered outdated on the server.

8. Device A upserts the reminders list from the fetched server record, but because the local title holds a newer timestamp, nothing changes. (Since not all fields were excluded from updates, this schedules another save.)

INSERT INTO "remindersLists" ("id", "color", "position", "title")
VALUES ('80ef064a-3dd8-4406-9a7b-548bedd18c65', 1167716351, 2, 'List changed on B') ON CONFLICT("id") DO UPDATE SET "position" = "excluded"."position"

9. Device A fetches changes from the server and attempts to apply them. Again, as the local reminders list’s title is newer, the one coming from the server isn’t applied.

SQLiteData (private.db) fetchedDatabaseChanges
  ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
  ┃ action      ┃ zoneName                            ┃ ownerName        ┃
  ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
  │ ✅ Modified │ co.pointfree.SQLiteData.defaultZone │ __defaultOwner__ │
  └─────────────┴─────────────────────────────────────┴──────────────────┘

SQLiteData (private.db) fetchedRecordZoneChanges
  ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Modified │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └─────────────┴────────────────┴─────────────────────────────────────────────────────┘

INSERT INTO "remindersLists" ("id", "color", "position", "title")
VALUES ('80ef064a-3dd8-4406-9a7b-548bedd18c65', 1167716351, 2, 'List changed on B')
ON CONFLICT("id") DO UPDATE SET "color" = "excluded"."color", "position" = "excluded"."position"

10. Device A sends the “changed” reminder list to the server.

SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action   ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │
  └──────────┴────────────────┴─────────────────────────────────────────────────────┘

11. Both devices end up with a synced reminders list, but the inserted reminder never makes it to the server. Since it was rejected in the original batch and never retried on its own, it remains unsynced until it’s modified again.

The chosen example in the Reminders sample app may look a bit constructed, but in my project, creating a child record while modifying its parent (with a high likelihood of hitting a conflict) is a common and intentional pattern. This is just a minimal repro to demonstrate the problem. The key issue is that the child insert is permanently dropped unless it’s modified again, which leads to data divergence similar to #274.

Checklist

• I have determined whether this bug is also reproducible in a vanilla SwiftUI project.
• I have determined whether this bug is also reproducible in a vanilla GRDB project.
• If possible, I've reproduced the issue using the main branch of this package.
• This issue hasn't been addressed in an existing GitHub issue or discussion.

SQLiteData version information

main

Sharing version information

2.7.4

GRDB version information

7.8.0

Destination operating system

iOS 26

Xcode version information

26.1 (17B54)

Swift Compiler version information

swift-driver version: 1.127.14.1 Apple Swift version 6.2 (swiftlang-6.2.0.19.9 clang-1700.3.19.1)
Target: arm64-apple-macosx15.0

[lukaskubanek]

Here’s a short video showing the bug in action using two Simulators. The iPhone Air on the left represents device A and the iPhone 16 on the right device B.

sqlitedata-315.mp4

Notes:

• The reminders list title doesn’t refresh in the detail view, so I had to navigate back and forth to update it.
• I also had to force-quit the apps to trigger a sync, but that shouldn’t affect the underlying issue.

[lukaskubanek]

After looking further, this doesn’t seem specific to parent-child relationships. I can reproduce the same behavior with unrelated records batched together. For example, if I combine the conflicting rename of a reminders list with inserting a new, unrelated list, I get:

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action         ┃ recordType     ┃ recordName                                          ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ 🛑 Save failed │ remindersLists │ 80ef064a-3dd8-4406-9a7b-548bedd18c65:remindersLists │ serverRecordChanged (14) │
  │ 🛑 Save failed │ remindersLists │ 9cab7075-4ba5-40d4-a230-c170a0a8cb97:remindersLists │ batchRequestFailed (22)  │
  └────────────────┴────────────────┴─────────────────────────────────────────────────────┴──────────────────────────┘

This looks very much like the batch is treated atomically per CKSyncEngine.RecordZoneChangeBatch.atomicByZone:

/// If set to true, the sync engine will modify these records atomically by zone.
///
/// If this is true, and if any record change fails, then any other changes from that zone 
/// in this batch will also fail with ``CKError/batchRequestFailed``.
///
/// Records that exist in different zones will not be modified together atomically.
public var atomicByZone: Bool

However, when checking in the codebase, this property is false for the batches created in SyncEngine.nextRecordZoneChangeBatch(…), so I’m a bit puzzled why the atomic behavior is effectively enabled.

I’m aware that you need to save a CKShare together with its shared record in a single atomic operation using CKDatabase.modifyRecords(saving:deleting:savePolicy:atomically:), where atomically defaults to true. But that doesn’t appear to be what’s happening here for records synced via CKSyncEngine.

Maybe you have an idea why this send is still treated as atomic.

But regardless of the underlying cause, the failed records should still be handled according to Apple’s guidance for CKError.Code.batchRequestFailed: handle the specific record errors first, and then retry the records that failed only due to the batch. Right now those records are not retried, which is what leads to the drop.

[lukaskubanek]

I wondered why I haven’t run into this issue in my own implementation, so I checked, and even though my CKSyncEngine setup looks very similar to SQLiteData’s, the batches do not seem to be treated atomically. I reproduced the same scenario described above and got the following:

<SentRecordZoneChanges
  savedRecords=[
    <CKRecord List 7B048B5C-5403-48A7-8DEF-3DFF959A10AB etag=r1>
  ]
  failedRecordSaves=[
    <FailedRecordSave
      record=<CKRecord List BA4A403F-11C8-43B5-AB1A-81BD6AD3ACD8 etag=px>
      error=<CKError 'Server Record Changed' (14/2004);
        message = 'client oplock error updating record';
        clientEtag = px;
        serverEtag = qz;
        op = 8C521FE285BA3109;
        uuid = 5413A2E3-1157-4EE6-BB77-7AC7B1940FE6>
    >
  ]
>

Batching two records and hitting .serverRecordChanged on one did not produce .batchRequestFailed on the other, and the unrelated record was saved successfully. In other words, the batch was not treated as atomic. I also verified that explicitly setting atomicByZone to true on the batch does make the batch atomic and reproduces the .batchRequestFailed behavior.

So the question is what causes SQLiteData’s SyncEngine to enforce atomic saves even when atomicByZone is false?

---

I found another pointer in CloudKit Dashboard, which shows true for the Atomic column of the custom zone used by SQLiteData.

At first I thought this might explain the behavior, but when I checked my own implementation, the custom zone I use also has true for Atomic. This seems to be the case for all custom zones. So this alone doesn’t explain why batches behave atomically in SQLiteData but not in my setup.

I poked around in the docs and found conflicting information about what this flag actually represents. I’m not sure whether:

• it’s simply a capability that can be used by APIs that explicitly opt into atomic writes such as CKModifyRecordsOperation or CKDatabase.modifyRecords(…), and possibly also the CKSyncEngine,
• or all custom zones implicitly enforce atomicity for every write.

I tend toward the first interpretation, because I was able to reproduce a clear counterexample in my own implementation. But that still doesn’t explain why SQLiteData behaves as if atomicity is always enabled even though atomicByZone is set to false.

Here are some relevant references:

• WWDC 2014: Advanced CloudKit - Intro to atomic commits
• CKRecordZone.Capabilities.atomic
• CKModifyRecordsOperation.isAtomic
• CKError.Code.batchRequestFailed

---

The more high-level question is whether atomicity is even desirable in this context. My feeling is that it isn’t. For regular sync operations, we can’t rely on meaningful batching or ordering guarantees from CKSyncEngine anyway. Records can arrive in unpredictable sequences, and the overall state will naturally be temporarily incomplete and invariants broken. If I’m not mistaken, SQLiteData already handles partially present data in a robust way, especially when it comes to enforcing foreign keys.

What really matters here is that non-conflicting records aren’t dropped. So to me there are two separate issues to address:

1. Disable atomicity for batched saves (and figure out how to reliably do that).
2. Handle .batchRequestFailed according to Apple’s docs. Records that failed only because another record in the batch failed should be retried rather than silently ignored.

[mbrandonw]

Hi @lukaskubanek, thanks for bringing up this topic and all the detailed information. I wanted to distill this problem down to its essentials without involving SQLiteData or even CKSyncEngine, and here is a repro that can be run in any CloudKit app to see the problem:

Task {
  let containerIdentifier = ModelConfiguration(groupContainer: .automatic).cloudKitContainerIdentifier!
  let container = CKContainer(identifier: containerIdentifier)

  let zone = CKRecordZone(zoneName: "my.custom.zone")
  let record1ID = CKRecord.ID(recordName: "record1", zoneID: zone.zoneID)
  let record2ID = CKRecord.ID(recordName: "record2", zoneID: zone.zoneID)

  do {
    _ = try await container.privateCloudDatabase.modifyRecordZones(saving: [zone], deleting: [])
    let (saveResults, deleteResults) = try await container.privateCloudDatabase
      .modifyRecords(
        saving: [],
        deleting: [record1ID, record2ID]
      )
    print("saveResults", saveResults)
    print("deleteResults", deleteResults)
    print("----------")
  } catch {
    print(error)
    print("------")
  }
  do {
    let record1 = CKRecord(recordType: "record1", recordID: record1ID)
    let record2 = CKRecord(recordType: "record2", recordID: record2ID)

    let (saveResults, deleteResults) = try await container.privateCloudDatabase.modifyRecords(
      saving: [record2, record1],
      deleting: []
    )
    print("saveResults", saveResults)
    print("deleteResults", deleteResults)
    print("----------")
  } catch {
    print(error)
    print("------")
  }

  do {
    let record1 = try await container.privateCloudDatabase.record(for: record1ID)
    let record2 = CKRecord(recordType: "record2", recordID: record2ID)
    record1["updated"] = true
    record2["updated"] = true

    let (saveResults, deleteResults) = try await container.privateCloudDatabase.modifyRecords(
      saving: [record1, record2],
      deleting: []
    )
    print("saveResults", saveResults)
    print("deleteResults", deleteResults)
    print("-------")
  } catch {
    print(error)
    print("------")
  }

  print("DONE")
}

This bit of code does 3 things:

• Creates a custom zone using modifyRecordZones and then deletes record1 and record2 from iCloud using modifyRecords. This delete is just to keep data from previous runs from interfering with later runs, so we always have a clean slate.
• Next we save record1 and record2 in iCloud, and this completes successfully.
• Finally, we fetch record1 and make some mutations to the record, and then we create record2 from scratch and make some mutations, and then we try saving that iCloud.

After the last step, both records fail to save:

[
  .failure(CKError(.serverRecordChanged)),
  .failure(CKError(.batchRequestFailed)),
]

One would expect that record1 saves successfully, and only record2 fails with .serverRecordChanged. So, as you noticed @lukaskubanek, it does seem that this custom zone is being treated atomically even though the zone's capabilities is not atomic.

If we change the modifyRecords to opt out of atomic, then it works as expected:

let (saveResults, deleteResults) = try await container.privateCloudDatabase.modifyRecords(
  saving: [record1, record2],
  deleting: [],
  atomically: false
)

[
  .success
  .failure(CKError(.serverRecordChanged)),
]

And further if we use the default zone (CKRecordZone.default()), then saves are automatically treated non-atomically. There's no need to specify it on modifyRecords calls. (@lukaskubanek: are you sure your code is using a custom zone and not the default? Can you share how you are defining your zone?)

The disappointing thing here is that I don't see any way to customize this behavior if using a CKSyncEngine. We cannot change their calls to modifyRecords to make them non-atomic, and I don't see any mention in the docs of customizing atomicity when using CKSyncEngine. So I think we have no choice but to assume that .batchRequestFailed can occur and deal with it ourselves. (Update: I do see that we should be able to customize this in RecordZoneChangeBatch, but just not sure if it's respected.)

Interestingly Apple's CKSyncEngine sample code does not show that one would need to handle this error, and they do use a custom zone.

[mbrandonw]

@lukaskubanek I am not able to reproduce this locally with SQLiteData or CKSyncEngine. The information I shared above was just me exploring how atomic operations work in general in CloudKit, but things in CKSyncEngine do seem to work correctly to me.

To test I wrote a custom function to execute in the CloudKitDemo app that forces the situation you are describing above. I invoke the following function when a button is tapped in that app:

func trigger() async {
  await withErrorReporting {
    // Step 1: Create a new counter
    let newCounter = try await database.write { db in
      try Counter.insert { Counter.Draft() }
        .returning(\.self)
        .fetchOne(db)!
    }

    // Step 2: Force sending all data to iCloud
    try await syncEngine.sendChanges()

    // Step 3: Grab the CKRecord for the newly inserted counter
    let newCounterLastKnownServerRecord = try await database.read { db in
      try SyncMetadata.find(newCounter.syncMetadataID).select(\.lastKnownServerRecord)
        .fetchOne(db)!!
    }

    // Step 4: Make a change to the counter directly on iCloud
    let container = CKContainer(identifier: ModelConfiguration(groupContainer: .automatic).cloudKitContainerIdentifier!)
    let serverCounter = try await container.privateCloudDatabase.record(for: newCounterLastKnownServerRecord.recordID)
    serverCounter.encryptedValues["count"] = Int.random(in: 1...1_000)
    let (saveResults, _) = try await container.privateCloudDatabase.modifyRecords(saving: [serverCounter], deleting: [])

    // Step 5: Make two changes to the local database: 1) decrement any counter besides the one
    //         created above (should succeed), and 2) increment the counter just created (should
    //         fail due to conflict)
    try await database.write { db in
      try Counter
        .where { $0.id.neq(newCounter.id) }
        .update { $0.count -= 1 }
        .execute(db)
      try Counter
        .find(newCounter.id)
        .update { $0.count += 1 }
        .execute(db)
    }

    try await syncEngine.sendChanges()
  }
}

That creates the circumstances you are describing above. The very last database.write makes a change to a record that should succeed as well as a change to a record that should be challenged with serverRecordChanged. And that is exactly what happens in my logs:

SQLiteData (private.db) nextRecordZoneChangeBatch: manual
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType ┃ recordName                                    ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ counters   │ 649b830b-cd8e-4283-86c3-3e5c80d1738b:counters │
  │ ➡️ Sending │ counters   │ f19bf74b-b597-4ce4-a4ae-13730774d5dc:counters │
  └────────────┴────────────┴───────────────────────────────────────────────┘
SQLiteData (private.db) stateUpdate
SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action         ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ f19bf74b-b597-4ce4-a4ae-13730774d5dc:counters │                          │
  │ 🛑 Save failed │ counters   │ 649b830b-cd8e-4283-86c3-3e5c80d1738b:counters │ serverRecordChanged (14) │
  └────────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

…

SQLiteData (private.db) willSendChanges
SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType ┃ recordName                                    ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ counters   │ 649b830b-cd8e-4283-86c3-3e5c80d1738b:counters │
  └────────────┴────────────┴───────────────────────────────────────────────┘
SQLiteData (private.db) stateUpdate
SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action   ┃ recordType ┃ recordName                                    ┃
  ┡━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved │ counters   │ 649b830b-cd8e-4283-86c3-3e5c80d1738b:counters │
  └──────────┴────────────┴───────────────────────────────────────────────┘

So I am not entirely sure what is going on with your set up, but at least for me CKSyncEngine is working as I would expect.

And FWIW, I am able to replicate the behavior you are seeing when I alter the batch in the SyncEngine to be atomic: batch?.atomicByZone = true. So it does seem like CKSyncEngine is respecting atomicity settings.

[lukaskubanek]

@mbrandonw Thanks for taking a look at this!

it does seem that this custom zone is being treated atomically even though the zone's capabilities is not atomic.

One would need to check the zone’s capabilities after saving it, which should include .atomic, as all custom zones come with it. You should be able to verify by checking the result of try await container.privateCloudDatabase.modifyRecordZones(saving: [zone], deleting: []).

One would expect that record1 saves successfully, and only record2 fails with .serverRecordChanged.

So, in this case, I’d expect to see the .batchRequestFailed error, as modifyRecords(…) defaults its atomically parameter to true, which explicitly marks this operation as atomic. And since the zone has the atomic capability, the operation should be executed as such.

And further if we use the default zone (CKRecordZone.default()), then saves are automatically treated non-atomically. There's no need to specify it on modifyRecords calls.

Yes, this is also expected.

are you sure your code is using a custom zone and not the default? Can you share how you are defining your zone?

I’m sure my code is using a custom zone and there really isn’t much to it. I defer the creation to the point where a record fails to save due to the missing zone, but other than that, it’s just a plain new record zone. Oh, and when it comes back as saved, it has the .atomic capability.

case .zoneNotFound, .userDeletedZone:
  let zone = CKRecordZone(zoneID: failedRecord.zoneID)
  syncEngine.state.add(pendingDatabaseChanges: [.saveZone(zone)])
  // Clear local record data
  // Signal retry of pending record changes

Interestingly Apple's CKSyncEngine sample code does not show that one would need to handle this error, and they do use a custom zone.

I remember studying this sample project back when working on my implementation but I quickly realized it does not cover many real-world edge cases. This has been pointed out several times in the issues, so I didn’t find the sample very helpful.

---

To test I wrote a custom function to execute in the CloudKitDemo app that forces the situation you are describing above. […] So I am not entirely sure what is going on with your set up, but at least for me CKSyncEngine is working as I would expect.

I’ve configured the CloudKitDemo sample app according to your instructions and I can confirm that it works for me just fine. It behaves the same way as my own implementation, meaning no unintended atomicity. I’m only seeing the issue in the Reminders example.

I peeked under the hood to check the CKModifyRecordsOperation that gets created by the CKSyncEngine by swizzling CKOperation.start() and filtering out CKModifyRecordsOperation objects. It does reflect the correct atomicity setting from the sync engine, yet I still run into the .batchRequestFailed error.

Here, two changes to the reminders lists are made.

SQLiteData (private.db) nextRecordZoneChangeBatch: manual
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ remindersLists │ 3ecd5376-d9b5-41b8-b420-5aebf0c48f3d:remindersLists │
  │ ➡️ Sending │ remindersLists │ c52c5f3c-9642-4e3f-8376-414203d82691:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘

This creates a CKModifyRecordsOperation configured in the following way:

CKModifyRecordsOperation
recordIDsToSave=[
  <CKRecordID: 0x6000002f8920; recordName=3ecd5376-d9b5-41b8-b420-5aebf0c48f3d:remindersLists, zoneID=co.pointfree.SQLiteData.defaultZone:__defaultOwner__>,
  <CKRecordID: 0x6000002f9220; recordName=c52c5f3c-9642-4e3f-8376-414203d82691:remindersLists, zoneID=co.pointfree.SQLiteData.defaultZone:__defaultOwner__>
]
recordIDsToDelete=[]
savePolicy=ifServerRecordUnchanged
isAtomic=false

And what follows is the atomic failure for the whole batch:

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action         ┃ recordType     ┃ recordName                                          ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ 🛑 Save failed │ remindersLists │ 3ecd5376-d9b5-41b8-b420-5aebf0c48f3d:remindersLists │ serverRecordChanged (14) │
  │ 🛑 Save failed │ remindersLists │ c52c5f3c-9642-4e3f-8376-414203d82691:remindersLists │ batchRequestFailed (22)  │
  └────────────────┴────────────────┴─────────────────────────────────────────────────────┴──────────────────────────┘

Since Tim asked for the exact error message over in the iOS Folks Slack, here it is. I don’t see anything in there that stands out.

<CKSyncEngine Private> failed sending changes for context <SendChangesContext reason=manual options=<SendChangesOptions scope=all group=CKSyncEngine-SendChanges-Manual groupID=9E7CE5465FB20289>>: Error Domain=CKErrorDomain Code=2 "Failed to send changes" UserInfo={NSLocalizedDescription=Failed to send changes, CKPartialErrors={
    "<CKRecordID: 0x6000002f8920; recordName=3ecd5376-d9b5-41b8-b420-5aebf0c48f3d:remindersLists, zoneID=co.pointfree.SQLiteData.defaultZone:__defaultOwner__>" = "<CKError 0x600000c88d20: \"Server Record Changed\" (14/2004); server message = \"client oplock error updating record\"; clientEtag = 4v; serverEtag = 4x; op = DAF3C063AC9933EB; uuid = A5A9641D-5762-4A58-9B9A-6D3CF75CE609; container ID = \"iCloud.eu.structuredpath.SQLiteData.Reminders\">";
    "<CKRecordID: 0x6000002f9220; recordName=c52c5f3c-9642-4e3f-8376-414203d82691:remindersLists, zoneID=co.pointfree.SQLiteData.defaultZone:__defaultOwner__>" = "<CKError 0x600000c69920: \"Batch Request Failed\" (22/2024); server message = \"Atomic failure\"; op = DAF3C063AC9933EB; uuid = 29004F91-3DF2-46F9-AB4E-BE73EB694F5D; container ID = \"iCloud.eu.structuredpath.SQLiteData.Reminders\">";
}}

So it does seem like CKSyncEngine is respecting atomicity settings.

This is also my conclusion: the client appears correct. The zone has the atomic capability, but the batch is not marked atomic, and the generated CKModifyRecordsOperation correctly reflects the atomicity flag. Yet CloudKit still treats the write as atomic. That is the puzzling part.

I’ll keep investigating and will look into the hints Tim shared in the iOS Folks Slack next.

[mbrandonw]

I’m only seeing the issue in the Reminders example.

I just updated the reminders app to perform the same kind of operation and it too seemed to work just fine:

SQLiteData (private.db) nextRecordZoneChangeBatch: manual
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ remindersLists │ 24480d2a-6061-4246-b4b6-e82903933af7:remindersLists │
  │ ➡️ Sending │ remindersLists │ 01fc2701-b150-4edc-abd0-bb168642d9d5:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘
SQLiteData (private.db) stateUpdate
SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action         ┃ recordType     ┃ recordName                                          ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ remindersLists │ 24480d2a-6061-4246-b4b6-e82903933af7:remindersLists │                          │
  │ 🛑 Save failed │ remindersLists │ 01fc2701-b150-4edc-abd0-bb168642d9d5:remindersLists │ serverRecordChanged (14) │
  └────────────────┴────────────────┴─────────────────────────────────────────────────────┴──────────────────────────┘
SQLiteData (private.db) stateUpdate
SQLiteData (private.db) didSendChanges
SQLiteData (private.db) willSendChanges
SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event      ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ remindersLists │ 01fc2701-b150-4edc-abd0-bb168642d9d5:remindersLists │
  └────────────┴────────────────┴─────────────────────────────────────────────────────┘
SQLiteData (private.db) stateUpdate
SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action   ┃ recordType     ┃ recordName                                          ┃
  ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved │ remindersLists │ 01fc2701-b150-4edc-abd0-bb168642d9d5:remindersLists │
  └──────────┴────────────────┴─────────────────────────────────────────────────────┘
SQLiteData (private.db) stateUpdate

Have you tried resetting your iCloud container just in case it somehow got into a bad state?

[lukaskubanek]

Uff, this one is really tough. I spent more time debugging and uncovered a few new details, but I still can’t pinpoint the root cause. At this point it feels like a CKSyncEngine quirk, and I may need help from Tim to understand what’s going on. I’ve shared some of the findings with him in this thread in the iOS Folks Slack.

Further Debugging

Here’s what I’ve tried so far:

1. I tested on real devices with my main Apple account instead of a Simulator with a test account. No change. The issue still occurs.

2. I created a brand-new CloudKit container and switched the app to it (both in the development environment). Same result. Still seeing the same issue.

3. I created a minimal example with no dependency on CKSyncEngine or SQLiteData, similar to the one you posted above, using CKDatabase.modifyRecords(…). I injected it into the Reminders sample app, used a new record type unrelated to SQLiteData and its tables, and tested both SQLiteData's custom zone and a fresh custom zone. All variants behaved correctly and non-atomicity was respected.

4. Next, I tried a minimal example based on this concept, applied again in the Reminders sample app. This setup uses SQLiteData and CKSyncEngine, with a manual CKDatabase.modifyRecords(…) update inserted in between. Surprisingly, this scenario also worked correctly for me.

5. Until this point, I had only been able to reproduce the issue in the Reminders sample app, so I wanted to rule out that it was tied to that specific project. I reproduced the original scenario in the CloudKitDemo sample app as well. The issue occurred, too. The conflicting save was treated as atomic.

6. Since Tim asked in the iOS Folks Slack whether this happens only with .serverRecordChanged, I changed the example to a conflicting deletion. Device B deletes a counter while device A updates that same counter and inserts a new one at the same time. Device B saves first, so the delete reaches the server before device A sends its changes. When device A sends, the inserted counter is saved successfully, while the updated counter that no longer exists on the server fails with .unknownItem. (The deleted counter is not recreated, but that is out of scope here.) So, when the batch contains a different partial error, the non-atomic save seems to work correctly.

SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event     ┃ recordType ┃ recordName                                    ┃
  ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ counters   │ 0cc4132f-ec86-4f80-b463-175dc039a63c:counters │
  │ ➡️ Sending │ counters   │ 250e71d9-4101-44d4-8a38-49d83e8691d1:counters │
  └───────────┴────────────┴───────────────────────────────────────────────┘

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error            ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ 0cc4132f-ec86-4f80-b463-175dc039a63c:counters │                  │
  │ 🛑 Save failed │ counters   │ 250e71d9-4101-44d4-8a38-49d83e8691d1:counters │ unknownItem (11) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────┘

<CKSyncEngine Private> failed sending changes for context <SendChangesContext reason=scheduled options=<SendChangesOptions scope=all group=CKSyncEngine-SendChanges-Automatic groupID=D22843897B0E5F2A>>: Error Domain=CKErrorDomain Code=2 "Failed to send changes" UserInfo={NSLocalizedDescription=Failed to send changes, CKPartialErrors={
    "<CKRecordID: 0x12303bee0; recordName=250e71d9-4101-44d4-8a38-49d83e8691d1:counters, zoneID=co.pointfree.SQLiteData.defaultZone:__defaultOwner__>" = "<CKError 0x1231fc900: "Unknown Item" (11/2003); server message = "recordChangeTag specified, but record not found"; op = F28E5CBE0A58A0E8; uuid = ADA394E0-9181-4AB9-B784-9F8842CF1108; container ID = "iCloud.eu.structuredpath.SQLiteData.CloudKitDemo">";
}}

Lessons Learned

• The issue does not appear to depend on device type, Apple account, CloudKit container, or custom record zone.
• The behavior seems specific to CKSyncEngine. Manual saves via CKDatabase.modifyRecords(…) do not trigger the issue, even when used alongside the sync engine.
• In my original report, I assumed the issue was limited to parent–child records. Further testing shows it also affects completely unrelated records batched together.
• The issue is reproducible in multiple SQLiteData-based sample apps (Reminders and CloudKitDemo), which makes it unlikely that it is caused by a particular schema or configuration in one app.
• When paired with an .unknownItem partial error, the batch behaves non-atomically as expected. This suggests the issue may be specific to .serverRecordChanged.
• I have not been able to reproduce the issue in my own CloudKit implementation that also uses CKSyncEngine. One notable difference is that SQLiteData stores values in encrypted form while my implementation does not, though it is unclear whether this plays a role.

[mbrandonw]

Hi @lukaskubanek, thanks for the detailed summary of everything you uncovered. I suppose at this point we have no choice but to start handling .batchRequestFailed just in case it is emitted. Should be a simple matter of re-enqueueing those records for saving again.

However, most important to me would be to get test coverage on this with our mock database, which means adding support for atomicity and further creating a way to force atomic to emulate this strange situation we are seeing. This will be something we look into soon.

[lukaskubanek]

I suppose at this point we have no choice but to start handling .batchRequestFailed just in case it is emitted. Should be a simple matter of re-enqueueing those records for saving again.

Yes, I think so as well. Though, I’ll still try out the suggestions from the iOS Folks Slack thread and report back.

[lukaskubanek]

@mbrandonw I have interesting news! 🙌

I ran the example with SQLiteData modified to use regular fields instead of encrypted ones as you’ve suggested in the iOS Folks Slack, and it behaved as expected. The save was not treated as atomic, as one record saved successfully while the conflicted one failed. It did, but this was short-lived. See my next comment.

SQLiteData (private.db) nextRecordZoneChangeBatch: scheduled
  ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ event     ┃ recordType ┃ recordName                                    ┃
  ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ➡️ Sending │ counters   │ 48e1c10d-80b3-48be-8817-0dfb9a0adc83:counters │
  │ ➡️ Sending │ counters   │ f8528f82-bc65-4eb9-98c5-4ac6e286829a:counters │
  └───────────┴────────────┴───────────────────────────────────────────────┘

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ 48e1c10d-80b3-48be-8817-0dfb9a0adc83:counters │                          │
  │ 🛑 Save failed │ counters   │ f8528f82-bc65-4eb9-98c5-4ac6e286829a:counters │ serverRecordChanged (14) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

I’ll share this with Tim next, but it seems the only option is to assume each save may fail as if treated atomically. This might also be useful when building this into the MockCloudDatabase.

[mbrandonw]

Hi @lukaskubanek, thanks for sharing this. Have you tried going back to encrypted values and testing again to make sure that is still broken? And further, have you tried updating your own sync engine code to use encrypted values to see if it reproduces there too?

Even if this somehow explains the batch failure on your end, I still don't understand how we are seeing different behavior.

[lukaskubanek]

@mbrandonw Oops, spoke too soon. I was too excited to see the test case succeed and should have waited before making the conclusion. Re-running the scenario this morning, I saw it failing again. It worked fine in some cases but failed in others. I assumed it depended on the concrete records and their state saved in CKSyncEngine. I then ran a few more rounds from a clean slate (removed app data & reset container) to understand what might be affecting it. This is my empirical observation:

• Saves to records with encrypted fields are always treated as atomic.
• Saves to records with regular fields are treated as non-atomic only in one specific case: when the conflicting record originates from another device and the .serverRecordChanged error is hit for the first time. When the record originates from the same device that hits the .serverRecordChanged error, or when that error has already occurred for the record in the past, the save becomes atomic again.

A typical run (from a single device’s perspective) looks like this:

1. Fetch the counter 15bec5a3-fc29-4d1e-941b-91184d814c0c from the server and trigger the conflict.

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ 4ab94139-a4fe-4981-a25a-dda4cb7e74ec:counters │                          │
  │ 🛑 Save failed │ counters   │ 15bec5a3-fc29-4d1e-941b-91184d814c0c:counters │ serverRecordChanged (14) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

2. Trigger the conflict on the same counter again.

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ 🛑 Save failed │ counters   │ 22d23b7d-b7b9-4c40-b154-cc19b9ba8438:counters │ batchRequestFailed (22)  │
  │ 🛑 Save failed │ counters   │ 15bec5a3-fc29-4d1e-941b-91184d814c0c:counters │ serverRecordChanged (14) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

3. Fetch another counter fa4b217a-c63c-411b-8210-50ef25df2d9c from the server and trigger the conflict.

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ ad8006aa-e770-47f1-a906-9a888a1c0cc2:counters │                          │
  │ 🛑 Save failed │ counters   │ fa4b217a-c63c-411b-8210-50ef25df2d9c:counters │ serverRecordChanged (14) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

4. Create a new counter fc8eb9a0-4599-42db-a3ef-4da8701d3558 locally, sync it, and trigger conflict.

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                    ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ 🛑 Save failed │ counters   │ fc8eb9a0-4599-42db-a3ef-4da8701d3558:counters │ serverRecordChanged (14) │
  │ 🛑 Save failed │ counters   │ c6c5a4cb-8636-4322-b0e8-ef7b3f708726:counters │ batchRequestFailed (22)  │
  └───────────────┴────────────┴───────────────────────────────────────────────┴──────────────────────────┘

This pattern seems quite consistent, though I’ve also seen cases where a single record failed with .batchRequestFailed while another was saved just fine:

SQLiteData (private.db) sentRecordZoneChanges
  ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃ action        ┃ recordType ┃ recordName                                    ┃ error                   ┃
  ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
  │ ✅ Saved       │ counters   │ 4a8fd6d7-759f-4ddc-a9e6-70dc7c2f365f:counters │                         │
  │ 🛑 Save failed │ counters   │ 3adee4eb-38c6-4970-8275-90b1cb7209aa:counters │ batchRequestFailed (22) │
  └───────────────┴────────────┴───────────────────────────────────────────────┴─────────────────────────┘

No idea what’s going on there, but something definitely seems off. This is probably as far as I can take it. I’ll share these findings (and the logs) with Tim in the iOS Folks Slack and see if he has any ideas.

Have you tried going back to encrypted values and testing again to make sure that is still broken?

Yes, after running the scenarios above, I cleared app data, reset the container, switched back to encrypted fields in SQLiteData, and re-ran the test case. As before, all saves with conflicts were treated as atomic. In other words, I still haven’t been able to hit a non-atomic save with a conflicting record when using pure CKSyncEngine.

And further, have you tried updating your own sync engine code to use encrypted values to see if it reproduces there too?

I updated my sync engine to use encrypted fields, and the issue immediately showed up:

<SentRecordZoneChanges
    savedRecords=[]
    failedRecordSaves=[
        <FailedRecordSave
            record=<CKRecord ListContent 92FB9E27…>
            error=<CKError 22/2024 "Batch Request Failed";
                   server message="Atomic failure">
        >,
        <FailedRecordSave
            record=<CKRecord List 1BD3C1B0…>
            error=<CKError 22/2024 "Batch Request Failed";
                   server message="Atomic failure">
        >,
        <FailedRecordSave
            record=<CKRecord List 58EA5EFE… etag=3>
            error=<CKError 14/2004 "Server Record Changed";
                   server message="client oplock error updating record";
                   clientEtag=3; serverEtag=6>
        >
    ]
    deletedRecordIDs=[]
    failedRecordDeletes=[]
>

This clearly suggests something is going on with encrypted fields. What I don’t understand is why I’m not seeing the issue with regular fields in my implementation, while SQLiteData does hit it.

Even if this somehow explains the batch failure on your end, I still don't understand how we are seeing different behavior.

Oh, so you weren’t able to reproduce the issue? Please note that none of the snippets you shared above triggered the behavior for me either, since they all rely on CKDatabase.modifyRecords(…). I can only reproduce the issue when running purely through CKSyncEngine. In particular, I’m testing by triggering this method from a counter row in the CloudKitDemo sample app and invoking it for the same counter on two devices in quick succession:

func trigger(for counterID: Counter.ID) async {
  withErrorReporting {
    try database.write { db in
      try Counter
        .find(counterID)
        .update { $0.count = 42 }
        .execute(db)
      
      try Counter
        .insert()
        .execute(db)
    }
  }
}

[lukaskubanek]

@mbrandonw There have been a few new developments in the iOS Folks Slack thread:

• Tim confirmed that the unexpected atomic failure is coming from the server.
• He also noted that the issue might be related to Advanced Data Protection. I do have it enabled, but I was able to reproduce the issue with a test account that has it disabled.
• I managed to reproduce the issue with CKDatabase.modifyRecords(…) as well, but using a different setup than the one you shared above, mainly involving two devices when creating the conflict.

This is the snippet I’m testing with in the CloudKitDemo sample app (after modifying the same record on a different device beforehand):

func trigger_modifyRecords(for counterID: Counter.ID) async {
  do {
    let containerIdentifier = ModelConfiguration(groupContainer: .automatic).cloudKitContainerIdentifier!
    let container = CKContainer(identifier: containerIdentifier)
    let cloudDatabase = container.privateCloudDatabase
    
    let counterRecord = try await database.read { db in
      let syncMetadataID = try Counter
        .find(counterID)
        .select(\.syncMetadataID)
        .fetchOne(db)!
      
      return try SyncMetadata
        .find(syncMetadataID)
        .select(\.lastKnownServerRecord)
        .fetchOne(db)!!
    }
    
    counterRecord.encryptedValues["count"] = 42
    
    let otherRecordID = CKRecord.ID(zoneID: counterRecord.recordID.zoneID)
    let otherRecord = CKRecord(recordType: "MyRecord", recordID: otherRecordID)
    
    let (saveResults, _) = try await cloudDatabase.modifyRecords(
      saving: [counterRecord, otherRecord],
      deleting: [],
      atomically: false
    )
    
    print(saveResults)
    print("----------")
  } catch {
    print(error)
    print("----------")
  }
}