After upgrade some aptly operations became extremely slow (hours vs minutes), suspect ext4 discard on QEMU

[volo-05]

Hi! After upgrading aptly we see a huge slowdown on a clone machine with (as far as we can tell) the same dataset and workload.

What changed

• Old aptly: 1.4.0+ds1-4+b4
• New aptly: 1.6.2

Symptoms (CI timings)

Before:

• Deploy pool and staging: ~5 min
• Deploy releases: ~5 min

After upgrade:

• Deploy pool and staging: ~1h 25m
• Deploy releases: >5h

How we run aptly

We run aptly api serve and our CI connects via a UNIX socket.
rootDir is on ext4 inside a QEMU/KVM VM.

Filesystem / storage

We noticed the filesystem is mounted with discard:
/dev/sda1 ... ext4 (rw,relatime,discard,...)

The virtual disk supports TRIM/UNMAP:

• lsblk -D shows DISC-GRAN=4K, DISC-MAX=1G, ROTA=0 (QEMU HARDDISK)
• /sys/block/sda/queue/discard_max_bytes = 1073741824
• /sys/block/sda/queue/discard_granularity = 4096

TRIM / discard test

With nodiscard, fstrim -av trims a lot of data:

• /: 26.2 GiB trimmed, real time ~12s

With discard, fstrim -av reports:

• /: 0 B trimmed, real time ~0.02s

This suggests continuous discard is active.

Question / request for guidance

1. Is discard known to hurt performance for LevelDB-heavy workloads used by aptly (compactions, table file churn, fsync)?
2. Are there recommended filesystem mount options for aptly databases?
   For example, should we avoid discard and use periodic fstrim instead?
3. Any aptly-side debug flags / logging / profiling you recommend to pinpoint why these steps became so slow after 1.6.2?

If you tell us what logs/metrics you need (aptly debug logs, strace snippet, LevelDB stats, etc.), we can provide them.

Thanks!

aptly config (redacted)

{
  "FileSystemPublishEndpoints": {},
  "S3PublishEndpoints": {
    "deb.wirenboard.com": {
      "acl": "public-read",
      "awsAccessKeyID": "<REDACTED>",
      "awsSecretAccessKey": "<REDACTED>",
      "bucket": "deb.wirenboard.com",
      "plusWorkaround": true,
      "region": "eu-west-1"
    }
  },
  "SwiftPublishEndpoints": {},
  "architectures": [],
  "dependencyFollowAllVariants": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowSource": false,
  "dependencyFollowSuggests": false,
  "downloadConcurrency": 4,
  "downloadSourcePackages": false,
  "downloadSpeedLimit": 0,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "internal",
  "ppaCodename": "",
  "ppaDistributorID": "ubuntu",
  "rootDir": "/home/jenkins/aptly-ng"
}

{
"rootDir": "/home/jenkins/aptly-ng",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": [],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": false,
"skipLegacyPool": true,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"skipContentsPublishing": false,
"FileSystemPublishEndpoints": {},
"S3PublishEndpoints": {},
"SwiftPublishEndpoints": {}
}

[volo-05]

Additional evidence from CI logs (same dataset, clone VM):

Even when snapshots are already published, publish checks became extremely slow after upgrade.

aptly 1.6.2:

• Publishing all/pool.latest: 09:59:56 -> “already published”: 10:05:37 (~5m41s)
• Publishing all/staging.latest: 10:05:42 -> 10:11:36 (~5m54s)
• Each following experimental publish also takes ~5m30s–5m40s before reporting “already published”.

aptly 1.4.0:

• Publishing all/pool.latest: 19:55:45 -> 19:55:48 (~2.6s)
• Publishing all/staging.latest: 19:55:48 -> 19:55:50 (~1.9s)
• Others are also ~1–2 seconds.

Also, our client debug logs show that repeated GET /api/snapshots/... requests that used to complete in <1s total now take ~20–30s total (many requests ~0.7–1.6s each), suggesting a general slowdown of API read/metadata operations (likely DB/FS I/O related).

[neolynx]

Hi @volo-05,

thanks for reporting. We fixed a lot of locking issues in the upcoming aptly 1.6.3, would you be able to give a CI build a try ?

https://www.aptly.info/download/ -> CI Builds for Debian

I hope this resolves the issue, otherwise we would need to debug a bit more...

[neolynx]

We believe this has been addressed in the 1.6.3 release: #1597 ! :-)

Could you give it a try ?

[volo-05]

Hi! We tested aptly 1.6.3 in our environment on a fresh clone of our production aptly data.

The good news: 1.6.3 is a major improvement compared to what we saw with 1.6.2.

Original 1.6.2 timings from our previous test were roughly:

• Deploy pool and staging: ~1h 25m
• Deploy releases: >5h

With aptly 1.6.3 on the updated clone we now get:

• Check packages: 47s
• Check firmwares: 20s
• Deploy pool and staging: 13m 52s
• Deploy releases: 54m 41s
• Deploy firmwares list: 39s

For comparison, the same Jenkins pipeline on our production aptly setup (aptly 1.4.0+ds1-4+b4) shows:

• Check packages: 33s
• Check firmwares: 19s
• Deploy pool and staging: 1m 23s
• Deploy releases: 1m 27s
• Deploy firmwares list: 39s

So 1.6.3 looks much better than 1.6.2, but the publish-related stages are still significantly slower than 1.4.0 in our environment:

• Deploy pool and staging: ~10x slower than 1.4.0
• Deploy releases: ~38x slower than 1.4.0

The clone was made from our current production aptly data. The workload/pipeline is the same, although the test was run on the cloned aptly environment rather than the production host.

So from our side: 1.6.3 definitely improves the situation, but there is still a large regression compared to 1.4.0, especially in the publish/deploy stages.

[MartinEmrich]

We experience a similar slowdown with the Debian Trixie version (1.6.1) vs. the Debian Bookworm version (1.5.0). I also just tried the latest 1.6.3, and it is still slow. It gets stuck for minutes "processing" a single package file (1.5GB, verified by regular looking with lsof).