Skip to content

fix(harness): prevent OOM in verifyChecksum using stream#1125

Merged
gorkem merged 1 commit into
kitops-ml:mainfrom
krsatyamthakur-droid:fix/oom-verifychecksum
Mar 17, 2026
Merged

fix(harness): prevent OOM in verifyChecksum using stream#1125
gorkem merged 1 commit into
kitops-ml:mainfrom
krsatyamthakur-droid:fix/oom-verifychecksum

Conversation

@krsatyamthakur-droid
Copy link
Copy Markdown

Summary

This PR fixes an Out-Of-Memory (OOM) risk in the harness download process.

Currently, pkg/lib/harness/llm_download.go uses os.ReadFile() inside verifyChecksum to load the entire downloaded harness file (llamafile.tar.gz or ui.tar.gz) into RAM all at once in order to compute the SHA-256 hash. In memory-constrained environments or with future larger harness binaries, this memory spike can trigger an OOM kill.

Changes Made

  • Replaced os.ReadFile and sha256.Sum256 with a streaming approach.
  • Implemented io.Copy to stream the file contents directly into a sha256.New() hasher.
  • This guarantees a small, constant memory footprint regardless of the downloaded file size.

Related Issue

Closes #1114

Testing Done

  • Successfully built kit locally.
  • Confirmed go test ./... passes locally.
  • Ensured my commit is DCO signed-off (-s).

AI-Assisted Contribution Checklist

  • I have read all AI-generated code. I understand what it does and why.
  • Testing: I have tested the code and verified its behavior.
  • Cleanup: I have removed verbose AI comments, unnecessary explanations, and boilerplate.
  • No Hallucinations: I have verified all imports, function calls, and APIs.

Resolves kitops-ml#1114 by replacing os.ReadFile with io.Copy to stream the file into the sha-256 hasher, rather than loading the whole file into memory first.

Signed-off-by: satyam kumar <krsatyamthakur@gamil.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR mitigates an OOM risk during harness downloads by changing checksum verification to hash files via streaming rather than reading the entire archive into memory.

Changes:

  • Replaced os.ReadFile + sha256.Sum256 with os.Open + sha256.New + io.Copy in verifyChecksum.
  • Computes the SHA-256 checksum with constant memory usage regardless of file size.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@gorkem gorkem merged commit 9f94511 into kitops-ml:main Mar 17, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High Memory Usage / OOM Risk in verifyChecksum due to os.ReadFile.

3 participants