Reimplement the feature of support setting data file size limit

[ahrtr]

Reimplement #929, in order to resolve #1108

@Elbehery please feel free to add test cases on top this PR. If you don't have bandwidth, please let me know; in that case, I will finish this PR next Monday or Tuesday. thx

cc @fuweid

[ahrtr]

cc @mattsains

[Copilot]

Pull request overview

This PR reimplements the “maximum data file size” (MaxSize) feature (from #929) and aims to address issue #1108 where hitting MaxSize during a write should not leave the DB client in a broken state.

Changes:

• Adds Windows-specific option validation to prevent InitialMmapSize > MaxSize from causing file expansion beyond the configured limit.
• Moves MaxSize enforcement earlier into DB.allocate() (before attempting mmap/remap) and removes the MaxSize check from DB.grow().
• Removes the Windows mmap()-time MaxSize guard that prevented truncating (growing) the file beyond MaxSize.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
db.go	Adds Windows option validation; shifts MaxSize enforcement into allocate() and removes the check from grow().
bolt_windows.go	Removes the MaxSize enforcement previously done inside the Windows mmap() implementation.

Comments suppressed due to low confidence (2)

db.go:1211

• allocate() still has a special-case branch for err == berrors.ErrMaxSizeReached coming back from db.mmap(minsz), but after moving the MaxSize check out of grow() and removing the Windows-side check, db.mmap() no longer appears to return ErrMaxSizeReached anywhere. This branch looks dead now and makes the error handling harder to follow; either remove it or switch to a single path that wraps/returns the mmap error consistently (or use errors.Is if you plan to return wrapped errors).

		if err := db.mmap(minsz); err != nil {
			if err == berrors.ErrMaxSizeReached {
				return nil, err
			} else {
				return nil, fmt.Errorf("mmap allocate error: %s", err)
			}
		}

bolt_windows.go:77

• Removing the MaxSize guard from the Windows mmap() path means opening/remapping can now grow the on-disk file beyond db.MaxSize (because Windows truncates the file to the mapping size). For example, opening a new DB with a small MaxSize can still truncate to the minimum mmap size (32KB) or a rounded-up mmapSize() value, exceeding the configured limit. There should still be a Windows-side check before db.file.Truncate(int64(sz)) to prevent file growth beyond MaxSize (while allowing mapping an already-larger existing file), or equivalent protection earlier in DB.mmap() before it unmaps the current mapping.

	if !db.readOnly {
		// Truncate the database to the size of the mmap.
		if err := db.file.Truncate(int64(sz)); err != nil {
			return fmt.Errorf("truncate: %s", err)
		}
		sizehi = uint32(sz >> 32)
		sizelo = uint32(sz)
	}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ahrtr]

@Elbehery please feel free to add test cases on top this PR.

Please feel free to apply this commit into your PR. thx

[ahrtr]

The idea is that we only perform the file size limitation check in one place (db *DB) allocate for all platforms. But for windows, we add an additional check when the clients open the db file in validateOption

[ahrtr]

The idea is that we only perform the file size limitation check in one place (db *DB) allocate for all platforms. But for windows, we add an additional check when the clients open the db file in validateOption

cc. @Elbehery @fuweid

[fuweid]

func TestDB_MaxSizeWithHighInitialMmapDoesNotGrowOnWrite(t *testing.T) {
	if runtime.GOOS == "windows" {
		t.SkipNow()
	}

	const (
		maxSize         = 64 * 1024 * 1024
		initialMmapSize = maxSize * 2
		valueSize       = 50 * 1024 * 1024
	)

	dbPath := filepath.Join(t.TempDir(), "db")
	db, err := bolt.Open(dbPath, 0600, &bolt.Options{
		MaxSize:         maxSize,
		InitialMmapSize: initialMmapSize,
	})
	require.NoError(t, err)
	defer func() {
		require.NoError(t, db.Close())
	}()

	value := bytes.Repeat([]byte("v"), valueSize)
	err = db.Update(func(tx *bolt.Tx) error {
		b, err := tx.CreateBucket([]byte("data"))
		if err != nil {
			return err
		}
		return b.Put([]byte("large-value"), value)
	})
	assert.ErrorIs(t, err, berrors.ErrMaxSizeReached)

	newSz := fileSize(dbPath)
	assert.Greater(t, newSz, int64(0), "unexpected new file size: %d", newSz)
	assert.LessOrEqual(t, newSz, int64(maxSize), "The size of the data file should not exceed MaxSize")
}

Please checkout this testcase.

The InitalMmapSize means db.datasz in linux platform.
When it's larger than MaxSize, it will bypass this condition and continue to grow.
It will large than MaxSize until it needs to re-mmap.

I think we still need to enforce the size limit in grow(). I’m not sure checking minsz > MaxSize every time the highest page id increases is the right place to do
it, since that path may be more performance-sensitive. grow() is where the file size is actually changed, so the check seems more precise there.

[ahrtr]

When it's larger than MaxSize, it will bypass this condition and continue to grow.
It will large than MaxSize until it needs to re-mmap.

Good catch & thx. Fixed. Please let me know if you still see any issue.

I think we still need to enforce the size limit in grow().

That's what the original PR #929 did. It has an issue #1108. That's the reason why we need to check the file size at early stage.

[ahrtr]

@Elbehery The test case above provided by @fuweid is a good test case. Please add them into your PR as well.

[Elbehery]

Ack 👍🏽

Thanks @fuweid 🙏🏽

[fuweid]

LGTM

Maybe we can add test in this pull request. Anyway, follow-up for testcase is fine to me

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, fuweid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ahrtr]

already integrated into #1210