Skip to content

OCI Image Specs support — resolve #5 vs main + finish read/write paths#13

Closed
sashml wants to merge 51 commits into
mainfrom
oci-image-specs-resolve
Closed

OCI Image Specs support — resolve #5 vs main + finish read/write paths#13
sashml wants to merge 51 commits into
mainfrom
oci-image-specs-resolve

Conversation

@sashml

@sashml sashml commented Jun 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Resolves PR #5 (OCI Image Specs support) against current main and finishes the implementation. This branch (oci-image-specs-resolve) is main + #5's OCI feature, conflict-free, building clean under -Werror, with the previously-incomplete paths now actually working.

What was wrong with #5 (found by building it under -Werror)

  • Conflicts with main across 20 files (main added encryption + Windows/MSVC + TOCTOU hardening since Add OCI Image Specs support #5 branched).
  • Did not compile with BFC_WITH_OCI=ON: missing _GNU_SOURCE (strdup/fmemopen), -Werror=format-truncation on path building.
  • Write path never worked: passed NULL to bfc_add_file (which rejects NULL) and never serialized the manifest/index to JSON.
  • Read path was a stub: bfc_get_oci_manifest/config, bfc_list_oci_layers returned BFC_OK without doing anything.
  • Wrong schema constant "2.0.1" — per the OCI image-spec schemaVersion MUST be 2.
  • Dead duplicate src/bfc_oci.c; a triple-pointer bug + invalid ... digests in tests/example (tripped the .. path-traversal guard).

Changes

Merge: union of main's encryption/portability work and #5's OCI feature (20 files).

Write path: bfc_create_from_oci_manifest/_index now serialize real OCI JSON via open_memstream.

Read path (new): implemented with libcjson

  • bfc_get_oci_manifest — parses schemaVersion/mediaType/config{digest,size}/layers[].digest
  • bfc_get_oci_config — parses architecture/os/created/author
  • bfc_list_oci_layers — parses layers[] into bfc_oci_layer_t[]

Dependency: libcjson is wired as the OCI feature's dep via pkg-config under BFC_WITH_OCI, mirroring the existing libzstd/libsodium optional-dep pattern. Build OCI with: apt install libcjson-dev (or vcpkg on Windows).

Correctness: schemaVersion"2"; truncation guards on all path snprintfs; removed dead src/bfc_oci.c; fixed the test triple-pointer + example digests.

Tests: added a write→read round-trip covering manifest, config and layers.

Docs: CALM + Mermaid C4 model of the OCI↔core boundary under docs/architecture/.

Architecture (critical parts)

OCI module ↔ BFC core boundary — the OCI layer only touches BFC's public API; gated behind BFC_WITH_OCI (default OFF):

                ┌───────────────────────────────────────────────────┐
   CLI / app ──►│  OCI module  src/lib/bfc_oci.c   [BFC_WITH_OCI]    │
                │   write: create_from_oci_manifest/_index, add_layer│
                │   read : get_oci_manifest/_config, list_oci_layers │
                └──────┬─────────────────────────────────┬──────────┘
          write path   │                       read path  │
        bfc_add_file   ▼                    bfc_stat +     ▼
                ┌───────────────────────────  bfc_read ───────────────┐
                │              BFC core  (bfc_reader / bfc_writer)     │
                └──────────────────────────────────────────────────────┘
   build JSON: open_memstream            parse JSON: libcjson (pkg-config)

Write path — struct → JSON → container entry:

  bfc_oci_manifest_t ──open_memstream──► {"schemaVersion":2,"mediaType":…,
        │                                 "config":{…},"layers":[{…}]}
        │                                        │ fmemopen
        ▼                                        ▼
   (caller-owned)                           bfc_add_file ──► "manifest.json"

Read path — container entry → JSON → struct (was a BFC_OK no-op, now real):

  "manifest.json" ──bfc_stat+bfc_read──► raw bytes ──cJSON_ParseWithLength──►
        DOM ──extract schemaVersion/mediaType/config/layers──► bfc_oci_manifest_t

Resulting BFC container layout for an OCI image:

  image.bfc
  ├── manifest.json            schemaVersion=2, mediaType, config{digest,size}, layers[]
  ├── config.json              architecture, os, created, author
  └── blobs/sha256/<digest>    layer blobs (bfc_add_oci_layer)

Verification

Config Build Tests
Default (OCI off) clean 11/11
Release + OCI + ZSTD (-Werror) clean 11/11 (incl. round-trip)
oci_example end-to-end runs, exit 0

BFC_WITH_OCI defaults OFF.

Enhance CI workflow with security permissions and GitHub Pages deploy…
- Introduced bfc_compress.h for compression context and result structures.
- Implemented compression and decompression functions in bfc_reader.c and bfc_writer.c.
- Added logic to handle compression type selection and thresholds in bfc_add_file.
- Enhanced bfc_create to initialize compression settings.
- Implemented unit tests for compression functionality in test_compress.c.
- Updated CMakeLists.txt to include new test file and link against ZSTD if enabled.
- Added functions to set and get compression settings in the BFC writer.
- Improved error handling for compression and decompression processes.
- Added `test_encrypt.c` to implement unit tests for encryption support, key management, data encryption/decryption, and error handling.
- Introduced `test_encrypt_integration.c` for integration tests focusing on encryption context lifecycle, key derivation edge cases, and large data encryption.
- Updated `CMakeLists.txt` to include the new encryption test files and link against libsodium if enabled.
- Temporarily disabled integration tests in `test_main.c` due to API mismatches.
Add unit tests for encryption functionality and integration tests
Reorder directory change and container opening in extract command for…
themoriarti and others added 21 commits September 12, 2025 17:02
- Add bfc_oci.h header with OCI data structures and function declarations
- Add bfc_oci.c implementation with OCI manifest, config, and layer support
- Add oci_example.c demonstrating OCI functionality
- Add OCI_SUPPORT.md documentation
- Update CMakeLists.txt to include OCI support option
- Add examples/CMakeLists.txt for building examples

This enables BFC to be used as a storage backend for OCI-compliant
container images, making it suitable for integration with container
runtimes and registries.
- Implemented symlink addition in the container with `bfc_add_symlink`.
- Added symlink extraction functionality in `extract_symlink`.
- Updated directory processing to handle symlinks during extraction.
- Enhanced info display to include symlink statistics.
- Modified file mode formatting to recognize symlinks.
Add comprehensive symlink support to BFC containers
- Add comprehensive OCI image manifest and config support
- Implement bfc_extract_to_oci function for extracting BFC containers to OCI format
- Add dynamic architecture and OS detection for cross-platform compatibility
- Create OCI example with proper CMakeLists.txt integration
- Update examples/README.md with OCI example documentation
- Add proper error handling and memory management
- Support for multiple architectures: x86_64, ARM64, RISC-V, PowerPC
- Support for multiple OS: Linux, Windows, macOS, BSD variants

This enables BFC to be used as an OCI image storage format while maintaining
compatibility with existing BFC functionality.
- Fix code formatting according to project standards
- Apply consistent indentation and spacing
- Sort includes alphabetically
- Ensure all C/C++ files follow clang-format conventions

This addresses the formatting issues reported in CI/CD pipeline.
Enable BFC to build successfully on FreeBSD by addressing platform-specific
compiler requirements and API differences.

Changes:
- Add -mcrc32 flag for CRC32 intrinsics on x86_64/amd64 architectures
- Detect FreeBSD's 'amd64' architecture identifier (in addition to x86_64/AMD64)
- Fix unused parameter warning in bfc_os_advise_nocache() on FreeBSD
- Fix benchmark_encrypt.c to use correct reader API (bfc_reader_set_encryption_password)
- Update documentation to mention FreeBSD support and pkgconf requirement

Technical details:
- FreeBSD's Clang requires both -msse4.2 and -mcrc32 for _mm_crc32_* intrinsics
- CMAKE_SYSTEM_PROCESSOR returns "amd64" on FreeBSD (not "x86_64")
- Changed from set_source_files_properties to target_compile_options for proper flag application

Tested on FreeBSD 14.3-RELEASE with Clang 19.1.7.
- Move src/bfc_oci.c to src/lib/bfc_oci.c for better organization
- Update src/lib/CMakeLists.txt to use correct path (bfc_oci.c instead of ../bfc_oci.c)
- Add BFC_WITH_OCI compile definition for targets
- Create comprehensive test suite in tests/unit/test_oci.c
  - Tests for validation functions (manifest, config)
  - Tests for creation functions (from manifest, from index)
  - Tests for retrieval and extraction functions
  - NULL pointer validation tests
  - Memory management tests
- Update test infrastructure to include OCI tests
- All changes applied from updated_changes.patch
Resolve conflicts across 20 files by unioning two orthogonal feature lines:
main's encryption subsystem + Windows/MSVC portability + TOCTOU hardening, and
the PR's OCI image-specs support. Remove dead duplicate src/bfc_oci.c (only
src/lib/bfc_oci.c is built).
- Add _GNU_SOURCE to bfc_oci.c/oci_example.c/test_oci.c (strdup, fmemopen)
- Implement manifest + index JSON serialization via open_memstream; the write
  path previously passed NULL to bfc_add_file (which rejects NULL) and never
  worked. bfc_create_from_oci_manifest/index now emit real OCI JSON.
- Read path now returns new BFC_E_NOSYS instead of a false BFC_OK; full read
  needs a JSON parser (follow-up).
- Guard all path-building snprintf against truncation (fixes -Werror=format-
  truncation and the reviewer's digest buffer-overflow concern).
- Correct OCI schemaVersion '2.0.1' -> '2' (per OCI image-spec it MUST be 2).
- Fix triple-pointer arg in test_oci.c; use valid 64-hex digests in the example
  (the '...' placeholders tripped the '..' path-traversal guard).
Replace the BFC_E_NOSYS stubs with working readers:
- bfc_get_oci_manifest reads manifest.json (bfc_stat + bfc_read) and parses
  schemaVersion/mediaType/config{digest,size}/layers[].digest with libcjson.
- bfc_get_oci_config parses architecture/os/created/author from config.json.
- bfc_list_oci_layers parses layers[] into a bfc_oci_layer_t array.
Wire libcjson as the OCI feature's dependency via pkg-config under BFC_WITH_OCI,
mirroring the existing libzstd/libsodium optional-dep pattern (lib, tests,
example). Add a write->read round-trip test covering manifest, config and
layers.
Document the OCI-module -> BFC-core boundary (gated BFC_WITH_OCI): bfc-oci.calm.json + bfc-oci-c4.md.
@sashml

sashml commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #5: fast-forwarded the conflict resolution + write/read implementation + fixes onto oci-image-specs-support (maintainer edit). #5 now carries this work, so closing this duplicate.

@sashml sashml closed this Jun 15, 2026
@sashml sashml deleted the oci-image-specs-resolve branch June 15, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants