Skip to content

Commit a6af1fc

Browse files
Eamon2009codeaddict-119ethos-cmddependabot[bot]
authored
Optimize training configuration and CI workflows (#87)
* Optimization (#82) * Modify training configuration parameters Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * Refactor training configuration and optimize CI workflows (#81) * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. * merge branch master of codeaddict (#77) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max <eamon5174@gmail.com> * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max <eamon5174@gmail.com> * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy <eamon112009@gmail.com> * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * merge branch master (#78) * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- Co-authored-by: Max <eamon5174@gmail.com> * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- Co-authored-by: Max <eamon5174@gmail.com> * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- Co-authored-by: Max <eamon5174@gmail.com> * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv Co-Authored-By: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-Authored-By: Eamon Sippy <eamon112009@gmail.com> * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Eamon <eamon112009@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Optimize CI workflow and Docker configurations with refactoring (#72) (#76) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp * docs:report [run_20260530_165216](~791 tok/s) (#60) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs: report [run_20260530_165216] (~791 tok/s) (#62) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 --------- * chore: clang-format configuration file based on LLVM (#63) * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. * Refactor core architecture and optimize CUDA features (#75) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- * Add CUDA kernels, optimize CI, and update documentation (#74) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. * CUDA header declarations for (LayerNorm) forward and backward (#66) * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces --------- * Add CUDA attention kernels, gradient norms, and CI improvements (#69) * exp(#58) * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * feat(ci): optimize workflow pipeline and update docker configurations * refactor(ci): optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * refactor : optimize workflow pipeline and update docker configurations * Added MIT LICENSE to this project Quadtrix.cpp * Refactor Dockerfile to use ARG for CUDA version * Refactor Dockerfile for backend dependencies * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * Delete .devops/Dockerfile.frontend * Delete .devops/Dockerfile.dev.frontend * refactor : Dockerfile.backend optimize workflow pipeline * refactor : Dockerfile.backend optimize workflow pipeline * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactored (CI): consolidated manual Docker build jobs into a matrix strategy to reduce duplication * refactor(ui): rewrite ThinkingIndicator to use inline styles and CSS keyframes * refactor : message bubble layout to use inline styles * refactor(ui): complete inline-style migration and update auto-scroll implementation * refactor(ui): complete inline-style migration for MessageAvatar component * refactor(ui): rewrite EmptyState component using pure inline styles * refactored(tensor): vectorize element-wise addition and scalar scaling using AVX/SSE - Added SIMD vectorization support (`__AVX__` and `__SSE__`) for element-wise `add`, `add_inplace`, and `scale` operations. - Maintained scalar fallback paths for non-vectorized bounds and platforms lacking hardware extensions. - Explicitly defined rule-of-five constructors (`default` and `noexcept` moves) within the `Tensor` struct layout. - Optimized vector initialization across the core construct layer via `std::move` and `std::vector::reserve`. * refactor(main): redesign training loop to log per-step and sample during evaluation - Replaced the periodic block evaluation layout with standard, per-step logging metrics (`loss`, `ms`, and `tok/s`). - Shifted initial validation loss calculation out of the iteration cycle to establish a zero-state baseline. - Restructured token streaming so that generations are triggered conditionally inside the training loop post-evaluation windows. - Streamlined architecture parameter reporting and consolidated command-line configuration visual prints. * feat: implement GPT training loop with multi-GPU and memory optimizations - Add advanced memory footprint optimization using forward-activation recomputation for LayerNorm and GeLU. - Optimize layer-wise activation buffer layout using a centralized `TensorSpec` registry to support large batch scaling. - Integrate cuBLASLt matmul fusions, optional cuDNN attention layers, and stochastic rounding options. - Fall back gracefully to `cudaMallocManaged` under heavy loads to prevent Outlier/OOM crashes. * Update README.md with new banner for qudtrix.cpp --------- * ci: add manual PR checks workflow with slash command support * feat(cuda): add attention forward backward kernel declarations (#64) * docs: report [run_20260530_165216] (~791 tok/s) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * docs:report [run_20260530_165216](~791 tok/s) (#61) Includes metrics for generalization gap, throughput (~791 tok/s), and gradient norms. Parameters: 6.68M | lr: 1e-3 | batch: 16 | steps: 6000 - Achieved best validation loss of 4.1319 at step 3900 * feat(cuda): add attention forward and backward kernel declarations Introduces the header declarations for `attention_forward` and `attention_backward` operations inside the `quadtrix::cuda` namespace. Configured with support for custom CUDA streams and head partitioning. --------- * feat(cuda): add checkpoint metadata struct and stub functions * feat(cuda): introduce core type definitions and error handling utilities - Defines `DType` and `DeviceKind` enums supporting standard types (F32, F16, BF16, I32, U8). - Implements `dtype_name` and `dtype_size` metadata helper functions. - Adds an explicit `Status` struct for non-throwing error propagation alongside `checked_mul` for safe allocation size computation. - Introduces `check_cuda` and `abort_on_cuda` error macros and handling mechanisms, exposed via the `QUADTRIX_CUDA_CHECK` macro. * feat(cuda): add TokenBatchView struct and DataLoader stub class * feat(cuda): add GeLU activation forward and backward declarations - Introduces the `GeluMode` enum to toggle between `Exact` and `Approximate` mathematical variants. - Declares the `gelu_forward` and `gelu_backward` kernel entrypoints. - Configures both signatures with optional stream execution and a default mode of `GeluMode::Approximate`. * feat(cuda): add gradient norm calculation and clipping interfaces * feat(cuda): add LayerNorm forward and backward kernel declarations * refactor(ci): organize workflow into push-triggered QA and manual docker builds Updated CI workflow to restrict branches for push events and improved input descriptions for image selection and push options. * Fix formatting and update CI workflow steps * Enhance CI with macOS binary build and release Added macOS binary build and release steps to CI workflow. * feat(docker): add Dockerfile for frontend application * feat(docker): add Dockerfile for frontend application * refactor(ci): remove release job from GitHub actions * ci: add unified release and docker build workflow * ci: add unified release and docker build workflow * Refactor macOS build workflow for arm64 architecture * Update release workflow to remove macOS x64 build Removed dependency on build-macos-x64 for the release job. * perf: update execution time benchmarks in csv * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * ci(docker): refactor image build workflow and add frontend job * Remove frontend job from Docker Images workflow * Update release workflow to remove s390x and add notes Removed s390x build configurations and added a step to write detailed release notes. * feat: add local orchestration script for frontend and backend servers Introduces a central Python execution script to concurrently manage and orchestrate the development environment for both the frontend and backend. - Detects system OS to invoke correct `npm` and `python` (virtualenv) binary variants. - Verifies existence of the local PyTorch `.pt` model checkpoint before starting. - Configures environment variables dynamically for Uvicorn (FastAPI) and Vite. - Handles cross-origin setups (CORS) linking ports interactively. - Gracefully handles process termination (`Ctrl+C`) by forwarding termination signals. - Automatically launches the frontend application in the system web browser. * chore(deps): bump actions/github-script from 7 to 9 (#71) Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... * feat(cuda): introduce log_message utility and LogLevel enum * feat(cuda): add cuBLAS handle wrapper and matmul operations * feat(cuda): implement core Tensor, TensorShape, and TensorView abstractions * refactor: untie embedding and lm_head weights and to quadtrix * feat(cuda): add NCCL communicator wrapper and all-reduce primitives * Update README.md with workflow badges Added badges for release, package, and CI workflows. * kernels: add AdamW optimization kernel with stochastic rounding Introduces the AdamW fused CUDA kernel including linear interpolation optimizations (`lerp`), multi-slice batching support via 2D grids, and `init_from_master` utility functions for low-precision parameter handling. * cudnn: implement cached SDPA forward graph using cuDNN frontend * feat(cuda): implement Packed128 memory vectorization utilities * feat: add distributed sharded DataLoader for binary token files * feat(multi-gpu): add foundational utilities for ZeRO sharding * feat(utils): add I/O and memory error-checking wrappers * feat : add PyTorch-compatible Mersenne Twister random utilities * README : Enhance README with header and workflow badges Updated README to include a header and badges for release, package, and CI workflows. * utils:`fopenCheck`, `freadCheck`, `fwriteCheck`, `fcloseCheck`, and `fseekCheck` with explicit crash details and project-specific troubleshooting hints. - Add cross-platform socket closure wrappers (`scloseCheck` and `closesocketCheck`) for Linux and Windows. * mfu: add GPU specifications database and utilities for MFU estimation * Modify project title in README.md Changed the project title to include 'llm.cpp' for clarity. * Update README to remove image and clean up content Removed image from README and adjusted formatting. --------- --------- --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor .dockerignore for improved clarity Removed unnecessary entries to streamline the build context. * Enhance .gitignore with additional exclusions Expanded .gitignore to include more file types and directories. * Modify training configuration parameters (#80) Updated training parameters including batch size, iterations, evaluation interval, learning rate, and dropout rate. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update main.py * Delete quadtrix_training_report.png * Delete docker-compose.yml * Delete docker-compose.gpu.yml * Delete docker-compose.dev.yml * Delete benchmark_results.csv * Delete SECURITY.md * refactor: switch tokenizer from gpt2 to tiktoken o200k * Delete contributing.md * Delete CUDA/llmcpp directory * Delete CUDA directory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max <eamon5174@gmail.com> Co-authored-by: codeenthusiasm23 <273188204+codeenthusiasm23@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: implement Head and MultiHeadAttention modules in C++ * feat: add core Tensor class with SIMD-accelerated math operations * add inference sampler parameters and repetition penalty * Linear layer with binary serialization Added Linear struct mimicking torch.nn.Linear with conditional bias support. * FeedForward network module Added FeedForward struct mirroring the standard Transformer position-wise MLP block * LayerNorm module with serialization Added LayerNorm struct mirroring torch.nn.LayerNorm(n_embd). * gradient and activation tracking structures for backpropagation * single Transformer Block with Pre-LN architecture Added Block struct mirroring the standard Transformer layer composition. * character-level DataLoader with batch sampling Added DataLoader struct for character-level tokenization and text processing. * Embedding layer with token and position mapping Added Embedding struct mirroring torch.nn.Embedding. * BPE tokenizer and training pipeline in DataLoader Upgraded DataLoader fr…
1 parent 3b5973e commit a6af1fc

24 files changed

Lines changed: 4759 additions & 706 deletions

‎.devops/docker-compose.dev.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
services:
2+
frontend:
3+
build:
4+
context: .
5+
dockerfile: .devops/Dockerfile.dev.frontend
6+
ports:
7+
- "5173:5173"
8+
volumes:
9+
- ./frontend:/app:delegated
10+
- /app/node_modules
11+
environment:
12+
VITE_API_BASE_URL: "http://localhost:3001"
13+
command: [ "npm", "run", "dev", "--", "--host", "0.0.0.0" ]
14+
healthcheck:
15+
test: [ "CMD", "wget", "-qO-", "http://localhost:5173/" ]
16+
interval: 15s
17+
timeout: 5s
18+
retries: 5
19+
20+
backend:
21+
volumes:
22+
- ./backend:/app/backend:delegated
23+
- ./engine:/app/engine:delegated
24+
- models:/models
25+
environment:
26+
LOG_LEVEL: DEBUG
27+
CORS_ORIGINS: "http://localhost:5173,http://localhost:3001"
28+
command:
29+
- python
30+
- -m
31+
- uvicorn
32+
- main:app
33+
- --host
34+
- "0.0.0.0"
35+
- --port
36+
- "3001"
37+
- --reload
38+
- --reload-dir
39+
- /app/backend
40+
41+
redis:
42+
ports:
43+
- "6379:6379"
44+
volumes:
45+
models:

‎.devops/docker-compose.gpu.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
services:
2+
backend:
3+
build:
4+
args:
5+
CUDA: "1"
6+
image: quadtrix/backend-cuda:latest
7+
deploy:
8+
resources:
9+
reservations:
10+
devices:
11+
- driver: nvidia
12+
count: all
13+
capabilities: [ gpu ]
14+
environment:
15+
CUDA_VISIBLE_DEVICES: "0"
16+
TORCH_CHECKPOINT_PATH: /models/best_model.pt
17+
18+
train-torch:
19+
build:
20+
args:
21+
CUDA: "1"
22+
image: quadtrix/backend-cuda:latest
23+
deploy:
24+
resources:
25+
reservations:
26+
devices:
27+
- driver: nvidia
28+
count: all
29+
capabilities: [ gpu ]
30+
environment:
31+
CUDA_VISIBLE_DEVICES: "0"
32+
QUADTRIX_TRAIN_DATA: /app/data/input.txt

‎.devops/docker-compose.yml‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
name: quadtrix
2+
3+
x-common-env: &common-env
4+
TZ: UTC
5+
PYTHONUNBUFFERED: "1"
6+
7+
services:
8+
9+
frontend:
10+
build:
11+
context: .
12+
dockerfile: .devops/Dockerfile.frontend
13+
args:
14+
VITE_API_BASE_URL: ""
15+
image: quadtrix/frontend:latest
16+
container_name: quadtrix-frontend
17+
restart: unless-stopped
18+
ports:
19+
- "5173:80"
20+
depends_on:
21+
backend:
22+
condition: service_healthy
23+
networks:
24+
- quadtrix-net
25+
healthcheck:
26+
test: [ "CMD", "wget", "-qO-", "http://localhost/" ]
27+
interval: 30s
28+
timeout: 5s
29+
retries: 3
30+
31+
backend:
32+
build:
33+
context: .
34+
dockerfile: .devops/Dockerfile.backend
35+
image: quadtrix/backend:latest
36+
container_name: quadtrix-backend
37+
restart: unless-stopped
38+
ports:
39+
- "3001:3001"
40+
environment:
41+
<<: *common-env
42+
API_PORT: "3001"
43+
CORS_ORIGINS: "http://localhost:5173,http://frontend"
44+
REDIS_URL: "redis://redis:6379/0"
45+
TORCH_CHECKPOINT_PATH: /models/best_model.pt
46+
LOG_LEVEL: INFO
47+
MAX_SESSIONS: "500"
48+
SESSION_TTL_HOURS: "24"
49+
volumes:
50+
- models:/models
51+
- ./engine:/app/engine:ro
52+
depends_on:
53+
redis:
54+
condition: service_healthy
55+
networks:
56+
- quadtrix-net
57+
healthcheck:
58+
test: [ "CMD", "curl", "-f", "http://localhost:3001/api/health" ]
59+
interval: 30s
60+
timeout: 10s
61+
start_period: 20s
62+
retries: 3
63+
64+
redis:
65+
image: redis:7-alpine
66+
container_name: quadtrix-redis
67+
restart: unless-stopped
68+
command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru
69+
volumes:
70+
- redis-data:/data
71+
networks:
72+
- quadtrix-net
73+
healthcheck:
74+
test: [ "CMD", "redis-cli", "ping" ]
75+
interval: 10s
76+
timeout: 5s
77+
retries: 5
78+
expose:
79+
- "6379"
80+
81+
cpp:
82+
build:
83+
context: .
84+
dockerfile: .devops/Dockerfile.cpp
85+
image: quadtrix/cpp:latest
86+
container_name: quadtrix-cpp
87+
88+
restart: "no"
89+
stdin_open: true
90+
tty: true
91+
volumes:
92+
- models:/models
93+
- ./data:/app/data:ro
94+
environment:
95+
<<: *common-env
96+
GPT_DATA_PATH: /app/data/input.txt
97+
GPT_MODEL_PATH: /models/best_model.bin
98+
networks:
99+
- quadtrix-net
100+
profiles:
101+
- cpp
102+
103+
train-cpp:
104+
build:
105+
context: .
106+
dockerfile: .devops/Dockerfile.cpp
107+
image: quadtrix/cpp:latest
108+
container_name: quadtrix-train-cpp
109+
restart: "no"
110+
volumes:
111+
- models:/models
112+
- ./data:/app/data:ro
113+
environment:
114+
<<: *common-env
115+
GPT_DATA_PATH: /app/data/input.txt
116+
GPT_MODEL_PATH: /models/best_model.bin
117+
command: [ "data/input.txt" ] # train mode (no --chat flag)
118+
networks:
119+
- quadtrix-net
120+
profiles:
121+
- train
122+
123+
train-torch:
124+
build:
125+
context: .
126+
dockerfile: .devops/Dockerfile.backend
127+
image: quadtrix/backend:latest
128+
container_name: quadtrix-train-torch
129+
restart: "no"
130+
volumes:
131+
- models:/models
132+
- ./engine:/app/engine
133+
- ./data:/app/data:ro
134+
environment:
135+
<<: *common-env
136+
QUADTRIX_TRAIN_DATA: /app/data/input.txt
137+
working_dir: /app
138+
command: [ "python", "engine/main.py" ]
139+
networks:
140+
- quadtrix-net
141+
profiles:
142+
- train
143+
144+
benchmark:
145+
build:
146+
context: .
147+
dockerfile: .devops/Dockerfile.cpp
148+
image: quadtrix/cpp:latest
149+
container_name: quadtrix-benchmark
150+
restart: "no"
151+
volumes:
152+
- models:/models
153+
- ./data:/app/data:ro
154+
- ./benchmark_results.csv:/app/benchmark_results.csv
155+
environment:
156+
<<: *common-env
157+
GPT_MODEL_PATH: /models/best_model.bin
158+
159+
command: [ "data/input.txt", "--generate" ]
160+
networks:
161+
- quadtrix-net
162+
profiles:
163+
- benchmark
164+
165+
volumes:
166+
models:
167+
driver: local
168+
redis-data:
169+
driver: local
170+
171+
networks:
172+
quadtrix-net:
173+
driver: bridge

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
frontend/** linguist-vendored

0 commit comments

Comments
 (0)