v1.3.0 - #24
Draft
FKKimura wants to merge 139 commits into
Draft
Conversation
…apter separately, run inference with vLLM
- Add `get_quant_config()` returning GPTQ-compatible config - Add `_build_quantization_bits()` and `finalize_quant_config_for_save()` - Add `create_inference_layer()` to build GPTQLinear from JointQResult
…wledge, verified with vllm
…ion knowledge, and add smoke test
Develop/v1 1 0
GPTQLinear weight packing only supports wbits in (2, 3, 4, 8), so JointQ with bits=1 must build/save the inference layer with pack_weights=False. - JointQ.validate_params: warn when bits=1 to remind callers to pass pack_weights=False at inference layer construction time - GPTQLinear.from_saved_state: load qweight/qzeros as unpacked tensors when wbits=1, matching the unpacked save format
…/lora-merge-v1.1.0 Resolved conflicts in: - onecomp/quantizer/jointq/_jointq.py - onecomp/runner.py
Add QuantizationProgressTracker and wire it through calibration, chunked calibration, multi-GPU phase 2, QEP general and arch-aware paths. Runner gains quantization_progress flag (default on). Includes unit tests for ETA formatting and thread-safe stepping. Co-authored-by: Cursor <cursoragent@cursor.com>
…wledge, verified with vllm
Raise clear error for unsupported QEP quantizers See merge request onecomp/onecomp-lab!71
feat: quantization progress logs with ETA
* refactoring : QuantizationProgressTracker * update CHANGELOG.md --------- Co-authored-by: FKKimura <50981196+FKKimura@users.noreply.github.com>
- QuantizationProgressTracker: suppress duplicate logs after completion, use `is not None` for lock check, expand Google-style docstrings - Demote per-layer / per-chunk INFO logs to DEBUG so the [progress] line is the single canonical per-step INFO signal (onecomp/quantizer/_quantizer.py, onecomp/runner_methods/chunked_quantization.py, onecomp/qep/_quantize_with_qep_arch.py) - Chunked path: switch to per-group progress (was per-layer × per-quantizer); drop chunk_progress / layer_progress arguments - Shorten tracker labels: "Quantization", "Chunked quantization", "QEP" - Per-block MSE log in QEP arch path: drop redundant `[INFO]` prefix, switch to lazy `%` formatting, fix "Layer N" -> "Block N" - Expand QuantizationProgressTracker tests (10 cases, incl. thread-safety and overflow suppression) - Update CHANGELOG for the new [progress] line and INFO -> DEBUG demotion
See merge request onecomp/onecomp-lab!72
Fix three OneComp-side bugs surfaced by Takane VLM (Cohere2Vision) work: the `save_quantized_model` flow dropping VLM auxiliary configs, the `load_quantized_model` flow leaving tied `lm_head` and other non-quantized modules in the wrong dtype after `load_state_dict(..., assign=True)`. Issue 1 (save_quantized_model misses VLM aux configs): - Replace the hard-coded allow-list (processor_config.json / preprocessor_config.json) with an extension-based whitelist of `*.json` / `*.jinja` (excluding weight shards, weight index files, `config.json`, and `generation_config.json`) - Extract source-model-dir resolution into `Runner._resolve_source_model_dir()` with a `snapshot_download( local_files_only=True)` fallback that warns on failure instead of aborting the save (onecomp/runner.py) - Skip files already produced by `model.save_pretrained` / `tokenizer.save_pretrained` and INFO-log both copied and skipped entries for traceability Issue 2 (tied lm_head left at fp16 after load): - Call `model.tie_weights()` after `load_state_dict(..., assign=True)` to restore `lm_head.weight is embed_tokens.weight` identity that `assign=True` breaks (onecomp/quantized_model_loader.py) - Guard the call on `tie_word_embeddings=True` and `isinstance(model.lm_head, nn.Linear)` to avoid touching quantized layers such as `GPTQLinear` - Walk one level of nested sub-configs (`text_config` / `language_config`) via `_should_retie_word_embeddings` so HF VLM configs that move `tie_word_embeddings` under `text_config` are also covered Issue 3 (non-quantized VLM modules left at fp16): - Make `_build_empty_model_from_config` honor `torch_dtype` / `dtype` from `config.json` when the caller did not pass an explicit dtype, so empty modules start in the saved checkpoint's dtype (onecomp/quantized_model_loader.py) - Add `_cast_fp16_to_target_dtype` as a safety net that casts any fp16 parameter / buffer left on non-quantized modules to `model.config.torch_dtype` after `tie_weights()`, returning the list of fully-qualified names that were converted for logging and tests - Skip `GPTQLinear` / `DoubleBinaryLinear` and limit the cast to fp16 so mixed-precision fp32 LayerNorms are not disturbed; mutate `p.data` in place so tied-weight identity is preserved Tests (tests/onecomp/runner/, 30 cases, CPU-only, no network): - test_save_quantized_aux_files.py (7): whitelist copy, existing-file skip, `snapshot_download` fallback, subdirectory exclusion, INFO log for skipped files - test_load_tied_embeddings.py (7): tiny LlamaForCausalLM with `tie_word_embeddings` True / False, identity restoration, forward dtype, `_should_retie_word_embeddings` top-level / nested / all-False / unrelated-sub-attrs cases - test_load_excluded_module_dtype.py (16): config-driven empty-model dtype, fp16 safety-net cast, fp32 preservation, quantized-layer skip, `_resolve_dtype_from_config` parsing edge cases, returned module-name list Update CHANGELOG.md (v1.1.1 Bug fixes) and verify each fix in isolation by reverting it and confirming the targeted test fails.
See merge request onecomp/onecomp-lab!69
Fix-tests-examples See merge request onecomp/onecomp-lab!70
Develop/v1-1-1
Resolved conflicts in: - CHANGELOG.md - example/vllm_inference/example_gptq_vllm_inference.py - onecomp/runner.py
…eature/ysmz/lora-save-load-vllm-infer_jointq
…/partial-quant-with-rotation-bug
feature: Add legacy vLLM environment definitions (v0.12.0 / v0.15.1)
…vllm-infer_jointq feat: LoRA adapter sidecar save/load support with vLLM-compatible inference + JointQ smoke test
…/partial-quant-with-rotation-bug
…rotation-bug fix: Derive Hadamard online-hook target types from the model for rotation + partial-quantization
add standalone package for vllm 0.24.x with ROCm See merge request onecomp/onecomp-lab!94
fix seed for quantizer-test See merge request onecomp/onecomp-lab!98
Remove legacy LoRA examples and complete README examples list See merge request onecomp/onecomp-lab!99
* cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * add test and chalgelog, and apply linter * Reverted the code under the example --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…Qwen3.6 (#35) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * apply linter * revert * Revert "revert" This reverts commit 0e006d6. --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…model save (#36) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * revert over-erased comment * add example file for qwen36, revert existing example, and update CHANGELOG * add new example file for qwen36 * apply pre-commit * revert blank line * revert blank line * revert qep option in example_gptq_vllm_inference.py * revert model_id in example, and fix comment to take users to example_gptq_vllm_qwen36_inference.py * fix typo --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…3.6 35B-A3B (#37) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * refactoring, linter, add test, and fix changelog * revert example modification * revert example modification * fix format * update docstring of Runner._exclude_moe_router_if_needed --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
* cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * bugfix: success save load * refactoring, linter, add test, and fix changelog * revert example modification * add test and fix changelog * revert example modification * fix format * bugfix: success save load * add test and fix changelog * add round trip test --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
#41) * [update] support rotated gptq vllm inference (TP = 1) * [fix] restrict online hadamard targeting to dense mlp.down_proj - share the Hadamard target predicate between preprocessing and vLLM - limit online Hadamard application to dense `mlp.down_proj` only - exclude MoE expert `down_proj` paths from the current rotation flow - keep preprocessing and vLLM behavior consistent with current MoE support scope * [fix] support rotated GPTQ vLLM inference with tensor parallel handling * [fix] register rotated linear method for vllm weight_loader_v2 dispatch - register RotatedLinearMethod via vllm's weight_loader_v2 allowlist so wrapped GPTQ / GPTQ-Marlin layers keep the same loading path as unwrapped vllm methods - add regression coverage for rotated down_proj dispatch and base process_weights_after_loading delegation - add unit test for the online Hadamard target predicate - document the TP>1 all_gather cost in _apply_tp_hadamard - clarify that rotation hooks currently target dense mlp.down_proj layers only * [test]Add smoke test with TP=2 * [test] fix dependency * [test]Add rotation unit tests and runner save-path coverage * [test]TP1 rotated GPTQ e2e coverage * [test] Move rotation helper tests into utils package * [test]make RotationMetadata.from_quant_config tests explicit * [test]cover missing rotation utility edge cases * [fix] enable DBF + rotation support in vLLM plugin - Add RotationMetadata support to DbfConfig - Wrap quantized layers with RotatedLinearMethod when rotation is configured - Expose layer dimensions in DoubleBinaryLinear * [test] add DBF rotation plugin integration tests and Runner coverage * [test] TP1 rotated DBF e2e coverage * [test] strengthen DBF rotation plugin unit tests - Add test_get_quant_method_wraps_dbf_linear_method_for_quantized_down_proj: verifies DBFLinearMethod (not UnquantizedLinearMethod) is the base when quantization_bits has a real entry, and that _dbf_mod_cfg/_dbf_prefix are set - Add test_prehook_is_installed_on_down_proj_after_process_weights: verifies _onecomp_hadamard_prehook_installed is True after process_weights_after_loading through the DBF quantized path - Add test_create_weights_raises_for_tensor_parallel_size_greater_than_one: verifies DBFLinearMethod.create_weights raises ValueError when get_tensor_model_parallel_world_size returns > 1 (monkeypatched) * [test] trim redundant rotated-vs-plain e2e quantize tests The saved-config metadata contrast (rotated flag + quant_method routing) is already covered by the cheap fake-config unit tests in test_runner_rotated_vllm_save.py, so the e2e config-comparison tests only re-verified that logic on top of a full quantization run. DBF: drop test_plain_dbf_uses_different_saved_load_path and rename TestRotatedVsPlainDBFVllmInference -> TestPlainDBFVllmInference. The plain generate smoke is kept because plain DBF still loads through the in-house "dbf" plugin. GPTQ: remove TestRotatedVsPlainGPTQVllmInference and the now-unused plain_quantized_model_dir fixture entirely, eliminating a second GPTQ quantization run. Plain GPTQ ("gptq") is served by vLLM's built-in handler and exercises no first-party code, so the smoke added little value. * [fix] fall back to naive path on GemLite inference failure with process-wide disable * [test] add GemLite fallback regression tests and strengthen assertions * [docs] document GemLite automatic fallback behavior and TRITON_CACHE_AUTOTUNING * [test] strengthen GemLite fallback tests: fused 2D scaling0, real naive, bias Cover the production-critical and previously-untested paths in DBFLinearMethod.apply()'s GemLite -> naive fallback: - Add test_fused_fallback_matches_real_naive (part_count=3). Uses a 2D (part_count, in_features) scaling0 so it exercises the `scaling0.ndim == 2` per-part-index branch in _compute_parts() that the fused qkv_proj actually hits. With only _apply_gemlite forced to fail, it runs the real _compute_parts / _apply_naive and verifies torch.cat(dim=-1) concat, per-part scaling2/scaling4/bp offset slicing, and the naive numerics against an independent reference. - Add test_bias_is_added_to_output covering the final `out + bias` path. - Replace caplog with a warning_spy fixture that patches the module logger directly, so the warning assertions no longer depend on the vLLM logger keeping propagate=True. * [docs] clarify that OOM is not caught by the GemLite fallback * [test] modularize vllm_plugins test infrastructure: shared conftest, session fixtures, try/finally cleanup * [test] consolidate LLM import in conftest and guard DBF integration test * [test] remove dead code in build_vllm_llm and document DBF skip gate Follow-up cleanup to the vllm_plugins conftest consolidation: - conftest.py: drop the unreachable `return LLM(...)` after pytest.skip() (pytest.skip raises NoReturn, so the line never executes and no type checker is configured to require it), and remove the stray blank line left between the vllm import and its except clause. - test_rotated_dbf_e2e.py: note in the module docstring that the file is skipped by default and enabled via RUN_DBF_INTEGRATION_TESTS=1, matching the convention in develop/v1-3-0's test_global_ptq_integration_dbf.py. * [docs] add v1.3.0 rotation/vLLM changelog and fix outdated rotation notes Add a CHANGELOG entry for the rotation-save-load-vllm-infer branch covering rotated GPTQ/DBF vLLM inference, the DBF GemLite automatic fallback, and the new tests. Correct the now-stale "rotation is not vLLM-servable" notes in the docs: rotation-preprocessed checkpoints are servable through the mixed_gptq and dbf plugins (dbf is TP1-only), while built-in gptq paths (RTN/JointQ) are out of scope. * [style] Apply pre-commit fixes * [fix(vllm)] mirror base method's weight_loader_v2 support in rotation wrapper * [docs] clarify rotation wrapper weight-loader dispatch in CHANGELOG --------- Co-authored-by: koohr <kou.ohira@compmind.co.jp>
* [update] Bitpack GPTQ weights and zeros immediately after quantization * [fix] disable GPTQ bitpacking for JointQ initialization * [fix] existing GPTQ tests to use unpacked quantize results by default * [fix] existing AutoBit tests to use unpacked quantize results by default * [update] Propagate AutoBit bitpack mode before child quantizer validation * [update] Add bitpack-mode flag (bitpack_on_quantize) default to base Quantizer * [update] Validate bitpack_on_quantize support in GPTQ params * [update] Add GPTQ bitpack equivalence test and example * [fix] Limit GPTQ wbits validation to 15 bits * [update] add test_gptq_bitpack.py for cover bitpack metadata, dequant, inference paths * [fix] Fail AutoBit validation for unsupported GPTQ bitpack wbits * [update] Update CHANGELOG.md (bitpack_on_quantize) * [style] Apply black formatter * [update] Update CHANGELOG.md (v1.3.0-wip merge into bitpack-mode branch) * [update] Add DBF bitpack_on_quantize flag and validation Add a DBF-specific bitpack_on_quantize flag (default True) to the DBF quantizer dataclass and validate it as a bool in validate_params(). DBF packs arbitrary shapes via padding, so no bit-width/shape constraint is imposed; only the bool type is checked. * [update] Add DBFResult packed-state metadata and unpack helpers Add packed-state metadata (dbf_A_is_packed, dbf_B_is_packed, dbf_A_original_shape, dbf_B_original_shape) and a get_unpacked_binary_factors() helper to DBFResult so callers can obtain the unpacked +/-1 factors regardless of storage. The helper returns float16 for both packed and unpacked storage so the representation difference never leaks to callers. Add shape-aware unpack_binary_matrix() (and pack_binary_matrix alias) to dbf_layer so the pack/unpack shape-restore logic lives in one place. Old results without the metadata fields are treated as unpacked via getattr defaults. * [update] Bitpack DBF binary factors right after quantization In DBF.quantize_layer(), when bitpack_on_quantize is enabled and DBF produced both binary factors, pack dbf_A / dbf_B into uint8 CPU tensors immediately after run_dbf() and record the packed-state metadata on the DBFResult. The unpacked references are dropped early to reduce RAM held during quantization. On DBF failure / is_dbf_quantized=False the factors are left untouched, preserving the existing dequant error behavior. * [update] Make DBF dequant path packed/unpacked agnostic compute_dequantized_weight() now recovers the +/-1 factors via get_unpacked_binary_factors(), so it returns the same weight whether dbf_A / dbf_B are stored packed or unpacked (pack/unpack is value-preserving). The reconstruction formula is unchanged. Route the existing in-layer unpack call sites (BitLinearPacked.forward and DoubleBinaryLinear._unpack_bp) through the shared unpack_binary_matrix helper to avoid duplicating the shape-restore logic. * [update] Avoid re-packing already-packed factors in DoubleBinaryLinear DoubleBinaryLinear.__init__() / from_quantization_result() now accept already-packed dbf_A / dbf_B (with their original shapes) and register them straight into bp1 / bp3 without an unpack/re-pack round-trip; unpacked inputs are packed as before. GemLite initialization builds a short-lived unpacked copy for packed inputs only (not retained). The saved checkpoint format (bp1 / bp3) and from_saved_state() are unchanged. * [update] Make existing DBF tests packed/unpacked agnostic Update the existing DBF quantizer test assertions that assumed unpacked float16 dbf_A / dbf_B: check_quantize_layer branches on the packed flags (uint8/1D when packed, float16/2D otherwise) and reconstructs the weight via get_unpacked_binary_factors(); check_equal_results and the apply-to-module helper compare/store factors through the same helper so they hold for both representations. No changes were needed in the blockwise post-processing optimizers or the vLLM DBF plugin, which work on the unchanged bp1 / bp3 packed buffers. * [update] Propagate AutoBit bitpack mode to child candidates and DBF fallback Add a bitpack_on_quantize flag (default True) to AutoBitQuantizer and propagate it to candidate quantizers that support on-quantize bitpacking: validate_params() syncs the flag onto GPTQ/DBF child candidates (via _sync_child_bitpack_on_quantize) before their own validation, _assign_all_dbf() forwards it to the all-DBF quantizer, and the inject_dbf() fallback path forwards it to every DBF it creates. A non-bool flag raises a clear ValueError. * [update] Add DBF bitpack representation and propagation tests Add DBF quantizer tests covering both representations: default flag is True; packing stores uint8 1D factors with shape metadata; disabling keeps unpacked float16; packed and unpacked dequantized weights are bit-identical (both from quantize_layer and hand-built results); pack_binary -> unpack_binary_matrix is bit-exact across shapes; missing original shape raises; from_quantization_result registers bp1/bp3 with no re-pack and forward matches the dequantized linear; and a non-bool flag raises. Add AutoBit tests that the flag is synced onto DBF candidates and forwarded to inject_dbf fallbacks (1-bit GPTQ on a 64-wide layer to stay under the DBF threshold). * [update] Smoke-test DBF packed results through downstream consumers QEP, LPCD, chunked / multi-GPU quantization, and the cumulative-error analyzer all consume DBF results only via compute_dequantized_weight().to(device).to(dtype) and never touch dbf_A / dbf_B directly, so no functional change is needed for the on-quantize bitpack. Add a regression smoke test exercising that exact consumption path (analyzer._update_weights) with a packed DBFResult, confirming it succeeds and yields weights identical to the unpacked representation (no packed state leaks to callers). * [update] Add DBF save/load round-trip and vLLM e2e/config smoke tests - pre_process pipeline: add DBF quantized/dequantized save/load round-trip cases for TinyLlama and Qwen3 (5 cases: GPTQ q/dq, DBF q/dq, RTN dq) - tests/vllm_plugins/dbf/test_dbf_e2e.py: DBF quantize -> save -> config verification -> vLLM generation smoke (slow, CUDA/vLLM gated) - tests/vllm_plugins/dbf/test_dbf_config.py: DBF plugin config parse/dispatch smoke (DbfConfig) * [add] Add DBF bitpack runner smoke tests Add DBF packed-result smoke coverage for QEP, LPCD, and chunked calibration calc_quant_error paths. The tests verify that each Runner path produces packed dbf_A / dbf_B results and consumes them through compute_dequantized_weight() without leaking storage details to callers. * [fix] Expose in_features/out_features on DoubleBinaryLinear register_online_hadamard_hooks -> get_hadK introspects module.in_features, but DoubleBinaryLinear only stored _bp1_shape/_bp3_shape, raising AttributeError when re-registering Hadamard hooks on saved DBF-quantized rotated models. Expose in_features/out_features like nn.Linear (derived from the original unpacked shapes; also set in from_saved_state). * [update] Use DBF factor pack helpers * [update] Add DBF bitpack equivalence example Quantize a layer with bitpack_on_quantize=False vs True under identical seeds and assert the dequantized weights match bit-exactly (compute_dequantized_weight -> torch.equal). The corresponding DBF equivalence test is in tests/onecomp/quantizer/dbf/test_dbf_bitpack_equivalence.py. * [update] Add DBF vLLM inference example Parallels example_gptq_vllm_inference.py for the DBF path: quantize TinyLlama with DBF, save, then load and generate with vLLM. * [update] Document bitpack_on_quantize in DBF and AutoBit algorithm docs Add a bitpack_on_quantize row to the parameters table in docs/algorithms/dbf.md (DBF) and docs/algorithms/autobit.md (propagation to candidate / injected-DBF quantizers). * [update] Register DBF bitpack and vLLM examples in README and vLLM inference guide * [fix] Log bitpack_on_quantize overrides in JointQ and AutoBit * [doc] Clarify GPTQ bitpack-on-quantize behavior - Document the breaking change in CHANGELOG - Add a comment explaining qzeros reshape after pack/unpack * [refactor] Inline GPTQ_MAX_BITS usage in pack support validation * [fix] Centralize GPTQ bit-packing width checks * [refactor] Clarify GPTQLinear packed state handling * [update] Update CHANGELOG.md (DBF bitpack-on-quantize) * [style] Apply black formatter * [docs] update CHANGELOG.md for GPTQ pack_weights validation * [fix] Use is_packable_wbits to decide weight packing * [fix] Log pack_weights fallbacks for unsupported GPTQ wbits * [fix] Normalize float GPTQ wbits and fix config bits truncation * [docs] update CHANGELOG.md for GPTQ wbits normalization * [chore] Remove redundant bitpack equivalence examples Equivalence is already covered by tests, and these examples do not demonstrate user-facing usage. * [style] Apply pre-commit fixes * [docs] Clarify bitpack-on-quantize support boundaries - document GPTQ quantize-time and save-time packing constraints - clarify AutoBit propagation, overrides, and fused-group restrictions - document DBF and JointQ bitpacking behavior - record these doc updates in the DBF and GPTQ bitpack changelog sections * [docs] Sync example references after bitpack example cleanup - Remove references to the deleted DBF and GPTQ equivalence examples - Add the missing Qwen3.6 vLLM example to README and the inference guide --------- Co-authored-by: cm_ysmz <yohei.shimizu@compmind.co.jp> Co-authored-by: sikoji <seiichiro.kojio@compmind.co.jp>
… FusedMoE kernel support) (#39) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * bugfix: success save load * succeeded in inferecing MoE quantized model with vLLM * refactoring, linter, add test, and fix changelog * revert example modification * add test and fix changelog * revert example modification * fix format * bugfix: success save load * add test and fix changelog * update CHANGELOG * succeeded in inferecing MoE quantized model with vLLM * update CHANGELOG * add test and fix changelog * update comment for review no.4, 6, 7, and 8 * generalize moe detection * comment2: fallback when group_size is None * comment3: add test * comment5: allow rtn fallback when every children quantizer GPTQ * missed adding __init__.py * lazy import is_moe_expert_g_idx_key in runner --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.