feat: native tokenization engine for BaseModel

[ptomecek]

Summary

Replace dask-based tokenization with a standalone, composable engine that works with any pydantic BaseModel.

API

• model.model_token — deterministic hex digest of a model's data (and optionally behavior)
• __ccflow_tokenizer__ ClassVar — swap the tokenizer engine per-class or globally
• __ccflow_tokenizer_deps__ ClassVar — declare standalone function dependencies for behavior hashing

Architecture

Two independent axes of composition:

• FunctionCollector (which functions to hash) × SourceTokenizer (how to hash them)
• DefaultTokenizer() — data only
• DefaultTokenizer.with_bytecode() — data + bytecode-hashed own methods (default for behavior)
• DefaultTokenizer.with_ast() — data + AST-normalized own methods

What changed

File	Change
ccflow/utils/tokenize.py	Core engine — SourceTokenizer/FunctionCollector ABCs, normalize_token singledispatch, DefaultTokenizer
ccflow/base.py	model_token property, __ccflow_tokenizer__ ClassVar, cache invalidation
ccflow/callable.py	ModelEvaluationContext.model_token — strips transparent evaluator layers
ccflow/evaluators/common.py	Simplified cache_key() to use model_token directly
ccflow/utils/__init__.py	Export new types
pyproject.toml	Remove dask from tokenization path
docs/wiki/Tokenization.md	Comprehensive wiki page
ccflow/tests/utils/test_tokenize.py	182 test cases

Extension points

1. normalize_token.register(MyType) — custom type handlers via singledispatch
2. __ccflow_tokenize__() — custom canonical form for any object
3. FunctionCollector / SourceTokenizer subclasses — custom strategies
4. Field(default_factory=...) — inject external state (package versions, file checksums, env vars) into tokens

Limitations

• No transitive dependency tracking (only own methods + explicit deps)
• Bytecode tokens change across Python minor versions
• AST tokens change on variable renames
• See wiki page for full false-hit/false-miss tables

[github-actions]

Test Results

854 tests  +206   852 ✅ +206   1m 44s ⏱️ -2s
  1 suites ±  0     2 💤 ±  0 
  1 files   ±  0     0 ❌ ±  0 

Results for commit bac1375. ± Comparison against base commit 5adf933.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 93.63745% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.65%. Comparing base (5adf933) to head (bac1375).

Files with missing lines	Patch %	Lines
ccflow/tests/utils/test_tokenize.py	93.99%	74 Missing and 5 partials ⚠️
ccflow/utils/tokenize.py	91.91%	18 Missing and 6 partials ⚠️
ccflow/base.py	96.29%	0 Missing and 1 partial ⚠️
ccflow/callable.py	95.65%	0 Missing and 1 partial ⚠️
ccflow/evaluators/common.py	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@                     Coverage Diff                     @@
##           pit/evaluator_cache_key     #195      +/-   ##
===========================================================
- Coverage                    95.98%   95.65%   -0.34%     
===========================================================
  Files                          140      141       +1     
  Lines                         9797    11445    +1648     
  Branches                       568      619      +51     
===========================================================
+ Hits                          9404    10948    +1544     
- Misses                         275      367      +92     
- Partials                       118      130      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[NeejWeej]

Maybe we could narrow model_token down.
Something like:

• model_token means data token only
• behavior becomes separate: behavior_version() or model_behavior_token
• cache key combines them
  That removes a lot of conceptual confusion.
  So for a callable model:
  model.model_token # explicit field-data snapshot
  model.behavior_version() # code semantics
  Then the cache-key generator composes both.

[ptomecek]

Maybe we could narrow model_token down. Something like:

• model_token means data token only
• behavior becomes separate: behavior_version() or model_behavior_token
• cache key combines them
  That removes a lot of conceptual confusion.
  So for a callable model:
  model.model_token # explicit field-data snapshot
  model.behavior_version() # code semantics
  Then the cache-key generator composes both.

Part of me wonders if we should even both with caching/having it on the model. We could remove any access from the model itself, and just have everything go through the existing tokenizer classes. It would then be the responsibility of the evaluator to do any caching.

The drawback of that is that I think it's then more difficult for people to debug issues because it's harder to reproduce exactly what was used (i.e. from a notebook)...