Add model analyzer scripts and setup

[brendandahl]

Introduce Python scripts for analyzing ONNX and TFLite models to help understand their structure and features.

Added a local virtual environment setup to manage dependencies without affecting the global system.

[rmahdav]

Maybe add (or onnx files path in this example) here at the end for more clarity?

[rmahdav]

Interesting add-on to the benchmark! Do you have any sample analysis result? And how do you think the analysis result will help us improving the benchmark?

[brendandahl]

I'm planning to run this on more of the models that we're thinking of adding to see if they cover anything new. I was also curious what are the most common operations and data types.

Here's a sample from one:

=== Model: /usr/local/google/home/dahlb/projects/webai-compute-benchmark/resources/transformers-js/models/AdamCodd/vit-base-nsfw-detector/onnx/model_q4.onnx ===
Inputs:
  - Name: pixel_values, Shape: [batch_size, num_channels, height, width], Type: FLOAT
Outputs:
  - Name: logits, Shape: [batch_size, 2], Type: FLOAT
Structure:
  - Total operations (including subgraphs): 626
  - Unique operation types: 23
  - Subgraphs: 0
  - Operations breakdown (top 15):
    - Add: 159
    - Mul: 73
    - MatMulNBits: 72
    - ReduceMean: 50
    - Reshape: 49
    - Transpose: 49
    - Div: 37
    - Sub: 25
    - Pow: 25
    - Sqrt: 25
    - MatMul: 24
    - Softmax: 12
    - Erf: 12
    - Concat: 3
    - Shape: 2
    - ... and 8 more types
Features:
  - Uses explicit quantization/integer ops: True
  - Filename implies quantization: True

I've also collected some overall data:

=== TFLite Operation Usage (Number of models using each operation) ===
Analyzed 4 models.
CONV_2D: 4
DELEGATE: 4
DEPTHWISE_CONV_2D: 4
PAD: 4
ADD: 4
MAX_POOL_2D: 2
LOGISTIC: 2
RESIZE_BILINEAR: 2
RESHAPE: 2
TRANSPOSE_CONV: 2
MUL: 2
HARD_SWISH: 2
CONCATENATION: 1
AVERAGE_POOL_2D: 1
FULLY_CONNECTED: 1
MEAN: 1
MINIMUM: 1
SUB: 1

=== ONNX Operation Usage (Number of models using each operation) ===
Analyzed 16 models.
Add: 16
Reshape: 16
Mul: 16
Concat: 15
Shape: 15
ReduceMean: 15
Div: 15
MatMul: 15
Transpose: 15
Softmax: 15
Sqrt: 14
Sub: 14
Pow: 14
Unsqueeze: 14
Gather: 14
Erf: 12
Slice: 12
Cast: 11
Expand: 10
Constant: 9
Where: 9
ConstantOfShape: 8
Squeeze: 8
Range: 8
Equal: 7
Conv: 7
MatMulNBits: 6
Less: 5
Neg: 4
Split: 4
Gemm: 4
Tile: 4
Clip: 3
Log: 3
Greater: 3
Sin: 3
Pad: 3
Sigmoid: 3
Cos: 3
Resize: 3
Tanh: 3
Relu: 3
LessOrEqual: 2
And: 2
Abs: 2
Identity: 2
ScatterND: 2
ConvTranspose: 2
MatMulBnb4: 2
BatchNormalization: 2
If: 2
Min: 2
GatherElements: 1
Ceil: 1
Sign: 1
Exp: 1
Round: 1
NonZero: 1
STFT: 1
ReduceSum: 1
LSTM: 1
LeakyRelu: 1
GreaterOrEqual: 1
Atan: 1
CumSum: 1
LayerNormalization: 1
Floor: 1
GlobalAveragePool: 1
DynamicQuantizeLinear: 1
MatMulInteger: 1
InstanceNormalization: 1
ConvInteger: 1
Einsum: 1
Mod: 1
Not: 1
OneHot: 1
AveragePool: 1
Flatten: 1
ArgMax: 1

[brendandahl]

Letting gemini do some analysis:

prompt:

In script/model-analyzer there are two scripts for analyzing the onnx and tflite models in this repo. To run:
- `./script/model-analyzer/venv/bin/python ./script/model-analyzer/analyze_tf.py resources --all-ops`
- `./script/model-analyzer/venv/bin/python ./script/model-analyzer/analyze_onnx.py resources --all-ops`
Run the analyzers on all the models in the repo and provide a report that covers:
- What operations are well covered
- What data types are well covered
- What models have a lot of overlap
- Any other interesting observations you note

Model Analysis Report

Overview

This report summarizes the analysis of ONNX and TFLite models found in the repository, specifically within the resources directory. A total of 16 ONNX models and 4 TFLite models were analyzed.

Well Covered Operations

ONNX Models

The ONNX models are heavily dominated by Transformer architectures and attention mechanisms, leading to high coverage of the following operations:

• Core Arithmetic & Math: Add, Mul, Div, Pow, Sqrt, Erf.
• Tensor Manipulation: Unsqueeze, Reshape, Concat, Transpose, Gather, Shape, Slice.
• Linear Algebra: MatMul, Gemm.
• Reductions & Activations: ReduceMean, Softmax.
• Quantized Operations: MatMulNBits (4-bit) and MatMulBnb4 are well represented across several optimized models.

TFLite Models

The TFLite models are primarily vision-based (MediaPipe and MobileNet), showing high coverage of:

• CONV_2D (Convolution)
• ADD
• DEPTHWISE_CONV_2D
• HARD_SWISH
• MUL
• MEAN

Well Covered Data Types

ONNX Models

• FLOAT (float32): The primary data type for activations and standard weights.
• INT64: Widely used for token IDs and masks (input_ids, attention_mask).
• BOOL: Used for control flow flags (e.g., use_cache_branch).
• Quantized Types: While not explicitly listed as types in the summary, the use of MatMulNBits, MatMulBnb4, and ConvInteger implies heavy coverage of 4-bit and 8-bit quantized weights.

TFLite Models

• float32: All 4 analyzed TFLite models are float models. No quantized TFLite models were found in the resources directory.

Models with High Overlap

ONNX Transformers

There is massive overlap among the Transformer-based models:

• Alibaba-NLP/gte-base-en-v1.5
• mixedbread-ai/mxbai-rerank-base-v1
• Xenova/UAE-Large-V1
• Xenova/whisper-small (encoder and decoder)
• Xenova/flan-t5-small (encoder and decoder)
• Xenova/mobileclip_s0 (text model)
• Marqo/marqo-fashionSigLIP (text model)

These models all share the same basic building blocks: large numbers of Gather for embeddings, MatMul (or MatMulNBits) for projections, and sequences of ReduceMean/Sub/Pow/Add/Sqrt/Div for Layer Normalization.

TFLite Vision Models

• HandLandmarkDetector.tflite and HandDetector.tflite have extremely similar operation profiles, both relying heavily on CONV_2D and DEPTHWISE_CONV_2D.

Interesting Observations

• Quantization Prevalence: A large portion of the ONNX models are quantized (indicated by _q4, _bnb4, _uint8 in filenames and specific ops like MatMulNBits). This suggests a strong focus on size and performance optimization.
• Manual vs. Fused Operations: Many ONNX models appear to implement Layer Normalization manually using basic operations rather than using a fused LayerNormalization node (except for Kokoro-82M).
• TFLite Uniformity: The TFLite models are very uniform, all being float32 and targeting similar vision tasks with similar architectures.

[brendandahl]

I would also be fine with this living in my branch only, since it's largely vibe coded and for research only.

[rmahdav]

Do we need this allocation for the analysis? I think this line will fail for models with dynamic shapes e.g. [?, 224, 224, 3]. Maybe we can remove it if it is not necessary/needed, or somehow handle the dynamic shape?

[rmahdav]

I think there are more quantization related operations, meybe we have have a more comprehensive list? The below is AI suggested list, I haven't check to see if it is complete and correct:

Suggested change

	has_quantization = any(op.startswith("Quantize") or "MatMulInteger" in op or "DynamicQuantize" in op or op == "MatMulNBits" for op in op_counts)
	quant_ops = {
	"QuantizeLinear", "DequantizeLinear", "QLinearConv",
	"QLinearMatMul", "MatMulInteger", "ConvInteger",
	"DynamicQuantizeLinear", "MatMulNBits"
	}
	has_quantization = any(
	any(q_op in op for q_op in quant_ops)
	for op in op_counts
	)

[rmahdav]

I added a few comments. The output and analysis are so interesting and I would love to have the script merged into the repository.