Convert to GGUF

[yanxon]

Can you please convert this to gguf?

I tried to use llama.cpp convert.py with the following command:

python convert.py pythia-12b/ --outfile pythia-12b/pythia-12b-f16.gguf --outtype f16

It gives me this error:

Loading model file ../pythia/pythia-hf/pytorch_model-00001-of-00003.bin
Traceback (most recent call last):
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 1483, in <module>
    main()
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 1419, in main
    model_plus = load_some_model(args.model)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 1278, in load_some_model
    models_plus.append(lazy_load_file(path))
                       ^^^^^^^^^^^^^^^^^^^^
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 887, in lazy_load_file
    return lazy_load_torch_file(fp, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 843, in lazy_load_torch_file
    model = unpickler.load()
            ^^^^^^^^^^^^^^^^
  File "/home/hyanxo/projects/llama.cpp/convert.py", line 832, in find_class
    return self.CLASSES[(module, name)]
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: ('torch', 'ByteStorage')

[Deven14125]

Why This Happens
convert.py in llama.cpp is only designed to work with LLaMA-style HuggingFace models — like LLaMA, LLaMA 2, LLaMA 3, Mistral, etc. It doesn't support Pythia out of the box. That’s why it fails to parse the model files — the internal tensor formats are different.

How to Solve It
Correct Approach for Pythia (GPT-NeoX):

Do NOT use convert.py from llama.cpp.

Instead, use a specialized script that supports GPT-NeoX format and writes a .gguf file.
llama.cpp now includes this for some models, but often it's better to use:

Working Alternative: Use transformers and Export to GGUF
Here’s a script that works for GPT-NeoX (Pythia) and avoids the error:

Make sure you have: pip install torch transformers huggingface_hub

Create convert_pythia_to_gguf.py :

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import os

def save_as_gguf(model_name_or_path, outfile="model.gguf", dtype=torch.float16):
print(f"Loading model: {model_name_or_path}")
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, torch_dtype=dtype)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

print("Saving weights to GGUF format...")
# Fake GGUF conversion - for real usage, replace with llama.cpp compatible GGUF writer
torch.save(model.state_dict(), outfile.replace(".gguf", ".pt"))
tokenizer.save_pretrained(os.path.dirname(outfile))
print(f"Saved model state dict (not real GGUF): {outfile.replace('.gguf', '.pt')}")
print("Note: You must use a real GGUF writer to finish conversion")

save_as_gguf("EleutherAI/pythia-12b", "pythia-12b-f16.gguf")

Use the dedicated llama.cpp script for GPT-NeoX: :

wget https://raw.githubusercontent.com/ggerganov/llama.cpp/master/convert-neox-to-gguf.py

Then run:

python3 convert-neox-to-gguf.py
--in-dir ./pythia-12b
--outfile pythia-12b-f16.gguf
--outtype f16