Skip to main content

Model conversion smoke tests

This runner converts selected Hugging Face models, generates a short continuation, and evaluates MMLU. It does not compare outputs or logits against Transformers and is not a numerical parity validator.

Run from the repository root:

export EOLE_MODEL_DIR=/path/to/models
# Set HF_TOKEN if any selected checkpoint requires authentication.
bash recipes/model-validator/run.sh > model-validator.log 2>&1

Before running, edit the models array in run.sh to select a small set that fits your hardware and disk capacity. The default list is a historical catalogue, not a current compatibility guarantee. Each entry must be an HF repository ID; append |quant to enable runtime bitsandbytes NF4 quantization for that model. Install bitsandbytes for those entries. Conversion still stores the original weights, so quantization does not reduce conversion storage requirements.

Generation and MMLU use GPU 0. Results go into outputs/<HF-repository-ID>/, and failures are recorded in error_log.txt. Existing conversion directories can be written again, so use dedicated model storage. This checks execution, not output quality; see MMLU for its evaluation method.