AI Changelog Aggregator

examples : add debug utility/example (#18464)

examples : add debug utility/example

This commit introduces a new example named llama-debug which is a utility that is intended to be used to assist with developing/debugging a converted model.

The motivation for this utilitiy is to assist in model conversion work to verify that the model produces the expected outputs. It is intended to replace logits.cpp in examples/model-conversion.

Example usage:

./build/bin/llama-debug \
    -m models/Qwen2.5-0.5B-Instruct.gguf \
    --prompt "Hello, my name is" \
    --save-logits
...
Model add_bos: false
Input prompt: "Hello, my name is"
Token ids (5):
Hello(9707) ,(11)  my(847)  name(829)  is(374)
Data saved to data/llamacpp-Qwen2.5-0.5B-Instruct.bin
Data saved to data/llamacpp-Qwen2.5-0.5B-Instruct.txt
Prompt saved to data/llamacpp-Qwen2.5-0.5B-Instruct-prompt.txt
Tokens saved to data/llamacpp-Qwen2.5-0.5B-Instruct-tokens.bin

For more details about the options available for this example, please refer to examples/debug/README.md.

throw runtime error instead of logging error
remove params.warmup and enable the warmup/nowarmup option
model-conversion : remove logits.cpp

This commit removes logits.cpp in favor of using llama-debug for generating logits and embeddings.

examples : remove model-conversion directory

This was missed in the previous commit.

model-conversion : add support for saving prompt and token ids

This commit add support for storing the prompt and the token ids for the prompt when running the original models.

The motivation for this is that this will allow us to compare the prompt and the tokens generated for the prompt when verifing the converted model. Currently it is possible that even if the same prompt is used that the tokens generated are different if there is a difference in the tokenization between the original and converted model which would currently go unnoticed (the verification will most likely fail but it might not be obvious why).

squash! model-conversion : add support for saving prompt and token ids

fix pyright errors.

model-conversion : add compare_tokens utility

This commit adds a script to compare token outputs between original and converted models.

Example usage:

(venv) $ ./scripts/utils/compare_tokens.py pytorch-gemma-3-270m-it llamacpp-gemma-3-270m-it-bf16

Comparing tokens between:
  Original : pytorch-gemma-3-270m-it (6 tokens)
  Converted: llamacpp-gemma-3-270m-it-bf16 (6 tokens)

✅ All 6 tokens match!

And there is a verbose flag that will also print out the prompts:

(venv) $ ./scripts/utils/compare_tokens.py pytorch-gemma-3-270m-it llamacpp-gemma-3-270m-it-bf16 -v

Original model prompt (pytorch-gemma-3-270m-it):
  prompt: Hello, my name is
n_tokens: 6
token ids: 2, 9259, 236764, 1041, 1463, 563

Converted model prompt (llamacpp-gemma-3-270m-it-bf16):
  prompt: Hello, my name is
n_tokens: 6
token ids: 2, 9259, 236764, 1041, 1463, 563

Comparing tokens between:
  Original : pytorch-gemma-3-270m-it (6 tokens)
  Converted: llamacpp-gemma-3-270m-it-bf16 (6 tokens)

✅ All 6 tokens match!

model-conversion : add token comparison to verifiction scripts

This commit add the calling of the compare_tokens function in compare-logits.py and semantic_check.py to ensure that the token ids that the tokenizers procoduce are the same before proceeding with verifying the logits/embeddings.

Placing them in the existing scripts instead calling them separately ensures that the token comparison is always done prior to the logit/embedding verifications.

Follow up commit/pr could refactor the causal logits verification into a single script instead of the two that exist now. This would reduce the code and make it consistent with the embeddings verficiation which only has a single script.

debug : use llama_model_n_embd_out

This commit updates the debug example to use the new function llama_model_n_embd_out instead of llama_model_n_embd.

The motivation for this change is to support late interation retriever models, like LFM2-ColBert-350M, where the output embeddings are down projected to a lower dimension.

debug : add print_usage function

This commit adds a print_usage function that is passed to the common_params_parse.

The motivation for this is that this enables a specific usage message which will be printed after all the options, for example:

example usage:

  Print tensors:

  ./build/bin/llama-debug -m model.gguf -p "Hello my name is" --verbose

  The tensors to be printed can be filtered with --tensor-filter option.

  Save logits/embeddings:

  ./build/bin/llama-debug -m model.gguf -p "Hello my name is" --save-logits

  Add --embedding to save embeddings

macOS/iOS:

Linux:

Windows:

openEuler: