AI Changelog Aggregator

llama : remove write/read of output ids/logits/embeddings (#18862)

llama : remove write/read of output ids/logits/embeddings

This commit removes the write/read of output ids, logits and embeddings from the llama context state.

Refs: https://github.com/ggml-org/llama.cpp/pull/18862#issuecomment-3756330941

completion : add replying of session state

This commit updates the session handing in the completion tool to handle the that logits are no longer stored in the session file. Instead, we need to replay the last token to get the logits for sampling.

common : add common_prompt_batch_decode function

This commit adds a new function which is responsible for decoding prompt and optionally handle the saving for session data.

update save-state.cpp to use llama_state_load_file

This commit updates the save-load-state example to utilize the new llama_state_load_file function for loading the model state from a file. And it also replays the last token after loading since this state is now stored before the last token is processed.

examples : set n_seq_max = 2 for ctx3

This commit updates the save-load-state example to set the n_seq_max parameter to 2 when initializing the ctx3 context.

The motivation for this change is that using 1 as n_parallel/n_seq_max the context only supports one sequence, but the test laster tries to use a second sequence which results in the following error:

main : loaded state with 4 tokens
main : seq 0 copied, 225760 bytes
main : kv cache cleared
find_slot: seq_id=1 >= n_seq_max=1 Try using a bigger --parallel value
state_read_meta: failed to find available cells in kv cache

This seems to only happen for recurrent/hybrid models.

macOS/iOS:

Linux:

Windows:

openEuler: