What's logits_all meaning in perplexity?

[Septa2112]

Reading perplexity.cpp, I found the below code in main function.

int main(int argc, char ** argv) {
    ...
    params.n_ctx = 512;
    params.logits_all = true;
    ...
}

But I can’t understand params.logits_all = true; very well.

The option --all-logits must be used with llama-perplexity. But when running llama-perplexity, logits_all is always set to true by params.logits_all = true;, so I found:

1. The option --all-logits doesn’t seem to have any effect on llama-perplexity?
2. I tried changing this value to false, but didn’t see any change in the output of llama-perplexity.

Perhaps someone can help answer my question? Thanks.

[ngxson]

To measure the perplexity, you need to know the logits of all tokens. Without logits_all, you can only receive the logits of last token in sequence (which is useful in generation mode).

The reason why params.logits_all does not have any effect in llama-perplexity is because the logits it set in batch:

https://github.com/ggerganov/llama.cpp/blob/cfac111e2b3953cdb6b0126e67a2487687646971/examples/perplexity/perplexity.cpp#L604

params.logits_all maybe used in another example so it's being kept, but marked as deprecated in common.h

[Septa2112]

Thanks a lot!