Skip to content

imatrix : handle partial entries #7833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 9, 2024
Merged

Conversation

ggerganov
Copy link
Member

fix #7816

Print warning messages when imatrix entries have zero counts:

compute_imatrix: tokenizing the input ..
compute_imatrix: tokenization took 22899.3 ms
compute_imatrix: computing over 4918 chunks with batch_size 256
compute_imatrix: 12.00 seconds per pass - ETA 16 hours 23.53 minutes
[1]6.6864,[2]9.1522,[3]9.9931,[4]8.8414,
save_imatrix: entry '             blk.17.ffn_down_exps.weight' has partial data (93.75%) - skipping
save_imatrix: entry '             blk.17.ffn_gate_exps.weight' has partial data (93.75%) - skipping
save_imatrix: entry '               blk.17.ffn_up_exps.weight' has partial data (93.75%) - skipping
save_imatrix: entry '             blk.16.ffn_down_exps.weight' has partial data (73.44%) - skipping
save_imatrix: entry '             blk.16.ffn_gate_exps.weight' has partial data (73.44%) - skipping
save_imatrix: entry '               blk.16.ffn_up_exps.weight' has partial data (73.44%) - skipping
save_imatrix: entry '             blk.15.ffn_down_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '             blk.14.ffn_down_exps.weight' has partial data (60.94%) - skipping
save_imatrix: entry '               blk.14.ffn_up_exps.weight' has partial data (60.94%) - skipping
save_imatrix: entry '              blk.1.ffn_down_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '             blk.13.ffn_gate_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '             blk.13.ffn_down_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '             blk.11.ffn_down_exps.weight' has partial data (59.38%) - skipping
save_imatrix: entry '               blk.11.ffn_up_exps.weight' has partial data (59.38%) - skipping
save_imatrix: entry '             blk.15.ffn_gate_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '               blk.15.ffn_up_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '             blk.14.ffn_gate_exps.weight' has partial data (60.94%) - skipping
save_imatrix: entry '             blk.12.ffn_down_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '             blk.12.ffn_gate_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '               blk.12.ffn_up_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '              blk.3.ffn_down_exps.weight' has partial data (98.44%) - skipping
save_imatrix: entry '                blk.1.ffn_up_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '               blk.13.ffn_up_exps.weight' has partial data (62.50%) - skipping
save_imatrix: entry '              blk.1.ffn_gate_exps.weight' has partial data (78.12%) - skipping
save_imatrix: entry '              blk.3.ffn_gate_exps.weight' has partial data (98.44%) - skipping
save_imatrix: entry '                blk.3.ffn_up_exps.weight' has partial data (98.44%) - skipping
save_imatrix: entry '              blk.2.ffn_down_exps.weight' has partial data (96.88%) - skipping
save_imatrix: entry '              blk.2.ffn_gate_exps.weight' has partial data (96.88%) - skipping
save_imatrix: entry '             blk.11.ffn_gate_exps.weight' has partial data (59.38%) - skipping
save_imatrix: entry '                blk.2.ffn_up_exps.weight' has partial data (96.88%) - skipping
save_imatrix: warning: storing only 306 out of 336 entries

save_imatrix: stored collected data after 10 chunks in imatrix.dat

Such entries are not stored in the output matrix to prevent errors when using the imatrix. To prevent this from happening, provide larger and more diverse training data

@ggerganov ggerganov force-pushed the gg/imatrix-partial-data branch from 175a179 to 5a21852 Compare June 8, 2024 09:40
@ggerganov ggerganov merged commit e95beeb into master Jun 9, 2024
57 of 68 checks passed
@ggerganov ggerganov deleted the gg/imatrix-partial-data branch June 9, 2024 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: QWEN2 MoE imatrix contains nan's after generating it
1 participant