Why published quantization benchmarks disagree
Two publishers can report different values without either value being a transcription error. A different reference checkpoint, corpus, context length, runtime or metric definition changes the experiment.
Check whether a result compares against BF16, Q8 or another reference. Distinguish mean KLD, tail KLD, perplexity, top-1 agreement and downstream task scores. A copied benchmark is not an independent replication; follow the source and copied-claim flags.
If methodology is missing or results conflict, report the uncertainty and show each claim with its provenance. Do not average incompatible scores or infer a winner from the smallest number. A controlled rerun is needed to resolve experimental disagreement.