LLM Quantization Types: a Cheat Sheet

Trying to run a large language model locally but not sure which file to download? Q4_K_M, IQ3_S, Q5_K_M - these aren’t random strings. They describe the quantization format, which determines response quality and how much memory the model will use.