· 6 min read
Quantizing LLMs for in-car inference
GPTQ, AWQ and GGUF are not three versions of the same thing. Here is how they actually differ, and what breaks when you move an LLM onto automotive-grade silicon.

Most people writing about large language models have never had to fit one inside a power budget. You call an API, you get tokens back, and the hard parts — memory bandwidth, thermal headroom, cold-start latency — belong to somebody else.
In a car, they belong to you.
An in-cabin assistant has constraints that cloud deployments simply do not have. The model has to respond while the vehicle has no connectivity. It shares silicon with systems that are not allowed to miss a deadline. And it runs inside a thermal envelope that was designed years before anyone put a transformer in it.
Thermal envelope
Cockpit SoC
- CPUllama.cpp lands here
- GPUbusy drawing the HMI
- NPUwhere the speed is
Shared memory (LPDDR)
Split is illustrative
- 01No connectivityThe model has to answer while the car is offline. There is no cloud fallback.
- 02Deadline-critical neighboursThe cluster and driver assistance share the chip and cannot be starved.
- 03A fixed thermal envelopeDesigned before transformers. When the die heats up, the clocks come down.
Quantization is how you get there. But the three names you see everywhere — GPTQ, AWQ, GGUF — get used interchangeably in a way that causes real damage, because one of them is not the same kind of thing as the other two.
The distinction nobody makes clearly
GPTQ and AWQ are algorithms. They decide which numbers to throw away.
GGUF is a container format. It decides how the numbers are stored on disk.
Treating GGUF as a third quantization method is like comparing JPEG compression to a ZIP file. Related, but not the same category — and confusing them is how teams end up benchmarking a format against an algorithm and drawing conclusions from noise.
Algorithm
Decides which numbers to throw away
- GPTQRounds layer by layer, using second-order (Hessian) information to keep error small.
- AWQFinds the channels that matter most and protects them before rounding.
Container
Decides how the numbers are stored
- safetensorsHugging Face checkpoints. GPTQ and AWQ weights usually ship this way.
- GGUFThe llama.cpp file. Also carries its own k-quant schemes (Q4_K_M…).
- Vendor formatWhat the NPU toolchain compiles to. It quantizes on the way in.
Analogy: JPEG is an algorithm, ZIP is a container. GGUF is closer to ZIP.
GPTQ
Post-training quantization, layer by layer, using second-order (Hessian) information to decide how to round weights so downstream error stays small. It needs a calibration set, it was designed with GPU inference in mind, and it typically targets 4-bit weights.
Its weakness is that calibration data matters more than people expect. Quantize on generic web text, deploy on domain-specific in-cabin phrasing, and the degradation shows up exactly where you least want it.
AWQ
Activation-aware weight quantization. The observation underneath it is that not all weights matter equally — a small fraction of channels, identifiable by looking at activation magnitudes, carry disproportionate importance. AWQ scales those channels before quantizing so they survive the rounding.
In practice it tends to hold accuracy better than GPTQ at the same bit width, and it is generally friendlier to deploy because it does not reorder weights.
GPTQ
Error-compensating, column by column
- Already rounded to 4-bit
- Being rounded now
- Adjusted to absorb the error
AWQ
Activation-aware, salient channels first
- Salient channel (~1%), scaled before rounding
- Everything else, rounded as usual
GGUF
The llama.cpp container. It carries a family of k-quant schemes (Q4_K_M, Q5_K_M and
friends) that mix precision across tensors — more bits for the layers that need them,
fewer for the ones that don't.
- q4-bit
- k4-bit
- v4-bit
- o4-bit
- gate4-bit
- up4-bit
- down4-bit
- q4-bit
- k4-bit
- v6-bit
- o4-bit
- gate4-bit
- up4-bit
- down6-bit
- output6-bit
Its real advantage is portability. One file, a runtime that builds nearly anywhere, CPU inference with optional GPU offload. If you need something running on a bench today, GGUF gets you there fastest.
python convert_hf_to_gguf.py ./llama-3-8b \
--outtype f16 --outfile llama-3-8b-f16.gguf
./build/bin/llama-quantize \
llama-3-8b-f16.gguf llama-3-8b-Q4_K_M.gguf Q4_K_MWhere automotive silicon changes the answer
Here is the part that does not transfer from a laptop benchmark.
On a Qualcomm SoC, the interesting compute is not the CPU. It's the NPU — and the vendor
runtime wants the model to arrive through its own toolchain, in its own quantization
path. A Q4_K_M GGUF that runs beautifully on a workstation may be the wrong artifact
entirely, because the format that is easiest to produce is not the format the accelerator
can actually consume.
NPU pathWhat the car wants
- Checkpoint
- Calibrate on in-cabin text
- GPTQ / AWQ 4-bit
- Vendor toolchain
- NPU
Bench pathFastest to get running
- Checkpoint
- Convert to GGUF
- Q4_K_M
- llama.cpp
- CPU
A Q4_K_M file can’t go straight to the NPU. It has to be unpacked and quantized again by the vendor toolchain: two roundings, and the error compounds.
That reframes the whole decision. The question stops being "which quantization method has the best perplexity" and becomes:
- What can the target runtime ingest without a lossy conversion step?
- What survives the conversion the vendor toolchain performs on the way in?
- What fits in the memory the platform will actually give you at runtime — not peak, but what's left after everything else on the SoC has taken its share?
- What holds up when the die is hot and the clocks come down?
Point four is the one that catches teams out. Benchmarks get run on cold hardware. Cars do not stay cold.
Here is roughly what that looked like for Llama 3 8B on a Snapdragon automotive cockpit SoC.
Time to first tokenseconds · lower is better
- FP16 baseline
- too slow
- GPTQ 4-bit
- < 1 s
- AWQ 4-bit
- < 1 s
- GGUF Q4_K_M
- ~2–4 s
Sustained throughputtokens/sec · higher is better
- FP16 baseline
- too slow
- GPTQ 4-bit
- ~10–20
- AWQ 4-bit
- ~10–20
- GGUF Q4_K_M
- ~3–6
On-disk sizeGB · computed
- FP16 baseline
- ~16 GB
- GPTQ 4-bit
- ~5.5 GB
- AWQ 4-bit
- ~5.5 GB
- GGUF Q4_K_M
- ~4.9 GB
| Variant | Runs on | On-disk | Time to first token | Tokens/sec (sustained) | Quality vs FP16 |
|---|---|---|---|---|---|
| FP16 baseline | Device, but impractically slow | ~16 GB | Too slow to use | Too slow to use | Reference |
| GPTQ 4-bit | NPU, via vendor toolchain | ~5.5 GB | < 1 s | ~10–20 | Dipped on in-cabin phrasing |
| AWQ 4-bit | NPU, via vendor toolchain | ~5.5 GB | < 1 s | ~10–20 | Closest to FP16 |
| GGUF Q4_K_M | CPU (llama.cpp) | ~4.9 GB | ~2–4 s | ~3–6 | Close, but slow |
Three things stood out.
The 4-bit variants are all roughly the same size, so size stopped being the deciding factor almost immediately. What separated them was where they could run: the GPTQ and AWQ builds reached the NPU through the vendor toolchain and were several times faster than the GGUF build, which was the easiest to get working and stayed on the CPU.
Find a fast charger on my route.
The nearest fast charger on your route is at the next service area, 18 km ahead. Want me to add it as a stop
Find a fast charger on my route.
Thinking…
AWQ held closest to FP16. GPTQ showed exactly the calibration problem described above: calibrated on generic text, it slipped on the phrasing people actually use in a car.
from pathlib import Path
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
MODEL = "path/to/llama-3-8b"
model = AutoAWQForCausalLM.from_pretrained(MODEL)
tokenizer = AutoTokenizer.from_pretrained(MODEL)
# Calibrate on what drivers actually say, not generic web text.
prompts = Path("calibration/in_cabin_prompts.txt")
in_cabin_samples = prompts.read_text().splitlines()
model.quantize(
tokenizer,
quant_config={
"zero_point": True,
"q_group_size": 128,
"w_bit": 4,
"version": "GEMM",
},
calib_data=in_cabin_samples,
)
model.save_quantized("llama-3-8b-awq-in-cabin")And the cold numbers were not the real numbers. After the SoC had been running for a while, sustained throughput dropped by roughly 20–30%, which is the difference between a reply that feels conversational and one that feels like waiting.
What I would tell someone starting this
Measure sustained throughput, not burst. Measure after the device has been running for ten minutes. And measure the thing your users actually feel — time to first token — not the number that looks best in a table.
Then pick the artifact your runtime can consume without a lossy hop, and only then start arguing about algorithms.
The biggest win wasn't choosing GPTQ or AWQ. It was getting the model into the format the NPU actually wanted.