quant blocked fp8

[CUHKSZzxy]

Usage

1. quantize

lmdeploy lite blocked_fp8 ${model_path} --work-dir ${quantized_model_path} --quant-dtype fp8

2. test case

NOTE: We can use either pytorch or turbomind backend for FP8 inference. Here we take pytorch backend as an example.

from lmdeploy import pipeline, PytorchEngineConfig

model_path = "OpenGVLab/InternVL3_5-8B-FP8"

if __name__ == '__main__':
    engine_config = PytorchEngineConfig(tp=1)
    pipe = pipeline(model_path, backend_config=engine_config)
    response = pipe(["Hi, pls intro yourself", "Shanghai is"])
    print(response)

Accuracy

Dataset: OCRBench
Model: InternVL3.5-8B (FP8), InternVL3_5-30B-A3B (FP8)

Backend	InternVL3.5-8B	InternVL3.5-8B-FP8	InternVL3_5-30B-A3B	InternVL3_5-30B-A3B-FP8
TurboMind	84.3	84.1	88.8	88.4
PyTorch	84.3	84.2	88.7	88.1

Tested with VLMEvalKit.

Checklist

• Align the quantization config with QWen3 / InternS1 FP8
• Add documents for blocked FP8
• Verify the FP8 model accuracy
• Fix quantizations for MOE models
• Check whether weight_scale_inv modification affects other quant methods / modules

[lvhan028]

These configurations are model-specific. We should adopt a more maintainable approach.

[CUHKSZzxy]

I checked the vLLM FP8 compressor example, and noticed that the ignored patterns are indeed model-specific. Currently, these patterns are passed as an input argument named ignore in the quantization recipe.

https://github.com/vllm-project/llm-compressor/tree/main/examples/quantization_w8a8_fp8

https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/qwen2vl_example.py#L20

How about we also expose this as a configurable input argument, allowing users to define their own ignore patterns as needed?

[lvhan028]

@RunningLeon As discussed with @CUHKSZzxy, we propose a new --skip-pattern config.py option for custom skip patterns, alongside lmdeploy's internal defaults.
what's your opinion?

[RunningLeon]

Personally, if only passing skip patterns, a config file is not necessary.

[lvhan028]

Changing scale to weight_scale_inv might affect w8a8 quantized model inference.

[lvhan028]

@RunningLeon @grimoire any good ideas?