Adapt to open source inference framework

[Siegfried-qgf]

Have you considered incorporating this work into an open source inference framework, such as vLLM?

[preminstrel]

Yeah, this is a good idea!