Skip to content

[FEATURE] - Add rpm_limit argument #93

@patrickfleith

Description

@patrickfleith

Is your feature request related to a problem? Please describe.

Gemini models offer free generations but are capped in terms of RPM, RPD.

  • LLM providers or the generate method should offer to deliberately slow down the generation process to accomodate low RPM.
  • This could be down via an rpm_limit argument.

Imagine two providers: OpenAI + Gemini:

  • OpenAI is not rate limited
  • Gemini we want to leverage the firee tier so it is rate limited: say 10 RPM.
    We should set rpm_limit=10 either in GeminiProvider class or 'LLMProvideror ingenerate()`. I have the feeling it should be at provider level. TBC.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions