base_api

Module Contents

Classes

APITemplateParser

Intermidate prompt template parser, specifically for API models.

BaseAPIModel

Base class for API model wrapper.

TokenBucket

A token bucket for rate limiting.
class lagent.llms.base_api.APITemplateParser(meta_template=None)

Intermidate prompt template parser, specifically for API models.

Parameters:

meta_template (Dict) – The meta template for the model.

class lagent.llms.base_api.BaseAPIModel(model_type, query_per_second=1, retry=2, template_parser=APITemplateParser, meta_template=None, *, max_tokens=512, top_p=0.8, top_k=None, temperature=0.8, repetition_penalty=0.0, stop_words=None)

Bases: lagent.llms.base_llm.BaseModel

Base class for API model wrapper.

Parameters:

  • model_type (str) – The type of model.

  • query_per_second (int) – The maximum queries allowed per second between two consecutive calls of the API. Defaults to 1.

  • retry (int) – Number of retires if the API call fails. Defaults to 2.

  • meta_template (Dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

  • template_parser (APITemplateParser) –

  • max_tokens (int) –

  • top_p (float) –

  • top_k (float) –

  • temperature (float) –

  • repetition_penalty (float) –

  • stop_words (Union[List[str], str]) –

is_api: bool = True
class lagent.llms.base_api.TokenBucket(rate)

A token bucket for rate limiting.

Parameters:

rate (float) – The rate of the token bucket.

get_token()

Get a token from the bucket.