base_llm

Module Contents

Classes

LMTemplateParser	Intermidate prompt template parser, specifically for language models.
BaseLLM	Base class for model wrapper.
AsyncLLMMixin
AsyncBaseLLM	Base class for model wrapper.

class lagent.llms.base_llm.LMTemplateParser(meta_template=None)

Intermidate prompt template parser, specifically for language models.

参数:

meta_template (list of dict, optional) – The meta template for the model.

meta_template = None

class lagent.llms.base_llm.BaseLLM(path, tokenizer_only=False, template_parser=LMTemplateParser, meta_template=None, *, max_new_tokens=512, top_p=0.8, top_k=40, temperature=0.8, repetition_penalty=1.0, stop_words=None)

Base class for model wrapper.

参数:

• path (str) – The path to the model.

• max_new_tokens (int) – Maximum length of output expected to be generated by the model. Defaults to 512.

• tokenizer_only (bool) – If True, only the tokenizer will be initialized. Defaults to False.

• meta_template (list of dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

• template_parser (LMTemplateParser) –

• top_p (float) –

• top_k (float) –

• temperature (float) –

• repetition_penalty (float) –

• stop_words (Union[List[str], str]) –

path

tokenizer_only = False

template_parser

eos_token_id = None

gen_params

abstract generate(inputs, **gen_params)

Generate results given a str (or list of) inputs.

参数:

• inputs (Union[str, List[str]]) –

• gen_params (dict) – The input params for generation.

返回:

A (list of) generated strings.

返回类型:

Union[str, List[str]]

eg.

batched = True if isinstance(inputs, str):

inputs = [inputs] batched = False

response = [‘’] if batched:

return response

return response[0]

abstract stream_generate(inputs, **gen_params)

Generate results as streaming given a str inputs.

参数:

• inputs (str) –

• gen_params (dict) – The input params for generation.

返回:

A generated string.

返回类型:

str

chat(inputs, session_ids=None, **gen_params)

Generate completion from a list of templates.

参数:

• inputs (Union[List[dict], List[List[dict]]]) –

• gen_params (dict) – The input params for generation.

• session_ids (Union[int, List[int]]) –

Returns:

abstract stream_chat(inputs, **gen_params)

Generate results as streaming given a list of templates.

参数:

• inputs (Union[List[dict]) –

• gen_params (dict) – The input params for generation.

Returns:

abstract tokenize(prompts)

Tokenize the input prompts.

参数:

prompts (str | List[str]) – user’s prompt, or a batch prompts

返回:

prompt’s token ids, ids’ length and requested output length

返回类型:

Tuple(numpy.ndarray, numpy.ndarray, numpy.ndarray)

update_gen_params(**kwargs)

class lagent.llms.base_llm.AsyncLLMMixin

abstract generate(inputs, session_ids=None, **gen_params)

Async:

参数:

• inputs (Union[str, List[str]]) –

• session_ids (Union[int, List[int]]) –

返回类型:

str

Generate results given a str (or list of) inputs.

参数:

• inputs (Union[str, List[str]]) –

• gen_params (dict) – The input params for generation.

• session_ids (Union[int, List[int]]) –

返回:

A (list of) generated strings.

返回类型:

Union[str, List[str]]

eg.

batched = True if isinstance(inputs, str):

inputs = [inputs] batched = False

response = [‘’] if batched:

return response

return response[0]

abstract stream_generate(inputs, **gen_params)

Async:

参数:

inputs (str) –

返回类型:

List[str]

Generate results as streaming given a str inputs.

参数:

• inputs (str) –

• gen_params (dict) – The input params for generation.

返回:

A generated string.

返回类型:

str

async chat(inputs, session_ids=None, **gen_params)

Generate completion from a list of templates.

参数:

• inputs (Union[List[dict], List[List[dict]]]) –

• gen_params (dict) – The input params for generation.

• session_ids (Union[int, List[int]]) –

Returns:

abstract stream_chat(inputs, **gen_params)

Async:

参数:

inputs (List[dict]) –

Generate results as streaming given a list of templates.

参数:

• inputs (Union[List[dict]) –

• gen_params (dict) – The input params for generation.

Returns:

abstract tokenize(prompts)

Async:

参数:

prompts (Union[str, List[str], List[dict], List[List[dict]]]) –

Tokenize the input prompts.

参数:

prompts (str | List[str]) – user’s prompt, or a batch prompts

返回:

prompt’s token ids, ids’ length and requested output length

返回类型:

Tuple(numpy.ndarray, numpy.ndarray, numpy.ndarray)

class lagent.llms.base_llm.AsyncBaseLLM(path, tokenizer_only=False, template_parser=LMTemplateParser, meta_template=None, *, max_new_tokens=512, top_p=0.8, top_k=40, temperature=0.8, repetition_penalty=1.0, stop_words=None)

Bases: AsyncLLMMixin, BaseLLM

Base class for model wrapper.

参数:

• path (str) – The path to the model.

• max_new_tokens (int) – Maximum length of output expected to be generated by the model. Defaults to 512.

• tokenizer_only (bool) – If True, only the tokenizer will be initialized. Defaults to False.

• meta_template (list of dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

• template_parser (LMTemplateParser) –

• top_p (float) –

• top_k (float) –

• temperature (float) –

• repetition_penalty (float) –

• stop_words (Union[List[str], str]) –