huggingface

Module Contents

Classes

HFTransformer	Model wrapper around HuggingFace general models.
HFTransformerCasualLM	Model wrapper around HuggingFace general models.
HFTransformerChat	Model wrapper around HuggingFace general models.

class lagent.llms.huggingface.HFTransformer(path, tokenizer_path=None, tokenizer_kwargs=dict(), tokenizer_only=False, model_kwargs=dict(device_map='auto'), meta_template=None, stop_words_id=None, **kwargs)

Bases: lagent.llms.base_llm.BaseLLM

Model wrapper around HuggingFace general models.

Adapted from Internlm (https://github.com/InternLM/InternLM/blob/main/

chat/web_demo.py)

参数:

• path (str) – The name or path to HuggingFace’s model.

• tokenizer_path (str) – The path to the tokenizer. Defaults to None.

• tokenizer_kwargs (dict) – Keyword arguments for the tokenizer. Defaults to {}.

• tokenizer_only (bool) – If True, only the tokenizer will be initialized. Defaults to False.

• model_kwargs (dict) – Keyword arguments for the model, used in loader. Defaults to dict(device_map=’auto’).

• meta_template (Dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

• stop_words_id (Union[List[int], int]) –

logits_processor

stopping_criteria

prefix_allowed_tokens_fn = None

additional_eos_token_id = []

tokenize(inputs)

Tokenize the input prompts.

参数:

• prompts (str | List[str]) – user’s prompt, or a batch prompts

• inputs (str) –

返回:

prompt’s token ids, ids’ length and requested output length

返回类型:

Tuple(numpy.ndarray, numpy.ndarray, numpy.ndarray)

generate(inputs, do_sample=True, **kwargs)

Return the chat completions in non-stream mode.

参数:

• inputs (Union[str, List[str]]) – input texts to be completed.

• do_sample (bool) – do sampling if enabled

返回:

(a list of/batched) text/chat completion

stream_generate(inputs, do_sample=True, **kwargs)

Return the chat completions in stream mode.

参数:

• inputs (Union[str, List[str]]) – input texts to be completed.

• do_sample (bool) – do sampling if enabled

返回:

status, text/chat completion, generated token number

返回类型:

tuple(Status, str, int)

stream_chat(inputs, do_sample=True, **kwargs)

Return the chat completions in stream mode.

参数:

• inputs (List[dict]) – input messages to be completed.

• do_sample (bool) – do sampling if enabled

返回:

the text/chat completion

class lagent.llms.huggingface.HFTransformerCasualLM(path, tokenizer_path=None, tokenizer_kwargs=dict(), tokenizer_only=False, model_kwargs=dict(device_map='auto'), meta_template=None, stop_words_id=None, **kwargs)

Bases: HFTransformer

Model wrapper around HuggingFace general models.

Adapted from Internlm (https://github.com/InternLM/InternLM/blob/main/

chat/web_demo.py)

参数:

• path (str) – The name or path to HuggingFace’s model.

• tokenizer_path (str) – The path to the tokenizer. Defaults to None.

• tokenizer_kwargs (dict) – Keyword arguments for the tokenizer. Defaults to {}.

• tokenizer_only (bool) – If True, only the tokenizer will be initialized. Defaults to False.

• model_kwargs (dict) – Keyword arguments for the model, used in loader. Defaults to dict(device_map=’auto’).

• meta_template (Dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

• stop_words_id (Union[List[int], int]) –

class lagent.llms.huggingface.HFTransformerChat(template_parser=APITemplateParser, **kwargs)

Bases: HFTransformerCasualLM

Model wrapper around HuggingFace general models.

Adapted from Internlm (https://github.com/InternLM/InternLM/blob/main/

chat/web_demo.py)

参数:

• path (str) – The name or path to HuggingFace’s model.

• tokenizer_path (str) – The path to the tokenizer. Defaults to None.

• tokenizer_kwargs (dict) – Keyword arguments for the tokenizer. Defaults to {}.

• tokenizer_only (bool) – If True, only the tokenizer will be initialized. Defaults to False.

• model_kwargs (dict) – Keyword arguments for the model, used in loader. Defaults to dict(device_map=’auto’).

• meta_template (Dict, optional) – The model’s meta prompt template if needed, in case the requirement of injecting or wrapping of any meta instructions.

chat(inputs, do_sample=True, **kwargs)

Return the chat completions in stream mode.

参数:

• inputs (Union[List[dict], List[List[dict]]]) – input messages to be completed.

• do_sample (bool) – do sampling if enabled

返回:

the text/chat completion