Chapter 3, Hugging Face Transformers about generative_ai_with_langchain

[benman1]

@NPPprojects! Hi again!
I struggled to make this work - the pipeline was very slow for me until I switched to GPU on Google Colab. I've tried with different versions of LangChain and transformers, but the only thing that I've found to work was adapting the HuggingFacePipeline class.

This here works for me with the current LangChain version:

from typing import Any, List, Mapping, Optional

from langchain.llms import HuggingFacePipeline
from langchain_core.outputs import Generation, LLMResult
from langchain_core.callbacks import CallbackManagerForLLMRun
from langchain_community.llms.utils import enforce_stop_tokens

VALID_TASKS = ("text2text-generation", "text-generation", "summarization")


class HFP(HuggingFacePipeline):
      def _generate(
        self,
        prompts: List[str],
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> LLMResult:
        # List to hold all results
        text_generations: List[str] = []

        for i in range(0, len(prompts), self.batch_size):
            batch_prompts = prompts[i : i + self.batch_size]

            # Process batch of prompts
            responses = self.pipeline(batch_prompts)

            # Process each response in the batch
            for j, response in enumerate(responses):
                if isinstance(response, list):
                    # if model returns multiple generations, pick the top one
                    response = response[0]

                if self.pipeline.task == "text-generation":
                    try:
                        from transformers.pipelines.text_generation import ReturnType

                        remove_prompt = (
                            self.pipeline._postprocess_params.get("return_type")
                            != ReturnType.NEW_TEXT
                        )
                    except Exception as e:
                        logger.warning(
                            f"Unable to extract pipeline return_type. "
                            f"Received error:\n\n{e}"
                        )
                        remove_prompt = True
                    if remove_prompt:
                        text = response[len(batch_prompts[j]) :]
                        # ["generated_text"]
                    else:
                        text = response
                        # ["generated_text"]
                elif self.pipeline.task == "text2text-generation":
                    text = response["generated_text"]
                elif self.pipeline.task == "summarization":
                    text = response["summary_text"]
                else:
                    raise ValueError(
                        f"Got invalid task {self.pipeline.task}, "
                        f"currently only {VALID_TASKS} are supported"
                    )
                if stop:
                    # Enforce stop tokens
                    text = enforce_stop_tokens(text, stop)

                # Append the processed text to results
                text_generations.append(text)

        return LLMResult(
            generations=[[Generation(text=text)] for text in text_generations]
        )


hfp = HFP(pipeline= generate_text)
llm_chain = LLMChain(prompt=prompt, llm=hfp)
question = "What is electroencephalography?"
print(llm_chain.run(question))

[benman1]

Hi @NPPprojects,

Modifying the code produced the following error:

text = response[0]["generated_text"][len(prompt) :]
TypeError: string indices must be integers

That's why I wrote the HFP class, see above ;)

It should be easy enough to adapt it to any previous LC version. You are right though - depending on the task, you might not need any LC abstraction at all.

[benman1]

Hi @NPPprojects. Thanks for reporting this. I remember there were quite a few changes to LLMChain and local pipelines, and I might have missed testing this code for the pinned version.

I haven't run this yet, but have you tried wrapping the transformers pipeline with a HuggingFacePipeline?

from langchain.llms import HuggingFacePipeline
hfp = HuggingFacePipeline(pipeline= generate_text)

Later, instead of using generate_text you'd use hfp, like this:

llm_chain = LLMChain(prompt=prompt, llm=hfp)

Please let me know if this works - I'll try in the evening.

As for the transformers library, it's not directly included in the requirements, but it's required by sentence-transformers, which is included in the requirements.

[NPPprojects]

Hi @NPPprojects. Thanks for reporting this. I remember there were quite a few changes to LLMChain and local pipelines, and I might have missed testing this code for the pinned version.

I haven't run this yet, but have you tried wrapping the transformers pipeline with a HuggingFacePipeline?

from langchain.llms import HuggingFacePipeline
hfp = HuggingFacePipeline(pipeline= generate_text)

Later, instead of using generate_text you'd use hfp, like this:

llm_chain = LLMChain(prompt=prompt, llm=hfp)

Please let me know if this works - I'll try in the evening.

As for the transformers library, it's not directly included in the requirements, but it's required by sentence-transformers, which is included in the requirements.

Modifying the code produced the following error:

text = response[0]["generated_text"][len(prompt) :]
TypeError: string indices must be integers

Thrown by huggingface_pipeline.py

Currently just going through the textbook and testing the samples before I update to the latest LangChain version. This is a pretty rudimentary function where I'm not even too certain LangChain is all that useful as something like this works fine, without introducing any abstraction:

generate_text = pipeline(
    model="aisquared/dlite-v1-355m",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
    framework="pt",
    token= 1500
)
#No Langchain Abstraction
question = "What is electroencephalography?"
template = f"""Question: {question}
Answer: Let's think step by step."""
print(generate_text(template))