googlecloudplatform / generative-ai

Sample code and notebooks for Generative AI on Google Cloud

Generative AI

Welcome to the Google Cloud Generative AI repository.

This repository contains notebooks, code samples, sample apps, and other resources that demonstrate how to use, develop and manage generative AI workflows using Generative AI on Google Cloud, powered by Vertex AI.

For more Vertex AI samples, please visit the Vertex AI samples Github repository.

Using this repository

	Description	Contents
gemini/	Discover Gemini through starter notebooks, use cases, function calling, sample apps, and more.	Sample notebooks, apps, use cases
search/	Use this folder if you're interested in using Vertex AI Search, a Google-managed solution to help you rapidly build search engines for websites and across enterprise data. (Formerly known as Enterprise Search on Generative AI App Builder)	Sample apps, use cases
conversation/	Use this folder if you're interested in using Vertex AI Conversation, a Google-managed solution to help you rapidly build chat bots for websites and across enterprise data. (Formerly known as Chat Apps on Generative AI App Builder)	Sample apps, use cases
language/	Use this folder if you're interested in building your own solutions from scratch using Google's language foundation models (Vertex AI PaLM API). text-bison model chat-bison model textembedding-gecko model code-bison model code-gecko model codechat-bison model prompt design deploying apps tuning models grounded responses langchain with Google's foundation models	Sample notebooks, apps, use cases
vision/	Use this folder if you're interested in building your own solutions from scratch using features from Imagen on Vertex AI (Vertex AI Imagen API). These are the features that Imagen on Vertex AI offers: Image generation Image editing Visual captioning Visual question answering	Sample notebooks, apps, use cases
speech/	Use this folder if you're interested in building your own solutions from scratch using features from Chirp, a version of Google's Universal Speech Model (USM) on Vertex AI (Vertex AI Chirp API).	Sample notebooks, apps, use cases
setup-env/	Instructions on how to set up Google Cloud, the Vertex AI Python SDK, and notebook environments on Google Colab and Vertex AI Workbench.	Setup instructions
RESOURCES.md	Learning resources (e.g. blogs, Youtube playlists) about Generative AI on Google Cloud	Resources (e.g. videos, blogposts, learning paths)

Related Repositories

• Google Cloud Applied AI Engineering
• Generative AI for Marketing using Google Cloud
• Generative AI for Developer Productivity
• Vertex AI Core
  • Vertex AI Samples
  • MLOps with Vertex AI
  • Developing NLP solutions with T5X and Vertex AI
  • AlphaFold batch inference with Vertex AI Pipelines
  • Serving Spark ML models using Vertex AI
  • Sensitive Data Protection (Cloud DLP) for Vertex AI Generative Models (PaLM2)
• Conversational AI
  • Contact Center AI Samples
  • Reimagining Customer Experience 360
• Document AI
  • Document AI Samples
• Duet AI
  • Cymbal Superstore
• Cloud Databases
  • GenAI Databases Retrieval App
• Other
  • ai-on-gke
  • ai-infra-cluster-provisioning
  • solutions-genai-llm-workshop
  • terraform-genai-doc-summarization
  • terraform-genai-knowledge-base
  • genai-product-catalog
  • solutionbuilder-terraform-genai-doc-summarization
  • solutions-viai-edge-provisioning-configuration
  • mis-ai-accelerator
  • dataflow-opinion-analysis

Contributing

Contributions welcome! See the Contributing Guide.

Getting help

Please use the issues page to provide suggestions, feedback or submit a bug report.

Disclaimer

This repository itself is not an officially supported Google product. The code in this repository is for demonstrative purposes only.

generative-ai's Issues

Is Chat-bison model good for batch evaluation?

I only saw the tutorial on batch evaluation for base text-bison model, but not the chat-bison model. But the feature page does mention chat-bison has Batch requests capability. Is it turned on now?

Custom TF Model with Generative AI (PaLM API)

Hi,

I'm wondering if I could actually use my custom Tensorflow ML model with PaLM API (Gen AI), for example I saved tf model in .h5 format and import it in PaLM AI and expect questions that were asked are answered according to the custom model I trained.

I've been reading the official documentation, however, most are explaining the use of text models, etc instead of a custom model.

Evaluation of Palm models via bAbi tasks

Do Palm models really ‘understand’ the language? Especially the logical reasoning aspect?

Folks at Facebook AI Research came up with a set of seemingly easy, toy-ish, childish tasks (called ‘bAbi’) to gauge the ‘understanding’ of a Question-Answering system, like any chatbot. Tried them on Palm today and here is the result:

Prompt:

Mary went to the bathroom.
John moved to the hallway.
Mary travelled to the office.
Where is Mary?

Bard Response:

Mary is in the office.

The passage states that Mary went to the bathroom, then John moved to the hallway, and then Mary travelled to the office. This means that Mary is currently in the office. The bathroom and the hallway are both in the same building, but the office is a separate location. Therefore, Mary is not in the bathroom or the hallway, she is in the office.

Can we do set a benchmark test-set to evaluate efficaly of Palm models against these tasks?

Here is evaluation of ChatGPT against the same tasks.

Here is a script to evaluate any (publicly available) LLM against the bAbi tasks. Would you want to incorporate the script? @polong-lin @rominirani

Having issue on `EnterpriseSearchRetriever` with Structure Datastore

Hi All:

I have created an Gen App with Structure Datastore with bigquery source, from the Gen App Builder, i can see the data processing is complete.

But when i use question_answering.ipynb example with my Structure datastore SEARCH_ENGINE_ID, when i execute the run function, i get 401 error. Anyone meet this as well?

Links to colab and workbench need updating

The links in several notebooks go to an old repo layout structure such https://github.com/GoogleCloudPlatform/generative-ai/blob/main/prompt-engineering/ideation.ipynb

mattrobn

Getting Error with custom embedding function

[~/.local/lib/python3.10/site-packages/chromadb/db/index/hnswlib.py](https://localhost:8080/#) in add(self, ids, embeddings, update)
    137         """Add or update embeddings to the index"""
    138 
--> 139         dim = len(embeddings[0])
    140 
    141         if self._index is None:

IndexError: list index out of range

Getting `TypeError` when using LangChain 0.0.316 in `search/retrieval-augmented-generation/examples/question_answering.ipynb`

When trying the example in search/retrieval-augmented-generation/examples/question_answering.ipynb I am getting: TypeError: TextGenerationModel.predict() got an unexpected keyword argument 'candidate_count'.

After changing to version 0.0.310 as per this issue the notebook seems to run as expected. But it misses GoogleVertexAIMultiTurnSearchRetriever.

Vertex AI Conversation app not giving useful answers

The agent behind the demo app at https://vertex-ai-conversation.web.app/ is not giving useful answers:

ImportError: cannot import name 'resourcemanager' from 'google.cloud' (unknown location)

When I tried to run "intro_palm_api.ipynb" with colab, import vertexai failed with following error:
pip install google-cloud-resource-manager did not improve the error, too.
How can we resolve this issue?

---

ImportError Traceback (most recent call last)
in <cell line: 1>()
----> 1 import vertexai
2
3 PROJECT_ID = "[your-project-id]" # @param {type:"string"}
4 vertexai.init(project=PROJECT_ID, location="us-central1")

8 frames
/usr/local/lib/python3.10/dist-packages/google/cloud/aiplatform/utils/resource_manager_utils.py in
20
21 from google.auth import credentials as auth_credentials
---> 22 from google.cloud import resourcemanager
23
24 from google.cloud.aiplatform import initializer

ImportError: cannot import name 'resourcemanager' from 'google.cloud' (unknown location)

---

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

can't open Question Answering with Large Documents using LangChain notebook in Colab

when you try to run the Question Answering with Large Documents using LangChain https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/document-qa/question_answering_large_documents_langchain.ipynb notebook in Colab by clicking on the Run in Colab link https://screenshot.googleplex.com/3z2uwdNUMpNKXCb from the example in github, you get Notebook not found message: https://screenshot.googleplex.com/5d7EzzPFWgL4yoa

If you click on the Authorize with Github link in that dialog and enter your github credentials you just get back to the Notebook not found message again: https://screenshot.googleplex.com/5d7EzzPFWgL4yoa

FWIW, I have been able to run other notebooks in this repo in Colab (e.g. https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/ideation.ipynb and https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/text_classification.ipynb) with no issues.

404 errors for table of content

All the links in this Table of Contents returns a 404 error.

Incorrect notes: "This is more efficient than Map-Reduce but less efficient."

In "question_answering_large_documents_langchain.ipynb", notes say: This is more efficient than Map-Reduce but less efficient.

This needs to be rewritten.

Ll

Vector Search notebook not functional

DocAI OCR only reads the first few pages of the doc and the rest of the entire document is empty

In addition, after going through the chunking process, pdf_data_sample.chunks is completely empty. No content is stored.

QA DocAI can't work because of NaN's (and little update request)

In this example:
https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-qa/question_answering_documentai_vector_store_palm.ipynb

1. You have to update the import of google-cloud-documentai-toolbox from 0.10.1 of the document to 0.10.3a, if not, it doesn't go ;-)

2. It crashes at one point:

pdf_data_sample["embedding"] = pdf_data_sample["chunks"].apply(
    lambda x: embedding_model_with_backoff([x])
)

Investigating, it comes from the chunks, there are "NaNs" in the chunks ... and there are because when handling the PDF, it is only able to grab the first page, the rest are all "null" values.

Any solution / workaround ?

Breaking Bug in Sample App: Conversation Web App

Bug report

The chat widget on https://vertex-ai-conversation.web.app/ does not work.

404 on https://www.gstatic.com/dialogflow-console/fast/messenger-cx/prod/df-messenger.min.js?v=1

also Uncaught TypeError: Cannot read properties of null (reading 'querySelector') but that's perhaps due to the missing JS file above.

getting_started_tuning.ipynb: large-language-model-tuner failed Accelerators are not supported for this project (Qwiklabs)

Step: large-language-model-tuner
Lab: generative-ai/language/examples/tuning/getting_started_tuning.ipynb
`
Error message

com.google.cloud.ai.platform.common.errors.AiPlatformException: code=INVALID_ARGUMENT, message=List of found errors: 1.Field: job_spec.worker_pool_specs[0].machine_spec.accelerator_count; Message: Accelerators are not supported for this project. , cause=null; Failed to create custom job.Project number: 979501391966, Job id: 235253706902208512, Task id: 3184303321783468032, Task name: large-language-model-tuner, Task state: DRIVER_SUCCEEDED, Execution name: projects/979501391966/locations/europe-west4/metadataStores/default/executions/8194865760206430625; Failed to create external task or refresh its state. Task:Project number: 979501391966, Job id: 235253706902208512, Task id: 3184303321783468032, Task name: large-language-model-tuner, Task state: DRIVER_SUCCEEDED, Execution name: projects/979501391966/locations/europe-west4/metadataStores/default/executions/8194865760206430625; Failed to handle the pipeline task. Task: Project number: 979501391966, Job id: 235253706902208512, Task id: 3184303321783468032, Task name: large-language-model-tuner, Task state: DRIVER_SUCCEEDED, Execution name: projects/979501391966/locations/europe-west4/metadataStores/default/executions/8194865760206430625

RuntimeError: Job failed with:
code: 9
message: "The DAG failed because some tasks failed. The failed tasks are: [tuning-graph].; Job (project_id = qwiklabs-gcp-03-4942b3f44777, job_id = 235253706902208512) is failed due to the above error.; Failed to handle the job: {project_number = 979501391966, job_id = 235253706902208512}"

https://console.cloud.google.com/vertex-ai/locations/europe-west4/pipelines/runs/tune-large-model-20230810182936?project=qwiklabs-gcp-03-4942b3f44777

Qwiklab: https://explore.qwiklabs.com/focuses/72727

Images not uploaded properly in the tutorial

Team,
Some pictures are not loading properly in the following link. Can you double check please?
https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/intro_generative_ai_studio.md
Thanks.

Generative AI Studio API Vision QA

In the current notebooks custom models are used. Is it possible to access the generative ai vision qa via api?

typo at create_index

I have found a typo at line 95, "treeah" should be replaced with "tree-ah" =D
https://github.com/GoogleCloudPlatform/generative-ai/tree/main/language/use-cases/document-qa/utils/matching_engine_utils.py

Random Seed

OpenAI has a way to set the seed. Is there a way to do this with gemini models?

Error running Prompt Design/Ideation in Workbench

I ran https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/ideation.ipynb in Colab, no issues.

However, when I tried to run https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/prompt-design/ideation.ipynb in Workbench (starting with clicking the Workbench link https://screenshot.googleplex.com/BkHN3iBbAZEVfUY), I got an error.

Here is what happened after clicking the open in Workbench link:

• Opens up Console in Vertex, select existing project genai-test-project-jun18 and set a name for the notebook https://screenshot.googleplex.com/327KmVYqEVCbQt5
• I click Confirm and after about a minute, I get this message https://screenshot.googleplex.com/tAngTWU4oe7dmaU
• I click Confirm and the notebook opens in Jupyter
• I ran the notebook cell by cell, OK up until this cell:

prompt = "Generate a marketing campaign for sustainability and fashion"

print(
    generation_model.predict(
        prompt, temperature=0.2, max_output_tokens=1024, top_k=40, top_p=0.8
    ).text
)

running this cell, I get the following errror:

---------------------------------------------------------------------------
_InactiveRpcError                         Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:65, in _wrap_unary_errors.<locals>.error_remapped_callable(*args, **kwargs)
     64 try:
---> 65     return callable_(*args, **kwargs)
     66 except grpc.RpcError as exc:

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:946, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    944 state, call, = self._blocking(request, timeout, metadata, credentials,
    945                               wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:849, in _end_unary_response_blocking(state, call, with_call, deadline)
    848 else:
--> 849     raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.PERMISSION_DENIED
	details = "Permission 'aiplatform.endpoints.predict' denied on resource '//aiplatform.googleapis.com/projects/genai-test-project-jun18/locations/us-central1/publishers/google/models/text-bison@001' (or it may not exist)."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.124.95:443 {grpc_message:"Permission \'aiplatform.endpoints.predict\' denied on resource \'//aiplatform.googleapis.com/projects/genai-test-project-jun18/locations/us-central1/publishers/google/models/text-bison@001\' (or it may not exist).", grpc_status:7, created_time:"2023-07-13T14:46:42.615806389+00:00"}"
>

The above exception was the direct cause of the following exception:

PermissionDenied                          Traceback (most recent call last)
Cell In[6], line 4
      1 prompt = "Generate a marketing campaign for sustainability and fashion"
      3 print(
----> 4     generation_model.predict(
      5         prompt, temperature=0.2, max_output_tokens=1024, top_k=40, top_p=0.8
      6     ).text
      7 )

File /opt/conda/lib/python3.10/site-packages/vertexai/language_models/_language_models.py:251, in TextGenerationModel.predict(self, prompt, max_output_tokens, temperature, top_k, top_p)
    229 def predict(
    230     self,
    231     prompt: str,
   (...)
    236     top_p: float = _DEFAULT_TOP_P,
    237 ) -> "TextGenerationResponse":
    238     """Gets model response for a single prompt.
    239 
    240     Args:
   (...)
    248         A `TextGenerationResponse` object that contains the text produced by the model.
    249     """
--> 251     return self._batch_predict(
    252         prompts=[prompt],
    253         max_output_tokens=max_output_tokens,
    254         temperature=temperature,
    255         top_k=top_k,
    256         top_p=top_p,
    257     )[0]

File /opt/conda/lib/python3.10/site-packages/vertexai/language_models/_language_models.py:287, in TextGenerationModel._batch_predict(self, prompts, max_output_tokens, temperature, top_k, top_p)
    279 instances = [{"content": str(prompt)} for prompt in prompts]
    280 prediction_parameters = {
    281     "temperature": temperature,
    282     "maxDecodeSteps": max_output_tokens,
    283     "topP": top_p,
    284     "topK": top_k,
    285 }
--> 287 prediction_response = self._endpoint.predict(
    288     instances=instances,
    289     parameters=prediction_parameters,
    290 )
    292 return [
    293     TextGenerationResponse(
    294         text=prediction["content"],
   (...)
    297     for prediction in prediction_response.predictions
    298 ]

File /opt/conda/lib/python3.10/site-packages/google/cloud/aiplatform/models.py:1559, in Endpoint.predict(self, instances, parameters, timeout, use_raw_predict)
   1546     return Prediction(
   1547         predictions=json_response["predictions"],
   1548         deployed_model_id=raw_predict_response.headers[
   (...)
   1556         ),
   1557     )
   1558 else:
-> 1559     prediction_response = self._prediction_client.predict(
   1560         endpoint=self._gca_resource.name,
   1561         instances=instances,
   1562         parameters=parameters,
   1563         timeout=timeout,
   1564     )
   1566     return Prediction(
   1567         predictions=[
   1568             json_format.MessageToDict(item)
   (...)
   1573         model_resource_name=prediction_response.model,
   1574     )

File /opt/conda/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py:602, in PredictionServiceClient.predict(self, request, endpoint, instances, parameters, retry, timeout, metadata)
    597 metadata = tuple(metadata) + (
    598     gapic_v1.routing_header.to_grpc_metadata((("endpoint", request.endpoint),)),
    599 )
    601 # Send the request.
--> 602 response = rpc(
    603     request,
    604     retry=retry,
    605     timeout=timeout,
    606     metadata=metadata,
    607 )
    609 # Done; return the response.
    610 return response

File /opt/conda/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py:113, in _GapicCallable.__call__(self, timeout, retry, *args, **kwargs)
    110     metadata.extend(self._metadata)
    111     kwargs["metadata"] = metadata
--> 113 return wrapped_func(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:67, in _wrap_unary_errors.<locals>.error_remapped_callable(*args, **kwargs)
     65     return callable_(*args, **kwargs)
     66 except grpc.RpcError as exc:
---> 67     raise exceptions.from_grpc_error(exc) from exc

PermissionDenied: 403 Permission 'aiplatform.endpoints.predict' denied on resource '//aiplatform.googleapis.com/projects/genai-test-project-jun18/locations/us-central1/publishers/google/models/text-bison@001' (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "aiplatform.googleapis.com"
metadata {
  key: "permission"
  value: "aiplatform.endpoints.predict"
}
metadata {
  key: "resource"
  value: "projects/genai-test-project-jun18/locations/us-central1/publishers/google/models/text-bison@001"
}
]

'from utils.matching_engine' fails

Need to ghclone:
!ghclone https://github.com/GoogleCloudPlatform/generative-ai/tree/main/language/examples/document-qa/utils

Then this works

Import custom Matching Engine packages

from utils.matching_engine import MatchingEngine
from utils.matching_engine_utils import MatchingEngineUtils

Chat calls fail with TypeError: send_message() got an unexpected keyword argument 'context'

Notebook: intro_langchain_palm_api.ipynb
LangChain version: 0.0.208
Vertex AI SDK version: 1.26.0

chat([HumanMessage(content="Hello")])

fails with TypeError: send_message() got an unexpected keyword argument 'context'

Stack:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[53], line 1
----> 1 chat([HumanMessage(content="Hello")])

File ~/.local/lib/python3.9/site-packages/langchain/chat_models/base.py:208, in BaseChatModel.__call__(self, messages, stop, callbacks, **kwargs)
    201 def __call__(
    202     self,
    203     messages: List[BaseMessage],
   (...)
    206     **kwargs: Any,
    207 ) -> BaseMessage:
--> 208     generation = self.generate(
    209         [messages], stop=stop, callbacks=callbacks, **kwargs
    210     ).generations[0][0]
    211     if isinstance(generation, ChatGeneration):
    212         return generation.message

File ~/.local/lib/python3.9/site-packages/langchain/chat_models/base.py:102, in BaseChatModel.generate(self, messages, stop, callbacks, tags, **kwargs)
    100 except (KeyboardInterrupt, Exception) as e:
    101     run_manager.on_llm_error(e)
--> 102     raise e
    103 llm_output = self._combine_llm_outputs([res.llm_output for res in results])
    104 generations = [res.generations for res in results]

File ~/.local/lib/python3.9/site-packages/langchain/chat_models/base.py:94, in BaseChatModel.generate(self, messages, stop, callbacks, tags, **kwargs)
     90 new_arg_supported = inspect.signature(self._generate).parameters.get(
     91     "run_manager"
     92 )
     93 try:
---> 94     results = [
     95         self._generate(m, stop=stop, run_manager=run_manager, **kwargs)
     96         if new_arg_supported
     97         else self._generate(m, stop=stop)
     98         for m in messages
     99     ]
    100 except (KeyboardInterrupt, Exception) as e:
    101     run_manager.on_llm_error(e)

File ~/.local/lib/python3.9/site-packages/langchain/chat_models/base.py:95, in <listcomp>(.0)
     90 new_arg_supported = inspect.signature(self._generate).parameters.get(
     91     "run_manager"
     92 )
     93 try:
     94     results = [
---> 95         self._generate(m, stop=stop, run_manager=run_manager, **kwargs)
     96         if new_arg_supported
     97         else self._generate(m, stop=stop)
     98         for m in messages
     99     ]
    100 except (KeyboardInterrupt, Exception) as e:
    101     run_manager.on_llm_error(e)

File ~/.local/lib/python3.9/site-packages/langchain/chat_models/vertexai.py:136, in ChatVertexAI._generate(self, messages, stop, run_manager, **kwargs)
    134 for pair in history.history:
    135     chat._history.append((pair.question.content, pair.answer.content))
--> 136 response = chat.send_message(question.content, **params)
    137 text = self._enforce_stop_words(response.text, stop)
    138 return ChatResult(generations=[ChatGeneration(message=AIMessage(content=text))])

TypeError: send_message() got an unexpected keyword argument 'context'

Link not working

In the. file we have the following:

This notebook focuses on using the Python SDK to call the Vertex AI PaLM API. For more information on using Generative AI Studio without writing code, you can explore [Getting Started with the UI instructions](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/getting-started/getting_started_ui.md)

This link is not working.

Output answer in the same language as the one being asked

Hello,

Is it possible to make the chatbot answer in the same language as the inquired one?

How would different language indexing work?

With the current app, will multi-lingual site content be correctly indexed and answered? For example, asked in LangA --> answered in LangA cotent or answered in all the available languages of the same content?

The ideal case is that when being asked with LangA, the LangA site content will be highlighted and other language contents will also be showed underneath for reference purposes.

Thanks

NameError: name 'TextGenerationModel' is not defined

As I'm presently creating a workshop for the Google Developer Group (GDG), I'm facing a few challenges. Specifically, I'm having trouble executing this particular notebook: language/intro_palm_api.ipynb.

Despite having installed google-cloud-aiplatform, I'm attempting to run the following

import pandas as pd
import seaborn as sns
from IPython.display import Markdown, display
from sklearn.metrics.pairwise import cosine_similarity
from vertexai.preview.language_models import (ChatModel, InputOutputTextPair,
                                              TextEmbeddingModel,
                                              TextGenerationModel)

generation_model = TextGenerationModel.from_pretrained("text-bison@001")

Unfortunately, I encounter the subsequent error:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 generation_model = TextGenerationModel.from_pretrained("text-bison@001")

NameError: name 'TextGenerationModel' is not defined

Could you provide some guidance on how to rectify this issue?

No gemini samples in 'dev' branch to contribute.

The new samples are unavailable in the 'dev' branch.

google-cloud-aiplatform version errors

google-cloud-aiplatform==1.26 is needed to run langchain.llms.VertexAI but it gives error in chat.send_message as
is_blocked=safety_attributes.get("blocked", False),
AttributeError: 'list' object has no attribute 'get'

googleapis/python-aiplatform#2337 suggests soution to be downgrade to google-cloud-aiplatform==1.25 but then the LangChain does not work

Need version that resolves both.

Model tuning does not work

Like most of the code uploaded by Google developers , your model tuning code that uses the stackoverflow data fails miserably giving the below errors.

{
  "summary": "Found 7 errors in your file. See 'errors' field for specific details.\nValidated 4000 examples for tokenization. Found 7 examples where either 'input_text' or 'output_text' exceeds the model token limits. See 'tokenization_issues' field for some specific examples.\nValidated 1000 examples for RAI. Found 43 examples that has RAI issues. See 'rai_issues' field for some specific examples.\n",
  "max_user_input_token_length": 8177,
  "tokenization_issues": [
    "Row: 122. Token limit exceeded for 'input_text' [tokens: 15851|limit: 8192] or 'output_text' [tokens: 24|limit: 1024]",
    "Row: 362. Token limit exceeded for 'input_text' [tokens: 13474|limit: 8192] or 'output_text' [tokens: 19|limit: 1024]",
    "Row: 391. Token limit exceeded for 'input_text' [tokens: 10643|limit: 8192] or 'output_text' [tokens: 34|limit: 1024]",
    "Row: 528. Token limit exceeded for 'input_text' [tokens: 9351|limit: 8192] or 'output_text' [tokens: 17|limit: 1024]",
    "Row: 840. Token limit exceeded for 'input_text' [tokens: 16309|limit: 8192] or 'output_text' [tokens: 33|limit: 1024]",
    "Row: 868. Token limit exceeded for 'input_text' [tokens: 20337|limit: 8192] or 'output_text' [tokens: 51|limit: 1024]",
    "Row: 1535. Token limit exceeded for 'input_text' [tokens: 8969|limit: 8192] or 'output_text' [tokens: 26|limit: 1024]"
  ],
  "rai_issues": [
    "Row: 15. RAI violation. High scores for categories Finance",
    "Row: 46. RAI violation. High scores for categories Finance",
    "Row: 275. RAI violation. High scores for categories Finance",
    "Row: 401. RAI violation. High scores for categories Finance",
    "Row: 444. RAI violation. High scores for categories Health",
    "Row: 503. RAI violation. High scores for categories Finance",
    "Row: 558. RAI violation. High scores for categories Finance",
    "Row: 571. RAI violation. High scores for categories Health",
    "Row: 848. RAI violation. High scores for categories Finance",
    "Row: 934. RAI violation. High scores for categories Finance",
    "... there are more cases ..."
  ],
  "errors": [
    "Row: 122. exceeds token limit",
    "Row: 362. exceeds token limit",
    "Row: 391. exceeds token limit",
    "Row: 528. exceeds token limit",
    "Row: 840. exceeds token limit",
    "Row: 868. exceeds token limit",
    "Row: 1535. exceeds token limit"
  ],
  "max_user_output_token_length": 79
}

Unable to use matching engine in collab!

I am not able to use
Import custom Matching Engine packages
from utils.matching_engine import MatchingEngine
from utils.matching_engine_utils import MatchingEngineUtils

Throwing error as No module named 'utils' after installing utils its giving error as No module named 'utils.matching_engine'.

gs:// missing from question_answering_documents_langchain_matching_engine.ipynb

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/document-qa/question_answering_documents_langchain_matching_engine.ipynb

From ->

! gsutil mb -l us-central1 $ME_EMBEDDING_DIR

To ->

! gsutil mb -l us-central1 gs://$ME_EMBEDDING_DIR

and

from ->

! gsutil cp embeddings_0.json {ME_EMBEDDING_DIR}/init_index/embeddings_0.json

To ->

! gsutil cp embeddings_0.json gs://{ME_EMBEDDING_DIR}/init_index/embeddings_0.json

Input Tokens Count in ChatBison

Im using ChatBison to get answers from a piece of text , suppose I give input text using palm.chat() and then ask questions using palm.reply(). Does the conversation also include previous historical text as input again and again ?

Run Quantum Computing Cell Error

Commit 8a28ee4
appears to remove the cell which prompts the model to ask about quantum computing.

This makes the lab "Generative AI with Vertex AI: Getting Started" difficult to complete as the assessment check message errors and returns "Please run the Quantum Computing cell" which no longer exists as it has been removed in the above commit.

Looking through commit history and re-adding this prompt to the cell fixed this issue for me.

The prompt seems to have to be Quantum Computing related to pass the assessment check.

None the less, the current error message on the assessment check must be clearer than it currently is without the Quantum computing cell.

Fix for me...

Unable to deploy Data Checker Python Notebook

Deploying data checker python notebook fails with 404 error. Looks like link is referring to incorrect url,

Thanks
Jignesh

Establish consistency standards in notebooks around (colab vs workbench)

Sometimes you have code blocks which are commented and you say to uncomment them, sometimes you have codeblocks which are uncommented and you say to comment them.

• https://github.com/search?q=repo%3AGoogleCloudPlatform%2Fgenerative-ai+workbench+comment&type=code
• https://github.com/search?q=repo%3AGoogleCloudPlatform%2Fgenerative-ai+workbench+uncomment&type=code

Instead, can we establish shared conventions for notebook set up, and ideally a runtime conditional allowing it to work in workbench and colab without editing. The fewer things a user needs to do to execute the colab, the better.

Other things to try to standardize on:

• auto-restart of the runtime after package install, if restart needed: https://g.co/bard/share/0df557a3a623
• params for anything you expect a user to edit, just to get the colab to run (eg: no private data stores)
• public sample data for anything you need to use (eg: no private data stores)

The following quota exhausted: `MatchingEngineStreamUpdateThroughput`

Hello,

Could you please help on this error running a notebook step ?

Notebook

Question Answering with Documents using LangChain 🦜️🔗 and Vertex AI Matching Engine
here: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-qa/question_answering_documents_langchain_matching_engine.ipynb

We get the same result on Google Colab and on a Vertex AI User managed notebook.

Step

doc_ids = me.add_texts(texts=texts, metadatas=metadatas)

Error

status = StatusCode.RESOURCE_EXHAUSTED
details = "The following quota exhausted: MatchingEngineStreamUpdateThroughput."
debug_error_string = "UNKNOWN:Error received from peer ipv4:[REDACTED]:443 {grpc_message:"The following quota exhausted: MatchingEngineStreamUpdateThroughput.", grpc_status:8, created_time:"2023-10-17T19:08:25.548707816+00:00"}"

Questions

1. The default region quota for MatchingEngineStreamUpdateThroughput is 300 KB, which quota increase value should we ask for ?
2. Would you explain how to evaluate the needed value ?
3. Would it be possible to improve the notebook to prevent the quota increase need ?

Thanks in advance for your help
Ludovic

MatchingEngine workbooks don't work (intro and vector-search)

In intro-textemb... workbook fails on this cell:

create index

my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
display_name = f"embvs-tutorial-index-{UID}",
contents_delta_uri = BUCKET_URI,
dimensions = 768,
approximate_neighbors_count = 20,
)

vector-search... workbook fails on this cell:

create Index

my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
display_name = f"vs-quickstart-index-{UID}",
contents_delta_uri = BUCKET_URI,
dimensions = 768,
approximate_neighbors_count = 10,
)

Stacktrace looks like this:

---

_InactiveRpcError Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:65, in wrap_unary_errors..error_remapped_callable(*args, **kwargs)
64 try:
---> 65 return callable(*args, **kwargs)
66 except grpc.RpcError as exc:

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:1161, in _UnaryUnaryMultiCallable.call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
1155 (
1156 state,
1157 call,
1158 ) = self._blocking(
1159 request, timeout, metadata, credentials, wait_for_ready, compression
1160 )
-> 1161 return _end_unary_response_blocking(state, call, False, None)

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:1004, in _end_unary_response_blocking(state, call, with_call, deadline)
1003 else:
-> 1004 raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered."
debug_error_string = "UNKNOWN:Error received from peer ipv4:64.233.191.95:443 {grpc_message:"Internal error encountered.", grpc_status:13, created_time:"2023-11-14T23:25:07.40184234+00:00"}"

The above exception was the direct cause of the following exception:

InternalServerError Traceback (most recent call last)
Cell In[16], line 2
1 # create index
----> 2 my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
3 display_name = f"embvs-tutorial-index-{UID}",
4 contents_delta_uri = BUCKET_URI,
5 dimensions = 768,
6 approximate_neighbors_count = 20,
7 )

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index.py:523, in MatchingEngineIndex.create_tree_ah_index(cls, display_name, contents_delta_uri, dimensions, approximate_neighbors_count, leaf_node_embedding_count, leaf_nodes_to_search_percent, distance_measure_type, description, labels, project, location, credentials, request_metadata, sync, index_update_method, encryption_spec_key_name)
511 algorithm_config = matching_engine_index_config.TreeAhConfig(
512 leaf_node_embedding_count=leaf_node_embedding_count,
513 leaf_nodes_to_search_percent=leaf_nodes_to_search_percent,
514 )
516 config = matching_engine_index_config.MatchingEngineIndexConfig(
517 dimensions=dimensions,
518 algorithm_config=algorithm_config,
519 approximate_neighbors_count=approximate_neighbors_count,
520 distance_measure_type=distance_measure_type,
521 )
--> 523 return cls._create(
524 display_name=display_name,
525 contents_delta_uri=contents_delta_uri,
526 config=config,
527 description=description,
528 labels=labels,
529 project=project,
530 location=location,
531 credentials=credentials,
532 request_metadata=request_metadata,
533 sync=sync,
534 index_update_method=index_update_method,
535 encryption_spec_key_name=encryption_spec_key_name,
536 )

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/base.py:817, in optional_sync..optional_run_in_thread..wrapper(*args, **kwargs)
815 if self:
816 VertexAiResourceNounWithFutureManager.wait(self)
--> 817 return method(*args, **kwargs)
819 # callbacks to call within the Future (in same Thread)
820 internal_callbacks = []

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index.py:209, in MatchingEngineIndex._create(cls, display_name, contents_delta_uri, config, description, labels, project, location, credentials, request_metadata, sync, index_update_method, encryption_spec_key_name)
205 gapic_index.labels = labels
207 api_client = cls._instantiate_client(location=location, credentials=credentials)
--> 209 create_lro = api_client.create_index(
210 parent=initializer.global_config.common_location_path(
211 project=project, location=location
212 ),
213 index=gapic_index,
214 metadata=request_metadata,
215 )
217 _LOGGER.log_create_with_lro(cls, create_lro)
219 created_index = create_lro.result(timeout=None)

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/index_service/client.py:592, in IndexServiceClient.create_index(self, request, parent, index, retry, timeout, metadata)
587 metadata = tuple(metadata) + (
588 gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),
589 )
591 # Send the request.
--> 592 response = rpc(
593 request,
594 retry=retry,
595 timeout=timeout,
596 metadata=metadata,
597 )
599 # Wrap the response in an operation future.
600 response = gac_operation.from_gapic(
601 response,
602 self._transport.operations_client,
603 gca_index.Index,
604 metadata_type=index_service.CreateIndexOperationMetadata,
605 )

File /opt/conda/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py:113, in _GapicCallable.call(self, timeout, retry, *args, **kwargs)
110 metadata.extend(self._metadata)
111 kwargs["metadata"] = metadata
--> 113 return wrapped_func(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:67, in wrap_unary_errors..error_remapped_callable(*args, **kwargs)
65 return callable(*args, **kwargs)
66 except grpc.RpcError as exc:
---> 67 raise exceptions.from_grpc_error(exc) from exc

InternalServerError: 500 Internal error encountered.

Is this possibly a change in the grpcapi that broken the python client?

steps to creating datastore missing in data_store_checker notebook

data_store_checker.ipynb does not show the steps to create the data store. This might make the notebook difficult to learn from for those new to the topic.

Solution: we should add the steps to creating the datastore.

Contributing Guide link on README.md is not working

Visit README.md and click on Contributing Guide link : https://github.com/GoogleCloudPlatform/generative-ai/CONTRIBUTING.md

Missing `PreviewTextGenerationModel`

From ->

from vertexai.language_models import TextGenerationModel

To ->

from vertexai.language_models import TextGenerationModel
from vertexai.language_models._language_models import _PreviewTextGenerationModel

setup-env/ link on README.md file is incorrect

The current link for setup-env in README.md file is incorrect. It should be https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env

Ensemble LLMs to get better results: Suggestion for new example

Each LLM model has its own strength, has its own diverse corpus. Rather than relying on one LLM at a time, to get response, why not employ multiple LLMs and then get the best result by ranking the responses?

Steps for any problem:

• Have multiple LLMs in ensemble (for-loop)
• Let each LLM respond to the given prompt, collect responses from each
• Let each LLM rank the responses from all the LLMs.
• Aggregate the ranks, or some way of majority, find the best response.

Json file based QA model

I'm trying to replicate the document based question and answering model but instead of the reading and storing the document in the chromoDB. I want to read the Json file and store it in the chromoDB. If user requested any questions then it should reply from the json file. So, the json file is the source for QA model. The size of json file is huge so I need to do chunking also.

How to query a public endpoint in matching engine.

**I have created a public endpoint to deploy the indexes.

Like this:**

my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name=DISPLAY_NAME,
    contents_delta_uri=EMBED_URI, **//Path the embedding bucket**
    dimensions=ME_DIMENSIONS,
    approximate_neighbors_count=150,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
)

Then deployed the index to public endpoint

my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name=f"index_endpoint_for_demo_new",
    public_endpoint_enabled= True
)

DEPLOYED_INDEX_ID = f"index_endpoint_for_demo_new"
my_index_endpoint = my_index_endpoint.deploy_index(
    index=my_index, deployed_index_id=DEPLOYED_INDEX_ID
)

my_index_endpoint.deployed_indexes

Now I have created a vector store from the deployed endpoint and created index

vector_store = MatchingEngine.from_components(
    project_id=PROJECT_ID,
    region=REGION,
    gcs_bucket_name=EMBED_URI,
    index_id="#####",
    endpoint_id="#####",
)

Now when I am don't similarity search like this:

vector_store.similarity_search("American Home Shield Corporation", k=2)

It's throwing error

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "DNS resolution failed for :10000: unparseable host:port"
	debug_error_string = "UNKNOWN:DNS resolution failed for :10000: unparseable host:port {created_time:"2023-08-07T08:51:47.728727082+00:00", grpc_status:14}"
>

How to solve this issue?

How to run mutiple text model requests to summarize files in parallel

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/document-summarization/summarization_large_documents.ipynb

Read the PDF file and create a list of pages

reader = PyPDF2.PdfReader(pdf_file)
pages = reader.pages

Create an empty list to store the summaries

initial_summary = []

Iterate over the pages and generate a summary for each page

for page in tqdm(pages):

# Extract the text from the page and remove any leading or trailing whitespace
text = page.extract_text().strip()

# Create a prompt for the model using the extracted text and a prompt template
prompt = initial_prompt_template.format(text=text)

# Generate a summary using the model and the prompt
summary = model_with_limit_and_backoff(prompt=prompt, max_output_tokens=1024).text

# Append the summary to the list of summaries
initial_summary.append(summary)

it should be a sync code to run one after one.
how to make it in parallel?

Add Documentation on Policies and Permissions Needed

In order to get this demo to work I had to make two changes:

1.) set constraints/iam.allowedPolicyMemberDomains to allow all
2.) Give cloudrun service account (by default the default compute service account) discoveryengine.viewer permissions

Quota exceeded no matter what the quota is in langchain notebook example

When attempting to run https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-qa/question_answering_documents_langchain.ipynb in cell 18

doc_ids = me.add_texts(texts=texts, metadatas=metadatas)

I get a

ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/quotas.

My quota is not exceeded however (and I increased it) when checking the quota dashboard in GCP.

I attempted to limit the scope using

doc_ids = me.add_texts(texts=texts[0], metadatas=metadatas[0])

and receive the same error.

gunicorn failure on cloud run example

Hello,

I'm running the search webapp following the readme. The cloud run is successfully built but raises errors when deployed:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker worker.init_process() File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py", line 95, in init_process super().init_process() File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process self.load_wsgi() File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi self.wsgi = self.app.wsgi()

The cloud run web page displays "service not available".

I tried several verion of gunicorn through the Dockerfile, but it didn't help.

Any idea?
Thanks.

Getting Started with LangChain and Vertex AI PaLM API notebook has broken links to LangChain documentation

The Getting Started with LangChain + Vertex AI PaLM API is awesome. However, a few links to the Langchain Chat documentation are broken.

• Link to https://python.langchain.com/docs/modules/model_io/models/chat/getting_started.html in Chat Messages cell 404s. Maybe should be https://python.langchain.com/docs/modules/model_io/models/chat/
• Link to https://python.langchain.com/docs/modules/model_io/models/text_embedding/examples/google_vertex_ai_palm.html in Text Embedding Model 404s. Maybe should be https://python.langchain.com/docs/modules/data_connection/text_embedding/
• Link to https://python.langchain.com/en/latest/modules/prompts/output_parsers.html in Output Parsers also 404s, but there doesn't seem to be a good landing spot in LangChain docs (e.g. Python Guide link here https://docs.langchain.com/docs/components/prompts/output-parser in the LangChain docs also 404s).
• Link to https://python.langchain.com/docs/modules/data_connection/vectorstores/examples/matchingengine.html in VectorStores 404s. Maybe should be https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/matchingengine
• Link to https://python.langchain.com/en/latest/modules/memory/getting_started.html in Memory cell 404s. Maybe should be https://python.langchain.com/docs/modules/memory/. In same cell https://python.langchain.com/en/latest/modules/memory/how_to_guides.html 404s but I could not find an obvious link in the current LangChain docs to replace this one.
• Link to https://python.langchain.com/en/latest/modules/chains/how_to_guides.html in Chains intro 404s. Maybe should be https://python.langchain.com/docs/modules/chains/. In same cell, https://python.langchain.com/docs/modules/chains/generic/sequential_chains.html also 404s. Maybe should be https://python.langchain.com/docs/modules/chains/foundational/sequential_chains
• Link to https://python.langchain.com/docs/modules/chains/index_examples/summarize.html 404s in Summarization Chain cell, Maybe should be https://python.langchain.com/docs/modules/chains/popular/summarize
• Link to https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa_with_sources.html 404s in Question/Answering Chain cell. Maybe should be https://python.langchain.com/docs/modules/chains/popular/chat_vector_db
• not a doc link but https://abc.xyz/investor/static/pdf/20230203_alphabet_10K.pdf also 404s, which causes the first code cell in Question/Answering Chain to fail, Not sure what alternative to recommend

# Ingest PDF files
from langchain.document_loaders import PyPDFLoader

# Load GOOG's 10K annual report (92 pages).
url = "https://abc.xyz/investor/static/pdf/20230203_alphabet_10K.pdf"
loader = PyPDFLoader(url)
documents = loader.load()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-83-8cded97b8ca8>](https://localhost:8080/#) in <cell line: 6>()
      4 # Load GOOG's 10K annual report (92 pages).
      5 url = "https://abc.xyz/investor/static/pdf/20230203_alphabet_10K.pdf"
----> 6 loader = PyPDFLoader(url)
      7 documents = loader.load()

1 frames
[~/.local/lib/python3.10/site-packages/langchain/document_loaders/pdf.py](https://localhost:8080/#) in __init__(self, file_path)
     57 
     58             if r.status_code != 200:
---> 59                 raise ValueError(
     60                     "Check the url of your file; returned status code %s"
     61                     % r.status_code

ValueError: Check the url of your file; returned status code 404