aws-samples / amazon-transcribe-live-meeting-assistant

Live Meeting Assistant (LMA) with Amazon Transcribe, Amazon Bedrock, and Knowledge Bases for Amazon Bedrock

Companion AWS blog post: Live Meeting Assistant with Amazon Transcribe, Amazon Bedrock, and Knowledge Bases for Amazon Bedrock

See CHANGELOG for latest features and fixes.

Introduction

You’ve likely experienced the challenge of taking notes during a meeting while trying to pay attention to the conversation. You’ve probably also experienced the need to quickly fact-check something that’s been said, or look up information to answer a question that’s just been asked in the call. Or maybe you have a team member that always joins meetings late, and expects you to send them a quick summary over chat to catch them up.

Then there are the times that others are talking in a language that’s not your first language, and you’d love to have a live translation of what people are saying to make sure you understand correctly.

And after the call is over, you usually want to capture a summary for your records, or to send to the participants, with a list of all the action items, owners, and due dates.

All of this, and more, is now possible with our newest sample solution, Live Meeting Assistant (LMA).

Check out the following demo to see how it works.

LMABlogDemoV3.mp4

Solution overview

The Live Meeting Assistant (LMA) sample solution captures speaker audio and metadata from your browser-based meeting app (Zoom, Teams, and Chime currently), or audio only from any other browser-based meeting app, softphone or audio source. It uses Amazon Transcribe for speech to text, Knowledge Bases for Amazon Bedrock for contextual queries against your company's documents and knowledge sources, and Amazon Bedrock for customizable transcription insights and summaries.

Everything you need is provided as open source in this GitHub repo. It’s straightforward to deploy in your AWS account. When you’re done, you’ll wonder how you ever managed without it!

The following are some of the things LMA can do:

• Live transcription with speaker attribution - LMA is powered by Amazon Transcribe ASR models for low-latency, high-accuracy speech to text. You can teach it brand names and domain-specific terminology if needed, using custom vocabulary and custom language model features in Amazon Transcribe.

  

• Live translation - It uses Amazon Translate to optionally show each segment of the conversation translated into your language of choice, from a selection of 75 languages.

  

• Context aware meeting assistant - It uses Knowledge Bases for Amazon Bedrock to provide answers from your trusted sources, or a Bedrock LLM if you don't have or need a knowledge base, using the live transcript as context for fact-checking and follow-up questions. To activate the assistant, just say “Okay, Assistant,” choose the ASK ASSISTANT! button, or enter your own question in the UI.

  

• On demand summaries of the meeting - With the click of a button on the UI, you can generate a summary, which is useful when someone joins late and needs to get caught up. The summaries are generated from the transcript by Amazon Bedrock. LMA also provides options for identifying the current meeting topic, and for generating a list of action items with owners and due dates. You can also create your own custom prompts and corresponding options.

  

• Automated summary and insights - When the meeting has ended, LMA automatically runs a set of large language model (LLM) prompts on Amazon Bedrock to summarize the meeting transcript and extract insights. You can customize these prompts as well.

  

• Meeting Recording - The audio is (optionally) stored for you, so you can replay important sections on the meeting later.

  

• Inventory list of meetings - LMA keeps track of all your meetings in a searchable list.

  

• Browser extension captures audio and meeting metadata from popular meeting apps - The browser extension captures meeting metadata — the meeting title and names of active speakers — and audio from you (your microphone) and others (from the meeting browser tab). As of this writing, LMA supports Chrome for the browser extension, and Zoom, Teams and Chime for meeting apps (with others coming soon). Standalone meeting apps don’t work with LMA — instead, launch your meetings in the browser.

  

You are responsible for complying with legal, corporate, and ethical restrictions that apply to recording meetings and calls. Do not use this solution to stream, record, or transcribe calls if otherwise prohibited.

Prerequisites

You need to have an AWS account and an AWS Identity and Access Management (IAM) role and user with permissions to create and manage the necessary resources and components for this application. If you don’t have an AWS account, see How do I create and activate a new Amazon Web Services account?

If you want LMA to use your own trusted documents to power the context aware meeting assistant then you will use a Knowledge Base on Amazon Bedrock. You choose to have LMA either:

1. Automatically create a new knowledge base and populate it from documents in an S3 bucket in your AWS account, or from publicly accessible Web pages. Be prepared to specify your document location when you deploy LMA.
2. Or, use an existing knowledge base that you have previously created and populated. See Create a knowledge base.
   If you choose not to have LMA use your own documents, then no knowledge base is needed, and the LMA meeting assistant relies directly on the 'world knowledge' of the LLM model.

Finally, LMA uses Amazon Bedrock LLM models for its live meeting assistant and meeting summarization features. Before proceeding, if you have not previously done so, you must request access to the following Amazon Bedrock models:

• Amazon: All Titan Embeddings models (Titan Embeddings G1 - Text, and Titan Text Embeddings V2)
• Anthropic: All Claude 3 models (Claude 3 Sonnet and Claude 3 Haiku)

Deploy the CloudFormation stack

We’ve provided pre-built AWS CloudFormation templates that deploy everything you need in your AWS account.

If you’re a developer and you want to build, deploy, or publish the solution from code, refer to the Developer README.

Complete the following steps to launch the CloudFormation stack:

1. Log in to the AWS Management Console.

2. 2. Choose Launch Stack for your desired AWS Region to open the AWS CloudFormation console and create a new stack.

   Region	Easy Deploy Button
   US East (N. Virginia)
   US West (Oregon)

3. For Stack name, use the default value, LMA.

4. For Admin Email Address, use a valid email address—your temporary password is emailed to this address during the deployment.

5. For Authorized Account Email Domain, use the domain name part of your corporate email address to allow users with email addresses in the same domain to create their own new UI accounts, or leave blank to prevent users from directly creating their own accounts. You can enter multiple domains as a comma separated list.

6. For MeetingAssistService choose BEDROCK_KNOWLEDGE_BASE (Use Existing), BEDROCK_KNOWLEDGE_BASE (Create), or BEDROCK_LLM (if you do not need a knowledge base)

   1. If you select BEDROCK_KNOWLEDGE_BASE (Use Existing), then:
      • For Bedrock Knowledge Base Id (existing), enter your existing Knowledge base ID (for example, JSXXXXX3D8). You can copy it from the Amazon Bedrock Knowledge bases console.

        

   2. If you select BEDROCK_KNOWLEDGE_BASE (Create), a new knowledge base is created for you automatically:
      • To optionally populate your new knowledge base with documents from an Amazon S3 bucket:
        1. For S3 bucket with knowledge base source documents, enter the name of an existing Amazon S3 bucket containing the documents you want to ingest into your new knowledge base.
        2. For S3 prefix(es) for your content, enter any S3 prefixes (paths) to the documents in the S3 bucket, or leave it blank to ingest all documents in the bucket.
      • To optionally populate your new knowledge base with web site content from public web pages:
        1. For Publicly accessible URLs for web crawling, enter a comma separated list of web site Urls
        2. For Web crawling sync scope, select the choice that best matches how you want the web crawler to navigate out from the Urls you specified.

7. For all other parameters, use the default values. If you want to customize the settings later, for example to add your own lambda functions, to use custom vocabularies and language models to improve accuracy, enable PII redaction, and more, you can update the stack for these parameters.

8. Check the acknowledgement boxes, and choose Create stack.

The main CloudFormation stack uses nested stacks to create the following resources in your AWS account:

• • Amazon Simple Storage Service (Amazon S3) to hold build artifacts and call recordings
• An AWS Fargate task with an Application Load Balancer providing a websocket server running code to consume stereo audio streams and relay to Amazon Transcribe, publish transcription segments in Kinesis Data Streams, and create and store stereo call recordings.
• Amazon Kinesis Data Stream to relay call events and transcription segments to the enrichment processing function.
• Meeting assist resources including the QnABot on AWS solution stack which interacts with Amazon OpenSearch service and Amazon Bedrock.
• AWS AppSync API, which provides a GraphQL endpoint to support queries and real-time updates
• Website components including S3 bucket, Amazon CloudFront distribution, and Amazon Cognito user pool
• A downloadable pre-configured browser extension application for Chrome browsers.
• Other miscellaneous supporting resources, including AWS Identity and Access Management (IAM) roles and policies (using least privilege best practices), Amazon - Virtual Private Cloud (Amazon VPC) resources, Amazon EventBridge event rules, and Amazon CloudWatch log groups.

The stacks take about 35-40 minutes to deploy. The main stack status shows CREATE_COMPLETE when everything is deployed. You may want to skip ahead to review “Processing flow overview” while you wait for it, and then come back here when it’s deployed.

Set your password

After you deploy the stack, you need to open the LMA web user interface and set your password.

1. Open the email you received, at the email address you provided, with the subject “Welcome to Live Meeting Assistant!”

2. Open your web browser to the URL shown in the email. You’re directed to the login page.

   

3. The email contains a generated temporary password that you use to log in and create your own password. Your username is your email address.

4. Set a new password.
   Your new password must have a length of at least eight characters, and contain uppercase and lowercase characters, plus numbers and special characters.

Follow the directions to verify your email address, or choose Skip to do it later.

You’re now logged in to LMA.

You also received a similar email with the subject “QnABot Signup Verification Code.” This email contains a generated temporary password that you use to log in and create your own password in the QnABot designer. You use QnABot designer only if you want to customize LMA meeting assistant easy buttons and prompts. Your username for QnABot is “Admin”. You can set your permanent QnABot Admin password now, or keep this email safe in case you want to customize things later.

Download and install the Chrome browser extension

For the best meeting streaming experience, install the LMA browser plugin - currently available for Chrome.

1. Choose Download Chrome Extension to download the browser extension zip file (lma-chrome-extension.zip)

   

2. Right click and expand the zip file (lma-chrome-extension.zip) to create a local folder named lma-chrome-extension.

3. Open Chrome and paste the link chrome://extensions into the address bar.

   

4. Enable Developer mode.

5. Choose Load unpacked, navigate to the lma-chrome-extension folder (which you unzipped from the download), and click select. This loads your extension.

6. Pin the new LMA extension to the browser tool bar as shown below. You will use it often to stream your meetings.

   

Start using LMA

LMA provides two streaming options:

1. Use the Chrome browser extension to stream audio and speaker metadata from your meeting browser app. It currently works with Zoom, Teams and Chime, but we hope to add more meeting apps.
2. Use the LMA Stream Audio tab to stream audio from your microphone and any Chrome browser-based meeting app, softphone, or audio application. We show you how to use both options in the following sections.

Option 1: Use the Chrome browser extension to stream a Zoom call

1. Open the LMA extension and login with your LMA credentials.

   

2. Join or start a Zoom meeting in your web browser (do not use the separate Zoom client). If you already have the Zoom meeting page loaded, please reload it.

   

   The LMA extension automatically detects that Zoom is running in the browser tab, and populates your name and the meeting name.

   

3. Tell others on the call that you are about to start recording the call using LMA and obtain their permission. Do not proceed if participants object.

4. Choose Start Listening.

5. Read and accept the disclaimer. Choose Allow on the popup asking you to share the browser tab.

   

   The LMA extension automatically detects and displays the active speaker on the call. If you are alone in the meeting, invite some friends to join, and observe that the names they used to join the call are displayed in the extension when they speak, and attributed to their words in the LMA transcript.

   

6. Choose Open in LMA to see your live transcript in a new tab.

7. Choose your preferred transcript language, and interact with the meeting assistant using the wake phrase "OK Assistant!" or the Meeting Assist Bot pane on the right. The ASK ASSISTANT button is fun to try – it asks the meeting assistant service (Bedrock knowledge base or Bedrock LLM) to suggest a ‘good response’ based on the transcript of the recent interactions in the meeting. Your mileage may vary, so experiment!

   

8. When you are done, choose Stop Streaming to end the meeting in LMA. Within a few seconds the automated end-of-meeting summaries appear, and the audio recording becomes available. You can continue to use the bot after the call has ended.

   

Option 2: Use the LMA UI Stream Audio tab to stream from your microphone and any browser based audio application

The browser extension is the most convenient way to stream metadata and audio from the supported meeting web apps. However, you can also use LMA to stream from any browser based softphone, meeting app, or any other audio source playing in your Chrome browser, using the very convenient Stream Audio tab that is built into the LMA UI.

1. Open any audio source in a Chrome browser tab. For example, this could be a softphone (such as Google Voice), another meeting app, or for demo purposes, you can simply play a local audio recording or a YouTube video in your browser to emulate another meeting participant. If you just want to try it, open the following YouTube video in a new tab.

   

2. In the LMA App UI, choose Stream Audio (no extension) to open the Stream Audio tab.

   

3. For Meeting Topic, enter a name for your meeting. When the recording starts it will be appended to a timestamp to create a unique meeting identifier.

4. For Meeting owner (microphone), enter a name for yourself (applied to audio from your microphone)

5. For Participants (stream), enter the name(s) of the other participants (applied to the incoming audio source).

6. Choose Start Streaming

7. Choose the browser tab you opened earlier, and choose Allow to share.

8. To see your in-progress meeting use the "Open in progress meeting" link. NOTE: you may have a wait a few seconds for the meeting to start streaming before the meeting page is available.

   

9. You can also access the meeting from the LMA UI tab where you will see it "In Progress".

   

10. You can choose to mute or unmute your own microphone during the stream.

    

11. To stop the recording click on the "Stop Streaming" button.

12. Once you do that a link to the meeting that has just been recorded will appear at the bottom of the screen.

    

Use the Stream Audio feature to stream from any softphone app, meeting app, or any other streaming audio playing in the browser, along with your own audio captured from your selected microphone. Always obtain permission from others before recording them using LMA, or any other recording application.

Processing flow overview

How did LMA transcribe and analyze your meeting? Let’s look at how it works. The following diagram shows the main architectural components and how they fit together at a high level.

The LMA user joins a meeting in their browser, enables the LMA browser extension, and authenticates using their LMA credentials. If the meeting app (for example, Zoom.us) is supported by the LMA extension, the user's name, meeting name, and active speaker names are automatically detected by the extension. If the meeting app is not supported by the extension, then the LMA user can manually enter their name and the meeting topic—active speakers’ names will not be detected.

After getting permission from other participants, the LMA user chooses Start Listening on the LMA extension pane. A secure WebSocket connection is established to the preconfigured LMA stack WebSocket URL, and the user's authentication token is validated. The LMA browser extension sends a START message to the WebSocket containing the meeting metadata (name, topic, and so on), and starts streaming two-channel audio from the user's microphone and the incoming audio channel containing the voices of the other meeting participants. The extension monitors the meeting app to detect active speaker changes during the call, and sends that metadata to the WebSocket, enabling LMA to label speech segments with the speaker's name.

The WebSocket server running in Fargate consumes the real-time two-channel audio fragments from the incoming WebSocket stream. The audio is streamed to Amazon Transcribe, and the transcription results are written in real time to Kinesis Data Streams.

Each meeting processing session runs until the user chooses Stop Listening in the LMA extension pane, or ends the meeting and closes the tab. At the end of the call, the function creates a stereo recording file in Amazon S3 (if recording was enabled when the stack was deployed).

An AWS Lambda function, the Call Event Processor, fed by Kinesis Data Streams, processes and optionally enriches meeting metadata and transcription segments. The Call Event Processor integrates with the Meeting Assist services. LMA meeting assist is powered by Amazon Lex, Bedrock knowledge bases, and Amazon Bedrock large language models using the open source QnABot on AWS solution for answers based on FAQs and as an orchestrator for request routing to the appropriate AI service. The Call Event Processor also invokes the Transcript Summarization lambda when the call ends, to generate a summary of the call from the full transcript.

The Call Event Processor function interfaces with AWS AppSync to persist changes (mutations) in DynamoDB and to send real-time updates to the LMA user's logged in web clients (conveniently opened by choosing the Open in LMA option shown in the browser extension.)

The LMA web UI assets are hosted on Amazon S3 and served via CloudFront. Authentication is provided by Amazon Cognito.

When the user is authenticated, the web application establishes a secure GraphQL connection to the AWS AppSync API, and subscribes to receive real-time events such as new calls and call status changes for the meetings list page, and new or updated transcription segments and computed analytics for the meeting details page. When translation is enabled, the web application also interacts securely with Amazon Translate to translate the meeting transcription into the selected language.

The entire processing flow, from ingested speech to live webpage updates, is event driven, and so the end-to-end latency is small—typically just a few seconds.

Monitoring and troubleshooting

AWS CloudFormation reports deployment failures and causes on the relevant stack Events tab. See Troubleshooting CloudFormation for help with common deployment problems. Look out for deployment failures caused by limit exceeded errors; the LMA stacks create resources that are subject to default account and Region service quotas, such as elastic IP addresses and NAT gateways. When troubleshooting CloudFormation stack failures, always navigate into any failed nested stacks to find the first nested resource failure reported—this is almost always the root cause.

Amazon Transcribe has a default limit of 25 concurrent transcription streams, which limits LMA to 25 concurrent meetings in a given AWS account or Region. Request an increase for the number of concurrent HTTP/2 streams for streaming transcription if you have many users and need to handle a larger number of concurrent meetings in your account.

LMA provides runtime monitoring and logs for each component using CloudWatch:

• Websocket processing and transcribing Fargate task – On the Amazon Elastic Container Service (Amazon ECS) console, navigate to the Clusters page and open the LMA-WEBSOCKETSTACK-xxxx-TranscribingCluster function. Choose the Tasks tab and open the task page. Choose Logs and View in CloudWatch to inspect the WebSocket transcriber task logs.
• Call Event Processor Lambda function – On the Lambda console, open the AISTACK-CallEventProcessor function. Choose the Monitor tab to see function metrics. Choose View logs in CloudWatch to inspect function logs.
• AWS AppSync API – On the AWS AppSync console, open the CallAnalytics-LMA API. Choose Monitoring in the navigation pane to see API metrics. Choose View logs in CloudWatch to inspect AWS AppSync API logs.

For QnABot on AWS for Meeting Assist, refer to the Meeting Assist README, and the QnABot solution implementation guide for additional information.

Cost assessment

LMA provides a websocket server using Fargate (0.25vCPU) and VPC networking resources costing about $0.014/hr (~$10/mth) - see Fargate pricing.

Meeting Assist is enabled using QnABot and Knowledge bases for Amazon Bedrock. You create your own Knowledge base which you use for LMA and potentially other use cases – see Amazon Bedrock pricing for more, or you can choose, when you deploy, to use a Bedrock LLM without a knowledge base. Additional AWS services used by the QnABot solution as configured (with Opensearch node count default of '1') cost about $0.14/hour ($100/mth) – see QnABot on AWS solution costs.

The remaining solution costs are based on usage.

The usage costs add up to about $0.17 for a 5-minute call, although this can vary based on options selected (such as translation), number of LLM summarizations and total usage because usage affects Free Tier eligibility and volume tiered pricing for many services. For more information about the services that incur usage costs, see the following:

• AWS AppSync pricing
• Amazon Translate pricing
• Amazon Cognito Pricing
• Amazon DynamoDB pricing
• AWS Lambda Pricing
• Amazon S3 pricing
• Amazon Transcribe Pricing
• Amazon Bedrock pricing
• QnABot on AWS costs

To explore LMA costs for yourself, use AWS Cost Explorer or choose Bill Details on the AWS Billing Dashboard to see your month-to-date spend by service.

Customize your deployment

Use the following CloudFormation template parameters when creating or updating your stack to customize your LMA deployment:

• To use your own S3 bucket for meeting recordings, use Call Audio Recordings Bucket Name and Audio File Prefix.
• To redact PII from the transcriptions, set Enable Content Redaction for Transcripts to true, and adjust Transcription PII Redaction Entity Types as needed. For more information, see Redacting or identifying PII in a real-time stream.
• To improve transcription accuracy for technical and domain-specific acronyms and jargon, set Transcription Custom Vocabulary Name to the name of a custom vocabulary that you already created in Amazon Transcribe and/or set Transcription Custom Language Model Name to the name of a previously created custom language model. For more information, see Improving Transcription Accuracy.
• To transcribe meetings in a supported language other than US English, chose the desired value for Language for Transcription. You can also choose to have Transcribe identify the primary language, or even multiple languages used during the meeting by setting Language for Transcription to identify-language or identify-multiple-languages and provide a list of languages with an optional preferred language - see Language identification with streaming transcriptions.
• To customize transcript processing, optionally set Lambda Hook Function ARN for Custom Transcript Segment Processing to the ARN of your own Lambda function. For more information, see Using a Lambda function to optionally provide custom logic for transcript processing.
• To customize the Meeting Assist capabilities based on the QnABot on AWS solution, Amazon Lex, Amazon Bedrock, and Bedrock Knowledge base integration, see the Meeting Assist README.
• To customize Transcript Summarization by configuring LMA to call your own Lambda function, see Transcript Summarization LAMBDA option.
• To customize Transcript Summarization by modifying the default prompts or adding new ones, see Transcript Summarization.
• To change the retention period, set Record Expiration In Days to the desired value. All call data is permanently deleted from the LMA DynamoDB storage after this period. Changes to this setting apply only to new calls received after the update.

LMA is an open-source project. You can fork the LMA GitHub repository, enhance the code, and send us pull requests so we can incorporate and share your improvements!

Update an existing LMA stack

1. Log into the AWS console if you are not already. Note: If you are logged in as an IAM user, ensure your account has permissions to create and manage the necessary resources and components for this application.
2. Select your existing LMA stack
3. Choose Update
4. Choose Replace current template
5. Use one of the published template below for your region, or use the Template URL output of the publish.sh script if you have build your own artifacts from the repository:

Region name	Region code	Template URL
US East (N. Virginia)	us-east-1	https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/lma/lma-main.yaml
US West (Oregon)	us-west-2	https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/lma/lma-main.yaml

6. Choose Next and review the stack parameters.
7. Choose Next two more times.
8. Check the blue boxes for creating IAM resources, and choose Update stack to start the update.

Update the Chrome browser extension

Make sure you have the latest version of the browser extension installed in your browser after you update the LMA stack.

1. Remove your existing Amazon Live Meeting Assistant extension in the chrome://extensions/ page.

2. Follow the steps documented in Download and install the Chrome browser extension to install the latest extension from your updated LMA deployment.

Clean Up

Congratulations! 🎉 You have completed all the steps for setting up your live call analytics sample solution using AWS services.

To make sure you are not charged for any unwanted services, you can clean up by deleting the stack created in the Deploy section and its resources.

When you’re finished experimenting with this sample solution, clean up your resources by using the AWS CloudFormation console to delete the LMA stacks that you deployed. This deletes resources that were created by deploying the solution. The recording S3 buckets, the DynamoDB table and CloudWatch Log groups are retained after the stack is deleted to avoid deleting your data.

Live Call Analytics: Companion solution

Our companion solution, Live Call Analytics and Agent Assist (LCA), offers real-time transcription and analytics for contact centers (phone calls) rather than meetings. There are many similarities - in fact LMA has been built using an architecture and many components derived from LCA.

Conclusion

The Live Meeting Assistant sample solution offers a flexible, feature-rich, and customizable approach to provide live meeting assistance to improve your productivity during and after meetings. It uses Amazon AI/ML services like Amazon Transcribe, Amazon Lex, Knowledge Bases for Amazon Bedrock, and Amazon Bedrock LLMs to transcribe and extract real-time insights from your meeting audio.

The sample LMA application is provided as open source—use it as a starting point for your own solution, and help us make it better by contributing back fixes and features via GitHub pull requests. Browse to the LMA GitHub repository to explore the code, choose Watch to be notified of new releases, and check the README for the latest documentation updates.

For expert assistance, AWS Professional Services and other [AWS Partners(https://aws.amazon.com/partners/)] are here to help.

We’d love to hear from you. Let us know what you think in the comments section, or use the issues forum in the LMA GitHub repository.

Contributing

Your contributions are always welcome! Please have a look at the contribution guidelines first. 🎉

Security

See CONTRIBUTING for more information.

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

amazon-transcribe-live-meeting-assistant's Issues

The 'ASK ASSISTANT!' user custom pompt feature isn't functioning when KB_ID is null

Describe the bug
The 'ASK ASSISTANT!' user custom pompt feature isn't functioning through the LMA GUI for me; it shows the error message: 'Default message - you'll see this only if the lambda hook failed'

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
This is caused by the KB_ID not specified to the Cloudformation template during stack deployment time. KB_ID should be a required parameter for the Cloudfomation template.

Screenshots

Additional context
Add any other context about the problem here.

Bedrock KB source links for S3 documents should be click to open

Describe the bug

Sources are linked as s3 uris, like s3://bucket/object
They should be converted to signed URLs so they can be opened in the browser.

Intermittent: "Default message - you'll see this only if the lambda hook failed" response from meeting assistant

Describe the bug

Sometime the meeting assistant responds: "Default message - you'll see this only if the lambda hook failed"

On investigation, the root cause is reflected by this error in the CW Logs for function QNA-lma6-BedrockKB-LambdaHook:

Amazon Bedrock KB Exception:  An error occurred (ValidationException) when calling the RetrieveAndGenerate operation: 1 validation error detected: Value at 'retrieveAndGenerateConfiguration.knowledgeBaseConfiguration.generationConfiguration.promptTemplate.textPromptTemplate' failed to satisfy constraint: Member must have length less than or equal to 4000

which is caused by the prompt size being too long for the Bedrock API call.. due to too much transcript text being included in the prompt.

The transcript by default is truncated to last 12 turns - Qnabot setting LLM_CHAT_HISTORY_MAX_MESSAGES - setting this to a lower value resolves the issue.. but we should defend against it happening at all by checking and truncating prompt length regardless of MAX_MESSAGES (since individual messages can be long)

To Reproduce
Steps to reproduce the behavior:

Generate a transcript with long turns.. eg where one person is presenting..

After several long turns, invoke the meeting assistant via either 'OK Assistant' voice prompt or bot button..

It will respond with "Default message - you'll see this only if the lambda hook failed"

Expected behavior

It should not fail.

Screenshots

Additional context
Add any other context about the problem here.

Stack deployment failure in AISTACK, due to node package checksum problem.

Describe the bug

As of July 15, LMA deployment fails with nested AISTACK reporting a CodeBuild failure.

On investigation, the Code Buidl error is:

npm ERR! code EINTEGRITY
npm ERR! sha512-+nKZ39+nvK7Qq6i0PvWWRA4j/EkfWOtkP/YhMtupm+lJIiHxUrgTr1CcKv1nBk1rHtkRRQ3O2+Ih/q/sA+FXZA== integrity checksum failed when using sha512: wanted sha512-+nKZ39+nvK7Qq6i0PvWWRA4j/EkfWOtkP/YhMtupm+lJIiHxUrgTr1CcKv1nBk1rHtkRRQ3O2+Ih/q/sA+FXZA== but got sha512-oLDq3jw7AcLqKWH2AhCpVTZl8mf6X2YReP+Neh0SJUzV/BdZYjth94tG5toiMB1PPrYtxOCfaoUCkvtuH+3AJA==. (2409319 bytes)

Seems to be related to SheetJS/sheetjs#1009

To Reproduce
Steps to reproduce the behavior:
Deploy LMA v0.1.4

Unable to see data for a meeting, when meeting name includes '/'

Important: Bug reports submitted here should relate specifically to the open source sample amazon-transcribe-live-meeting-assistant project. Do not submit bugs related directly to any AWS service.
If your issue relates to Amazon Transcribe, Amazon Bedrock or Amazon Bedrock Knowledge Base - setup, user provisioning, document ingestion, accuracy, or other aspects of the service, then first check the service documentation, and then reproduce the problem using the relevant AWS service console before engaging AWS support directly. Thank you.

Describe the bug
Unable to see data for a meeting, when meeting name includes '/'

To Reproduce
Steps to reproduce the behavior:

1. Go to Live Meeting Assistant
2. Click on 'Stream Audio'
3. Include '/' in meeting name
4. See error by clicking on meeting after recording is completed

Expected behavior
See details of meetings, similar to those that do not include '/' in meeting name. If entering '/' is prohibited, should see a message stating the same.

Screenshots

Additional context
Add any other context about the problem here.

Updating Participant Name on Stream Audio Page does not reflect in the meeting transcript

Describe the bug
For meetings created using the Stream Audio option the streamed audio always shows as "Other Participant" in the transcript even if you have set it to a different value.

To Reproduce
Steps to reproduce the behavior:

1. Go to Stream Audio (no extension), https:///#/stream
2. Set Participant Names(s) to a different value than "Other Participant"
3. Click Start Streaming
4. Navigate to 'Meetings' and choose the meeting for the stream
5. Observe that the stream audio still shows as "Other Participant" regardless of what it was set to.

Expected behavior
The Participant Names set on the Stream Audio meeting information page should show in the corresponding transcript.

Screenshots
Here is the field in question:

Additional context
There is also a spelling error in Participant Names(s), it should be Participant Name(s)

Browser extension intermittently silently fails to authenticate with web socket resulting in meeting not being transcribed

Describe the bug
A clear and concise description of what the bug is.

Browser extension intermittently silently fails to authenticate with web socket resulting in meeting not being transcribed. When trying to open the meeting in LMA, the meeting page does not exist (is blank)

To Reproduce
Steps to reproduce the behavior:

Open browser extension
Log into LMA
Let it sit for several hours
Start a call, and choose 'Start Listening"
Chose "Open in LMA" - page may be blank because call has silently failed to stream.

Observe Cloudwatch logs for Farget websocket server task for AUTH failed message

Expected behavior

LMA browser extension should not silently fail to authorize to WS server

The LMA App in Stream Audio Mode is transcribing even when I am on mute!

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

The LMA application in stream only mode listens to the microphone and picks up surrounding speech/audio and then transcribes the same even when I am on mute within the Microsoft Teams application. This will lead to bad transcriptions and unwanted data especially when there are background chatter. (office environments)

Describe the solution you'd like
A clear and concise description of what you want to happen.

When on mute, I dont want the application to listen in and start transcribing. I would also ideally want a way to control when the application can listen in.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
NA

Additional context
Add any other context or screenshots about the feature request here.
NA

Multi-language configuration

In most european countries we use both english and local language (in my case, Spain, spanish, es-ES). Seting up LMA with a static language configuration that cannot be updated after deployment makes it unusable for such countries, as both languages can be used depending on the audience.

I would like to either be able to change (manually) the language on the meeting/LMA interface or (in future releases) automatically detect it, if possible.

TEST ALL in QnABot is continuously putting file version into the S3 bucket

Important: Bug reports submitted here should relate specifically to the open source sample amazon-transcribe-live-meeting-assistant project. Do not submit bugs related directly to any AWS service.
If your issue relates to Amazon Transcribe, Amazon Bedrock or Amazon Bedrock Knowledge Base - setup, user provisioning, document ingestion, accuracy, or other aspects of the service, then first check the service documentation, and then reproduce the problem using the relevant AWS service console before engaging AWS support directly. Thank you.

Describe the bug
"TestAllStepLambda" continues to run even when not in use. The "TestAll-2024-05-07-14-02-813.csv" file is continuously put into the S3 bucket "lma-qnabot-cudpdtligjfk-testallbucket-*", and S3 versioning keeps growing.

To Reproduce

Deploy Amazon Transcribe Live Meeting Assist - LMA (v0.1.1). I have not tried other version.

Screenshots

Meeting assistant bot voice output doesn't work.

Describe the bug

The meeting assistant bot pane has a voice interaction mode, where user clicks the microphone to speak the question, and the meeting assistant is supposed to speak back the response, in addition to showing it in text. However it fails to respond with voice.

On investigation, the reason is that the voice SSML in the bot response is missing the tags.

    "messages": [
        {
            "contentType": "SSML",
            "content": "Amazon Redshift is a fully managed, petabyte-scale data warehouse service provided by Amazon Web Services (AWS)."
        }
    ]

To Reproduce

Click the mic in the meeting assist pane, and as a question to the assistant. It should respond with voice, but does not.

By contrast, try again, this time saying 'Start Meeting Assist" - this time the response comes from a built-in QnABot answer (rather than from the Bedrock LLM plugin) and the response is now spoken audibly as expected.

Expected behavior

Bot should deliver voice responses for voice inputs.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

CustomVocabulary does not work when combined with identify-language or identify-multiple-languages

Describe the bug

CustomVocabulary does not work when combined with identify-language or identify-multiple-languages

This message is logged by WebSocket transcriber:

BadRequestException: You must specify the custom vocabularies in the vocabularyNames parameter when using language identification or multiple language identification.

On looking at API spec, when using identify-language or identify-multiple-languages, any Custom Vocabularies must be specified as VocabularyNames - not VocabularyName

To Reproduce
Steps to reproduce the behavior:

Choose identify-language or identify-multiple-languages, AND specific a CustomVocabulary.

Expected behavior

Template parameter CustomVocabulary should accept a list of vocab names for use as VocabularyNames when using identify-language or identify-multiple-languages.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Web URL missing from assistant response sources from Bedrock KB webcrawler

Describe the bug

Assistant responses sourced from web crawler data sources are missing the URL to the source document.

To Reproduce
Steps to reproduce the behavior:

• configure a web crawler data source
• in meeting assistant bot, as a question based on source web pages
• observe missing link - no url

Expected behavior

Answer should include clickable link.

Screenshots

Additional context
Add any other context about the problem here.

Possibly issue with boto3 client version... The URL is not included in the bedrock response.

Add support for es-ES

Transcribe does have support for es-ES but it is not present as a selectable language on the template dropdown when deploying through cloudformation (lma-main.yaml). Modifying the dropdown should suffice to enable this.

The "Stream Audio" tab stops working after a stack update when AssistantWakePhraseRegEx is modified

Describe the bug
The "Stream Audio" tab stops working after a stack update when AssistantWakePhraseRegEx is modified. It does not display disclaimer, and then asserts that the websocket server is not enabled.

To Reproduce
Steps to reproduce the behavior:

1. Update LMA stack, and change the value of the AsistantWakePhrase parameter
2. After stack is updated, try using the LMA Stream Audio tab.

Expected behavior
Stream Audio tab should display disclaimer, and start streaming.

Screenshots

Additional context
Root cause: Updating AsistantWakePhrase parameter causes SSM Param CFN-LMASettingsParameter value to be replaced, overwriting the WSEndpoint and AssistantWakePhraseRegEx setting. The customer resource Lambda UpdateWSEndpointSettingsExec does not get re-run since none of its properties have changed, so the values never get re-added to the replaced SSM param.

"Open in LMA" button does not open the meeting details page in LMA

Describe the bug
Clicking on meeting link fails to open the meeting details page.

To Reproduce
Steps to reproduce the behavior:
Login to the Amazon Live Meeting Browser extension
Update "Meeting Topic" to include '&' anywhere in the text
Click on "Start Listening" button
Click on "Open in LMA" button
The meeting details page should not open (shows a blank page)

Expected behavior
The meeting details page should open in the LMA application.

Additional context
Root cause:
The meeting topic field is used in the url path. Presence of & in the topic string breaks the url.

Longer CloudFormation stack names cause errors in length of Lambda function names

Important: Bug reports submitted here should relate specifically to the open source sample amazon-transcribe-live-meeting-assistant project. Do not submit bugs related directly to any AWS service.
If your issue relates to Amazon Transcribe, Amazon Bedrock or Amazon Bedrock Knowledge Base - setup, user provisioning, document ingestion, accuracy, or other aspects of the service, then first check the service documentation, and then reproduce the problem using the relevant AWS service console before engaging AWS support directly. Thank you.

Describe the bug
When you use a longer, more descriptive CloudFormation stack name like "LiveMeetingAssistant" instead of "LMA", the stacks fail to deploy because the name of the stack is used to generate the names of the Lambda functions. The names of these Lambda functions quickly exceed the 64 character limit. See screenshot at the end.

To Reproduce
Steps to reproduce the behavior:

1. Go to the README and Launch Stack button to launch the CloudFormation stack
2. Enter LiveMeetingAssistant as the stack name instead of the default LMA
3. Leave all other values default and create the stack
4. Within 5-10 minutes, the stack will fail with an error message similar to the screenshot above

Expected behavior
It would be nice if the CloudFormation stack names can be a little longer and more descriptive, especially when multiple stacks are deployed in a single AWS account.

Screenshots

Additional context
N/A

Chime active speaker name not detected correctly sometimes

Describe the bug
Chime active speaker name not detected and recored correctly sometimes

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
Chime active speaker name should be detected correctly and recored

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

User Input Values for 'Your Name' and 'Meeting Topic' Overwritten in Meeting Details from the browser extension

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
The expected behavior is to initially auto-populate fields with information from the Chime meeting. However, if a user manually overwrites these fields, they should not be auto-populated again from the Chime meeting.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

'Export' button for exporting call summary and call transcript

Is your feature request related to a problem? Please describe.

Yes, currently there is no 'Export' button to share call summary and call transcript with attendees. It needs to be done by either Print (as a PDF) or Copy/Paste.

Describe the solution you'd like

An 'Export' button for exporting the call summary and call transcript as an output to share with attendees.

Describe alternatives you've considered

Currently using copy/paste

Additional context
Add any other context or screenshots about the feature request here.

Active speaker on microphone is wrongly assigned to incoming audio from meeting when speaking concurrently

Describe the bug

Active speaker on microphone is wrongly assigned to incoming audio from meeting when speaking concurrently

To Reproduce

Join meeting twice from browser (using incognito tab) as two different users
Use LMA to stream one of the sessions.
Mute mic on one meeting and talk.. Then reverse the mute, and talk. Each time speaker is correctly attributed.
Now unmute mic on both meetings and talk. Now you see the same transcription twice - expected as from the perspective of LMA, it is received once in the mic, and also as the second participant on the incoming meeting audio in the meeting app.. BUT, the speaker name for the incoming meeting audio is not correctly assigned to the other user. Both audio is assigned to same speaker.

Expected behavior

Since microphone audio and incoming meeting audio are on different audio channels, LMA should be able to differentiate them for speaker assignment, and never assign incoming audio to the active speaker on the mic.

Screenshots

The segments below should be assigned to different speakers. (Inaccurate transcription due to audio feedback.. ignore that)

Additional context
Add any other context about the problem here.

Assistant fails when Bedrock KB article is sourced from new KB web crawler data source connector

Describe the bug

Assistant queries fail with error message:
Default message - you'll see this only if the lambda hook failed

CW logs for the QNABedrockKnowledgeBaseFunction show exception

[ERROR] TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/var/task/qna_bedrockkb_lambdahook_function.py", line 279, in handler
    event = format_response(event, kb_response, query)
  File "/var/task/qna_bedrockkb_lambdahook_function.py", line 198, in format_response
    title = os.path.basename(url)
  File "<frozen posixpath>", line 142, in basename

The code is expecting only s3Location.. needs to be more dynamic.