cerebriumai / examples Goto Github PK

Send Result: {'run_id': '71d6c236-fe7a-961f-9548-120375ac8377', 'result': {'error': "HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/models (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc7730c5490>: Failed to establish a new connection: [Errno 111] Connection refused'))"}, 'run_time_ms': 7026.292324066162}

18-realtime-voice-agent

Hi everyone!

Can someone explain how GreedyLLMAggregator and VADGate are used in 18-realtime-voice-agent? I noticed there’s no inference happening in main.py

Thanks a lot for your help!

In the tutorial Running LLaMA 3 8B with TensorRT-LLM on Serverless GPUs, you mentioned a GitHub link for the trt-llm source code, but it seems to no longer exist:

https://github.com/CerebriumAI/examples/tree/master/15-tensor-trt

My primary issue is the claim of reaching ~4500 output tokens per second on a single Nvidia A100. I'm only getting around 200 tokens/second. Am I missing something, or did you apply any optimization techniques to achieve the higher rate?

Is there any way to connect this to a phone call (via twilio perhaps) instead of using webRTC? Thanks!

After sucessfully building the project, I'm getting an error when running it:

Live Replica View log:

Jun 29 08:22:29 PM
 2024-06-29 13:22:29.791 | DEBUG | pipecat.services.deepgram:run_tts:65 - Generating TTS: [Hello, I'm FastBot, how can I help you today?]
Jun 29 08:22:29 PM
 2024-06-29 13:22:29.794 | ERROR | pipecat.services.deepgram:run_tts:94 - Voice exception: Cannot connect to host 127.0.0.1:8082 ssl:default [Connection refused]

Run log

Jun 29 08:30:53 PM
WebSocketException in AsyncLiveClient.start: [Errno 111] Connection refused

I believe both errors are connected because on the code it adresses:

main.py

stt = DeepgramSTTService(
            name="STT",
            api_key=None,
            url='ws://127.0.0.1:8082/v1/listen'
)
tts = ClearableDeepgramTTSService(
            name="Voice",
            aiohttp_session=session,
            api_key=None,
            voice=deepgram_voice,
            base_url="http://127.0.0.1:8082/v1/speak"
)

How can I check if the deepgram server is running ok locally, or debugging it?

run cerebrium deploy
then result
TypeError: init() got an unexpected keyword argument 'docker_base_image_url'