hibobmaster / matrix-stt-bot

A simple matrix bot that transcribes your voice to text message

Introduction

This is a simple matrix bot that transcribes your voice to text message using faster-whisper, a reimplementation of OpenAI's Whisper model using CTranslate2.

Feature

1. Liberate your hands: support automatically speech to text transcribtion
2. Support E2EE Room
3. Self host your service without privacy problem

Installation and Setup

1. Edit config.json or .env with proper values
2. Edit compose.yaml
3. Launch the container

Here is a guide to make bot works on E2E encrypted room.

For explainations and complete parameter list see: https://github.com/hibobmaster/matrix-stt-bot/wiki

1. Create config.json

Tips: set a non-exist room_id at the first time to prevent bot handling historical message which may mess your room up.

{
    "homeserver": "https://matrix.org",
    "user_id": "@xxxx:matrix.org",
    "password": "xxxxxxxxxx",
    "device_id": "GMIAZSVFF",
    "room_id": "!xxxxxxxx:xxx.xxx.xxx",
    "model_size": "base",
    "import_keys_path": "element-keys.txt",
    "import_keys_password": "xxxxxxxxxxxx"
}

2. Create compose.yaml

services:
  app:
    image: ghcr.io/hibobmaster/matrix-stt-bot:latest
    container_name: matrix-stt-bot
    restart: always
    # build:
    #   context: .
    #   dockerfile: ./Dockerfile
    # env_file:
    #   - .env
    volumes:
      # use env file or config.json
      - ./config.json:/app/config.json
      # use touch to create an empty file stt_db, for persist database only
      - ./stt_db:/app/db
      # import_keys path
      - ./element-keys.txt:/app/element-keys.txt
      # store whisper models that program will download
      - ./models:/app/models
    networks:
      - matrix_network

networks:
  matrix_network:

Get your E2E room keys here: Notice: If you deploy matrix_chatgpt_bot along with this project, remember do not use the same database name.

3. Launch for the first time

docker compose up

You will get notice: INFO - start import_keys process

After INFO - import_keys success, please remove import_keys configuration!!!

Wait a second, to see if stt_db has finished syncing_progress (The space occupied is about 100kb and above?)

Then Ctrl+C stop the container

4. Edit config.json again

Remove import_keys_path and import_keys_password options

Set a correct room_id or remove it if you hope the bot to work in the rooms it is in.

Tips: if bot exits because RuntimeError: Unable to open file 'model.bin' in model, try rm -r models then relaunch the container.

5. Finally

Launch the container

docker compose up -d

Demo

Thanks

1. https://github.com/guillaumekln/faster-whisper
2. https://github.com/poljar/matrix-nio
3. https://github.com/8go/matrix-commander

matrix-stt-bot's Issues

Encrypted messages are in format m4a and are silently not processed

The code checks for audio/ogg before decoding, so audio/m4a is ignored.
I'm going to submit a PR for this.

Error validating event: 'file' is a required property

Hey,

Thanks so much for your work. I really appreciate this!

I've just setup the bot and encryption seems to be working correctly. But there is a problem with the message events. The schema requires the file attribute of the event which is according to the matrix spec required for encrypted messages. But somehow even though the message should be encrypted it is not there. I don't know if it is a problem with the bot or my matrix server instance but it happens with whatever audio message I send to the bot.

This is the output of the bot:

Error validating event: 'file' is a required property

Failed validating 'required' in schema['properties']['content']:
    {'properties': {'body': {'type': 'string'},
                    'file': {'properties': {'hashes': {'properties': {'sha256': {'type': 'string'}},
                                                       'type': 'object'},
                                            'iv': {'type': 'string'},
                                            'key': {'properties': {'alg': {'type': 'string'},
                                                                   'k': {'type': 'string'}},
                                                    'required': ['alg',
                                                                 'k'],
                                                    'type': 'object'},
                                            'url': {'type': 'string'}},
                             'required': ['url', 'hashes', 'iv', 'key'],
                             'type': 'object'},
                    'info': {'properties': {'thumbnail_file': {'properties': {'hashes': {'properties': {'sha256': {'type': 'string'}},
                                                                                         'type': 'object'},
                                                                              'iv': {'type': 'string'},
                                                                              'key': {'properties': {'alg': {'type': 'string'},
                                                                                                     'k': {'type': 'string'}},
                                                                                      'required': ['alg',
                                                                                                   'k'],
                                                                                      'type': 'object'},
                                                                              'url': {'type': 'string'}},
                                                               'required': ['url',
                                                                            'hashes',
                                                                            'iv',
                                                                            'key'],
                                                               'type': 'object'}},
                             'type': 'object'},
                    'msgtype': {'enum': ['m.image',
                                         'm.audio',
                                         'm.video',
                                         'm.file'],
                                'type': 'string'}},
     'required': ['body', 'file', 'msgtype'],
     'type': 'object'}

On instance['content']:
    {'body': 'audio.ogg',
     'info': {'duration': 68000,
              'mimetype': 'audio/ogg; codecs=opus',
              'size': 152320},
     'msgtype': 'm.audio',
     'org.matrix.msc1767.audio': {'duration': 68000,
                                  'waveform': [0,
                                               986,
                                               408,
                                               646,
                                               0,
                                               663,
                                               833,
                                               68,
                                               408,
                                               969,
                                               34,
                                               0,
                                               476,
                                               255,
                                               0,
                                               697,
                                               340,
                                               272,
                                               0,
                                               0,
                                               374,
                                               0,
                                               102,
                                               0,
                                               374,
                                               85,
                                               442,
                                               0,
                                               476,
                                               0,
                                               0,
                                               85,
                                               272,
                                               0,
                                               204,
                                               238,
                                               476,
                                               0,
                                               170,
                                               0,
                                               442,
                                               119,
                                               289,
                                               0,
                                               0,
                                               476,
                                               238,
                                               595,
                                               0,
                                               0,
                                               391,
                                               0,
                                               0,
                                               646,
                                               374,
                                               238,
                                               0,
                                               0,
                                               0,
                                               0,
                                               0,
                                               799,
                                               34,
                                               0]},
     'org.matrix.msc3245.voice': {},
     'url': 'mxc://xxx.xxx/PFsnoEkSeCsjQgngRdprpoeS'}

Any idea on how to fix this?

Thanks in advance!

Getting stuck at "INFO - Login via password"

Hi! Thanks for providing this repo. Very cool idea that I would like to experiment with for deaf people.

I am using the Yunohost package synapse_ynh. I followed the instructions and edited the config.json according to my server dates. Unfortunately when starting the the docker image with sudo docker compose up this is the output (won't change in 1 hour):

[+] Running 1/0
 ✔ Container matrix-stt-bot  Created                                                                                                                                0.0s 
Attaching to matrix-stt-bot
matrix-stt-bot  | 2023-07-19 18:15:05,645 - INFO - Bot started!
matrix-stt-bot  | 2023-07-19 18:15:11,946 - INFO - Login via password

I restarted the system and double checked the login credentials. Using Cinny I can login just fine.

This is my current config.json

{
    "homeserver": "https://matrix.MYDOMAIN.de",
    "user_id": "@botwhisper:MYDOMAIN.de",
    "password": "************",
    "device_id": "GMIAZSVFF",
    "room_id": "!mDqFzeYMAuCTDOlrBh:MYDOMAIN.de",
    "model_size": "medium",
    "import_keys_path": "element-keys.txt",
    "import_keys_password": "************"
}

Any idea? How can I get more verbose logging on why the login doesn't work?

Matrix store and olm account is not loaded.

Hello! I have a problem related to e2ee.
I log in using an access token. I followed the instructions and edited the config.json and compose.yaml. I initially ran the Docker image using sudo docker compose without any problems, but after I tried to add e2ee, I found the following error message:

matrix-stt-bot  | 2023-12-04 09:34:41,890 - INFO - Bot started!
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - Login via access_token
matrix-stt-bot  | 2023-12-04 09:34:43,052 - INFO - start import_keys process, this may take a while...
matrix-stt-bot  | Traceback (most recent call last):
matrix-stt-bot  |   File "/app/src/bot.py", line 632, in <module>
matrix-stt-bot  |     asyncio.run(main())
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
matrix-stt-bot  |     return runner.run(main)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
matrix-stt-bot  |     return self._loop.run_until_complete(task)
matrix-stt-bot  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
matrix-stt-bot  |     return future.result()
matrix-stt-bot  |            ^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/app/src/bot.py", line 613, in main
matrix-stt-bot  |     await bot.import_keys()
matrix-stt-bot  |   File "/app/src/bot.py", line 537, in import_keys
matrix-stt-bot  |     resp = await self.client.import_keys(
matrix-stt-bot  |                  ^^^^^^^^^^^^^^^^^^^^^^^^
matrix-stt-bot  |   File "/usr/local/lib/python3.11/site-packages/nio/client/base_client.py", line 115, in inner
matrix-stt-bot  |     raise LocalProtocolError("Matrix store and olm account is not loaded.")
matrix-stt-bot  | nio.exceptions.LocalProtocolError: Matrix store and olm account is not loaded.