balena-io-experimental / meetbot Goto Github PK

Currently once hubot leaves a meet the meeting listing is removed from the UI.
Would be nice to have a toggle or a "past meetings" section where you can go and grab old transcripts for example if you forgot to copy the link.

be able to send a command in chat to play a youtbue video / playlist for music

Meetbot(s) could pass back the information to the collective intelligence (Our server) over which group of people are currently in which meet? Maybe we keep these to scheduled meetings only as I don't like the idea of snooping into ad-hoc meetings.

The information can then be listed in the dashboard and probably can be searched too.

Switch to audio-only to reduce bandwidth and improve performance even further.

Stenographer has broken the meet transcription feature. While debugging, I found that even while having multiple users on the meet then the transcription was being generated using the same name and image of the person. The data received from the problematic caption event is mentioned below.

{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'Hello, checking the transcript,  checking the transcript. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:01:28.044Z',
  id: 'dea879a21a26'
}
{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'He transcript from another user. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:02:28.043Z',
  id: 'dea879a21a26'
}
{
  image: 'https://lh3.googleusercontent.com/a-/AOh14GigH5zYWQA4OaIiQhNQhsqzrrOjBUvvbWDdrTuv=s40-p-k-no-mo',
  person: 'Vipul Gupta',
  text: 'Checking the transcript now.  Why is the photo?  Same text same. ',
  startedAt: '2022-05-10T16:01:15.043Z',
  endedAt: '2022-05-10T16:03:07.043Z',
  id: 'dea879a21a26'
}

Meetbot only emits a left event when the participants reaches 0. There are however other ways meetbot could leave a meet and it should also emit a 'left' event:

• meetbot gets kicked from the meet
• the rest API tells the bot to leave

Meetbot could still list old meets, and provide drive links. The drive can limit access only to people who can see it.

IMHO not that clear what this is, maybe add a hover tip to help understanding what this is?

Meet recordings are only available after the recording has ended. Puppeteer has this deprecated project to record pages https://github.com/puppeteer/recorder. Perhaps we could use something like this so that someone can join the meet and go back and see the meeting record immediately.

Clean up and properly format CLI output messages. Add better logging.

Example: https://github.com/balena-io-playground/meetbot/blob/0a824f03066bb0d34b3d9cbb4501857436038604/src/meetbot/features/record.ts#L15

Would make it easier to maintain since you won't need to go looking for the strings (given that Google Meet UI changes often) and also a better notion of how "hacky" things are.

Since we are storing all bots in the BOTS map object, we reach the max limit of bots pretty fast which leads to meetbot not able to join any other meetings.

14.03.22 21:35:58 (+0530)  main  queuing: starting the recording...
14.03.22 21:35:58 (+0530)  main  Current bot queue size: 5

It would be super useful if we could summarize in a couple of paragraphs/bullet points the topics discussed in the call.

I think that would be better instead of having some trivial configuration.

• Some balena projects have the lock file versioned, others don' t
• How to deal with yarn and npm being used simultaneously by different people?

We talked to Stathis about this, and the quick and easy way to add CI to this repo is to move it to an org that has CI access. Playground doesn't and CI access can't be activated for specific repos, it's an org-level thing.

So maybe we do a product call and this goes into people-os probably?

reping is not clear what is being pinged again. Since you are trying to get links a command titled /links makes the most sense.

Modify the command to also send all links it has created (voice and chat transcripts)

I saw the bot with latest code always said participants were zero. I think the selector is wrong.

@somombo has some guidelines we can use for conversion and use of the code in TS

At times, you can't find the current status of what hubot's next calendar event will be. This can be rectified locally at least by writing the calendarEvents to a file which will make debugging easier.

My PR #37 is using .env for secrets in production which isn't the best way to go about them but it's better that they are all in one place and at least documented.

I don't really like how complex the Google authentication is. This function checks for credentials but the GoogleCalendar implementation also performs some authentication (creating as token). It's become hard to follow how authentication is done so if a token.json file is what all these google APIs use then I think 1 single place should create it and the rest of the code doesn't have to always check if it exists and create if not...not worth doing in this PR but wanted to mention my thoughts about this.

Originally posted by @20k-ultra in #56 (comment)

Starting/Stopping the recording when there are more than 2 people in the call seems like a nice tweak to meetbot recording behavior.
Also prevents the over extending of the calls.

FD: https://www.flowdock.com/app/rulemotion/r-beginners/threads/yUhHN4GmeRDIY9GL9LW2pBXgAin

Google Meet has implemented this settings tab in the closed captions that is breaking the Google Doc feature with the following error. We probably need the stenographer to ignore this settings tab and keep sending out caption events properly.

(node:68795) UnhandledPromiseRejectionWarning: Error: Invalid requests[0].insertInlineImage: The URL should not be empty.
    at Gaxios._request (/home/vipulgupta2048/work/voicebot/node_modules/gaxios/build/src/gaxios.js:129:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async OAuth2Client.requestAsync (/home/vipulgupta2048/work/voicebot/node_modules/google-auth-library/build/src/auth/oauth2client.js:368:18)

When this error occurs, here's what we see in the Google Docs https://docs.google.com/document/d/1AA01PnHlW4QjzwtOClqZSsJdiA0Vo05A9ytOkWIxJVQ/edit?pli=1

Short term fix until we add proper ci. Miguel only you can do it :)

• #17
• #41
• #39
• #40

title

Secure API endpoints using the OAuth2 API so that only members logged in to the Balena organization are allowed.

References:

• https://github.com/jaredhanson/passport-google-oauth2

In the initial iteration, we did write the code for listing all admin events and then scheduling meetbot to join said calendar events when they begin.

Code for fetching events is below: https://github.com/balena-io-playground/meetbot/blob/66c7868b6ff6f0ee7033347299ea0bc865539805/src/google/calendar.js
#38 will be deleting the code above

At the moment, Google Meet doesn't have an option to join meetings again. For recurring meets having the same meet link every week. This will be a problem.

The bot will say it will go to a page even if it's not a meet url. We could make it smarted to know it can't join a meet if it isn't for the meet.google.com domain.

We hit navigation timeout quite a lot, and it is great to wrap these in a retry. I was thinking to use this https://www.npmjs.com/package/async-retry

28.12.21 19:56:37 (+0530)  main  ERROR in meet! Exiting... https://meet.google.com/oxd-hqer-qno TimeoutError: Navigation timeout of 30000 ms exceeded
28.12.21 19:56:37 (+0530)  main      at /usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:106:111
28.12.21 19:56:37 (+0530)  main      at async FrameManager.waitForFrameNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:128:23)
28.12.21 19:56:37 (+0530)  main      at async Frame.waitForNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:441:16)
28.12.21 19:56:37 (+0530)  main      at async Page.waitForNavigation (/usr/src/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1218:16)

so we can start to collaborate better async via PRs

Consider running the bot on a Raspberry Pi 4. Alternatively, ask ops for a spare NUC in one of the offices.

On one meetbot run, the docs transcript was not created and the log is below. Maybe Somo and Andrei can look into handling this for next time.

Running caption feature..
07.12.21 19:49:27 (+0530)  main  Attached Chat Saver
07.12.21 19:49:27 (+0530)  main  Introducing hubot in the meet - feature
07.12.21 19:49:27 (+0530)  main  deactivating Jellyfish integration because of missing JELLYFISH_AUTH_TOKEN env
07.12.21 19:49:27 (+0530)  main  Running raw caption feature..
07.12.21 19:49:27 (+0530)  main  Running record feature..
07.12.21 19:49:27 (+0530)  main  Running storage feature..
07.12.21 19:49:27 (+0530)  main  Attached Transcript Streamer
07.12.21 19:49:27 (+0530)  main  Current bot queue size: 1
07.12.21 19:49:28 (+0530)  main  typing out email
07.12.21 19:49:30 (+0530)  main  typing out password
07.12.21 19:49:31 (+0530)  main  doing 2FA login
07.12.21 19:49:42 (+0530)  main  going to Meet after signing in
07.12.21 19:49:46 (+0530)  main  turn off cam using Ctrl+E
07.12.21 19:49:49 (+0530)  main  turn off mic using Ctrl+D
07.12.21 19:49:52 (+0530)  main  Changing layout to Spotlight mode
07.12.21 19:49:57 (+0530)  main  turn on captions
07.12.21 19:49:57 (+0530)  main  Chatlog joined a meeting https://meet.google.com/ipq-rsdm-vgm
07.12.21 19:49:58 (+0530)  main  Joined the meeting:  https://meet.google.com/ipq-rsdm-vgm
07.12.21 19:49:58 (+0530)  main  queuing: starting the recording...
07.12.21 19:49:58 (+0530)  main  (node:33) UnhandledPromiseRejectionWarning: Error: The service is currently unavailable.
07.12.21 19:49:58 (+0530)  main      at Gaxios._request (/usr/src/node_modules/gaxios/build/src/gaxios.js:129:23)
07.12.21 19:49:58 (+0530)  main      at runMicrotasks (<anonymous>)
07.12.21 19:49:58 (+0530)  main      at processTicksAndRejections (internal/process/task_queues.js:95:5)
07.12.21 19:49:58 (+0530)  main      at async OAuth2Client.requestAsync (/usr/src/node_modules/google-auth-library/build/src/auth/oauth2client.js:368:18)
07.12.21 19:49:58 (+0530)  main  (Use `node --trace-warnings ...` to show where the warning was created)
07.12.21 19:49:58 (+0530)  main  (node:33) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
07.12.21 19:49:58 (+0530)  main  (node:33) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
07.12.21 19:49:59 (+0530)  main  PAGE LOG: Autofocus processing was blocked because a document already has a focused element.
07.12.21 19:49:59 (+0530)  main  Meeting ipq-rsdm-vgm (2021-12-07T14:19:57.962Z) Chat is available at 
07.12.21 19:49:59 (+0530)  main  https://docs.google.com/document/d/1hJSzybzqc7s5VZUZa7EtxkrkVlZls7DG6yJpiHJRjhE
07.12.21 19:49:59 (+0530)  main  setting chat transcript url tohttps://docs.google.com/document/d/1hJSzybzqc7s5VZUZa7EtxkrkVlZls7DG6yJpiHJRjhE
07.12.21 19:50:00 (+0530)  main  open people list to activate feature
07.12.21 19:50:01 (+0530)  main  PAGE LOG: [stenographer] loaded - Debug: false
07.12.21 19:50:01 (+0530)  main  PAGE LOG: [stenographer] init called
07.12.21 19:50:01 (+0530)  main  captions are on
07.12.21 19:50:02 (+0530)  main  streaming captions
07.12.21 19:50:03 (+0530)  main  open chat section and send a message to all
07.12.21 19:50:05 (+0530)  main  close chat section again
07.12.21 19:50:07 (+0530)  main  starting the recording...
07.12.21 19:50:15 (+0530)  main  open chat section and send a message to all
07.12.21 19:50:19 (+0530)  main  close chat section again

There is an event called end which sounds like the left but it is only used when conveying a runtime issue has caused meetbot to stop in an unrecoverable fashion. This is not the same as left because the bot left on purpose. We should consider changing it's name to error.

• Add button to leave the meet
• Add link to chat transcripts (The links would be represented as - Voice // Chat )
• Rename column "Transcripts URL" to "Transcripts"
• Remove button to join the call, since there is already a link to the meet below the ID (Make that link clickable)
• #26
• #25
• Clean up code / refactor
• #24

From #56, we have made meetbot's meet history persistent. Now, https://bit.ly/getmeetbot shows all meetings that happened in order for it to surface transcripts better. This caused a regression where as the meet queue has become persistent. The meetbot logic to track active meetbots wasn't patch to actually track active meetbots. Rather, it's tracking both active and dead meetbots together.

This leads to an error where we hit our set limit of MAX_BOTS (Atm set to 10) and then no meetbots can be deployed. We need to fix how we calculate active number of meetbots in order for the MAX_BOTS limit to function.

Workarounds:

1. To prevent the bot queue from filling up we can restart the meetbot container couple of days to reset it
2. Take MAX_BOTS limit to 100 or something

Triggering the debug mode or a certain log level if decide to work on #13 will click pictures of each step to debug steps easily on meetbot run

Context is lost for folks that join after hubot has sent the transcript links. With the /reping command, people can get access to both the past chat as well as the transcripts. Maybe this makes the chat a little more noisy with all the links but it will kill the dialogue:

"Oh hey Ryan, we sent the links before you joined. Let me send them (all) again."

Meetbot failing to "sign in to Google" and doesn't work due to this precautionary check by Google shown when signing in.

Workarounds,

• We can handle this by clicking the button and signing in when the screen shows "Verify it's you"
• OR, we can terminate the running browser session and let meetbot sign in into a new browser instance preferably without cookies or an authentication notice. That what I will do by restarting the container,

The error:

07.03.22 20:28:37 (+0530)  main  we're already logged in
07.03.22 20:28:37 (+0530)  main  going to Meet after signing in
07.03.22 20:29:15 (+0530)  main  Unrecoverable bot error occured: waiting for XPath `//*[contains(text(),'call_end')]|//*[contains(text(),'Join now')]` failed: timeout 30000ms exceeded
07.03.22 20:29:15 (+0530)  main  Leaving the meeting: https://meet.google.com/yhc-fmfq-hxx
07.03.22 20:29:52 (+0530)  main  typing out email
07.03.22 20:30:22 (+0530)  main  Unrecoverable bot error occured: waiting for selector `input[type="email"]` failed: timeout 30000ms exceeded
07.03.22 20:30:22 (+0530)  main  Leaving the meeting: https://meet.google.com/yhc-fmfq-hxx
07.03.22 20:30:42 (+0530)  main  typing out email
07.03.22 20:31:12 (+0530)  main  Unrecoverable bot error occured: waiting for selector `input[type="email"]` failed: timeout 30000ms exceeded

Impact:
Meetbot doesn't work until this is fixed or workarounds are deployed

The FD message command will prompt hubot to get a meet and also add meetbot to that meet right from FD.
We already got a PR that works in the direction to enable this: https://github.com/people-os/hubot-as-mainbot/pull/712