audioslides / audioslides.io

Use Amazon Polly, Google Slides and FFMpeg to create videos that can be updated at anytime by anyone. This project is written in Elixir.

AudioSlides.IO

Articles

• Produce Easy-to-Update Video Courses with Speech Synth by Robin Böhm

tl;dr

Generate small videos with spoken text from Google Slides.

Using Amazon Polly, Google Slides and FFMpeg to create videos that can be updated at anytime by anyone. This project is written in Elixir.

The Prototype

For our prototype we decided to give Amazon Polly a try. It has a good and simple HTTP-API that allows you to convert text to speech really easily.

For the visual layer we just used Google Slides because they also provide a really good REST-API that allows you to easily export PNG of a slide. It’s also possible to get the speaker notes via the same API that could be the input for the Amazon Polly transformation.

The last step is to combine the generated voice output with the exported png image and produce a small video sequence. For this we just used a handy command line interface called FFMPEG. So the basic processing would look something like this:

Example Input & Output

As shown before we need a Google Presentation to start from. My input will be a short slide deck about the new release of Angular version 5.

Google Slides as Input

Generated Video as Output

How to start the project

To start your Phoenix server:

• Install dependencies with mix deps.get
• Create and migrate your database with mix ecto.create && mix ecto.migrate
• Install Node.js dependencies with cd assets && npm install
• Start Phoenix endpoint with mix s

Now you can visit localhost:4000 from your browser.

Use with docker

Build the container

docker build -t audioslides .

Run via docker compose

Init the database

docker-compose run web mix ecto.setup

Run database + project

docker compose up

How to test

Run all tests

mix t

Run all test with integration test(ffmpeg, write files)

mix test.integration

audioslides.io's Issues

feat(google_slides): allow to use url as google presentation input

feat(aws/polly): handle amazon errors

(CaseClauseError) no case clause matching: 
{:ok, %HTTPoison.Response{body: "{\"message\":\"The security token included in the request is invalid.\"}", headers: [{"x-amzn-RequestId", "7b046fac-cd2d-11e7-b2af-914568816213"}, {"x-amzn-ErrorType", "UnrecognizedClientException:http://internal.amazon.com/coral/com.amazon.coral.service/"}, ...
...
...
}
    (platform) lib/platform/speech/aws/polly.ex:137: Platform.Speech.AWS.Polly.get_binary_speech/2

Response is :ok but there is an x-amzn-ErrorType in this response.

Maybe we should create two functions?

get_binary_speech/2
get_binary_speech!/2

feat(web/lesson): add a visible boolean datafield for lessons

If someone create an audioslide and it's still in progress, it shouldn't visible to the public.

feat(course/comments): Allow comments for lessons

We should use https://disqus.com/

fix(speech/polly): Handle SSML Error on speaker notes with editor information

Valid example:

<speak>Some Text to speak</speak>

Invalid example:

<speak>Some Text to speak</speak>
Extra text for editor

expected
Use the string inside of the speak tags and ignore all other.

feat(slide-editor): add a speak-preview widget with shortcuts

Context: Create or edit text for a slide. Regenerate whole video takes too long.
As content-creator we want to:

• Play the whole text of the current slide as preview
• Play the selected sentence of the current slide as preview
• Use keyboard shortcuts to trigger these functions

feat(web): create a default image for non-synced slides

ATM there is a random slides of our angular course

bug(build): coveralls updates disabled

I destroyed it but don't know how! 🤔

Maybe @sbrink could take a look at it?

feat(endpoint): redirect to https

open website with http -> redirect to https

style(course): create a basic design for courses

Adapt design from: https://angularjs.de/tutorials/

feat(slide-editor): create a basic slide editor

Add a component that enables to edit speaker notes inside AudioSlides that will be synced to the google API.

• Bidirectional sync with google API
• Text-Input to edit speaker-notes
• Direct link to Google Slide (URL + #slide_id=%%)
• Button-Group with shortcuts for common tags (pause, prosody, .. )
• Add syntax highlighting

feat(video-generation): update status in UI via websocket

Track current state of the image-, audio- and video-generation via websocket in the web-ui.

let exampleTypes = `
     NEEDS_UPDATE -> UPDATING -> UP_TO_DATE
  `;

let exampleState = {
  lesson_id: 123,
  video_state: "NEEDS_UPDATE",
  slides: [
    {
      slide_id: 123,
      video_state: "UP_TO_DATE",
      audio_state: "UP_TO_DATE",
      image_state: "UP_TO_DATE",
    }
  ]
}

• Video Module should return a stream that updates on every generation
• Connect Stream to lesson_controller/generate_video and push to socket
• Add processing icons to view layer

feat(brunch): enable typescript for the frontend

chore(polly): change region from us-east-1 to eu-central-1

• change url parameter in get_signed_url/1 in polly.ex
• change vcr_cassettes for the http calls (or refactor to Mox)

feat(test): create docker container for integration tests

test --include integration:true should run on a docker container

feat(video): add a state UI for generated content

Following states should be visible for the user:

• Last sync with Google Presentation at %%
• Generated Slide-Audio is up-to-date
• Generated Slide-Image is up-to-date
• Generated Slide-Video is up-to-date
• Generated Lesson-Video is up-to-date
• The duration of a slide-video

feat(course/progress): add progress tracking for a course

Show progress for course and lesson.
First implementation should be only a simple value that is defined via edit.
Next steps: track technical tasks like SEO, title length, ...

Example:

feat(newsletter): add a basic newsletter form

use Mailchimp