acecentre / pasco

Phrase Auditory Scanning COmmunicator - AAC App for iOS and the Web

pasco (Phrase Auditory Scanning COmmunicator)

pasco is a development project by the Ace Centre to create a AAC app to support people who require auditory scanning only. This may be with one or several Text to Speech (TTS) languages or recorded speech. The app supports auditory cues in a headphone and main speech out of the main speaker of a device. Currently only these features are supported in iOS - other platforms are due to be supported in the future.

Getting Started

You can try out a limited version of pasco on the web at app.pasco.chat. Alternatively download the iOS app here.

To learn more about pasco, Read pasco documentation.

Differences between versions

Feature	iOS	Web
Cue / Main voice splitting between headphone and speaker	✅	❎
Import/Export of Vocab tree	✅	✅
Offline support	✅	✅
In-App Editing	✅	✅
In-App Voice Recording	✅	❎
Adjustable font size and Colour Themes	✅	✅
Switch Access (1-4)	✅	✅
On-Screen navigation tool, Resizable and Draggable	✅	✅
Webhook support (e.g. IFTT, Zapier)	✅	✅
(Beta) Scroll-wheel navigation mode	✅	✅
Offline voices	✅	❎

Developing

Prerequisites

• Node

If developing the iOS build you will also need:

• A mac
• XCode

How to install?

npm install

To test the web build:

npm run dev

Start Web Development Server

gulp dev

Web Production Build

gulp build-prod

Cordova Builds

Initially create a cordova build from cordova-template.

## You may also want to clean the previous build folder
# gulp clean-cordova-build
gulp init-cordova-build
cd ./builds/cordova-main-build/
npm install
cordova platform add ios
## For ios9 build
# gulp clean-cordova-build --build-name ios9-build
# gulp init-cordova-build --build-name ios9-build

Build for development with live-reload.

gulp dist-to-cordova-build-dev-watch
## for iOS 9
# gulp dist-to-cordova-build-dev-watch --build-name ios9-build --target es5
## then open another terminal for cordova build
cd ./builds/cordova-<build-name>/
cordova run --live-reload

Build for production

gulp dist-to-cordova-build-prod
cd ./builds/cordova-<build-name>/
cordova build ios --release

Having problems building?

Try this:

npm install --upgrade cordova@latest
npm install --upgrade cordova-ios@latest
npm install --upgrade ios-deploy
npm install cordova-icon

Notes

• configuration file is at html/config.json. modes are auto and switch.
• the app detects a number of keystrokes. To get it to work with 1 or more switches the app needs a switch box that sends these keystrokes:

ArrowUp or W -> Previous # on switch mode
ArrowDown or S -> Next # on switch mode
ArrowRight or D -> Go into or select
ArrowLeft or A -> Go out

Roadmap

Our next milestone is: https://github.com/AceCentre/pasco/milestone/4

Built With

• FontAwesome - For some of the icons
• Responsive Voice - The web framework used for TTS on the web

Contributing

We would 💛 your contributions. Please feel free to fork the project and we would to see your Pull Requests. Take a look at the issue queue if you want something to get started on. If GitHub isn't your thing and you want to give us feedback - drop us a line

Authors

• Will Wade - Initial work, and Project Lead - WillWade - GitHub
• Hossein Amin - Most of the hard programming work - HoseeinAmin - GitHub
• Selma Al Zarrouk - Adding tests + Fixing bugs
• Gavin Henderson - A little bit of coding

Releasing to App Store Gotchas

Signing for "pasco" requires a development team. Select a development team in the Signing & Capabilities editor.

This is pretty straight forward. Go into the signing and capabilities tab and select Ace Centre North as the team. Make sure you do it for Debug and Release.

The app icon is the cordova icon

Run cordova-icon --icon=model/icon.png from the cordova folder

The build and version numbers are weird

You have to manually bump the version numbers

Archive is greyed out under product

You have to select 'Any iOS Device' as your build target

pasco has conflicting provisioning settings. pasco is automatically signed for development, but a conflicting code signing identity iPhone Distribution has been manually specified. Set the code signing identity value to "Apple Development" in the build settings editor, or switch to manual signing in the Signing & Capabilities editor.

Try untick and retick 'automatically manage signing'

License

This project is licensed under the GNU GPL v3 - see the LICENSE.txt file for details

Acknowledgments

• A massive thanks to our Clients who have inspired the need for this
• Paul Pickford who has helped fund the project through his eBay Sales
• Francis, Alli and Students at Lancasterian School who have helped test the project and give really useful feedback

pasco's Issues

Spelling - Basic implementation

(NB: note similarities to discussion from #14 )

When we work with indviduals on paper this is usually done by:

1 . "A to H", "I to M", "N to Z" (or a different arrangement of letters). The user selects the block they want e.g. "A to H?" "Yes".
2. And then the partner says "A, B , C, D, E, F, G, H". The user selects the letter
3. And the partner reads out the letters selected e.g. "H e".

If the user finishes the branch there it will say out "He".

That would be the simplest mode of operation I think we need.

Navigation tool: ability to change size

Should be able to change size of navigation tool. Not sure best way of doing this - maybe a slider in config.

Meta-data for Auditory Cue (and in brackets)

Need a simple - and more complex way of having a different auditory cue..

• So simple method: Text in brackets after each item e.g. "I want to go somewhere (go somewhere)"
• Advanced method: Meta tag e.g. <meta auditory-cue:"some string here">

Config: drop down with default trees (markdown files).

A drop down available in the config before the language section which would allow a user to load up different default files. E.g.

• Adult - care needs
• Adult - general chatting
• Child - Starting off auditory scanning

Selecting would put contents in the markdown text area.

Small visual bug in chrome

Just running it as is in Chrome there is a bug with the appearance not clearing out when you go back...

Otherwise great!

Auto save Config

Instead of save button - would be nice if config auto-saved any changes.

Tidy up config section

Minor tweaks. Put all speech options under heading "Speech". Use radio button to ask "Provide rapid first time run cue? Yes / No" and hitting yes displays the options for the voice etc.

Bug: iOS on-screen keyboard keeps appearing

The iOS keyboard needs to not show at all when in the run mode. In config mode it does need to appear when tapping in areas such as pitch and font size.

See video: https://www.dropbox.com/s/tthe6yv1oeeare9/IMG_0791.MOV?dl=0

Organise voices under languages

When selecting a voice - a small play symbol next to hear the voice - pitch, rate etc. Will have to just have fixed string to play for different languages.

Controls (arrow keys) not working in firefox or Safari (MacOS)

Again - low priority - but strangely these aren't working in Firefox or Safari on MacOS.

Odd

Need credits and help link

Need some way of adding in a credits screen (Notably where we state that we are using responsive voice as part of their EULA).

May also be wise to link off to a help link (just to https://pasco.chat/#help )

Not urgent - but probably a good idea before submitting app.

UI: On-screen navigation size.

Could do with being slightly larger

Need an on-screen way of navigating (non-keyboard/switch)

From @willwade on June 19, 2017 7:44

I'm thinking of a way for a carer to facilitate someone - if they cant press any buttons..

So:

• In config: Under method:

  • On-screen (alongside switch scanning options of auto/manual)

• And if selected the keyboard detection goes away and this kind of element appears on-screen:

https://thenounproject.com/search/?q=arrow%20keys&i=1074190

Which allows the user to navigate the tree with the arrows.. (or similar - arrow key image!)

Copied from original issue: hosseinzoda/audscan#5

Make on-screen navigation slightly lighter

The black surround of the on-screen navigation (arrows) is a little to clear. The current highlighted text should be the clearest thing on-screen.

Propose just making the outline lighter.

Audio recordings: A way of importing audio by file system

As well as recording in the app - we need a way of browsing local file system for audio recordings

iOS: Act as a handler for .md files

not super urgent at all.. Use case:

• Use MindNode to develop a mindmap.
• Export to Markdown
• Send Markdown to relevant app - e.g. pasco

Defaults: Config

From @willwade on June 19, 2017 9:17

Default to Auto, Space to enter.
Delay at first item: 1000

Copied from original issue: hosseinzoda/audscan#7

Format of language file..

I realise this may be dramatic but...

I'm wondering before we go much further the format of the language tree needs to change. Reason being I'm now thinking this: https://gist.github.com/willwade/8a91165148bf537ad34233b82a4f78fb is easier to read than https://raw.githubusercontent.com/AceCentre/pasco/master/html/trees/en-GB-Pragmatic_Phrases.md

@hosseinamin - how hard would it be to cope with this change? Too hard?

(NB: This is a nice bit of ruby that converts the markdown trees we already have to this format: https://gist.github.com/willwade/2c771fdcb4adfc0c1588324bf59c4d11 - from http://brettterpstra.com/2013/08/18/markdown-to-mind-map/)

No speech on web iOS (safari / chrome)

It's not major - the main port we need to focus on is the iOS app - but I'm not getting any speech output on iOS web - either chrome or safari

indicator to show further elements available

For those supporting someone a visual indicator suggesting there is more available would be interested.. Something like this maybe??

Bug: Config: Auditory and cue voice wrong way round

From @willwade on June 19, 2017 9:16

Regular voice IS cue voice
And Cue voice IS regular voice (i.e. just swap headings around!)

Copied from original issue: hosseinzoda/audscan#6

Bug: Flickering on start (iOS)

So we have a wierd bug. Use scanning - and you will get the first item flickering.

change pasco to source sans pro

For the "pasco" in the header it would be nice if either an image or using text using font source sans pro: http://pasco.chat -
https://fonts.google.com/specimen/Source+Sans+Pro

Audio recordings: Have a way of a way of exporting the vocabulary md file with Audio recordings

Not sure best way of doing this - but need a way of exporting the markdown file - with the Audio recordings. A Zip?

[iOS specific] ability to record voice for a node

Bug: Web version - Error: auditory_cue_first_run_voice_options.pitch should be a number

Go to app.pasco.chat.
Go to config. The error "Error: auditory_cue_first_run_voice_options.pitch should be a number" comes up and you cant save.

<meta data-onselect-restart-in-branch-here>

So Put automatic scanning on and try and spell a word using the alphabet

e.g.

## I shall spell it (spell)<meta data-onselect-restart-here>

### a

### b

### c

Its great - after selecting a letter it restarts in that menu. However the user has no way of escaping out and going to the top of the tree. So maybe one or both of these meta functions is required:

• <meta data-onselect-restart-in-branch-here> (so hitting a letter would restart in the level above)
• <meta data-onselect-go-up-level> (so placing this at the end of the alphabet would allow user to go back a level). e.g.

### z

### Go back <meta data-onselect-go-up-level>

If we can have only one I vote for the last one!

Bug: index.html - failure!

Getting this bug on the current build of the iOS version. All works but can't access any of the child nodes. This alert appears when trying to access a child node.

iOS - cancel on edit screen?

Not sure how we should implement this. Any ideas?

(Maybe when edit - remove cog - and replace with play symbol (play))

Ideas for new config

Not enjoying the way the menus are activated. Because we have so few options I'm toying with a UI that looks like this. Thoughts?

Audio: Different channel for cue and main voice

Need ability to have auditory cue coming from one channel (e.g. headphone jack) - and main audio coming from a different channel (e.g. the phone speaker or bluetooth speaker)

This maybe useful.. https://uihacker.blogspot.co.uk/2013/08/ios-force-audio-output-to-speakers.html

Configuration of style of text

From @willwade on May 22, 2017 11:58

This is predominantly for blind / Visually impaired users but it may be useful to have some nice on-screen menus.. So some configuration of the view of the text (colour, size, highlight) would be useful. Maybe easiest done just by configuring the css??

(e.g. http://markedstyle.com/styles )

w

Copied from original issue: hosseinzoda/audscan#3

Auditory cue voice.

From @willwade on May 22, 2017 11:45

Usually in these systems there is a need to have a different voice - usually with a lower volume when the user is listening through the options and a different (usually higher volume) when selecting them.

• So a setting to have a auditory_cue voice and auditory_cue volume parameter would be useful

Copied from original issue: hosseinzoda/audscan#2

Idea: meta-data-action-webhook-post (and maybe webhook-get?)

Just an idea..

On selecting an item with a meta-data-action-ifttt:command - selecting the item would trigger a IFTTT request (https://ifttt.com/maker_webhooks)

Default item in the quick-setup.html

Can we define (somewhere in the code) a default item to choose in the tree. Would allow someone to hit the Play item and get going. Lets say "Pragmatic Phrases" for now. Thanks!

Support recorded (mp4 or .wav) files for cue and main audio

This is a bit tricky.
Ideally we want to be able to support recorded speech files for both the main speech and audio.

Suggested ways of working:

• In config pass in a URL of a root path (e.g. dropbox folder that is publicly shared)

• Then app will look for files named the same as the message e.g " - People " (look for dropbox_path+"people.wav|mp4" etc). Or use the metadata function meta-data recorded-audio:file.mp4

• Download these and cache them ALL locally on initial run

• For cue files - use the volume provided by the cue voice.

• For main audio - use the volume provided by the main voice

UI: Config

From @willwade on June 19, 2017 9:19

Under Mode:
2 Options currently (3 if we ever do #5)

• "Auto-Scanning"
• "Manual-Scanning"

(a Question. If "Manual scanning" (currently "Switch") - are there keys also for navigating back stepping forward and selecting? Should we not be showing those too?)

Copied from original issue: hosseinzoda/audscan#8

Meta data for different actions on items

From @willwade on May 22, 2017 11:30

When a user selects the end of the branch it should speak it.

• It should return to the top of the tree by default
• IF However the item has a star in brackets after it then stay on that level.. e.g.

## I shall spell it

### a (*)

### b (*)

### c (*)

(so spelling it would be possible)

NOTE: IGNORE READING OUT ITEMS IN BRACKETS ;)

NOTE 2: There could be a better way of formatting this text to denote this. Any other better ideas welcome! (Maybe this is how we link to recorded audio files too. E.g. (## I shall spell it (f:somefile.mp3))

Copied from original issue: hosseinzoda/audscan#1

No add/remove handles in edit mode on iOS build

In the iOS build you don't see the small coloured dots in edit mode e.g. https://i.imgur.com/1qk8e0w.png

Allow user configurable colour schemes

We should allow users to configure colour schemes. This will allow users who are partially sighted to change the colours to whatever colour combination works for them best.

We should provide a set of base themes that are high contrast (for example) but we should also allow users to fully configure the colours.

Dimming previous items

Would be nice if all items previous in the tree would be dimmed - ie greyed out.

Rapid- Pre-Auditory Cue scan

For some people who are blind they can hear a short cue relatively quickly first. So we would like an option available in the configuration for "Initial rapid auditory cue" - so the first run of any tree is fast - and speed should be configurable.

I guess this is a fast auditory cue speed for first run through. Second and onwards of that tree should be normal speed.

Tweaks to "First Time Run (Cue Voice)" in Config

Some small tweaks:

• Only be available to turn on/off when Automatic Scanning is on.
• Turn OFF by default

Editor.

What are the best ways to edit the markdown for the non-technical person?

I have some ideas..

1. Either a cut-down in place editor that only has H1 fields and bullet points (e.g. Like this but with only h1, h2, h3, h4 and # https://github.com/jaredreich/pell/blob/master/README.md)
2. Something like this: http://gojs.net/latest/samples/mindMap.html * (see pic)

What were you thinking @hosseinamin ?

Note: We DO want the markdown format the main format - I'm not suggesting changing that!!

• See also https://github.com/dundalek/markmap

L10n - ability to change default language of app.

• Config menu items, help page will have to have different translations made.
• Change language in Config.
• Will also need an option for R-L or L-R for pasco operation.

First run start

On First run:

• Ask user to set default language
• Ask user if language will be R-L or L-R
• Ask user to set cue, main voice
• Ask user to select default vocabulary file (See #26 )
• Ask user if going to be using a switch and whether Single or two-switch scanning.

iOS - white screen for help screen on iOS

On the current build we are getting a white screen with no content on the help button.

Odd

Navigation tool: Ability to move

If the navigation tool (the 4 way arrow) is on - a user should be able to press and drag it to somewhere on the screen - and remember that location between opening/closing.

UI: Change colours/theme

• We should have an option to turn on a high-contrast colour theme
  (e.g. using a pallete like Yellow text on Black background) - along with the current view.

Either:

• Choose your own colours (not great though - people make it look a mess)
• Have an option for "High contrast theme". (e.g. https://color.adobe.com/High-contrast-color-theme-349965/)
• Or provide a range of pre-selected colour themes with high-contrast being one of them.

UI: A way of text being enlarged

Text should be scalable in size..

• Either it should use iOS in-built dynamic text sizing..

• Or failing that a way in-app to increase font size. Note really the main highlighted item is the key one to increase in size..