TalkGuidanceGPT is a Chrome extension designed to make web browsing more accessible for visually impaired users. By utilizing GPT technology, this extension intelligently analyzes DOM elements and provides detailed information about them when users navigate using the Tab key.
HTML 0.40%CSS 6.50%TypeScript 87.51%JavaScript 5.59%
For now, the response speed is around 3~4s, which is not sufficient. And most of the time spent is from awaiting openai response. This means that we need to reduce the text length from both our side and GPT side.
Now when the prompt is failed, or internet failed, it won't feedback anything. To improve this, add fallback mechanism to our system:
error handling: add error handling in prompt(other place that need network), when error happen, try to get as much information as possible, and report it.
fallback translator: now we are using gpt to translate, if network failed, we can have a fallback translator(try to find the library) or just hard code some failback text in multi language, to let user know that what is wrong now.
Now when we are ask GPT to introduce an element, we only give gpt the information of this element, we can give GPT more information to make the introduction has more context:
title of the website.
cloest sematic parent tag of this element, like: , , ...
When users go to the new page, they don't know if the page is ready or not, we need to have voice instructions for it. Events that can add voice instruction are:
Jump to a new page.
Page loading success.
Page loading failed.
Notifications and alerts, when website show notifications and alerts, report to user.