clean99 / talkguidancegpt

TalkGuidanceGPT is a Chrome extension designed to make web browsing more accessible for visually impaired users. By utilizing GPT technology, this extension intelligently analyzes DOM elements and provides detailed information about them when users navigate using the Tab key.

Hi there 👋

I'm KohHom(许峰)⛰️,

• 🔭 I’m currently working at TikTok
• 🌱 I’m currently learning sicpjs
• 💬 Ask me about anything
• 📫 How to reach me: [email protected](Open for any remote/overseas opportunities(especially open source))
• ⚡ Fun fact: I eat OMAD(One Meal a Day) as a VEGAN🥬 No more because I want to gain weight lol
• 

talkguidancegpt's Issues

Add voice instruction in page event

When users go to the new page, they don't know if the page is ready or not, we need to have voice instructions for it. Events that can add voice instruction are:

1. Jump to a new page.
2. Page loading success.
3. Page loading failed.
4. Notifications and alerts, when website show notifications and alerts, report to user.
5. Browser tab switching, read the title of the tab

Optimise Network request delay

current delay:
baidu search bar:

google search bar:

For now, the response speed is around 3~4s, which is not sufficient. And most of the time spent is from awaiting openai response. This means that we need to reduce the text length from both our side and GPT side.

add collect important information to make the gpt introduce more percisely

Now when we are ask GPT to introduce an element, we only give gpt the information of this element, we can give GPT more information to make the introduction has more context:

1. title of the website.
2. cloest sematic parent tag of this element, like:
   , , ...

Add fall back text

Now when the prompt is failed, or internet failed, it won't feedback anything. To improve this, add fallback mechanism to our system:

1. error handling: add error handling in prompt(other place that need network), when error happen, try to get as much information as possible, and report it.
2. fallback translator: now we are using gpt to translate, if network failed, we can have a fallback translator(try to find the library) or just hard code some failback text in multi language, to let user know that what is wrong now.

Abort network request when user tab to a new element

Now when user tab to new element, it will keep loading the previous element information, which could cause bad experience and misunderstand.