A real-time sneeze detection application that listens to your microphone and responds with "bless you" spoken aloud in your chosen language.

Stack: React + TypeScript frontend, Python + FastAPI backend, TensorFlow/Keras CNN for sneeze detection, WebSockets for real-time communication.

1. Browser captures microphone audio via AudioWorklet (dedicated audio thread)
2. Raw PCM audio streams over WebSocket to the FastAPI backend
3. Backend maintains a 1-second sliding window, runs CNN inference every 100ms
4. On sneeze detection, backend sends the "bless you" phrase back to the client
5. Browser speaks the phrase aloud using the Web Speech API

Latency target: < 200ms from sneeze to spoken response.

• Python 3.11+
• Node.js 20+
• Docker (optional)

Download the ESC-50 dataset and organize it:

cd backend
pip install -r requirements.txt
python scripts/download_dataset.py

This downloads ESC-50, extracts sneeze and negative samples into data/sneeze/ and data/not_sneeze/.

Optionally, record your own sneezes for better personalization:

python scripts/download_dataset.py --record --n-recordings 10

cd backend
python -m app.model_trainer

Model saved to app/models/sneeze_model.keras (~1-2MB).

The app also works without a trained model using an energy-based fallback detector (less accurate but functional for demo purposes).

Evaluated on 2,598 test samples (580 sneeze, 2,018 not-sneeze):

Overall accuracy: 78% | Weighted F1: 0.76

Threshold analysis (sneeze class):

Status: The model needs improvement — sneeze recall is low, meaning many sneezes go undetected. The dataset (3,880 sneeze samples) is heavily augmented from a small base. Priority areas: more diverse real sneeze recordings, better negative sample curation, and architecture tuning.

cd backend
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

cd frontend
npm install
npm run dev

Open http://localhost:5173 in your browser.

docker-compose up --build

Then open http://localhost in your browser.

1. Open the app in your browser
2. Click "Start Listening" and grant microphone permission
3. Select your preferred language from the dropdown (18 languages supported)
4. Adjust sensitivity slider if needed (lower = more detections, higher = fewer false positives)
5. Sneeze near your microphone
6. Hear "bless you" spoken in your chosen language

Browser                          Server
┌──────────────────┐            ┌──────────────────┐
│  AudioWorklet    │            │  FastAPI          │
│  (audio thread)  │            │                  │
│       │          │            │  ┌────────────┐  │
│       ▼          │   WebSocket│  │ Ring Buffer │  │
│  PCM Float32  ───┼───────────►│  │ (1s window) │  │
│                  │            │  └──────┬─────┘  │
│  SpeechSynth  ◄──┼────────────│         ▼        │
│  (TTS output)    │   JSON     │  ┌────────────┐  │
│                  │   events   │  │ CNN Model   │  │
│  Canvas          │            │  │ (inference) │  │
│  (visualizer)    │            │  └──────┬─────┘  │
└──────────────────┘            │         ▼        │
                                │  sneeze_detected │
                                └──────────────────┘

cd backend
pytest tests/ -v

cd backend
python scripts/evaluate_model.py

sneeze-bless-you/
├── backend/
│   ├── app/
│   │   ├── main.py                 # FastAPI entry point
│   │   ├── websocket_handler.py    # WebSocket audio processing
│   │   ├── sneeze_detector.py      # CNN model architecture + inference
│   │   ├── model_trainer.py        # Training pipeline
│   │   ├── audio_features.py       # Mel spectrogram + feature extraction
│   │   ├── bless_you.py            # Multi-language phrases
│   │   ├── config.py               # App configuration
│   │   └── models/                 # Saved trained model
│   ├── data/                       # Training data
│   ├── scripts/                    # Dataset download, augmentation, evaluation
│   ├── tests/                      # Backend tests
│   └── requirements.txt
├── frontend/
│   ├── public/
│   │   └── audio-processor.worklet.js  # AudioWorklet (served as static)
│   ├── src/
│   │   ├── components/             # React components
│   │   ├── hooks/                  # Custom hooks (audio, WebSocket, TTS)
│   │   ├── utils/                  # Language config, audio buffer
│   │   ├── styles/                 # CSS
│   │   ├── App.tsx                 # Main application
│   │   └── main.tsx                # Entry point
│   └── package.json
├── docker-compose.yml
└── README.md

• HTTPS requirement: getUserMedia only works on HTTPS or localhost.
• Browser support: AudioWorklet requires Chrome 66+, Firefox 76+, Safari 14.1+, Edge 79+.
• False positives: Default threshold (0.7) prioritizes precision over recall. Adjust via the sensitivity slider.
• Model size: Custom CNN ~1-2MB. Fits easily in a container image.