Email Processor

A modern async Python application for Gmail email processing with OAuth2 authentication, PostgreSQL storage, and automated rule-based email management.

Features

Gmail OAuth2 Integration: Secure authentication with Gmail API
Async Architecture: Built with asyncio for high performance
PostgreSQL Storage: Reliable email storage with connection pooling
Rule-Based Processing: Automated email management with custom rules
Modern CLI: Rich terminal interface with Typer framework
Type Safety: Full mypy type checking with strict mode
Code Quality: Comprehensive linting with Ruff and pre-commit hooks

Prerequisites

Python 3.11+: Required for modern async features
Poetry: Dependency management and virtual environments
Docker & Docker Compose: For PostgreSQL database
Gmail API Credentials: OAuth2 client ID and secret

Quick Start

1. Clone and Setup

git clone <repository-url>
cd email-processor

2. Install Dependencies

# Install Poetry if you haven't already
curl -sSL https://install.python-poetry.org | python3 -

# Install project dependencies
poetry install

3. Environment Configuration

# Copy environment template
cp .env.example .env

# Generate encryption key
poetry run python -c "from cryptography.fernet import Fernet; print(f'ENCRYPTION_KEY={Fernet.generate_key().decode()}')"

Edit .env with your configuration:

# Application Configuration
ENVIRONMENT=development

# Database Configuration
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/email_processor
DB_POOL_MAX_SIZE=10
DB_POOL_MIN_SIZE=1
DB_ECHO=false

# Gmail OAuth Configuration (see Gmail API Setup section)
GMAIL_CLIENT_ID=your_gmail_client_id_here
GMAIL_CLIENT_SECRET=your_gmail_client_secret_here
OAUTH_REDIRECT_PORT=8080

# Security Configuration (generated above)
ENCRYPTION_KEY=your_base64_encoded_fernet_key_here

# Logging Configuration
LOG_LEVEL=INFO

# Gmail API Concurrency Control
GMAIL_MAX_CONCURRENT_REQUESTS=5

4. Start PostgreSQL Database

# Start PostgreSQL container
docker-compose up -d postgres

# Verify database is running
docker-compose ps

5. Run Database Migrations

# Run Alembic migrations to set up database schema
poetry run alembic upgrade head

6. Gmail API Setup

Create Google Cloud Project:
- Go to Google Cloud Console
- Create a new project or select existing one
Enable Gmail API:
- Navigate to "APIs & Services" > "Library"
- Search for "Gmail API" and enable it
Configure OAuth Consent Screen:
- Go to "APIs & Services" > "OAuth consent screen"
- Choose "External" user type
- Fill in required fields (App name, User support email, Developer contact)
- In "Scopes" section, add the following Gmail scopes:
  - https://www.googleapis.com/auth/gmail.readonly
  - https://www.googleapis.com/auth/gmail.modify
- Save and continue
Add Test Users (Required for unverified apps):
- In "Test users" section, click "Add Users"
- Add your Gmail address that you'll use for authentication
- This is required since the app won't be verified by Google
Create OAuth2 Credentials:
- Go to "APIs & Services" > "Credentials"
- Click "Create Credentials" > "OAuth client ID"
- Choose "Desktop application"
- Download the JSON file
Extract Credentials:
- From the downloaded JSON, copy client_id and client_secret
- Update your .env file with these values

7. Authenticate with Gmail

# Authenticate and fetch emails
poetry run python src/main.py gmail auth --email [email protected] --num-emails 100

This will:

Open your browser for OAuth authentication
Securely store access tokens
Fetch and store 100 emails in the database

Usage

Gmail Authentication

# Basic authentication
poetry run python src/main.py gmail auth

# Pre-select email and specify number of emails
poetry run python src/main.py gmail auth --email [email protected] --num-emails 50

# Short form
poetry run python src/main.py gmail auth -e [email protected] -n 200

Email Rule Processing

Create a rules configuration file (rules.json):

{
  "rules": [
    {
      "name": "Mark newsletters as read",
      "condition": "any",
      "criteria": [
        {
          "field": "from_address",
          "predicate": "contains",
          "value": "newsletter"
        },
        {
          "field": "subject",
          "predicate": "contains",
          "value": "unsubscribe"
        }
      ],
      "actions": [
        {"type": "mark_read"}
      ]
    },
    {
      "name": "Move old promotional emails to trash",
      "condition": "all",
      "criteria": [
        {
          "field": "from_address",
          "predicate": "contains",
          "value": "promotions"
        },
        {
          "field": "date_received",
          "predicate": "older_than_days",
          "value": 7
        }
      ],
      "actions": [
        {
          "type": "move",
          "value": "TRASH"
        },
        {"type": "mark_read"}
      ]
    }
  ]
}

Process emails with rules:

# Apply rules to inbox
poetry run python src/main.py gmail process --rules-file rules.json --email [email protected]

# Dry run to see what would happen
poetry run python src/main.py gmail process --rules-file rules.json --email [email protected] --dry-run

# Short form
poetry run python src/main.py gmail process -r rules.json -e [email protected]

Development

Code Quality Tools

# Format and lint code
poetry run ruff format .
poetry run ruff check . --fix

# Type checking
poetry run mypy src/

# Run pre-commit hooks
poetry run pre-commit run --all-files

Database Operations

# Start database
docker-compose up -d postgres

# Stop database
docker-compose down

# View database logs
docker-compose logs postgres

# Connect to database
docker-compose exec postgres psql -U postgres -d email_processor

# Reset database (WARNING: destroys all data)
docker-compose down -v
docker-compose up -d postgres
poetry run alembic upgrade head

Creating Database Migrations

# Generate new migration
poetry run alembic revision --autogenerate -m "Description of changes"

# Apply migrations
poetry run alembic upgrade head

# View migration history
poetry run alembic history

# Downgrade to previous migration
poetry run alembic downgrade -1

Project Structure

email-processor/
├── src/                          # Main application code
│   ├── config/                   # Configuration management
│   ├── db/                       # Database models and migrations
│   ├── gmail/                    # Gmail API integration
│   └── main.py                   # CLI entry point
├── .env.example                  # Environment template
├── docker-compose.yml            # PostgreSQL setup
├── pyproject.toml               # Project configuration
├── alembic.ini                  # Database migration config
└── README.md                    # This file

Configuration

Environment Variables

Variable	Description	Default
`ENVIRONMENT`	Application environment	`development`
`DATABASE_URL`	PostgreSQL connection string	`postgresql://postgres:postgres@localhost:5432/email_processor`
`GMAIL_CLIENT_ID`	OAuth2 client ID from Google Cloud	Required
`GMAIL_CLIENT_SECRET`	OAuth2 client secret from Google Cloud	Required
`ENCRYPTION_KEY`	Fernet key for token encryption	Required
`LOG_LEVEL`	Logging level	`INFO`
`OAUTH_REDIRECT_PORT`	Local OAuth redirect port	`8080`

Database Configuration

The application uses PostgreSQL 17 with the following default settings:

Host: localhost
Port: 5432
Database: email_processor
User: postgres
Password: postgres

Customize these in your .env file or docker-compose.yml.

SharathHuddar/Email-Processor

README