GitHubHelp

ichtrojan/deblock

★ 0Forks 0GoGitHub ↗Compare

README

Ethereum Transaction Monitor

Demo

High-performance Ethereum blockchain monitoring service that tracks transactions for 500,000+ addresses using native JSON-RPC methods.

Architecture

graph TB
    subgraph "Entry Point"
        Main[main.go]
    end

    subgraph "Core Components"
        Monitor[Monitor]
        EthClient[Ethereum Client]
        Storage[In-Memory Storage]
        Publisher[Output Publisher]
    end

    subgraph "Ethereum Network"
        RPC[JSON-RPC Provider]
    end

    subgraph "Output Destinations"
        Console[Console Output]
        Kafka[Kafka Topic]
    end

    subgraph "Persistence"
        StateFile[last_block.txt]
        AddressFile[addresses.csv]
    end

    Main --> Monitor
    Monitor --> EthClient
    Monitor --> Storage
    Monitor --> Publisher

    EthClient -->|eth_blockNumber| RPC
    EthClient -->|eth_getBlockByNumber| RPC

    Storage -->|saves/loads| StateFile
    Storage -->|loads| AddressFile

    Publisher --> Console
    Publisher --> Kafka

    style Main fill:#e1f5ff
    style Monitor fill:#fff4e1
    style Storage fill:#f0f0f0
    style RPC fill:#ffe1e1
Loading

System Flow

sequenceDiagram
    participant Main
    participant Monitor
    participant EthClient
    participant Storage
    participant Publisher

    Main->>Storage: Load addresses from CSV (500k)
    Main->>Storage: Get last processed block from file
    Main->>Monitor: Start monitoring

    loop Every 12 seconds
        Monitor->>EthClient: Get latest block number

        loop For each unprocessed block
            Monitor->>EthClient: Get block with transactions
            EthClient-->>Monitor: Block + Transactions

            loop For each transaction
                Monitor->>Monitor: Check if from/to in address set (O(1))

                alt Address matches
                    Monitor->>Storage: Check if already processed (in-memory)

                    alt Not processed
                        Monitor->>Storage: Mark as processed
                        Monitor->>Publisher: Publish output
                        Publisher-->>Console: JSON output
                        Publisher-->>Kafka: Event (optional)
                    end
                end
            end

            Monitor->>Storage: Save last processed block to file
        end
    end
Loading

Installation

go mod download

Usage

Environment Variables (Recommended)

Configure via environment variables (prevents exposing API keys in command history):

cp .env.example .env

Available Environment Variables:

  • ETH_RPC_URL - Ethereum RPC endpoint URL
  • STATE_PATH - State file path (default: ./last_block.txt)
  • ADDRESSES_FILE - CSV file with addresses (default: addresses.csv)
  • START_BLOCK - Starting block number (default: -1 for auto)
  • ENABLE_KAFKA - Enable Kafka output (true/false)
  • KAFKA_BROKERS - Kafka broker addresses
  • KAFKA_TOPIC - Kafka topic name

Basic Usage With .env Populated

go run main.go

With Command-Line Flags

go run main.go \
  -rpc "https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY" \
  -state "./last_block.txt" \
  -addresses "./addresses.csv" \
  -start-block 18000000

With Kafka

kafka

go run main.go \
  -rpc "YOUR_RPC_URL" \
  -kafka \
  -kafka-brokers "localhost:9092" \
  -kafka-topic "eth-transactions"

Address File Format

CSV file with userId,address format:

userId address
user1 0xdAC17F958D2ee523a2206206994597C13D831ec7
user2 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48

Output Format

{
  "userId": "user1",
  "from": "0xabc...",
  "to": "0xdef...",
  "amount": "1000000000000000000",
  "hash": "0x123...",
  "blockNumber": 18000000
}

Testing

Run all tests:

go test ./... -v

Run with coverage:

go test -cover ./... -v

Run specific package tests:

go test ./ethereum
go test ./storage
go test ./monitor
go test ./output

Block Reorganization (Reorg)

Current Implementation

This service does not handle block reorganizations. If a reorg occurs, some published transactions may not exist on the final chain, and some transactions may be missed. For a demonstration service monitoring a small set of addresses, this is an acceptable limitation. Reorgs typically only affect the most recent 1-3 blocks and occur infrequently.

Production Approach

In a production system, you would implement reorg detection by storing block hashes alongside block numbers in your state file. On each polling cycle, verify that previously processed blocks still have matching hashes. If a mismatch is detected, a reorg has occurred.

// Detection: Check if a previously seen block has changed
func (m *Monitor) detectReorg(blockNum int64) (bool, int64) {
    storedHash := m.storage.GetBlockHash(blockNum)
    currentHash := m.ethClient.GetBlockHash(blockNum)

    if storedHash != currentHash {
        return true, m.findCommonAncestor(blockNum)
    }
    return false, blockNum
}

// Recovery: Roll back and reprocess from the reorg point
func (m *Monitor) handleReorg(reorgPoint int64) {
    m.storage.ClearTransactionsAfter(reorgPoint)
    m.publisher.PublishReorgAlert(reorgPoint)
    m.currentBlock = reorgPoint
}

Production systems typically wait for 12 block confirmations before considering transactions finalized, as this represents Ethereum's practical finality threshold. Transactions would include a status field indicating whether they are pending, confirmed, or reorged. Reorg events should be published to a separate notification channel so downstream systems can handle invalidated transactions appropriately.

Contributors

ichtrojan

Issues