This project is a web scraper written in Go that fetches JSON data from multiple URLs concurrently and saves the data to Cassandra.

• Fetch JSON data from multiple URLs concurrently.
• Use goroutines and channels for concurrency.
• Use WaitGroup to wait for all goroutines to finish.
• Save the fetched data to Cassandra.
• Retry Mechanism
• Kafka integration for producing and consuming messages.

• Go 1.23.3 or higher
• Docker and Docker Compose
• github.com/joho/godotenv package
• github.com/confluentinc/confluent-kafka-go/v2 package

1. Install dependencies:

   go mod tidy

2. Create a .env file in the root directory with the following content:

   BASE_URL=https://jsonplaceholder.typicode.com/posts
   NUM_PAGES=10
   TIMEOUT=10000
   WORKERS=5
   RETRY_ATTEMPTS=3
   KAFKA_BROKER=kafka:9092
   KAFKA_TOPIC=posts
   MAX_MESSAGES=100
   MAX_DURATION=60000
   CASSANDRA_KEYSPACE=web_scraper

1. Build and run the Docker containers:

   docker-compose up --build

2. The fetched data will be saved to Cassandra.

To view the scraped posts stored in Cassandra, follow these steps:

1. Open a new terminal and connect to the Cassandra container:

   docker exec -it <cassandra_container_name> cqlsh

   Replace <cassandra_container_name> with the actual name of your Cassandra container. The default is cassandra.

2. Use the web_scraper keyspace:

   USE web_scraper;

3. Query the posts table to view the scraped posts:

   SELECT * FROM posts;

This will display the data in the posts table, allowing you to verify that the posts have been inserted correctly.

This project uses Kafka for message brokering. The docker-compose.yml file includes a Kafka service setup. Ensure you have Docker and Docker Compose installed.

The docker-compose.yml file should include the following services:

• zookeeper: Required by Kafka for broker coordination, fault tolerance, configuration management and distributed synchronization.
• kafka: The Kafka broker.
• web-scraper: The Go application.
• cassandra: The NoSQL storage system.

• Load environment variables from the .env file.
• Fetch JSON data from URLs concurrently using goroutines and channels.
• Produce and consume messages to/from Kafka.
• Save the fetched data to Cassandra.

• Fetch JSON data from a given URL.
• Decode the JSON data into a slice of Post structs.
• Send the data to a channel.

• Represents a post with UserID, ID, Title, and Body fields.

Refer to learning-notes.md for a cheatsheet on concurrency in Go, including usage of goroutines, channels, buffered channels, select statements, WaitGroup, and Mutex.

This project is licensed under the MIT License.