adrianhajdin / pricewise

Dive into web scraping and build a Next.js 13 eCommerce price tracker within a single video that teaches you data scraping, cron jobs, sending emails, deployment, and more.

A Ecom Price Tracking Application

Build this project step by step with our detailed tutorial on JavaScript Mastery YouTube. Join the JSM family!

📋 Table of Contents

1. 🤖 Introduction
2. ⚙️ Tech Stack
3. 🔋 Features
4. 🤸 Quick Start
5. 🕸️ Snippets
6. 🔗 Links
7. 🚀 More

🚨 Tutorial

This repository contains the code corresponding to an in-depth tutorial available on our YouTube channel, JavaScript Mastery.

If you prefer visual learning, this is the perfect resource for you. Follow our tutorial to learn how to build projects like these step-by-step in a beginner-friendly manner!

🤖 Introduction

Developed using Next.js and Bright Data's webunlocker, this e-commerce product scraping site is designed to assist users in making informed decisions. It notifies users when a product drops in price and helps competitors by alerting them when the product is out of stock, all managed through cron jobs.

If you're getting started and need assistance or face any bugs, join our active Discord community with over 27k+ members. It's a place where people help each other out.

⚙️ Tech Stack

• Next.js
• Bright Data
• Cheerio
• Nodemailer
• MongoDB
• Headless UI
• Tailwind CSS

🔋 Features

👉 Header with Carousel: Visually appealing header with a carousel showcasing key features and benefits

👉 Product Scraping: A search bar allowing users to input Amazon product links for scraping.

👉 Scraped Projects: Displays the details of products scraped so far, offering insights into tracked items.

👉 Scraped Product Details: Showcase the product image, title, pricing, details, and other relevant information scraped from the original website

👉 Track Option: Modal for users to provide email addresses and opt-in for tracking.

👉 Email Notifications: Send emails product alert emails for various scenarios, e.g., back in stock alerts or lowest price notifications.

👉 Automated Cron Jobs: Utilize cron jobs to automate periodic scraping, ensuring data is up-to-date.

and many more, including code architecture and reusability

🤸 Quick Start

Follow these steps to set up the project locally on your machine.

Prerequisites

Make sure you have the following installed on your machine:

• Git
• Node.js
• npm (Node Package Manager)

Cloning the Repository

git clone https://github.com/adrianhajdin/pricewise.git
cd pricewise

Installation

Install the project dependencies using npm:

npm install

Set Up Environment Variables

Create a new file named .env in the root of your project and add the following content:

#SCRAPER
BRIGHT_DATA_USERNAME=
BRIGHT_DATA_PASSWORD=

#DB
MONGODB_URI=

#OUTLOOK
EMAIL_USER=
EMAIL_PASS=

Replace the placeholder values with your actual credentials. You can obtain these credentials by signing up on these specific websites from BrightData, MongoDB, and Node Mailer

Running the Project

npm run dev

Open http://localhost:3000 in your browser to view the project.

🕸️ Snippets

cron.route.ts

import { NextResponse } from "next/server";

import { getLowestPrice, getHighestPrice, getAveragePrice, getEmailNotifType } from "@/lib/utils";
import { connectToDB } from "@/lib/mongoose";
import Product from "@/lib/models/product.model";
import { scrapeAmazonProduct } from "@/lib/scraper";
import { generateEmailBody, sendEmail } from "@/lib/nodemailer";

export const maxDuration = 300; // This function can run for a maximum of 300 seconds
export const dynamic = "force-dynamic";
export const revalidate = 0;

export async function GET(request: Request) {
  try {
    connectToDB();

    const products = await Product.find({});

    if (!products) throw new Error("No product fetched");

    // ======================== 1 SCRAPE LATEST PRODUCT DETAILS & UPDATE DB
    const updatedProducts = await Promise.all(
      products.map(async (currentProduct) => {
        // Scrape product
        const scrapedProduct = await scrapeAmazonProduct(currentProduct.url);

        if (!scrapedProduct) return;

        const updatedPriceHistory = [
          ...currentProduct.priceHistory,
          {
            price: scrapedProduct.currentPrice,
          },
        ];

        const product = {
          ...scrapedProduct,
          priceHistory: updatedPriceHistory,
          lowestPrice: getLowestPrice(updatedPriceHistory),
          highestPrice: getHighestPrice(updatedPriceHistory),
          averagePrice: getAveragePrice(updatedPriceHistory),
        };

        // Update Products in DB
        const updatedProduct = await Product.findOneAndUpdate(
          {
            url: product.url,
          },
          product
        );

        // ======================== 2 CHECK EACH PRODUCT'S STATUS & SEND EMAIL ACCORDINGLY
        const emailNotifType = getEmailNotifType(
          scrapedProduct,
          currentProduct
        );

        if (emailNotifType && updatedProduct.users.length > 0) {
          const productInfo = {
            title: updatedProduct.title,
            url: updatedProduct.url,
          };
          // Construct emailContent
          const emailContent = await generateEmailBody(productInfo, emailNotifType);
          // Get array of user emails
          const userEmails = updatedProduct.users.map((user: any) => user.email);
          // Send email notification
          await sendEmail(emailContent, userEmails);
        }

        return updatedProduct;
      })
    );

    return NextResponse.json({
      message: "Ok",
      data: updatedProducts,
    });
  } catch (error: any) {
    throw new Error(`Failed to get all products: ${error.message}`);
  }
}

generateEmailBody.ts

export async function generateEmailBody(
  product: EmailProductInfo,
  type: NotificationType
  ) {
  const THRESHOLD_PERCENTAGE = 40;
  // Shorten the product title
  const shortenedTitle =
    product.title.length > 20
      ? `${product.title.substring(0, 20)}...`
      : product.title;

  let subject = "";
  let body = "";

  switch (type) {
    case Notification.WELCOME:
      subject = `Welcome to Price Tracking for ${shortenedTitle}`;
      body = `
        <div>
          <h2>Welcome to PriceWise 🚀</h2>
          <p>You are now tracking ${product.title}.</p>
          <p>Here's an example of how you'll receive updates:</p>
          <div style="border: 1px solid #ccc; padding: 10px; background-color: #f8f8f8;">
            <h3>${product.title} is back in stock!</h3>
            <p>We're excited to let you know that ${product.title} is now back in stock.</p>
            <p>Don't miss out - <a href="${product.url}" target="_blank" rel="noopener noreferrer">buy it now</a>!</p>
            <img src="https://i.ibb.co/pwFBRMC/Screenshot-2023-09-26-at-1-47-50-AM.png" alt="Product Image" style="max-width: 100%;" />
          </div>
          <p>Stay tuned for more updates on ${product.title} and other products you're tracking.</p>
        </div>
      `;
      break;

    case Notification.CHANGE_OF_STOCK:
      subject = `${shortenedTitle} is now back in stock!`;
      body = `
        <div>
          <h4>Hey, ${product.title} is now restocked! Grab yours before they run out again!</h4>
          <p>See the product <a href="${product.url}" target="_blank" rel="noopener noreferrer">here</a>.</p>
        </div>
      `;
      break;

    case Notification.LOWEST_PRICE:
      subject = `Lowest Price Alert for ${shortenedTitle}`;
      body = `
        <div>
          <h4>Hey, ${product.title} has reached its lowest price ever!!</h4>
          <p>Grab the product <a href="${product.url}" target="_blank" rel="noopener noreferrer">here</a> now.</p>
        </div>
      `;
      break;

    case Notification.THRESHOLD_MET:
      subject = `Discount Alert for ${shortenedTitle}`;
      body = `
        <div>
          <h4>Hey, ${product.title} is now available at a discount more than ${THRESHOLD_PERCENTAGE}%!</h4>
          <p>Grab it right away from <a href="${product.url}" target="_blank" rel="noopener noreferrer">here</a>.</p>
        </div>
      `;
      break;

    default:
      throw new Error("Invalid notification type.");
  }

  return { subject, body };
}

globals.css

@tailwind base;
@tailwind components;
@tailwind utilities;

* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
  scroll-behavior: smooth;
}

@layer base {
  body {
    @apply font-inter;
  }
}

@layer utilities {
  .btn {
    @apply py-4 px-4 bg-secondary hover:bg-opacity-70 rounded-[30px] text-white text-lg font-semibold;
  }

  .head-text {
    @apply mt-4 text-6xl leading-[72px] font-bold tracking-[-1.2px] text-gray-900;
  }

  .section-text {
    @apply text-secondary text-[32px] font-semibold;
  }

  .small-text {
    @apply flex gap-2 text-sm font-medium text-primary;
  }

  .paragraph-text {
    @apply text-xl leading-[30px] text-gray-600;
  }

  .hero-carousel {
    @apply relative sm:px-10 py-5 sm:pt-20 pb-5 max-w-[560px] h-[700px] w-full bg-[#F2F4F7] rounded-[30px] sm:mx-auto;
  }

  .carousel {
    @apply flex flex-col-reverse h-[700px];
  }

  .carousel .control-dots {
    @apply static !important;
  }

  .carousel .control-dots .dot {
    @apply w-[10px] h-[10px] bg-[#D9D9D9] rounded-full bottom-0 !important;
  }

  .carousel .control-dots .dot.selected {
    @apply bg-[#475467] !important;
  }

  .trending-section {
    @apply flex flex-col gap-10 px-6 md:px-20 py-24;
  }

  /* PRODUCT DETAILS PAGE STYLES */
  .product-container {
    @apply flex flex-col gap-16 flex-wrap px-6 md:px-20 py-24;
  }

  .product-image {
    @apply flex-grow xl:max-w-[50%] max-w-full py-16 border border-[#CDDBFF] rounded-[17px];
  }

  .product-info {
    @apply flex items-center flex-wrap gap-10 py-6 border-y border-y-[#E4E4E4];
  }

  .product-hearts {
    @apply flex items-center gap-2 px-3 py-2 bg-[#FFF0F0] rounded-10;
  }

  .product-stars {
    @apply flex items-center gap-2 px-3 py-2 bg-[#FBF3EA] rounded-[27px];
  }

  .product-reviews {
    @apply flex items-center gap-2 px-3 py-2 bg-white-200 rounded-[27px];
  }

  /* MODAL */
  .dialog-container {
    @apply fixed inset-0 z-10 overflow-y-auto bg-black bg-opacity-60;
  }

  .dialog-content {
    @apply p-6  bg-white inline-block w-full max-w-md my-8 overflow-hidden text-left align-middle transition-all transform  shadow-xl rounded-2xl;
  }

  .dialog-head_text {
    @apply text-secondary text-lg leading-[24px] font-semibold mt-4;
  }

  .dialog-input_container {
    @apply px-5 py-3 mt-3 flex items-center gap-2 border border-gray-300 rounded-[27px];
  }

  .dialog-input {
    @apply flex-1 pl-1 border-none text-gray-500 text-base focus:outline-none border border-gray-300 rounded-[27px] shadow-xs;
  }

  .dialog-btn {
    @apply px-5 py-3 text-white text-base font-semibold border border-secondary bg-secondary rounded-lg mt-8;
  }

  /* NAVBAR */
  .nav {
    @apply flex justify-between items-center px-6 md:px-20 py-4;
  }

  .nav-logo {
    @apply font-spaceGrotesk text-[21px] text-secondary font-bold;
  }

  /* PRICE INFO */
  .price-info_card {
    @apply flex-1 min-w-[200px] flex flex-col gap-2 border-l-[3px] rounded-10 bg-white-100 px-5 py-4;
  }

  /* PRODUCT CARD */
  .product-card {
    @apply sm:w-[292px] sm:max-w-[292px] w-full flex-1 flex flex-col gap-4 rounded-md;
  }

  .product-card_img-container {
    @apply flex-1 relative flex flex-col gap-5 p-4 rounded-md;
  }

  .product-card_img {
    @apply max-h-[250px] object-contain w-full h-full bg-transparent;
  }

  .product-title {
    @apply text-secondary text-xl leading-6 font-semibold truncate;
  }

  /* SEARCHBAR INPUT */
  .searchbar-input {
    @apply flex-1 min-w-[200px] w-full p-3 border border-gray-300 rounded-lg shadow-xs text-base text-gray-500 focus:outline-none;
  }

  .searchbar-btn {
    @apply bg-gray-900 border border-gray-900 rounded-lg shadow-xs px-5 py-3 text-white text-base font-semibold hover:opacity-90 disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-40;
  }
}

index.scraper.ts

"use server"

import axios from 'axios';
import * as cheerio from 'cheerio';
import { extractCurrency, extractDescription, extractPrice } from '../utils';

export async function scrapeAmazonProduct(url: string) {
  if(!url) return;

  // BrightData proxy configuration
  const username = String(process.env.BRIGHT_DATA_USERNAME);
  const password = String(process.env.BRIGHT_DATA_PASSWORD);
  const port = 22225;
  const session_id = (1000000 * Math.random()) | 0;

  const options = {
    auth: {
      username: `${username}-session-${session_id}`,
      password,
    },
    host: 'brd.superproxy.io',
    port,
    rejectUnauthorized: false,
  }

  try {
    // Fetch the product page
    const response = await axios.get(url, options);
    const $ = cheerio.load(response.data);

    // Extract the product title
    const title = $('#productTitle').text().trim();
    const currentPrice = extractPrice(
      $('.priceToPay span.a-price-whole'),
      $('.a.size.base.a-color-price'),
      $('.a-button-selected .a-color-base'),
    );

    const originalPrice = extractPrice(
      $('#priceblock_ourprice'),
      $('.a-price.a-text-price span.a-offscreen'),
      $('#listPrice'),
      $('#priceblock_dealprice'),
      $('.a-size-base.a-color-price')
    );

    const outOfStock = $('#availability span').text().trim().toLowerCase() === 'currently unavailable';

    const images = 
      $('#imgBlkFront').attr('data-a-dynamic-image') || 
      $('#landingImage').attr('data-a-dynamic-image') ||
      '{}'

    const imageUrls = Object.keys(JSON.parse(images));

    const currency = extractCurrency($('.a-price-symbol'))
    const discountRate = $('.savingsPercentage').text().replace(/[-%]/g, "");

    const description = extractDescription($)

    // Construct data object with scraped information
    const data = {
      url,
      currency: currency || '$',
      image: imageUrls[0],
      title,
      currentPrice: Number(currentPrice) || Number(originalPrice),
      originalPrice: Number(originalPrice) || Number(currentPrice),
      priceHistory: [],
      discountRate: Number(discountRate),
      category: 'category',
      reviewsCount:100,
      stars: 4.5,
      isOutOfStock: outOfStock,
      description,
      lowestPrice: Number(currentPrice) || Number(originalPrice),
      highestPrice: Number(originalPrice) || Number(currentPrice),
      averagePrice: Number(currentPrice) || Number(originalPrice),
    }

    return data;
  } catch (error: any) {
    console.log(error);
  }
}

next.config.js

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    serverActions: true,
    serverComponentsExternalPackages: ['mongoose']
  },
  images: {
    domains: ['m.media-amazon.com']
  }
}

module.exports = nextConfig

tailwind.config.ts

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
    "./components/**/*.{js,ts,jsx,tsx,mdx}",
    "./app/**/*.{js,ts,jsx,tsx,mdx}",
  ],
  theme: {
    extend: {
      colors: {
        primary: {
          DEFAULT: "#E43030",
          "orange": "#D48D3B",
          "green": "#3E9242"
        },
        secondary: "#282828",
        "gray-200": "#EAECF0",
        "gray-300": "D0D5DD",
        "gray-500": "#667085",
        "gray-600": "#475467",
        "gray-700": "#344054",
        "gray-900": "#101828",
        "white-100": "#F4F4F4",
        "white-200": "#EDF0F8",
        "black-100": "#3D4258",
        "neutral-black": "#23263B",
      },
      boxShadow: {
        xs: "0px 1px 2px 0px rgba(16, 24, 40, 0.05)",
      },
      maxWidth: {
        "10xl": '1440px'
      },
      fontFamily: {
        inter: ['Inter', 'sans-serif'],
        spaceGrotesk: ['Space Grotesk', 'sans-serif'],
      },
      borderRadius: {
        10: "10px"
      }
    },
  },
  plugins: [],
};

types.ts

export type PriceHistoryItem = {
  price: number;
};

export type User = {
  email: string;
};

export type Product = {
  _id?: string;
  url: string;
  currency: string;
  image: string;
  title: string;
  currentPrice: number;
  originalPrice: number;
  priceHistory: PriceHistoryItem[] | [];
  highestPrice: number;
  lowestPrice: number;
  averagePrice: number;
  discountRate: number;
  description: string;
  category: string;
  reviewsCount: number;
  stars: number;
  isOutOfStock: Boolean;
  users?: User[];
};

export type NotificationType =
  | "WELCOME"
  | "CHANGE_OF_STOCK"
  | "LOWEST_PRICE"
  | "THRESHOLD_MET";

export type EmailContent = {
  subject: string;
  body: string;
};

export type EmailProductInfo = {
  title: string;
  url: string;
};

utils.ts

import { PriceHistoryItem, Product } from "@/types";

const Notification = {
  WELCOME: 'WELCOME',
  CHANGE_OF_STOCK: 'CHANGE_OF_STOCK',
  LOWEST_PRICE: 'LOWEST_PRICE',
  THRESHOLD_MET: 'THRESHOLD_MET',
}

const THRESHOLD_PERCENTAGE = 40;

// Extracts and returns the price from a list of possible elements.
export function extractPrice(...elements: any) {
  for (const element of elements) {
    const priceText = element.text().trim();

    if(priceText) {
      const cleanPrice = priceText.replace(/[^\d.]/g, '');

      let firstPrice; 

      if (cleanPrice) {
        firstPrice = cleanPrice.match(/\d+\.\d{2}/)?.[0];
      } 

      return firstPrice || cleanPrice;
    }
  }

  return '';
}

// Extracts and returns the currency symbol from an element.
export function extractCurrency(element: any) {
  const currencyText = element.text().trim().slice(0, 1);
  return currencyText ? currencyText : "";
}

// Extracts description from two possible elements from amazon
export function extractDescription($: any) {
  // these are possible elements holding description of the product
  const selectors = [
    ".a-unordered-list .a-list-item",
    ".a-expander-content p",
    // Add more selectors here if needed
  ];

  for (const selector of selectors) {
    const elements = $(selector);
    if (elements.length > 0) {
      const textContent = elements
        .map((_: any, element: any) => $(element).text().trim())
        .get()
        .join("\n");
      return textContent;
    }
  }

  // If no matching elements were found, return an empty string
  return "";
}

export function getHighestPrice(priceList: PriceHistoryItem[]) {
  let highestPrice = priceList[0];

  for (let i = 0; i < priceList.length; i++) {
    if (priceList[i].price > highestPrice.price) {
      highestPrice = priceList[i];
    }
  }

  return highestPrice.price;
}

export function getLowestPrice(priceList: PriceHistoryItem[]) {
  let lowestPrice = priceList[0];

  for (let i = 0; i < priceList.length; i++) {
    if (priceList[i].price < lowestPrice.price) {
      lowestPrice = priceList[i];
    }
  }

  return lowestPrice.price;
}

export function getAveragePrice(priceList: PriceHistoryItem[]) {
  const sumOfPrices = priceList.reduce((acc, curr) => acc + curr.price, 0);
  const averagePrice = sumOfPrices / priceList.length || 0;

  return averagePrice;
}

export const getEmailNotifType = (
  scrapedProduct: Product,
  currentProduct: Product
) => {
  const lowestPrice = getLowestPrice(currentProduct.priceHistory);

  if (scrapedProduct.currentPrice < lowestPrice) {
    return Notification.LOWEST_PRICE as keyof typeof Notification;
  }
  if (!scrapedProduct.isOutOfStock && currentProduct.isOutOfStock) {
    return Notification.CHANGE_OF_STOCK as keyof typeof Notification;
  }
  if (scrapedProduct.discountRate >= THRESHOLD_PERCENTAGE) {
    return Notification.THRESHOLD_MET as keyof typeof Notification;
  }

  return null;
};

export const formatNumber = (num: number = 0) => {
  return num.toLocaleString(undefined, {
    minimumFractionDigits: 0,
    maximumFractionDigits: 0,
  });
};

🔗 Links

Assets used in the project are here

🚀 More

Advance your skills with Next.js 14 Pro Course

Enjoyed creating this project? Dive deeper into our PRO courses for a richer learning adventure. They're packed with detailed explanations, cool features, and exercises to boost your skills. Give it a go!

Accelerate your professional journey with the Expert Training program

And if you're hungry for more than just a course and want to understand how we learn and tackle tech challenges, hop into our personalized masterclass. We cover best practices, different web skills, and offer mentorship to boost your confidence. Let's learn and grow together!

pricewise's Issues

invalid maxDuration value for Serverless Function "api/cron"

15:38:35.970 | Error: Builder returned invalid maxDuration value for Serverless Function "api/cron". Serverless Functions must have a maxDuration between 1 and 10 for plan hobby. : https://vercel.com/docs/concepts/limits/overview#serverless-function-execution-timeout

the code is: export const maxDuration = 300; // This function can run for a maximum of 300 seconds
but when it sets to 10, and setup cron-job it shows timeout error

Regarding mails of price drop.

It's sending welcome message when added a new product to track but it is not sending any price drop messages when the price goes below the lowest price.
I have even tried by taking lowest price more than the current price. But still it is not sending mails.

Fix cache revalidation after searching a new product for production deployments

Right now when the user searches a new product, the UI updates properly. But after reloading the page, the update no longer shows and instead you were treated with cached data. Checking Networks tab, it keeps saying HIT for Next.js Caching when it should be a MISS for the first page reload after searching.

I managed to fix this by adding "revalidatePath("/", "layout");" in actions > index.ts > scrapeAndStoreProduct() - right under the other revalidatePath(/products/${newProduct._id}); function we already called.

I'm hosting the app on an AWS EC2 instance. Tried on Vercel but since I'm on Starter Plan there I couldn't set maxDuration to 300 which wouldn't let my app function properly.

`outOfStock` variable always returning false even when the product is out of stock

The existing outOfStock variable will always return false even if the product is out of stock. Reason being, when any product page has "Currently unavailable." text, it is within two nested spans within the #availablity div.

To fix the problem, I'm using this code block and changed the named of the variable from outOfStock to isOutOfStock as it returns a Boolean.

const isOutOfStock =
      $(
        ".a-declarative span.a-size-medium.a-color-success" ||
          "#availablity span"
      )
        .text()
        .trim()
        .toLowerCase() === "currently unavailable.";

Mail functionality is not working

HeroCarousel

the Carousel tag is giving error.
the error says:- 'Carousel' cannot be used as a JSX component.
Its type 'typeof Carousel' is not a valid JSX element type.

pls fix the bug or suggest an alt.

Functions not working

Buttons not functioning.

Can't seem to search more than one item. (Shows Searching the returns error)(PERHAPS CODE NEEDS UPDATE)

interestingly enough the link has only purged for one item and saved that item. Any other item i try to search returns in the terminal
has anyone else ran into this error. my files are the same as adrian's however why can i only search 1 product. This one item has been saved to my database in MONGODB but no other item gets saved there.

my thinking based of the error it gives would be that it comes from the searchbar.tsx file as well as the lib>actions>index.tsx file.
I believe the this could be because in the searchbar.tsx file we define the constant "product" but do not use it or return the variable below in that same function.
the error could also be coming from the index .tsx file because of the newProduct function being defined but not used properly as well as the revalidatepath simply not acting as it should. I am not too quite knowledgable in this so this is all i know for know.

If any other person can search up more than 1 product please add your code snippet from your lib>actions>index.tsx file or where the source of the problem could be coming from,.

Emails not sent on deployed website

Hi there,nodemailer function not work on my deployed website. "sendEmail" function has been called and no error happened on log but no mail has been sent.It works perfectly on localhost server

Api CRON route takes multiple tries to succeed

Hi,

I have followed your course and used your gist to create the project. I did the deployment and testing with cron-job.org by setting a max duration of 10 since I am on hobby plan. When i access directly the api route, whether on local or on vercel, at localhost:3000/api/cron, i get a 500 error and only success in getting the data after clicking reload a few times.. cron-job.org isn't able to run it all.

any way to fix this ? I am posting the first result when accessing the link and the following result after accessing the link multiple times, I am not posting my code since it is sensibly a copy paste of yours. there is no difference.

By using console log all over the place, I know for sure it is the following line causing the issue (from the function scrapeAmazonProduct)

Thanks!