- Core Technologies
- Various Packages Used
- Custom ScScraper Library
- Mongoose Models
- updateDb Function
- GraphQL Mutations and Queries
- JWT
- Utilizing React's Context API
- Nodemailer Contact Form
- Rate-Limiting the API
- Custom Reusabale Components with Rebass
- Modals with Bootstrap
The official artist page for my best friend and upcoming DJ/producer Kuhlosul. This MERNG PWA includes a standard "About" page, a "Tracks" page, and a "Contact" page. There is also an admin panel which allows admins to complete various tasks such as editing the content on the About page, and updating the database with Kuhlosul's latest tracks (pulled from SoundCloud).
(Not all listed here)
- apollo-server-express
- axios
- bcrypt
- compression
- express-rate-limit
- express-slow-down
- helmet
- mongoose
- nodemailer
The ScScraper library is a small library specifically created for this application which can scrape data about a SoundCloud user, their tracks, and their tracks which have been posted to different accounts and added to their 'Playlists' page (this is the most popular method of making one's tracks posted elsewhere still show up on their main SoundCloud page).
The soundcloud-scraper library is slightly utilized for some of the methods within the ScScraper library; however, the vast majority of the logic is 100% custom.
ScScraper's roots lie in two very specific helper functions:
- loadProfileBody
This function takes in a SoundCloud username and a page on which to run the scrape. It utilized Puppeteer, and returns the HTML of the page based on the parameters specified.
- autoScroll
Used within the 'loadProfileBody' function, this asynchronous call simply scrolls to the bottom of the page before allowing the loadProfileBody function to return its final value of raw HTML.
These two functions are used notoriously within the actual ScScraper class methods. The return value of the loadProfileBody is then parsed and formatted into usable JSON content.
getTracks = async function (username, query = 'tracks', headless) {
try {
const body = await loadProfileBody(username, query, headless);
const $ = cheerio.load(body);
const tracks = [];
$('.sound__body').each((_i, elem) => {
const sound = {
title: $(elem).find('.soundTitle__title > span').text().trim(),
url:
'https://soundcloud.com' +
$(elem).find('.soundTitle__title').attr('href'),
};
tracks.push(sound);
});
return tracks;
} catch (err) {
throw new Error('Failed to pull tracks.');
}
};The small database of this app has three main models:
- Admin
- Track
- AboutPage
They are quite self-explanatory based on name alone. Most of the initial server-side work was done with the 'tracks' MongoDB collection.
The overall goal is to serve the end-user a list of tracks on the webpage. The problem is that scraping SoundCloud every time the user makes a request is extremely slow and inefficient due to the need to autoScroll each page before scraping it. Because of this, I decided to store the scraped tracks in a MongoDB database.
Because the data coming from ScScraper was more than necessary to fulfill Kuhlosul's vision of this app, I created a simplistic model for a Track:
const trackSchema = new Schema({
title: {
type: String,
},
thumbnail: {
type: String,
},
url: {
type: String,
},
genre: {
type: String,
},
publishedAt: {
type: Date,
},
});The updateDb function is a single asynchronous function which clears the tracks collection within the Db, scrapes SoundCloud for the freshest data, then creates a document within the 'tracks' collection for every single track.
With this function, we can utilize all of our app's server-side foundation logic in three lines of code:
const runUpdate = async() => {
const added = await updateDb();
return added; // Returns a Tracks.find({})
};Within the server.js file, this function is set on a setInterval. It automatically runs in the background every 12 hours to ensure the database is always up-to-date.
This application uses GraphQL entirely. I did create a 'routes' folder when initially creating the project, as I was considering having a hybrid of RESTful routes and GraphQL routes; however, I decided that would become messy and difficult to maintain.
There are 3 main queries, and 6 main mutations at the moment. This number will increase as updates roll out:
type Query {
tracks: [Track]
viewdashboard: AdminCheck
getAbout: About
}
type Mutation {
login(email: String!, password: String!): Auth
seed: [Track]
changePassword(password: String!): Admin
createAdmin(email: String!, password: String!): Admin
updateAbout(header: String!, body: String!): About
sendMessage(
email: String!
type: String!
subject: String!
body: String!
): Status
}The 'seed' mutation is the one which is used to manually run the updateDb function from the admin dashboard:
Two methods are used within the auth.js file of the server's utils folder in order to handle server-side authentication:
- signToken
This method takes in an 'Admin' object from our database, creates and signs a new token, and returns an object including the '_id', 'email', and the newly created web token. Because there is no ability for users to create accounts (except for admins creating new admin accounts), this method is only used in one place - the 'login' mutation resolver:
login: async (parent, { email, password }) => {
email = email.toLowerCase()
// grab the admin corresponding to the email sent in the request
const admin = await Admin.findOne({ email });
if (!admin) return new Error('No admin with this email found!');
// verify if the password is correct using custom Mongoose hook utilizing bcrypt
const correctPass = await admin.checkPassword(password);
if (!correctPass) return new AuthenticationError('Incorrect passoword!');
// create the 'Auth' object by passing our admin object into the signToken function
const token = signToken(admin);
// return the token object and the admin object
return { token, admin };
},In order to accommodate for the return of this resolver, it was necessary to create an 'Auth' type within my GraphQL typeDefs:
type Auth {
token: ID!
admin: Admin
}The token is set up to expire after 24 hours.
- authMiddleware
The authMiddleware method allows us to receive our token initially signed with signToken from the client. I decided to make it very flexible when writing it, and set it up to where the token can be placed in the request's body, query, or headers. Eventually I placed it in the headers.
After being pulled from the request, the JWT is then verified, and the decoded data is added to the request object to be used within our GraphQL context.
Adding the authMiddleware function to the ApolloServer:
const server = new ApolloServer({
typeDefs,
resolvers,
context: authMiddleware,
});Using the context within a resolver:
changePassword: async (parent, { password }, context) => {
// if no admin object within context, return custom error
if (!context.admin)
return new AuthenticationError('Failed to authenticate Admin');
// find the admin with the context.admin's _id and change the password
const withNewPassword = await Admin.findOneAndUpdate(
{ _id: context.admin._id },
{ password: password },
{ new: true }
);
return withNewPassword;
},Utilizing JWT decode, I created a custom class containing all of the necessary methods to handle client-side authentication, and exported it to be available to all of the app's components.
There are 6 total methods on this class:
- login
Takes a token as an argument. Adds the token to sessionStorage, then redirects the user to the dashboard.
- logout
Removes the token from sessionStorage and redirects the user to the homepage.
- getToken
Pulls the token from sessionStorage and returns it. This method is more of a helper method, as it is only used within the class itself, and not within any components.
- getProfile
Returns the decoded return value of the getToken method.
- isTokenExpired
Takes a token as an argument, compares its expiration date to the current date, then returns a boolean based on whether or not it's expired.
- loggedIn
Retrieves the existing token from sessionStorage, and returns a boolean based on whether or not the token exists, and whether or not it is expired (utilizing the isTokenExpired method).
Example the login method in use:
import Auth from '../../utils/auth';
const handleSubmit = async (e) => {
e.preventDefault();
try {
// Run our login GraphQL mutation, then run the login method
const { data } = await login({
variables: { ...formValues },
});
Auth.login(data.login.token);
setFormValues({
email: '',
password: '',
});
} catch (err) {
alert('There was an error logging in');
}
};Example of the loggedIn method in use:
// If the user is not logged in based on our checks, this link goes to the /login page
// Otherwise, it goes to the /dashboard page
<Link
to={Auth.loggedIn() ? '/dashboard' : '/login'}
style={{ height: '1.5rem', cursor: 'pointer' }}
>
<p style={{ fontSize: '1.5rem', cursor: 'pointer' }}>
Admin Dashboard
</p>
</Link>For a while, I was debating with myself on whether or not to use Redux, or Context API. Though I have experience with both, I eventually settled on the more modern technology - the useContext hook.
I created separate files for actions, reducers, and the context provider itself. At the moment, there is only one action; however, this will surely change in the future, and I am glad to have set this up early-on in the app's development.
'GlobalProvider.js' exports two main things:
- Custom 'useGlobalContext' hook
This removes the need to import 'useContext' every time the GlobalContext is needed.
- GlobalProvider
A stateful component which uses our reducer function defined in 'reducers.js'. A context provider is returned with the values being the current state of the useReducer, and the useReducer's dispatch function. Like the first export, this eliminates the need to import 'useReducer', as well as our reducer function, every time we want to mutate the global state.
We can see our global context in action within the 'Dropdown.js' component:
const { currentPage, dispatch } = useGlobalContext();
const handleItemClick = ({ target }) => {
if (target.id !== currentPage) window.scrollTo(0, 0);
dispatch({
type: SET_CURRENT_PAGE,
payload: target.id,
});
};One of Kuhlosul's visions for the application was to send the message from the contact form to a different email based on the type of message being sent. He has an email for general inquiries, and a separate one for promo emails. The logic for this functionality was handled entirely on the server-side. The request contains the form's information, which includes the value of the dropdown; therefore, a simple ternary operator was able to handle everything:
sendMessage: async (parent, { email, type, subject, body }) => {
try {
const to =
type === 'Promos' ? process.env.PROMO_EMAIL : process.env.MAIN_EMAIL;
// ...The nodemailer package made it fairly easy to handle the sendMessage request. I created a NoReply Outlook account, and used its credentials within the Nodemailer transporter. The documentation was wildly helpful in finding out exactly which options needed to be used in order to properly connect to and authenticate on various different types of SMTP servers.
After creating the contact form, I immediately realized that some rate limiting was in order to prevent the potential abuse of the site. On the client-side, specifically for the contact form, I created a class with just two main methods.
- saveSentTime
Saves an item in sessionStorage including the current date and time.
- canSend
Compares the current date to the date saved in sessionStorage. If the difference between the two is greater than or equal to a single day, return true.
Though it's nifty, this client-side-only solution is not very secure, and does not prevent the user from spamming the living daylights out of the server.
Using express-rate-limit and express-slow-down made it an extremely process to set up rate-limiting for all requests to the server:
const rateLimiter = new RateLimit({
// Store rateLimit information within the database
store: new MongoStore({
uri: process.env.MONGODB_URI || 'mongodb://localhost/masondb',
// Every 10 minutes, reset back to 0
expireTimeMs: 10 * 60 * 1000,
errorHandler: console.error.bind(null, 'rate-limit-mongo'),
statusCode: 429,
}),
// Maximum requests === 1000
max: 1000,
// Every 10 minutes,
windowMs: 10 * 60 * 1000,
});
const speedLimiter = SpeedLimit({
// Every 10 minutes,
windowMs: 10 * 60 * 1000,
// Begin delaying requests after the 600 cap has been reached within 10 minutes
delayAfter: 600,
// Delay each request thereafter by 300ms
delayMs: 300,
});Though Rebass is a great library of components as is, I realize that best practice is to create custom components which can later be modified to support a different dependency if needed. Because of this, some of this app's most reused components are custom components built (mostly) with Rebass components.
Example of the 'KInput.js' component:
export default function KInput({
id,
label,
labelColor,
type,
placeholder,
onChange,
color = 'white',
...props
}) {
return (
<Box style={{ width: '100%', minWidth: '200px', maxWidth: '1000px' }}>
<Label htmlFor={id} style={{ color: labelColor, fontStyle: 'italic' }}>
{label}
</Label>
<Input
id={id}
name={type}
type={type}
placeholder={placeholder}
style={{ color: color }}
onChange={onChange}
{...props}
/>
</Box>
);
}Bootstrap modal components within React are among some of the easier ones to work with, and streamlined the creation of the admin dashboard. All functions on the dashboard are done through/confirmed through a Bootstrap modal.
Matt Stephens
This project is protected by a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent