eonofrey / chess_scrape

Scrape of Chess.com for deeper analysis on games

Summary

This is a small side project in which I scrape all of my games on chess.com to get deeper stats like "What are my average number of moves per game in which I win and which I lose?", "What percent of the time do I win by timeout vs. win by checkmate. Does this percentage change as my ELO does?", "What day of the week do I play the most chess?" etc...

Sidenote: My ELO is a work in progress.

Scrape

First the scraper starts by logging into the site. It enters my username and password and clicks login.

Next, it navigates to the archive of every game I've played stored in a table shown below. I scrape the boxed data on the page, and iterate through every page in my history. In additon to that, it also grabs the link for every individual game, which I will then use to grab even more detials.

At this point I have a very clean dataframe with the date, link, moves, and result of every game I've played (below)

The next step is to iterate through the "Link" column and actually go into every individual chess game to get even more details. The individual games look like the screenshot below. On this page I scrape:

1. The name, elo, and country of the top player (red)
2. How the game concluded (timeout, resignation, checkmate, draw...) and the time of the game (purple)
3. The name, elo, and country of the bottom player (blue)

Originally I ran the scrape assuming I was always the bottom player because that's how it always appears to me on the site's interface. However, on the backend it's totally random who is the top player and who is the bottom player so I had to apply some logic to make sure the data flowed in properly from the scrape.

And the result is a datafrae that looks like the one shown below. I concatenate this to the previous dataframe to get my full dataset on my chess games.

Questions to Answer

"What is my "real" peak elo? (after it normalizes)"

Discarding the first few games where my elo artificially spiked to 1100+, my peak elo is 1074

"What day of the week do I play the most chess?"

Not surprisingly, I play the most chess on the weekends

"What countries have I played against? Which is the strongest, which is the weakest?"

Countries I've played the most

Those with the highest average elo

Those with the lowest average elo