joshuayoes / concerts-near-me Goto Github PK

After enabling dependabot, those pull requests should not update production facing playlists

Now that there are more playlists, we should be able to pass the --filter option to filter by venue name.

Some websites may be blocking IP traffic from Github Actions. Consider connecting github action to vpn

https://grrr.tech/posts/2021/github-actions-openvpn/

Currently, if any individual concert venue site is unavailable or errors, it throws an error and breaks the whole script. https://github.com/joshuayoes/concerts-near-me/runs/5474459139?check_suite_focus=true#step:7:38

If one site is unavailable, we want to make sure that the other playlists are still updated.

Find a way to automate the authentication workflow of confirming the user token in the browser.

Sometimes Spotify will return an 429 - Too Many Requests error: https://github.com/joshuayoes/concerts-near-me/runs/5632260322?check_suite_focus=true#step:7:516

Need to add an option to retry certain Spotify functions if they fail:

Venue scrapper should always return >1 artist names. If it doesn't something is wrong:
https://github.com/joshuayoes/concerts-near-me/runs/6080877566?check_suite_focus=true#step:7:308

Update command flow so that we don't update the playlist if 0 artist names are returned from the scrapper

The most common error in Github Actions is that the venue website times out when we want to scrape it:

• https://github.com/joshuayoes/concerts-near-me/runs/6074332348?check_suite_focus=true#step:7:82
• https://github.com/joshuayoes/concerts-near-me/runs/6060249665?check_suite_focus=true#step:7:303

There are a few possible reasons:

• The website provider is timing out our connection because it knows that it is not typical user browser traffic
• The GitHub Action runner has poor network connectivity (unlikely after many re-tries)

Currently, if anything at all goes wrong with the yarn start script, we retry the whole script. However, typically it's only one or two venue websites that are the issue.

With retry-step action, we can run a different command on re-runs after failure: https://github.com/marketplace/actions/retry-step#run-different-command-after-first-failure

We could set a list of error-ed venues on retry, and then run a new command with a filter to retry only the failed venues.

Figure out why CI will throw errors with network connectivity, but not locally
https://github.com/joshuayoes/concerts-near-me/runs/5507944276?check_suite_focus=true#step:7:82