nelsonic / arana Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 3.0 283 KB

:honeybee: with great power comes great responsibility

License: GNU General Public License v2.0

JavaScript 100.00%

arana's Introduction

10kh - Ten Thousand Hours

this is where I write about things that interest me.

The 10,000 Hour Rule

If you haven't read Malcolm Gladwell's 2008 book Outliers, I recommend it as a good holiday/weekend read:

The basic premise of the book is:

the key to success in any field is, to a large extent, a matter of practicing a specific task for a total of around 10,000 hours. http://en.wikipedia.org/wiki/Outliers_(book)

Over the last 15 years I've accumulated many (thousands of) hours in several fields of study/skill but not tracked my progress (in a central/systematic way)

I want to remedy that and help others do the same.

If you need a bit of inspiration listen to: https://www.youtube.com/results?search_query=Macklemore+Ten+Thousand+Hours 😉

arana's People

Contributors

Stargazers

Watchers

Forkers

simonlab bridgear cloudxtreme

arana's Issues

Create Docker File to Deploy this Anywhere!

And expand on: https://github.com/dwyl/learn-docker

cannot match when no url is set

Migrate Server to Digital Ocean

Need to add new ssh keys to DO.

How often should each page be scraped?

How long should we leave between scraping a page?

check history > if the page has recently been scraped, we should simply remove the task from the list.
if the task is already in the work-queue don't re-add it because that changes the order and means a popular repo might actually end up being de-prioritised...!

When scraping a personal profile we should automatically add a task to scrape their repositories and activity feed

Unless the profile is a zombie we should add a task to the queue to scrape the repositories and activity feed for the username immediately after scraping the profile page.

scraper switcher wrongly assuming repo page is an issues page

https://github.com/foundersandcoders/resolve-path (fork)

Bruno is doing it the hard/manual way...

https://github.com/brunocasanova

undefined :-(

If we get an undefined username the crawler enters a (death) loop

general purpose elasticsearch upsert all-the-things method

What if there was one method to save any type of record which intelligently knew what type of data it was saving and how to process it ... that's recorder 😉

Should we save all the issues a repo has to the repo record or only the first page and then the rest as links?

At the moment I'm thinking we should save the issue title and link to an issues array (of objects) and then store each issue as its own record.
what do you think @iteles?

recorder should add labels to the repo

need an add_labels method similar to add_people and add_stars etc.

Install Kibana to visualise records?

Detailed instructions:

Latest version of Redis (Node.js Module) does not include zscore!!

See: redis/node-redis#872

Use Sorted Sets instead of Lists for Work Queue and In Progress ?

Sorted Sets to Implement a Queue: http://stackoverflow.com/a/8928885/1148249

save list of repos for an organisation

Save all pages of repos for a given organisation

save issue to es using recorder

save an individual issue as a (its own) record in elastic search

Recorder (save-all-the-things) should save a repo record

Recorder should know what a repo record looks like and save it accordingly.

Use redis-connection instead of redis directly?

Given that we've made a better way of doing Redis, does it make sense to update this?

Scraper Fails when no url is set

Store how long the worker took to process the task?

Should we store the time it took to process the task?

Push a list of tasks to the front (top) of the queue?

We need a way of adding tasks to the front (top) of the work queue.
This will allow Hansel to request Arana to crawl a specific user (and and find all their repos, orgs & issues) which can then be displayed by Tudo post auth - see: dwyl/tudo#51

FLUSHDB (delete keys in redis) in teardown function for clean redis state

Given that we have an expectation of which is the next task in our work_queue test, we need to clear the Redis DB at the end of our test run (on localhost ... Travis handles this due to the ephemeral VM...)

should we have a regular expression to match the /followers or /following url?

at present we have: if(data.url.match(/\\/followers/)) but this matches: https://github.com/followers (a valid username) should we have a more sophisticated regular expression to only match where https://github.com/{username}/followers ?

Download the 36GB MySQL Data Dump on MacMini

http://ghtorrent.org/downloads.html
see: https://changelog.com/144/

work-queue empty?

From my understanding it seems that the program will try to get the next task/url from the work-queue even if this sorted set is empty (although the chance to have an empty list is small as the program add constantly new urls). So my question is what happen if the work-queue set is empty?