Updating has been stopped? about nipster

[eugeneware]

Hi @eirikb - the last update was in September last year. How can we help to keep this up and regularly updated?

[eirikb]

Hi @eugeneware.

Sorry but this is simply neglect on my part. Mostly laziness, and partly because of time.

I honestly don't know why my worker has stopped, when I look in the management console it is simply gone. I have a feeling it was never set up correctly to begin with.

The best solution for me now is to set up something on Azure like I had before. I get all services there for free.
But I'm in some desperate need of guidance on an architectural setup, e.g., which services I should use - it's been a while since I played around one Azure and a ton of new features have appeared.

What I have in mind now:

1. Schedule to update a list of npm packages once a day.
2. Schedule to move all packages to a queue for GitHub update once a day.
3. Listen to queue and make requests to GitHub to get repo info.
4. Schedule to build out the JSON-file once a day.

The services I think to use are: One web role for each step, set up to run on schedule, except for step 3 where it would be set up to listen on queue.
Store npm packages in own storage table. Store GitHub repo info in another storage table, with some identifier between the tables.

This is basically what I had before, except I replace SQL with storage tables (more light weight, simpler) and Web role instead of worker (can trigger on queue I think, and has schedule support).
However this does not use any of the new features. I also don't know if I must have a web site pr. web role to make them work properly.

[eugeneware]

Hi @eirikb - thanks for the feedback. Sounds like a great plan.

I've been doing quite a bit on docker and docker-compose which makes replicating production deployments a bit easier once you've hiked up the learning curve!

Are you planning to make the scraping stuff open source and available on github?

I don't have much azure experience, but you can run docker on azure I'm pretty sure, and then it may make it easier for people to help out with development and fixes too.

[eirikb]

An update:
Made a new service running on Azure.
Seems to work well.
Will start grinding through all packages soon, which will take a while since I want to start from scratch.

[eugeneware]

Great job @eirikb

Fingers crossed!

[eirikb]

Had to restart the GitHub grinding.
But made a couple of changes, first of the code is now available.
Secondly I've made a simple status-file. One of the Web Jobs will update this file every 15 minutes.
The file shows info about how many packages I have in each table, and in the GitHub Queue. The Queue is probably most important now, since when it hits zero I can create the job that publishes a new json-file.

[eugeneware]

Thanks @eirikb for sharing the source code! Let's hope it data crunches through OK!

[eirikb]

Built the last "builder" today, here is the output: http://data.nipstr.com/npm-datatables.json .
This will be the new file location. Haven't updated the gh-pages index to use this location yet, but I think that file should work.

[eirikb]

Now I have updated the page to use nipster.blob.core.windows.net/npm-datatables.json .
Closing issue for now.
Seems like the Web Job works.

[eugeneware]

Woo! Awesome work! On behalf of the node.js community - thank you! :-)