Comments (10)
Hi @eirikb - the last update was in September last year. How can we help to keep this up and regularly updated?
from nipster.
Hi @eugeneware.
Sorry but this is simply neglect on my part. Mostly laziness, and partly because of time.
I honestly don't know why my worker has stopped, when I look in the management console it is simply gone. I have a feeling it was never set up correctly to begin with.
The best solution for me now is to set up something on Azure like I had before. I get all services there for free.
But I'm in some desperate need of guidance on an architectural setup, e.g., which services I should use - it's been a while since I played around one Azure and a ton of new features have appeared.
What I have in mind now:
- Schedule to update a list of npm packages once a day.
- Schedule to move all packages to a queue for GitHub update once a day.
- Listen to queue and make requests to GitHub to get repo info.
- Schedule to build out the JSON-file once a day.
The services I think to use are: One web role for each step, set up to run on schedule, except for step 3 where it would be set up to listen on queue.
Store npm packages in own storage table. Store GitHub repo info in another storage table, with some identifier between the tables.
This is basically what I had before, except I replace SQL with storage tables (more light weight, simpler) and Web role instead of worker (can trigger on queue I think, and has schedule support).
However this does not use any of the new features. I also don't know if I must have a web site pr. web role to make them work properly.
from nipster.
Hi @eirikb - thanks for the feedback. Sounds like a great plan.
I've been doing quite a bit on docker and docker-compose which makes replicating production deployments a bit easier once you've hiked up the learning curve!
Are you planning to make the scraping stuff open source and available on github?
I don't have much azure experience, but you can run docker on azure I'm pretty sure, and then it may make it easier for people to help out with development and fixes too.
from nipster.
An update:
Made a new service running on Azure.
Seems to work well.
Will start grinding through all packages soon, which will take a while since I want to start from scratch.
from nipster.
Great job @eirikb
Fingers crossed!
from nipster.
Had to restart the GitHub grinding.
But made a couple of changes, first of the code is now available.
Secondly I've made a simple status-file. One of the Web Jobs will update this file every 15 minutes.
The file shows info about how many packages I have in each table, and in the GitHub Queue. The Queue is probably most important now, since when it hits zero I can create the job that publishes a new json-file.
from nipster.
Thanks @eirikb for sharing the source code! Let's hope it data crunches through OK!
from nipster.
Built the last "builder" today, here is the output: http://data.nipstr.com/npm-datatables.json .
This will be the new file location. Haven't updated the gh-pages index to use this location yet, but I think that file should work.
from nipster.
Now I have updated the page to use nipster.blob.core.windows.net/npm-datatables.json .
Closing issue for now.
Seems like the Web Job works.
from nipster.
Woo! Awesome work! On behalf of the node.js community - thank you! :-)
from nipster.
Related Issues (20)
- Include keywords in table so they affect the search HOT 3
- Debounce the search HOT 1
- many invalid packages HOT 1
- Keeping up to date? HOT 6
- Update registry more often HOT 16
- Add column tooltips HOT 2
- please put this on a domain HOT 8
- More prominent load indicator HOT 2
- More columns in the npm-datatables.json HOT 7
- Normalize repository URLs HOT 6
- Searching for term "git" HOT 7
- Description field renders HTML HOT 1
- Down HOT 1
- title of the color HOT 1
- nipstr.com is no longer working HOT 2
- backbone-forms HOT 1
- Fake duplicated packages. HOT 2
- please url-encode the #hash%20value ? HOT 7
- additional sorting HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nipster.