Comments (5)
Counter argument: Let's switch to the slimmer Promise-based got by Sindre:
"Created because request is bloated (several megabytes!)."
from blurts-server.
Updated the issue title, thanks for the suggestion! I haven't checked out got
yet but I trust your judgement.
from blurts-server.
Rough stab at get_hashsets.js:
const AppConstants = require("./app-constants").init();
const fs = require("fs");
const path = require("path");
const got = require("got");
const {name, version} = require("./package.json");
const BREACH_HASHSET_DIR = "breach_hashsets";
const HIBP_AUTH = `Bearer ${AppConstants.HIBP_API_TOKEN}`;
const HIBP_DEFAULT_HEADERS = {"User-Agent": `${name}/${version}`};
async function getBreaches() {
const url = `${AppConstants.HIBP_API_ROOT}/breaches`;
const headers = {...HIBP_DEFAULT_HEADERS};
try {
const {body} = await got.get(url, {json: true, headers});
if (!fs.existsSync(BREACH_HASHSET_DIR)) {
fs.mkdirSync(BREACH_HASHSET_DIR);
}
return body.map(async (breach) => await getBreachHashset(breach));
} catch(err) {
console.error(err);
}
}
async function getBreachHashset({IsActive, IsVerified, DataClasses, Name}) {
if (IsActive && IsVerified && DataClasses.includes("Email addresses")) {
const BREACH_HASHSET_ZIP = path.join(BREACH_HASHSET_DIR, `${Name}.zip`);
const url = `${AppConstants.HIBP_API_ROOT}/enterprisesubscriber/hashset/${Name}`;
const headers = {
...HIBP_DEFAULT_HEADERS,
Authorization: HIBP_AUTH
};
try {
const zipStream = fs.createWriteStream(BREACH_HASHSET_ZIP);
console.log(`Active, verified breach with email addresses: ${Name}`);
console.log(`Fetching ${url}...`);
return await got.stream(url, {headers}).pipe(zipStream);
} catch(err) {
console.log(`${err.message} -- ${url}`);
}
}
}
getBreaches();
... And I think we can maybe just use the following for the routes/oauth.js code:
return got.get(FxAOAuthUtils.profileUri, {json: true, headers: `Bearer ${user.accessToken}`})
.then(({body}) => {
const email = body.email;
gEmails.add(email);
res.send(`Registered ${email} for breach alerts. You may now close this window/tab.`);
})
.catch(err => res.send(err));
(and possibly even destructure body.email
a bit more succinctly to shave an extra line off that snippet)
.then(({body: email}) => {
old:
Lines 48 to 63 in 2900cc2
from blurts-server.
I briefly tried got.stream
for the hashsets, but it seems to fail harder on 404's and kill the process? I didn't dig into debugging it.
from blurts-server.
@groovecoder This may get us closer...
Using got.stream()
here to download the ZIP file of master branch (easier to fiddle w/ 404s by tweaking the URL):
const fs = require("fs");
const path = require("path");
const got = require("got");
const SRC_FILE = "https://github.com/mozilla/blurts-server/archive/master.zip";
const DEST_FILE = path.join(__dirname, "tmp", "master.zip");
async function request(inUrl, outPath) {
return await got.stream.get(inUrl)
.on('error', (error, body, {requestUrl, statusCode}) => {
console.error(`[${statusCode}] ${error.message}: ${requestUrl}`);
})
.on('response', ({requestUrl, statusCode}) => {
console.log(`[${statusCode}] Downloaded: ${requestUrl}`);
})
.pipe(fs.createWriteStream(outPath));
}
request(SRC_FILE, DEST_FILE);
> node get-master
# [200] Downloaded: https://github.com/mozilla/blurts-server/archive/master.zip
> node get-master
# [404] Response code 404 (Not Found): https://github.com/mozilla/blurts-server/archive/masterz.zip
from blurts-server.
Related Issues (20)
- Add lint:ts to lint targets in package.json (in nextjs branch)? HOT 1
- Add eslint-plugin-import plugin? HOT 1
- Update Coverage Badge in README HOT 2
- The title does not fit to the body in the result page HOT 1
- Localization issues in premium.ftl HOT 4
- Caz
- Update channels to match with nimbus ids HOT 1
- Poor JPEG images received at monitor.firefox.com when visitng with WebP accept header stripped HOT 2
- src/db/tables/featureFlags.ts uses `modified_at` whereas other tables use `updated_at`
- Non localizable string and placeholder issues in dashboard HOT 4
- Text overlap in landing page image (long localized category name)
- Style issue with hero image on new Mozilla Monitor home page (with long word; i18n)
- Gd
- Ganz
- Zcsng
- Ok
- Alternative solution for OneRep API key HOT 1
- Hudson Rock Cybercrime Intelligence Free Integration
- Add a CI job for validating Nimbus FML
- Bohari 0pur
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blurts-server.