mikeal / r2 Goto Github PK
View Code? Open in Web Editor NEWHTTP client. Spiritual successor to request.
License: Apache License 2.0
HTTP client. Spiritual successor to request.
License: Apache License 2.0
Much like #10, consider some ES6ish things:
defineProperty
code is unnecessarily verbose. ConsiderdefineProperty
to getters e.g. get json () { return this.response.then(resp => resp.clone().json()) }
or['json', 'blob', ...].forEach((n) => Object.defineProperty(this, n, {get: () => this.response.then(r => r.clone()[n]()})
ordefineProperties
setHeaders(obj = {})
and remove if (opts.headers)
makeBody
to a named function so that its name shows up when debuggingsetHeader (key, value) { this.setHeaders({ [key]: value }) }
_args
return this
, then change all the http method methods (get
, post
, ...) to return this._args(...args)
Due to transitive dependency usage of whatwg-url
(v5.0.0) via node-fetch
(v2.6.5) I got:
2021-10-11T02:26:37.263Z ERROR (node:8) [DEP0005] DeprecationWarning: Buffer() is deprecated due
to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
at showFlaggedDeprecation (buffer.js:194:11)
at new Buffer (buffer.js:278:3)
at utf8PercentDecode (/var/task/whatwg-url/lib/url-state-machine.js:105:17)
at parseHost (/var/task/whatwg-url/lib/url-state-machine.js:400:18)
at URLStateMachine.parseHostName (/var/task/whatwg-url/lib/url-state-machine.js:851:18)
at new URLStateMachine (/var/task/whatwg-url/lib/url-state-machine.js:562:44)
at Object.module.exports.basicURLParse (/var/task/whatwg-url/lib/url-state-machine.js:1258:15)
at new URLImpl (/var/task/whatwg-url/lib/URL-impl.js:17:27)
at Object.setup (/var/task/whatwg-url/lib/URL.js:187:17)
at new URL (/var/task/whatwg-url/lib/URL.js:25:18)
I haven't tested, but upgrading to node-fetch 3.0.0 might come in handy, I guess.
Hey there!
I belong to an open source security research community, and a member (@ranjit-git) has found an issue, but doesn’t know the best way to disclose it.
If not a hassle, might you kindly add a SECURITY.md
file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.
Thank you for your consideration, and I look forward to hearing from you!
(cc @huntr-helper)
This is most definitely a style preference, but I think many newcomers will be confused by the use of dynamically defined properties that return promises here, and I think they could lead to bugs due to their magicness.
Is there a rationale for preferring:
let html = await r2('https://www.google.com').text
over
let html = await r2('https://www.google.com').text()
Especially considering that the latter aligns with the Fetch API that users might be familiar with?
Since the promises returned here perform I/O, I think it's a stronger signal to the reader to use functions instead of properties.
It's a relatively well accepted maxim in languages that have had property methods for a while (eg, C#) that they should still be inexpensive wrappers to internal state instead of performing any real work – as this goes against users' expectations of what a property is.
Again, totally just a personal preference – but I think it falls well within the principle of least astonishment.
Hey,
I want to introduce you make-fetch-happen.
It's similar to node-fetch but more powerful in terms of retries and additional features in the backend side.
Do you think could be a good dependency to be used for this project?
With the old request, I can use streams to handle the 'response' event, and get connection properties because the HTTP connection is still open.
@mikeal how is this handled in r2, if it all? I know there's some conceptual overlap between promises and streams so I'm not sure if r2 can do this, can't do this,. or there's a better way of accessing connection properties.
req.on('response', function (res) {
var certificate = res.req.connection.getPeerCertificate && res.req.connection.getPeerCertificate(true)
Thanks!
There is room for improvment, in this case body cant be a json obj, and it needs to be manually converted to a string
This is regarding: https://github.com/mikeal/r2/blob/master/index.js#L73
If current code sees that you are passing json it overrides content-type header to be set to application/json
. But application/json is not the only JSON media type. There're plenty of others:
to name just a few
Hi,
When I receive a heavy json response (115Kb in my case)
const json = await r2.post('my_url', { headers: customHeaders }).json
my process is blocked
but if I write this
const res = await r2.post('my_url', { headers: customHeaders }).response
const json = await res.json()
It's works !
I work with node v9.5.0
node-fetch
allows Promise dependency injection (if you like using bluebird for example). It would be great if we could do the same thing with this wrapper.
Title says all. No mention about anything in the README nor in the package.json (except from node-fetch
which is a dependency.
Hi, thanks for the nice library!
In the code you create a resolvable()
only to resolve it with an already existing promise which is not something you typically have to do.
You can use fetch
as this.response
, the setTimeout
also isn't necessary as promises take care of reacting on the next tick anyway.
I'll write a PR to illustrate what I mean
Branch | Build failing 🚨 |
---|---|
Dependency | semantic-release |
Current Version | 15.9.8 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
semantic-release is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
At my wits end trying to get this POST to return properly.
What else does r2 do to the request vs fetch? Something is wrong. I've tried using different initialization methods and alternate response management to try to get a working result.
Universal variables for all four examples:
const url = 'https://example.com/path';
const params = { example: 'request data', other_data: 'test' };
const body = Object.keys(params).map(key => encodeURIComponent(key) + '=' + encodeURIComponent(params[key])).join('&');
const headers = {
'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
};
Broken, returns 400 error html:
const r2_example = await r2.post(url, {
headers,
body
});
console.log('r2', await r2_example.text);
Broken; alternate configuration, alternate response handling, still returning 400 error in html:
const r2_example_2 = await r2({
url,
body,
method: 'POST',
headers,
});
console.log('r2_2', await (await r2_example_2.response).text()); // 400 Bad Request
Working example with raw Header/Body
const fetch_example = await fetch(url, {
method: 'POST',
headers,
body
});
console.log('fetch', await fetch_example.text()); // Proper result
Working example with node-fetch.Headers (globals created by r2) and Buffer.from(string).
const fetch_example_with_headers = await fetch(url, {
method: 'POST',
headers: new global.Headers(headers),
body: Buffer.from(body)
});
console.log('fetch_example_with_headers', await fetch_example_with_headers.text()); // Proper result
I'm surprised it's not covered in docs or tests.
r2("http://localhost:3000/500").json
.catch(e => {
console.log("e:", e) // not getting there even for 500
})
.then(x => {
console.log("x:", x) // x is response.body here, how to get response.status?
})
Hi,
i was losing a lot of time in order to use r2 because i was getting a lot of problems. as @thechriswalker has noted, the r2
package on NPM is another one and so it creates confusion.
imho it's better if we find another name for this package that is free on NPM.
I find the default dependence of node-fetch is 2.x ,it's a alpha version , and i find in crawling some website (eg. https://www.baidu.com) ,the node-fetch 2.x inner function clone will in error that cannot output any result and not throw any error.
I test in the node-fetch 1.x ,it will ok .
3ks
Currently, failSet does not expose what property you tried to set.
My suggestion:
const failSet = (prop) => () => { throw new Error(`Cannot set read-only property R2.${prop}`) }
...
Object.defineProperty(this, 'json', {
get: () => this.response.then(resp => resp.clone().json()),
set: failSet('json')
})
...
this way, the error that will be thrown will allow for quicker and easier debugging, since you'll more easily know what to search for in your code.
let data = await r2('https://bibliotheques-specialisees.paris.fr/ark:/73873/pf0000855431/0001/v0001.simple.selectedTab=otherdocs').text;
console.log(data);
outputs nothing because the :
is misinterpreted as a port marker, when it's not a port, but
let data = await r2('https://bibliotheques-specialisees.paris.fr/ark\\:/73873/pf0000855431/0001/v0001.simple.selectedTab=otherdocs').text;
console.log(data);
works.
Furthermore the Promise
resolves correctly but with an empty .text
, thus leaving us without a way to handle it.
Hello,
I installed r2 on my computer with nodejs 8.4.0 but when I do a npm start I have this error.
/home/ngoura/projet/testnacim/node_modules/require-dir/index.js:93
if (!require.extensions.hasOwnProperty(ext)) {
^
TypeError: require.extensions.hasOwnProperty is not a function
at requireDir (/home/ngoura/projet/testnacim/node_modules/require-dir/index.js:93:37)
at Object.<anonymous> (/home/ngoura/projet/testnacim/node_modules/r2/index.js:19:1)
at Module._compile (module.js:573:30)
at Object.Module._extensions..js (module.js:584:10)
at Module.load (module.js:507:32)
at tryModuleLoad (module.js:470:12)
at Function.Module._load (module.js:462:3)
at Module.require (module.js:517:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/home/ngoura/projet/testnacim/app/controllers/proxy.controller.js:11:7)
So I can't use this package 😞
Basically, the title says it all: Which browsers does r2 support, i.e. on which browser are you able to use it?
Having to iterate over object to create Headers of them is obsolete.
Headers
constructor accepts a plain object and constructs Headers class instance of it.
const makeHeaders = obj => { let h = new Headers() for (let key in obj) { h.append(key, obj[key]) } return h }
becomes
const makeHeaders = obj => new Headers(obj)
after failing trying to run r2 on runkit, i fallback on working on my local machine but i still got an error:
╭─phra at kali in /tmp
╰─λ node -v 0 < 11:54:08
v8.4.0
╭─phra at kali in /tmp
╰─λ cat index.js 0 < 11:54:02
const r2 = require('r2')
const to = require('await-to-ts')
const assert = require('assert')
const URLS = [1,2,3,4,5,6,7,8,9,10].map(i => `https://jsonplaceholder.typicode.com/posts/${i}`)
(async () => {
const promises = URLS.map(url => r2(url))
const [err, res] = await to(promises)
assert.ifError(err)
res.forEach(console.log.bind(console))
})()⏎
╭─phra at kali in /tmp
╰─λ node index.js 0 < 11:54:05
/tmp/node_modules/r2/clayconfig.js:5
if(!config.raw.name) error( "package.json should have a name value")
^
ReferenceError: error is not defined
at Object.<anonymous> (/tmp/node_modules/r2/clayconfig.js:5:34)
at Module._compile (module.js:573:30)
at Object.Module._extensions..js (module.js:584:10)
at Module.load (module.js:507:32)
at tryModuleLoad (module.js:470:12)
at Function.Module._load (module.js:462:3)
at Module.require (module.js:517:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/tmp/node_modules/r2/index.js:15:14)
at Module._compile (module.js:573:30)
🚨 You need to enable Continuous Integration on all branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.
Since we did not receive a CI status on the greenkeeper/initial
branch, we assume that you still need to configure it.
If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/
.
We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial
branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.
const comms = {
requestData: async function (method, url, requestBody) {
return await r2[method](url, requestBody)
}
}
RequestData results in
Welcome to 3VOT
module.js:457
throw err;
^
Error: Cannot find module 'vinyl-source-stream'
at Function.Module._resolveFilename (module.js:455:15)
at Function.Module._load (module.js:403:25)
at Module.require (module.js:483:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (***/node_modules/r2/tasks/dist.js:10:20)
at Module._compile (module.js:556:32)
at Module._extensions..js (module.js:565:10)
at Object.require.extensions.(anonymous function) [as .js] (***/node_modules/babel-register/lib/node.js:152:7)
at Module.load (module.js:473:32)
at tryModuleLoad (module.js:432:12)
hi,
i'm trying to use r2 on runkit but i got this error: Error: Cannot find module 'chalk'
you can reproduce the issue here: https://runkit.com/phra/r2-chalk-missing
Hi @mikeal
Great job done, it is generally a great idea to keep your techs up-to-date. The only thing is that I couldn't find any documentations and guid towards using r2 or maybe migrating from request to r2. Are you planning on creating a documentation in the near future? And also, is the version 1.0.0 released or gonna be released soon? I plan on using it in a big project, I think it'd be a good test case with lots of feedback/issues.
Please document the support browser and Node versions.
At least for the browsers, I believe this to be very limited given there isn't any employment of a browser-side polyfill here if the Fetch API is not available. I think many people would be surprised by that when coming to the "spiritual successor of request
". Granted, that issue could be solved by incorporating something like #27 (using isomorphic-fetch
rather than just node-fetch
) if desired.
Hi,
I really wanted to use r2
as a modern alternative to request
, but I ran into issues with binary files. When I try to download for example our company logo, it all works fine:
> (async function () { console.log(await require('r2').get('https://wizcorp.jp/images/logo.png').arrayBuffer); console.log('done'); })();
output:
Promise {
<pending>,
domain:
Domain {
domain: null,
_events:
{ removeListener: [Function: updateExceptionCapture],
newListener: [Function: updateExceptionCapture],
error: [Function: debugDomainError] },
_eventsCount: 3,
_maxListeners: undefined,
members: [] } }
> ArrayBuffer { byteLength: 3874 }
done
But on a larger file (not sure if "large" has anything to do with it though), it just sort of hangs there:
(async function () { console.log(await require('r2').get('http://ipv4.download.thinkbroadband.com/5MB.zip').arrayBuffer); console.log('done'); })();
output:
Promise {
<pending>,
domain:
Domain {
domain: null,
_events:
{ removeListener: [Function: updateExceptionCapture],
newListener: [Function: updateExceptionCapture],
error: [Function: debugDomainError] },
_eventsCount: 3,
_maxListeners: undefined,
members: [] } }
>
Replacing .arrayBuffer
with .response
does seem to return a response object of some kind, but I don't know how to extract data from that:
> (async function () { console.log(await require('r2').get('http://ipv4.download.thinkbroadband.com/5MB.zip').response); console.log('done'); })();
output:
Promise {
<pending>,
domain:
Domain {
domain: null,
_events:
{ removeListener: [Function: updateExceptionCapture],
newListener: [Function: updateExceptionCapture],
error: [Function: debugDomainError] },
_eventsCount: 3,
_maxListeners: undefined,
members: [] } }
> Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]:
{ body:
PassThrough {
_readableState: [ReadableState],
readable: true,
domain: [Domain],
_events: [Object],
_eventsCount: 7,
_maxListeners: undefined,
_writableState: [WritableState],
writable: true,
allowHalfOpen: true,
_transformState: [Object] },
disturbed: false,
error: null },
[Symbol(Response internals)]:
{ url: 'http://ipv4.download.thinkbroadband.com/5MB.zip',
status: 200,
statusText: 'OK',
headers: Headers { [Symbol(map)]: [Object] } } }
16.3.0
to 16.4.0
.This version is covered by your current version range and after updating it in your project the build failed.
browserify is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.
The new version differs by 5 commits.
8980670
16.4.0
f871a85
Update changelog.markdown
52de2c4
Merge pull request #1916 from browserify/stream-http
5dc1bf2
Upgrade stream-http to v3
4a5ea7e
Add funding.yml
See the full diff
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
In https://github.com/mikeal/r2/blob/master/index.js#L14 there is a todo about supporting streams. Would using https://github.com/yoshuawuyts/from2-string work for this project or too large?
I saw 'TODO: formData API.' comment in index.js.
this means post formData is not yet made?
master
branch failed. 🚨I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.
You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.
Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.
Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master
branch. You can also manually restart the failed CI job that runs semantic-release.
If you are not sure how to resolve this, here is some links that can help you:
If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.
The npm token configured in the NPM_TOKEN
environment variable must be a valid token allowing to publish to the registry https://registry.npmjs.org/
.
If you are using Two-Factor Authentication, make configure the auth-only
level is supported. semantic-release cannot publish with the default auth-and-writes
level.
Please make sure to set the NPM_TOKEN
environment variable in your CI with the exact value of the npm token.
Good luck with your project ✨
Your semantic-release bot 📦🚀
Using request
, I can add options about encoding: encoding: 'latin1'
. Is it possible with r2
?
When I fetch some page like https://github.com, r2 failed and keep Promise pending.
node.js : v8.9.1
OS: Mac OS 10.12.6
r2: 2.0.0
r2('https://www.google.com').text.then(h => console.log(h)) // OK
r2('https://github.com').text.then(h => console.log(h)) // Failed
A potential security vulnerability has been disclosed for this repo https://huntr.dev/bounties/b0e92c01-2535-44ec-9808-cbba3a1cf0d7/ 3 months ago .
But i see it still not addressed .
Kindly validate the bug
One thing I liked with request, it's support of env vars for managing proxy behavior (https://github.com/request/request#controlling-proxy-behaviour-using-environment-variables).
It's very convenient, especially if you work under a corporate proxy 😢 , and the production is not.
Is there something like that planned for r2 ?
i'm the only r2 in this place
hi,
when i try to use r2 with node v8.x.x i got this error: TypeError: require.extensions.hasOwnProperty is not a function
you can reproduce the issue here: https://runkit.com/phra/r2-require-extensions
Branch | Build failing 🚨 |
---|---|
Dependency |
[semantic-release](https://github.com/semantic-release/semantic-release)
|
Current Version | 15.5.5 |
Type | devDependency |
This version is covered by your current version range and after updating it in your project the build failed.
semantic-release is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.
The new version differs by 1 commits.
4454d57
feat: allow to disable the publish plugin hook
See the full diff
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
Hi there,
I'm trying to pass a variable through headers: for example:
let headers = {'x-apikey': apiKey }
but when I run it, the log says that the apiKey value is an ilegal http header this variable (apiKey) return a random string code, but when I pasted the code instead of the variable it runs.
I want to know how pass the variable.
just became aware of fetch-h2 for http2
are you looking at http2 support?
That would be very helpful in typescript environment.
` let info = await r2(url(data[i]['ename']), { headers }).text
console.log(info) `
No output
` let info = await r2(url(data[i]['ename']), { headers }).response
info = await info.text()
console.log(info) `
The output is successful
node_modules\require-dir\index.js:93
if (!require.extensions.hasOwnProperty(ext)) {
TypeError: require.extensions.hasOwnProperty is not a function
Already fixed in aseemk/requireDir#45. require-dir
just needs an update
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.