Should `request size did not match content length` be considered an error in API about raw-body

[dougwilson]

Hi @Marak ! Yea, it could be seen either way. The check is mainly in there for mistakes and even request smuggling attack attempts. Since the validation only actually happens when you pass in the length option, would it be possible to just not pass that in? All other checks like limit are still intact even without the length option, which is purely there for this does not match check.

[Marak]

I was considering not passing in the length, but it would require we remove our req.headers['content-length'] header from the request ( raw-body being called from body-parser ). Removing this header would cause a bit of friction in our API since we want to reference that header value later.

The easiest thing on our end seems to be ignoring the length validation error. I understand why you have it turned on in raw-body by default. I wanted to ask to make sure removing the validation code wouldn't cause anything bad to happen. Seems like it should be OK.

Probably the best solution for us will be doing a fork commenting out the length check. I'm not sure what else I could do to fix it here. We are connecting HTTP streams to STDIO streams in a child process. The child process is usually responsible for parsing the request body. In some cases we have the parent process doing the request parse. In those cases, the additional parse in the child process was causing the length validation to fail.

[dougwilson]

Hi @Marak gotcha. Sorry, I didn't know you were using it though another module, otherwise I would have tried to think of another solution. I thought you may have been consuming this module directly based on no additional information + filed the issue here -- sorry! this module is used in quite a lot of other places, so hard to know though which you wouldn't been coming without context.

I agree that deleting req.headers['content-length'] is not a good solution, though of course all requests that come in chunk encoded wouldn't have that header anyway.

So to better understand your usage, you're saying that you're using body-parser in a child process, but sometimes also into the parent process? And this fails when it's invoked in the child process, presumably because the body was re-serialized without adjusting the content-length to reflect the new body, is that right?

[Marak]

@dougwilson - I think that is exactly right. Do you have any recommended work-around? Would it make sense to try and preserve the body so we can call the body-parser twice?

Really appreciate your prompt help on this. Sorry if I didn't provide enough information in the initial issue. I wasn't exactly sure the best place in the code to try and solve this.

[dougwilson]

It's no problem, happy to help through this, which even changing something in the modules is not yet out of the question :)

Yea, if you are not actually changing the body in the parent process, it may make sense to simple preserve the originally body rather than, for example, re-serializing it. For example, if this is JSON, I assume you are using bodyParser.json(), which will populate the req.body property with the result of JSON.parse of the body.

If that is the case, a potential solution is (pseudo):

// copy the incoming body into a Buffer instead of JSON.parse
app.use(bodyParser.raw({ type: 'application/json' })

// later on
app.use((req, res) => {
  if (Buffer.isBuffer(req.body)) {
    var json = JSON.parse(req.body.toString())
    // do whatever parent stuff with the json
  }

  // send the request to the child process now
  // req.body, if Buffer, would be the exact thing to send
  // instead of like JSON.stringify(req.body) if that is done now
})

[dougwilson]

You could even do this, to help make it a big more less changes:

app.use(bodyParser.raw({ type: 'application/json' })
app.use((req, res, next) => {
  var body = req.body
  if (Buffer.isBuffer(body)) {
    req.body = JSON.parse(body.toString())
    req.originalBody = body
  } else {
    req.body = {}
    req.originalBody = Buffer.alloc(0)
  }
  next()
})

[dougwilson]

This is all just idea spit-balling right now, so feel free to reject any of them :)

[dougwilson]

And there is the current "verify" trick to get the underlying raw Buffer as well:

app.use(bodyParser.json({
  verify: (req, res, buf) => req.originalBody = buf
}))
// then you'll have req.body as normal and req.originalBody you'd send to the child process

[Marak]

This information helps a lot, thank you.

I'm going to try using bodyParser.raw and attempt to preserve the original body.

Will post my results here.

[Marak]

I ended up just choosing to ignore the specific error by code and continue the request if it comes up.

Something like:

if (err.type !== 'request.size.invalid') {
  return res.json(err)
}

Seems to be the easiest solution. If this doesn't work I'll try to see if we can pass parsed state on request object or check for body as buffer.

Thanks again!