Batch multi-part response parsing (regex) issue about o.js

[DerGuteWolf]

Well, the general idea should better be to extract the boundary name from the header "Content-Type", value is usually "multipart/mixed; boundary=XYZ". "--XYZ" ist then the boundardy to use in the regexp.
AND per can a changeset inside a batchresponse, so if one extracted batch part has also a header "Content-Type", do the same logic recursivly. This will work much better than assuming some specific boundary names.

[janhommes]

Hi Chris,

Thanks for your input. We actual have the problem, that we cannot test any batch response because the odata test service does not allow batches. How does your plain response look like? I was using web api and there the --batchresponse keyword was used to seperate responses, but it might depend on the implementation.

Regards,
Jan

[cr00maui]

Here are the results of two batched GET requests. I can provide batched POSTs too if it helps. This was just quick and easy. Details of the source/provider in the headers. Must be quite challenging to code without the ability to test.

Request

POST /odata/$batch HTTP/1.1
Host: localhost:80
Content-Type: multipart/mixed; boundary=batch_36522ad7-fc75-4b56-8c71-56071383e77b
Cache-Control: no-cache

--batch_36522ad7-fc75-4b56-8c71-56071383e77b
Content-Type: application/http 
Content-Transfer-Encoding:binary

GET Clients(13) HTTP/1.1

--batch_36522ad7-fc75-4b56-8c71-56071383e77b
Content-Type: application/http 
Content-Transfer-Encoding:binary

GET Clients(11) HTTP/1.1

--batch_36522ad7-fc75-4b56-8c71-56071383e77b 

Response Headers

Cache-Control: no-cache
Content-Length: 3279
Content-Type: multipart/mixed; boundary=batchresponse_d599621c-0bfa-439c-ae1e-c20a25a5d087
Date: Tue, 11 Jul 2017 07:33:28 GMT
Expires: -1
OData-Version: 4.0
Pragma: no-cache
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

Response Body

--batchresponse_d599621c-0bfa-439c-ae1e-c20a25a5d087
Content-Type: application/http
Content-Transfer-Encoding: binary

HTTP/1.1 200 OK
Content-Type: application/json; odata.metadata=minimal; odata.streaming=true
OData-Version: 4.0

{
	"@odata.context": "http://localhost/odata/$metadata#Clients/$entity",
	"ClientId": 13,
	"Name": "John Smith"
}
--batchresponse_d599621c-0bfa-439c-ae1e-c20a25a5d087
Content-Type: application/http
Content-Transfer-Encoding: binary

HTTP/1.1 200 OK
Content-Type: application/json; odata.metadata=minimal; odata.streaming=true
OData-Version: 4.0

{
	"@odata.context": "http://localhost/odata/$metadata#Clients/$entity",
	"ClientId": 11,
	"Name": "Peter Parker"
}
--batchresponse_d599621c-0bfa-439c-ae1e-c20a25a5d087--

[janhommes]

Hey Chris

I checked your regex sugestions. Basically its a good idea, but we are using this complicated one because you can have complexer JSON response like the following:

HTTP/1.1 200 OK
Content-Type: application/json; odata.metadata=minimal; odata.streaming=true
OData-Version: 4.0

{
	"@odata.context": "http://localhost/odata/$metadata#Clients/$entity",
	"ClientId": 13,
	"Name": "John Smith",
        "More": {
          "Some": "more"
        }
}
--batchresponse_d599621c-0bfa-439c-ae1e-c20a25a5d087

When you have this response your regex is not working anymore. Thats why we use the regex until the --batchresponse_ part and afterwards substringing it. Normaly that should work, don't know what exactly is not working for your part because I checked it in a regex-coach and it marks the correct parts.

Regards
Jan

[cr00maui]

Sorry it took me so long to respond. I think I might have confused this a bit. Yes, the sample I posted above works fine. This one however is quite different, sorry for the confusion.

Request

--batch_b98d80d7-5929-4eb5-a442-28f9f1a18bda
Content-Type: multipart/mixed; boundary=changeset_5d16bad8-07e2-4c94-f537-0897f1024d1c

--changeset_5d16bad8-07e2-4c94-f537-0897f1024d1c
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID:01

POST http://target:8080/odata/ListDetails/ HTTP/1.1
Host: http://target:8080/odata/
Content-Type: application/json

{"ListId":19,"ClientId":12,"IsYes":false,"AttrUserId":1132,"SourceId":11}


--changeset_5d16bad8-07e2-4c94-f537-0897f1024d1c
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID:11

POST http://target:8080/odata/ListDetails/ HTTP/1.1
Host: http://target:8080/odata/
Content-Type: application/json

{"ListId":19,"ClientId":131,"IsYes":false,"AttrUserId":1132,"SourceId":11}


--changeset_5d16bad8-07e2-4c94-f537-0897f1024d1c--

--batch_b98d80d7-5929-4eb5-a442-28f9f1a18bda--

Response

--batchresponse_84059c2c-fddb-4d82-b835-38320e143c52
Content-Type: multipart/mixed; boundary=changesetresponse_b64175cf-4cbf-48d4-8a02-7f2e8a97e748

--changesetresponse_b64175cf-4cbf-48d4-8a02-7f2e8a97e748
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: 01

HTTP/1.1 201 Created
Location: http://target:8080/odata/ListDetails(67147)
Content-Type: application/json; odata.metadata=minimal
OData-Version: 4.0

{
  "@odata.context":"http://target:8080/odata/$metadata#ListDetails/$entity","ListDetailId":67147,"ListId":19,"ClientId":12,"IsYes":false,"AttrUserId":1132,"IsExported":false,"ExportDtTm":null,"ModifiedId":null,"ModifiedIdentityId":null,"ModifiedDtTm":null,"CreatedId":784,"CreatedIdentityId":null,"CreatedDtTm":"2017-09-04T17:52:20.7467198-10:00","SourceId":11,"IsError":false,"ErrorMsg":null
}
--changesetresponse_b64175cf-4cbf-48d4-8a02-7f2e8a97e748
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: 11

HTTP/1.1 201 Created
Location: http://target:8080/odata/MailingListDetails(67148)
Content-Type: application/json; odata.metadata=minimal
OData-Version: 4.0

{
  "@odata.context":"http://target:8080/odata/$metadata#ListDetails/$entity","ListDetailId":67148,"ListId":19,"ClientId":131,"IsYes":false,"AttrUserId":1132,"IsExported":false,"ExportDtTm":null,"ModifiedId":null,"ModifiedIdentityId":null,"ModifiedDtTm":null,"CreatedId":784,"CreatedIdentityId":null,"CreatedDtTm":"2017-09-04T17:52:21.4058573-10:00","SourceId":11,"IsError":false,"ErrorMsg":null
}
--changesetresponse_b64175cf-4cbf-48d4-8a02-7f2e8a97e748--
--batchresponse_84059c2c-fddb-4d82-b835-38320e143c52--

This is where I was running into trouble and where I now have updated the regex/substring to

var regex = /({[\s\S]*?--changesetresponse_)/g;
...
parseResponse(result[0].substring(0, result[0].length - 20), tempBase);

Perhaps the header needs to be interpreted to know whether to use "batch" or "changeset." I don't have enough experience with odata $batch support (even within our implementation) to really qualify "changesetresponse" vs "batchresponse" and perhaps if there are variations in vendor implementations, versions, etc.

Not sure if this is really helping (you)...

Best,

Chris

[janhommes]

Maybe we can generalize the regex and use this one: /({[\s\S]*?}\n--)/g

that would match change and batch only responses. What do you think?

[DerGuteWolf]

You shoud definitly parse the Content-Type and use the boundary from there to parse the body. This can then be kind of recursive, ie changeset inside batch.
Eg SAP OData implementation uses only GUIDs as boundaries.

[rriverak]

Hey, i have the same issue on Batch-Requests...
that works in my case, could that be problematic?

var regex = /({[\s\S]*?--)/g;
var result;
do {
    var result = regex.exec(ajaxRequest.responseText);
    if (result) {
        parseResponse(result[0].substring(0, result[0].length - 3), tempBase);
        dataArray.push(tempBase.data);
    }

} while (result);

Unfortunately, I do not know all the variations of batch-odata...

[janhommes]

Totally agree with you @DerGuteWolf .
But the Content-Type needs to parsed as well... so the complete logic of the parseResponse must be reviewed.

[janhommes]

#83 uses the Content-Type header to parse the $batch request.