Table Valued Parameter Performance Issue about node-red-contrib-mssql-plus

[harmonic7]

Thanks for the response and following this through further with tediousjs/node-mssql
And thanks also for your original assistance and interest.

[Steve-Mcl]

Hi @harmonic7 thanks for the compliments.

Regarding your slow down, obviously 50000 rows is gonna be slower however I suspect you are being impacted by the incredibly slow performance of JSONata when processing lots of rows. I have pointed out the slow performance of JSONata a few times on the node-red forum but I cant reach everyone ;). I realise the potential of JSONata and love the idea of it, but I just cant get to grips with the syntax & after seeing results like this, I feel its not worth investing my time learning it. 🤷

So - I recreated your flow (minus the database part) and added an alternative to the JSONata to generate the TVP and added a subflow to measure each operation....

Here are the results ...

This is the test flow i used...

[{"id":"e4f0493e.45eee8","type":"subflow","name":"flow-performance","info":"","category":"","in":[{"x":80,"y":100,"wires":[{"id":"25479dbf.0bfb82"}]}],"out":[{"x":440,"y":100,"wires":[{"id":"25479dbf.0bfb82","port":0}]}],"env":[{"name":"name","type":"str","value":"measure"},{"name":"operation","type":"str","value":"start"}]},{"id":"25479dbf.0bfb82","type":"function","z":"e4f0493e.45eee8","name":"do operation","func":"var name = msg.perfName || env.get(\"name\");\nvar operation = msg.perfOperation || env.get(\"operation\");\nvar measures = global.get(\"flow-performance\") || {};\nvar measure = measures[name] || {};\n\nfunction doOp(measure, op){\n    \n    if(operation === \"start\"){\n        measure.start = Date.now();//change to process.hrtime\n        measure.stop = null;\n        measure.durationMs = null;\n    } else if(operation === \"stop\") {\n        measure.stop = Date.now();//change to process.hrtime\n        measure.durationMs = measure.start ? measure.stop - measure.start : null;\n        msg._performance = measure;\n    }\n}\n\n\nif(operation === \"start\"){\n    doOp(measure, operation);\n} else if(operation === \"stop\") {\n    doOp(measure, operation);\n} else if(operation === \"msg.topic\") {\n    operation = msg.topic;\n    doOp(measure, operation);\n} else if(operation === \"msg.operation\") {\n    operation = msg.operation;\n    doOp(measure, operation);\n} else if(operation === \"msg.payload\") {\n    operation = msg.payload;\n    doOp(measure, operation);\n} else {\n    return msg;    \n}\nmeasures[name] = measure;\nglobal.set(\"flow-performance\", measures);\n\nreturn msg;","outputs":1,"noerr":0,"x":250,"y":100,"wires":[[]]},{"id":"740ff94c.16b5e8","type":"debug","z":"f9a4e92.e905418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":950,"y":80,"wires":[]},{"id":"2f567ddf.650172","type":"inject","z":"f9a4e92.e905418","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":80,"wires":[["3b8c48a7.e94808"]]},{"id":"dab1e303.f5e28","type":"change","z":"f9a4e92.e905418","name":"JSONata make tvp","rules":[{"t":"set","p":"payload","pt":"msg","to":"(\t    $rows := msg.payload.(\t        [\t            [Id, Name & \" Some concatenated value \" & Id]\t        ]\t    );\t    {\t        \"columns\" : [\t            {\"name\" : \"Id\", \"type\" : \"Int\"},\t            {\"name\" : \"Name\", \"type\" : \"NVarChar(2000)\"}\t        ],\t        \"rows\" : $rows\t    };\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":710,"y":80,"wires":[["c4fb859f.981da8"]]},{"id":"e06d6e8f.c65ce","type":"function","z":"f9a4e92.e905418","name":"fake data source","func":"var fakeData = []\nfor(let i = 0; i < 1000; i++){\n    fakeData.push({\n        \"Id\": i,\n        \"Name\": \"fake-data row \" + (i+1) + \".\"\n    })\n}\nmsg.payload = fakeData;\nreturn msg;","outputs":1,"noerr":0,"x":350,"y":80,"wires":[["458178b0.f702c8","1608f16.1d0cc0f"]]},{"id":"3b8c48a7.e94808","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"make_data","env":[{"name":"name","type":"str","value":"make_data"}],"x":215,"y":80,"wires":[["e06d6e8f.c65ce"]],"l":false},{"id":"caa9659e.a73418","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_jsonata"}],"x":575,"y":80,"wires":[["dab1e303.f5e28"]],"l":false},{"id":"c4fb859f.981da8","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_jsonata"},{"name":"operation","type":"str","value":"stop"}],"x":835,"y":80,"wires":[["740ff94c.16b5e8"]],"l":false},{"id":"52689c54.08ee24","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_func"}],"x":575,"y":200,"wires":[["14f4107b.9653a"]],"l":false},{"id":"4f0fc084.76059","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"format_data_func","env":[{"name":"name","type":"str","value":"format_data_func"},{"name":"operation","type":"str","value":"stop"}],"x":835,"y":200,"wires":[["d55dd790.9fade8"]],"l":false},{"id":"d55dd790.9fade8","type":"debug","z":"f9a4e92.e905418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":950,"y":200,"wires":[]},{"id":"458178b0.f702c8","type":"subflow:e4f0493e.45eee8","z":"f9a4e92.e905418","name":"make_data","env":[{"name":"name","type":"str","value":"make_data"},{"name":"operation","type":"str","value":"stop"}],"x":495,"y":80,"wires":[["caa9659e.a73418","52689c54.08ee24"]],"l":false},{"id":"14f4107b.9653a","type":"function","z":"f9a4e92.e905418","name":"Func make tvp","func":"var tvp = {\n    \"columns\":[\n        {\"name\":\"Id\",\"type\":\"Int\"},\n        {\"name\":\"Name\",\"type\":\"NVarChar(2000)\"}\n    ],\n    \"rows\":[]\n}\n\nvar rows = msg.payload;\n\ntvp.rows = rows.map(e => { \n    return [e.Id, e.Name + \" Some concatenated value \" + e.Id] \n})\nmsg.payload = tvp\nreturn msg;","outputs":1,"noerr":0,"x":700,"y":200,"wires":[["4f0fc084.76059"]]},{"id":"1608f16.1d0cc0f","type":"debug","z":"f9a4e92.e905418","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":370,"y":120,"wires":[]}]

My request to you...

• Would you please try using the function i wrote to generate your TVP parameter data and see if the speed is more acceptable?
• Would you use the subflow I wrote (its embedded in the test flow above) to measure the time it takes to run the 2 SQL parts of your original flow and feedback how long those elements take?

Question...

Are you on the node-red forum? Perhaps you would want to raise an issue on the performance of JSONata? (assuming I am correct on this matter of-course :) )

[harmonic7]

Hi @Steve-Mcl ,
Thanks so much for your reply.
I've done as you recommended and added your performance monitoring subflow into my original flow.
Here are the results for 6000 records:

{
    "format_data_jsonata": {
        "start": 1598264127165,
        "stop": 1598264127430,
        "durationMs": 265
    },
    "call_upsert_proc": {
        "start": 1598264127433,
        "stop": 1598264141976,
        "durationMs": 14543
    }
}

And for 7000 records:

{
    "format_data_jsonata": {
        "start": 1598264585046,
        "stop": 1598264585386,
        "durationMs": 340
    },
    "call_upsert_proc": {
        "start": 1598264585388,
        "stop": 1598264604205,
        "durationMs": 18817
    }
}

And for 8000 records:

{
    "format_data_jsonata": {
        "start": 1598264707228,
        "stop": 1598264707567,
        "durationMs": 339
    },
    "call_upsert_proc": {
        "start": 1598264707569,
        "stop": 1598264729709,
        "durationMs": 22140
    }
}

Actually, here are a few more timings as a graph. Although your function does help speed up the JSON transformation, the speed of writing to the database is the main concern here:

[Steve-Mcl]

Thanks for the feedback.

In order to understand where the time is taken I would need to perform tests against a database while debugging this node and the underlying packages.

• It could be mssql-plus node
• It could be the nodejs DB driver (tedious)
• It could be your database/procedure.

Next steps...

• Do you have a way of testing this in Management Studio - see how long the procUpsertTempDestination procedure is actually taking?
• Could you run query profiler and capture this procUpsertTempDestination procedures' performance as it runs from node-red?

Questions

• is this academic or do you have a real requirement for inputting such large quantities of data into a database table in single hits?
• what version of SQL server are you using and is it a direct/vm/docker install?

[harmonic7]

Thanks again @Steve-Mcl ,

Regarding testing in Management Studio - Here I've tested this in Azure Data Studio:

• As you can see, the stored proc isn't the issue.
• Regarding whether this is academic or not. I do have real requirements for processing > 50,000 rows and I'm investigating whether Node-Red and node-red-contrib-mssql-plus could be used in these scenarios.
• I'm using SQL Server 2017 running inside a Linux based docker container (Node-Red is also running inside a Linux based docker container).

[Steve-Mcl]

OK, well, interesting.

I will attempt to debug this later tonight - see where the bottleneck is.

What would be acceptable / realistic speeds (bearing in mind this is all running 3 layer deep in the matrix (OS --> Docker --> Nodejs) and the main driver is JavaScript :) ?

[harmonic7]

Thanks again, much appreciated!

The largest number of rows would be 250,000 and ideally this could be processed in under a minute (obviously the faster the better).

The stored proc on it's own (along with the DB processing UNION ALL statements) as below takes less than a second for approx. 250,000 rows:

[Steve-Mcl]

Initial testing environment ...

• SQL2017 running in linux container on a fairly low powered NAS
• node-red running native (on an Win10 i7 box)

1000 rows in TVP...

"call_upsert_proc":{"start":1598290509127,"stop":1598290509588,"durationMs":461}

5000 rows in TVP...

"call_upsert_proc":{"start":1598290711644,"stop":1598290715920,"durationMs":4276}

10000 rows in TVP...

"call_upsert_proc":{"start":1598290757500,"stop":1598290771535,"durationMs":14035}

50000 rows in TVP...

• I gave up after about 10 minutes.

Based on 10000 rows taking ~14s, we can assume 250000 rows would take 350s (5min 50s) - however, as we can see from the durationMs value its not linear and based on the fact I gave up on attempting to achieve 50000 rows after about 10m, I think its safe to say there is an issue somewhere.

So, looking under the hood i see that the mssql package is still using tedious V6 whereas tedious itself is on V9 and this issue on the tedious repo looks suspiciously like our problem.

The mssql package does have an alpha release with the updated tedious driver as a dependency (but it has been alpha for some time).

testing

test flow...

[{"id":"aa58b4fd.87c478","type":"subflow","name":"flow-performance (2)","info":"","category":"","in":[{"x":80,"y":100,"wires":[{"id":"711f2d64.d4a8b4"}]}],"out":[{"x":440,"y":100,"wires":[{"id":"711f2d64.d4a8b4","port":0}]}],"env":[{"name":"name","type":"str","value":"measure"},{"name":"operation","type":"str","value":"start"}]},{"id":"711f2d64.d4a8b4","type":"function","z":"aa58b4fd.87c478","name":"do operation","func":"var name = msg.perfName || env.get(\"name\");\nvar operation = msg.perfOperation || env.get(\"operation\");\nvar measures = global.get(\"flow-performance\") || {};\nvar measure = measures[name] || {};\n\nfunction doOp(measure, op){\n    \n    if(operation === \"start\"){\n        measure.start = Date.now();//change to process.hrtime\n        measure.stop = null;\n        measure.durationMs = null;\n    } else if(operation === \"stop\") {\n        measure.stop = Date.now();//change to process.hrtime\n        measure.durationMs = measure.start ? measure.stop - measure.start : null;\n        msg._performance = measure;\n    }\n}\n\n\nif(operation === \"start\"){\n    doOp(measure, operation);\n} else if(operation === \"stop\") {\n    doOp(measure, operation);\n} else if(operation === \"msg.topic\") {\n    operation = msg.topic;\n    doOp(measure, operation);\n} else if(operation === \"msg.operation\") {\n    operation = msg.operation;\n    doOp(measure, operation);\n} else if(operation === \"msg.payload\") {\n    operation = msg.payload;\n    doOp(measure, operation);\n} else {\n    return msg;    \n}\nmeasures[name] = measure;\nglobal.set(\"flow-performance\", measures);\n\nreturn msg;","outputs":1,"noerr":0,"x":250,"y":100,"wires":[[]]},{"id":"a59d1790.df3388","type":"change","z":"595a5dd5.a963a4","name":"JSONata make tvp","rules":[{"t":"set","p":"payload","pt":"msg","to":"(\t    $rows := msg.payload.(\t        [\t            [Id, Name & \" Some concatenated value \" & Id]\t        ]\t    );\t    {\t        \"columns\" : [\t            {\"name\" : \"Id\", \"type\" : \"Int\"},\t            {\"name\" : \"Name\", \"type\" : \"NVarChar(2000)\"}\t        ],\t        \"rows\" : $rows\t    };\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":2100,"wires":[["fc6b11d5.977b3"]]},{"id":"af8e26a1.7a8f28","type":"function","z":"595a5dd5.a963a4","name":"fake data source","func":"var fakeData = []\nfor(let i = 0; i < msg.payload; i++){\n    fakeData.push({\n        \"Id\": i,\n        \"Name\": \"fake-data row \" + (i+1) + \".\"\n    })\n}\nmsg.payload = fakeData;\nreturn msg;","outputs":1,"noerr":0,"x":370,"y":2100,"wires":[["1d617764.71a279"]]},{"id":"6f84d2cb.c048dc","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"make_data","env":[{"name":"name","type":"str","value":"make_data"}],"x":235,"y":2100,"wires":[["af8e26a1.7a8f28"]],"l":false},{"id":"79263b72.e037c4","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_jsonata"}],"x":635,"y":2100,"wires":[["a59d1790.df3388"]],"l":false},{"id":"fc6b11d5.977b3","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_jsonata"},{"name":"operation","type":"str","value":"stop"}],"x":895,"y":2100,"wires":[[]],"l":false},{"id":"593f19b9.8ba3e8","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"format_data_jsonata","env":[{"name":"name","type":"str","value":"format_data_func"}],"x":635,"y":2200,"wires":[["fc4aa8d7.902c88"]],"l":false},{"id":"264b0f07.18609","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"format_data_func","env":[{"name":"name","type":"str","value":"format_data_func"},{"name":"operation","type":"str","value":"stop"}],"x":895,"y":2200,"wires":[["57ec0829.02c6c8"]],"l":false},{"id":"1d617764.71a279","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"make_data","env":[{"name":"name","type":"str","value":"make_data"},{"name":"operation","type":"str","value":"stop"}],"x":515,"y":2100,"wires":[["593f19b9.8ba3e8"]],"l":false},{"id":"fc4aa8d7.902c88","type":"function","z":"595a5dd5.a963a4","name":"Func make tvp","func":"var tvp = {\n    \"columns\":[\n        {\"name\":\"Id\",\"type\":\"Int\"},\n        {\"name\":\"Name\",\"type\":\"NVarChar(2000)\"}\n    ],\n    \"rows\":[]\n}\n\nvar rows = msg.payload;\n\ntvp.rows = rows.map(e => { \n    return [e.Id, e.Name + \" Some concatenated value \" + e.Id] \n})\nmsg.payload = tvp\nreturn msg;","outputs":1,"noerr":0,"x":760,"y":2200,"wires":[["264b0f07.18609"]]},{"id":"3bc39ce6.807de4","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"2000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2140,"wires":[["6f84d2cb.c048dc"]]},{"id":"e6e3f53f.0530e8","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"5000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2180,"wires":[["6f84d2cb.c048dc"]]},{"id":"57ec0829.02c6c8","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"sql_proc","env":[{"name":"name","value":"call_upsert_proc","type":"str"}],"x":635,"y":2280,"wires":[["d9a32df5.572f5"]],"l":false},{"id":"a4a30e62.68b5f","type":"subflow:aa58b4fd.87c478","z":"595a5dd5.a963a4","name":"sql_proc","env":[{"name":"name","value":"call_upsert_proc","type":"str"},{"name":"operation","value":"stop","type":"str"}],"x":895,"y":2280,"wires":[["a2c44966.626868"]],"l":false},{"id":"d9a32df5.572f5","type":"MSSQL","z":"595a5dd5.a963a4","mssqlCN":"a51e405c.10f64","name":"","outField":"payload","returnType":0,"throwErrors":1,"query":"procUpsertTempDestination","modeOpt":"","modeOptType":"execute","queryOpt":"","queryOptType":"editor","paramsOpt":"","paramsOptType":"editor","params":[{"output":false,"name":"Data","type":"TVP(dbo.TempData)","valueType":"msg","value":"payload"}],"x":760,"y":2280,"wires":[["a4a30e62.68b5f"]]},{"id":"e2972976.a93108","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"10000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2220,"wires":[["6f84d2cb.c048dc"]]},{"id":"dec0068c.3f9a98","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"50000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2260,"wires":[["6f84d2cb.c048dc"]]},{"id":"4e7b9024.1ebeb","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"1000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2100,"wires":[["6f84d2cb.c048dc"]]},{"id":"d2678c61.550bd","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"250000","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":2300,"wires":[["6f84d2cb.c048dc"]]},{"id":"a2c44966.626868","type":"debug","z":"595a5dd5.a963a4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","x":990,"y":2280,"wires":[]},{"id":"82ee0ac5.174618","type":"comment","z":"595a5dd5.a963a4","name":"Not using JSONata - too slow over 50000 rows","info":"","x":800,"y":2060,"wires":[]},{"id":"a51e405c.10f64","type":"MSSQL-CN","z":"","tdsVersion":"7_4","name":"My SQL Server","server":"192.168.1.38","port":"1433","encyption":false,"database":"testdb","useUTC":false,"connectTimeout":"15000","requestTimeout":"240000","cancelTimeout":"5000","pool":"10","parseJSON":false,"enableArithAbort":true}]

progress

I modified my local install to use the alpha version of mssql (and thus tedious v9)...

• Edit .node-red/node_modules/node-red-contrib-mssql-plus/package.json
• change the dependency "mssql": "^6.x.x" to "mssql": "next"
• run npm update inside .node-red/node_modules/node-red-contrib-mssql-plus
• re-ran the test flow

5000 rows

"seconds":2
"milliseconds":857.523699

10000 rows

"seconds":5,
"milliseconds":760.4161

50000 rows

"seconds":28
"milliseconds":9.2825

250000 rows

2m 27s

Summary...

So, while its still above the 1 minute mark, it is an improvement.

I will watch the mssql for release of this update - its about the best i can do at this time.

[harmonic7]

Thanks again @Steve-Mcl ,

Interesting - I changed the mssql dependency as you suggested, then restarted the node-red container. Afterwards I got an error saying that the node couldn't connect because of a self-signed certificate error. So - I disabled encryption in the configuration node:

And now I can insert 250,000 rows in 6 seconds:

I'll now revert the dependency change and check to see whether it's just the encryption setting that made the huge difference.

I've now tried reverting the dependency change, but there wasn't the same performance boost (with encryption turned off).

So - it seems that the beta version of the mssql client library (or the newer version of tedious) is doing something very right (at least when encryption is disabled).

[Steve-Mcl]

Ahha, I assumed you "needed" encryption. If you are good without it, then that is great.

PS, did you manage to get 250000 rows into the DB using original mssql client lib? If so, how long did it take?

[harmonic7]

Hi again @Steve-Mcl ,

Well I do really need encryption, but I thought that at least there may be other ways forwards (like setting up a tunnel between the containers).

I didn't manage to get to 250000 rows with the original mssql client. When I switched back I found that the same performance issues were occurring so I didn't try past about 10,000.

However - I thought about things further yesterday and realised that I had only assumed that the encryption setting was the thing that slowed things down (because I couldn't connect to my sql instance without disabling it).

The resulted was that I forked this repo here, and pushed a change which exposed the trustServerCertificate setting. I then installed this manually and was able to use the latest mssql to connect to my SQL Server instance by enabling the trustServerCertificate.

I then ran the test with both encrypt and trustServerCertificate enabled and got the same great performance (7.4 seconds for 250,000 rows).

So - it wasn't the encryption settings after all, just the latest mssql library that did the trick.

I'll create a pull request for you to add this newly exposed setting. I realise that it also contains the adjustment to use the latest mssql, but hopefully you can just ignore the packages.json file adjustments if you do decide to merge my adjustment in.

Fingers crossed that the improvements in the latest version of mssql will make their way into the stable release soon.

[Steve-Mcl]

Thanks for the PR.
I am gonna comment on the mssql repo see if there is any plans to get this moving quicker.
Until then, i will leave the PR open & do it all at once.

[Steve-Mcl]

@harmonic7 There is a beta in NPM that includes your changes AND the package dep for mssql v7

NOTE: due to your PR being against master branch I could not merge it since I wanted to do this in a development branch so instead, I copied / pasted your changes into the new branch (thank you) please feel free to close the PR.

If you get the chance, please see https://discourse.nodered.org/t/updated-node-red-contrib-mssql-plus-0-7-0-beta-2-bulk-insert-support/37331 & provide feedback.

I will close this issue for now but feel free to re-open.

Cheers.