GithubHelp home page GithubHelp logo

bestlong / node-red-contrib-mssql-plus Goto Github PK

View Code? Open in Web Editor NEW
29.0 6.0 18.0 246 KB

A Node-RED node to read and write to Microsoft MS SQL Databases

License: MIT License

HTML 54.41% JavaScript 43.90% Dockerfile 0.42% Shell 1.24% TSQL 0.03%
mssql nodered node-red node-red-contrib hacktoberfest

node-red-contrib-mssql-plus's Introduction

node-red-contrib-mssql-plus

A Node-RED node to execute queries, stored procedures and bulk inserts in Microsoft SQL Server and Azure Databases SQL2000 ~ SQL2022.

Importantly, this package comes with pre-built linux drivers for communicating with the Azure & MS SQL services (using TDS protocol), removing the need to set-up environment level MSSQL (or similar) drivers.


Screen shot

image

Features include...

  • Connect to multiple SQL Servers and Azure databases from SQL2000 ~ 2022
  • Perform multiple queries in one go & get back multiple recordsets (depends on the queries sent)
  • Supports Stored Procedure execute
  • Supports Bulk Insert
  • Built in examples (node-red hamburger menu → import → examples → node-red-contrib-mssql-plus)
    • TVP - A demo of calling a stored procedure and passing in a table valued parameters
    • BULK - A demo of inserting a large amount of data in bulk mode
  • Use env vars in the config node for all fields (including credentials). e.g...
    • Server {{{SQL_IP}}}
    • Password {{{SQL_PW}}}
  • Use mustache in your SQL queries including msg, flow and global context. e.g...
    • SELECT TOP {{{payload.maxRows}}} * FROM [MyTable] WHERE Name = '{{{flow.name}}}' AND quantity <= {{{global.maxQty}}}
    • View the final query (mustache rendered into values) in msg.query to understanding what happened to your {{{mustache}}} parameters
  • Enter parameters in the UI or send parameters in via msg, flow or global variables for use in your SQL queries e.g...
    • SELECT * FROM [MyTable] WHERE Name = @name AND quantity <= @maxQty
    • View the final parameters (rendered with final values) in msg.queryParams that were used in the query to aid debugging
    • View output parameters values in msg.queryParams after the query has executed
  • Choose between throwing an error to the catch node or outputting an error property in msg.error
  • Additional properties are in the msg object (use a debug node with "complete msg object" set to see whats available)

Install

Easiest

Use the Manage Palette > Install option from the menu inside node-red

Harder

Alternatively in your Node-RED user directory, typically ~/.node-red, run

npm install node-red-contrib-mssql-plus

Usage

Please refer to the built in help in the info panel in node red.

Sample flow

Demonstrating mustache rendering, input parameters, mutliple queries, print info...

  • Payload: {"count": 5, "age": 35}
  • Parameters
    • name, varchar(20), stephen
    • age, int, msg.payload.age
  • Query:
      PRINT @name
      SELECT TOP {{{payload.count}}} * 
      FROM testdb.dbo.[MyTable] WHERE Name = @name
      SELECT TOP {{{payload.count}}} * 
      FROM testdb.dbo.[MyTable] WHERE Age = @age
      PRINT 'complete'
  • After Rendering Mustache:
      PRINT @name
      SELECT TOP 5 * 
      FROM testdb.dbo.[MyTable] WHERE Name = @name
      SELECT TOP 5 * 
      FROM testdb.dbo.[MyTable] WHERE Age = @age
      PRINT 'complete'

flow...

[{"id":"61625aaf.479d84","type":"inject","z":"595a5dd5.a963a4","name":"{\"count\": 5, \"age\": 35}","topic":"","payload":"{\"count\": 5, \"age\": 35}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":220,"y":320,"wires":[["6e09980a.127878"]]},{"id":"6e09980a.127878","type":"MSSQL","z":"595a5dd5.a963a4","mssqlCN":"a51e405c.10f64","name":"","outField":"payload","returnType":"1","throwErrors":"0","query":"PRINT @name\n\nSELECT TOP {{{payload.count}}} * \nFROM testdb.dbo.[MyTable] WHERE Name = @name\n\nSELECT TOP {{{payload.count}}} * \nFROM testdb.dbo.[MyTable] WHERE Age = @age\n\nPRINT 'complete'","modeOpt":"","modeOptType":"query","queryOpt":"","queryOptType":"editor","paramsOpt":"","paramsOptType":"editor","params":[{"output":false,"name":"name","type":"VarChar(20)","valueType":"str","value":"stephen"},{"output":false,"name":"age","type":"int","valueType":"msg","value":"payload.age"}],"x":260,"y":380,"wires":[["babb6d0.5ae7e9"]]},{"id":"babb6d0.5ae7e9","type":"debug","z":"595a5dd5.a963a4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":270,"y":440,"wires":[]},{"id":"a51e405c.10f64","type":"MSSQL-CN","z":"","tdsVersion":"7_4","name":"My SQL Server","server":"192.168.1.38","port":"1433","encyption":false,"database":"testdb","useUTC":false,"connectTimeout":"15000","requestTimeout":"15000","cancelTimeout":"5000","pool":"5","parseJSON":false}]

Other

This node based on node-red-contrib-mssql.

Thanks to Redconnect.io.

node-red-contrib-mssql-plus's People

Contributors

alexgv99 avatar belgianbeer avatar bestlong avatar cjmuller avatar dhartveld avatar fitz0019 avatar iba-jsykes avatar jturner-22 avatar martin-doyle avatar mrauter avatar redconnect avatar retailos avatar steve-mcl avatar tmdoit avatar wrathagom avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

node-red-contrib-mssql-plus's Issues

Issue with persistent login credentials

I am receiving this error message:
17 Aug 08:29:22 - [info] [MSSQL-CN:SQLSExpress2019] Error connecting to server : localhost\SQLEXPRESS, database : NodeRedTest, port : 1433, user :
ConnectionError: Login failed for user ''.

The MSSQL-CN doesn't appear to be saving or rather reading the login credentials once deployed. Before I deploy, if I check the credentials they are there, but after deploying they are gone. I see an encrypted value in the flows__cred.json file.

I have this working on my work machine just fine (windows 10), but for some reason on my home dev box (windows 10) is not reading the info back in and thus not passing the login upon running the flow.

I've done the following so far:

  • I've started node-red in a cmd window in Admin mode
  • Made sure I can connect to SQL Server in other ways (ODBC)
  • Updated node-red
  • Updated your recent push on the MSSQL-plus node
  • Deleted and recreated the connection info
  • Using a very simple flow (inject -> mssql -> debug)
  • Verified username and password is correct

Obviously, it is something on my home development machine, but any suggestions would be helpful.
Thank you,
Greg

Possible to include connection option for ApplicationIntent?

At first glance I don't see the ability to define the ApplicationIntent. This is needed for Availability Groups with read-only replicas. The connection string value would be "ApplicationIntent=ReadOnly". I don't mind working on a PR as I believe this is supported in Tedious, but want to make sure I'm not overlooking the ability to do this currently.

batch data load is failing with Invalid column type from bcp client for colid 2.

Hi Steve,

Thank you for your response and real apologies for not reverting you sooner. As we had to go for demo, we used the static file load at the time of starting DB and progressed to other places.

Back to bulk insert now and this looks pretty good. I'm generating an array of around 98000+ records and pushing it in sql-server via batch mode. Array does get created quite quickly though while pushing the data in sql, it's throwing:
Invalid column type from bcp client for colid 2.

Which seems a bummer as I'm creating table along with the bulk insert, which means schema is exactly the same. Kept all my fields as nullable too so it shouldn't be the issue. Would you have any recommendation to look for in fixing this issue?

Tarn: opt max. must be an Integer > 0

Hi
I am running NodeRed 1.0.3 together with node-red-contrib-mssql-plus 0.4.0.
When executing queries on the server, I get the error message: 'Tarn: opt max. must be an Integer > 0' right below the node.
Tried to google it but did not find anything useful.
DB is SQL Server Standard 2017.
25 03

Thanks for your help!
klib

Table Valued Parameter Performance Issue

Firstly, thanks so much for your fantastic work.

I wanted to see how this node performed, and so I tried a simple test with 50,000 rows, however had to limit things. 1000 rows were inserted using a TVP fairly quickly (about a second or so), however more than that seemed to get slower exponentially.

I guess that perhaps the slowness is coming from actually building the sql table before inserting it. Perhaps you may have some insights as to how to increase performance.

Here's the test:

I setup two tables TempSource and TempDestination, a table type, and a stored proc to populate the TempDestination table. I also populated 49,999 rows of dummy data into TempSource:

-- Setup some testing tables
CREATE TABLE TempSource (
    Id INT PRIMARY KEY,
    Name NVARCHAR(2000)
)
GO

CREATE TABLE TempDestination (
    Id INT PRIMARY KEY,
    Name NVARCHAR(2000)
)
GO

-- Insert values into the source
SET NOCOUNT ON

DECLARE @Counter INT = 10000

BEGIN TRANSACTION
WHILE @Counter < 50000
BEGIN
    INSERT INTO TempSource VALUES(@Counter, 'The Object Name ' + CAST(@Counter AS NVARCHAR(20)))
    SET @Counter = @Counter + 1
END
COMMIT

-- Create Table Type
CREATE TYPE TempData AS TABLE (
    Id INT PRIMARY KEY,
    Name NVARCHAR(2000)
)
GO

-- Create procedure to 'Upsert'
-- Actually for now it just truncates and re-populates.
CREATE PROCEDURE procUpsertTempDestination
    @Data dbo.TempData READONLY
AS
BEGIN
    TRUNCATE TABLE TempDestination
    INSERT INTO TempDestination
    SELECT * FROM @Data
END

I then setup the following flow (which also used some JSONata transform to adjust the data, as I was testing the performance of this also):

[
    {
        "id": "dd45cc9.283d83",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": ""
    },
    {
        "id": "184967eb.124df8",
        "type": "MSSQL",
        "z": "dd45cc9.283d83",
        "mssqlCN": "f288f4bc.61043",
        "name": "",
        "outField": "payload",
        "returnType": 0,
        "throwErrors": 1,
        "query": "SELECT  TOP 5000 *\nFROM TempSource",
        "modeOpt": "",
        "modeOptType": "query",
        "queryOpt": "",
        "queryOptType": "editor",
        "paramsOpt": "",
        "paramsOptType": "editor",
        "params": [],
        "x": 360,
        "y": 80,
        "wires": [
            [
                "f92b847c.535cb8"
            ]
        ]
    },
    {
        "id": "cbe8113e.97571",
        "type": "debug",
        "z": "dd45cc9.283d83",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 890,
        "y": 80,
        "wires": []
    },
    {
        "id": "569a0c98.c79404",
        "type": "inject",
        "z": "dd45cc9.283d83",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 170,
        "y": 80,
        "wires": [
            [
                "184967eb.124df8"
            ]
        ]
    },
    {
        "id": "f92b847c.535cb8",
        "type": "change",
        "z": "dd45cc9.283d83",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "(\t    $rows := msg.payload.(\t        [\t            [Id, Name & \" Some concatenated value \" & Id]\t        ]\t    );\t    {\t        \"columns\" : [\t            {\"name\" : \"Id\", \"type\" : \"Int\"},\t            {\"name\" : \"Name\", \"type\" : \"NVarChar(2000)\"}\t        ],\t        \"rows\" : $rows\t    };\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 580,
        "y": 80,
        "wires": [
            [
                "cbe8113e.97571",
                "18dec6ec.7177a9"
            ]
        ]
    },
    {
        "id": "18dec6ec.7177a9",
        "type": "MSSQL",
        "z": "dd45cc9.283d83",
        "mssqlCN": "f288f4bc.61043",
        "name": "",
        "outField": "payload",
        "returnType": 0,
        "throwErrors": 1,
        "query": "procUpsertTempDestination",
        "modeOpt": "",
        "modeOptType": "execute",
        "queryOpt": "",
        "queryOptType": "editor",
        "paramsOpt": "",
        "paramsOptType": "editor",
        "params": [
            {
                "output": false,
                "name": "Data",
                "type": "TVP(dbo.TempData)",
                "valueType": "msg",
                "value": "payload"
            }
        ],
        "x": 900,
        "y": 140,
        "wires": [
            [
                "20300723.19c61"
            ]
        ]
    },
    {
        "id": "20300723.19c61",
        "type": "debug",
        "z": "dd45cc9.283d83",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1010,
        "y": 240,
        "wires": []
    },
    {
        "id": "f288f4bc.61043",
        "type": "MSSQL-CN",
        "z": "",
        "tdsVersion": "7_4",
        "name": "",
        "server": "sqldata",
        "port": "1433",
        "encyption": true,
        "database": "sdb",
        "useUTC": true,
        "connectTimeout": "30000",
        "requestTimeout": "30000",
        "cancelTimeout": "5000",
        "pool": "50",
        "parseJSON": true,
        "enableArithAbort": true
    }
]

Note that the first SQL node has a TOP 5000 limit in the query - this can be removed to get the full 49999 records for testing.

Thanks again!

error connecting to ms sql server 2019 linux / docker

Hi
i've no success to connect to a linux ms sql server in a docker container
besides, i can connect from a windows client to the ms sql server and also from Azure Data Studio (MacOS)

what i noticed is, that the windows client shows me no instance name for the linux ms sql server, just a servername (mssqlserver)
that page says, that the linux version has no instance name:
https://stackoverflow.com/questions/41695344/how-can-i-create-named-instance-of-microsoft-sql-server-on-ubuntu
could that be the problem?

error msg with instance name 'mssqlserver': Failed to connect to ip\mssqlserver
error msg w/o an instance name: invalid syntax near '-'.

thanks

Examples of using a connection node? Cannot even find the node in the palette

Sorry for perhaps being a noob question, but how can I setup a common connection that opens in the beginning of a flow. Will be used for a lot of operations/queries and then closed at the end of the flow. I cannot find an example and honestly don't know how to implement it. I cannot even find the CN-node in my palette (it shows up as installed but I cannot find it to drag it into my flow). Have tried in Azure web app, Azure Ubuntu VM and on my Home Assistant with the same result :-( Can I be missing some prerequisite
chrome_ksOmyon3GV
chrome_8sRrMaIINA

?

ConnectionError: Connection is closed.

Using:
Raspberry Pi 3 B+
Latest Raspbian Image
Node-Red v0.19.2
node-red-contrib-mssql-plus v0.0.12
mssql v3.3.0

I have total of four mssql-plus nodes in a flow. None of them are activated simultaneously.
Only one of the nodes is throwing this error. The other three function normally.

Restarting Node-Red didn't solve the issue.

Another note, I have an identical Node-Red instance running on a different Raspberry Pi that works normally.

ECONNCLOSED: Connection is closed - Concurrent SQL connections result in first succeeding and all other failing

Recently forced to move from age old node-red install version (think 0.something) and node-red-contrib-mssql.

Flows imported fine but with node-red-contrib-mssql no longer maintained (deleted?) we have moved to node-red-contrib-mssql-plus. Now we see where concurrent table reads/writes work in the older Node they fail in mssql-plus.

A few of our flows for example read from multiple tables in a single DB, then join the results for an external report or email.

permit additional options in the parameter value

Add jsonata & timestamp to parameter value options.

Also, for consideration...

  • now Date (for Date types)
  • now DateTime (for DateTime types)
  • now DateTime2 (for DateTime2 types)

This simplifies things like a datetime to be set to now or getting info from a JS object (using jsonata)

image

Problem connecting to mssql express

I have a local install of mssql express but I cannot connect. I tried windows authentication, then I tried creating a user, but I cannot connect either way. I have checked that mssql express service is running, and I even set the TCP/IP port to 1433, not that this should matter. So far I have not found a way to connect. To get started, I just want to create an inject node with a simple insert query into an existing table. Can you help?

MSSQL 2017 with JSON feature add "JSON_F52E2B61-18A1-11d1-B105-00805F49916B" prefix to JSON resultset

Hi,

I try use this node with MSSQL 2017 and JSON feature. Problem is that MSSQL add MS specific prefix to JSON result. Like below. String is always same.

"JSON_F52E2B61-18A1-11d1-B105-00805F49916B": "{\"ProductCode\":\"0040001\",\"ProductName\":\"Brother DK11209 osoitetarra 29x62mm\"}"

It changes JSON to JavaScript Objects says Nick O'Leary, see thread
https://discourse.nodered.org/t/http-response-json-without-backslashes/19376/7?u=timnis

In node-mssql there is option "config.parseJSON = true" to strip out this MS prefix, could it be also possible to add to this node?

More info
https://tediousjs.github.io/node-mssql/#json-support

Using split in combination with MSSQL-plus and azureSQL

First of all. Great work with this node. Easy to use and still versatile.
I might however have push it to it's limit in this scenario and like to discuss it.

This is the scenario that im testing.
image

I want to be able to run a query for each row (object in my array) but im facing the issue
Resource ID : 1. The request limit for the database is 30 and has been reached. See 'http://go.microsoft.com/fwlink/?LinkId=267637' for assistance.

From what i can understand the node is opening a connection for each query that runs and that would totally make sense.
In this case it does however create problems with limits towards Azure.

What i think could solve this problem would be if we actually open the connection --> run alla queries --> close connection.
Something like this
image

I have found a couple of workarounds like concatenate the complete query and then run. That technically solves the issue but makes the flow harder to follow and maintain.

Is there a way to control the opening and closing of the connection manually?

A example flow to recreate the issue
flows.zip

suggestion: be able to handle multiple SELECT queries

Hi,

Just a quick suggestion I'd love to see it being implemented.

Add the ability to handle multiple SELECT queries, e.g.

msg.payload =

SELECT * FROM Customers;
SELECT * FROM CustomersDetails;

In Studio Management or Navicat this will execute 2 SELECT queries and return each one.

Would be great to have a similar behaviour here, maybe having the node return these as an array:

msg.payload = [
[query 1 results array],[query 2 results array]
]

SQL Parameter Escaping may be required?

Hi There, I see that the payload.Name doesn't get escaped before inserting into the database:

INSERT INTO TempDestination VALUES ({{{payload.Id}}}, '{{{payload.Name}}}')

For instance, if we have payload.Name containing a first character of ', then the payload.Name is populated into the database as an empty string.

Stored Procedures

Hi,

Is there any way of supporting the execution of stored procedures.

I notice in the Node.JS MSSQL that stored procedures are called in a different way, is there support for it node-red-contrib-mssql-plus.

Best Regards,

Glyn

Error "Incorrect syntax near ')'." in Bulk Insert when Columns are from msg.paramQuery

image
I get this error when passing Columns parameter from specific msg.xx

This is the configuration:
image

this is the content of the msg.complete

{"paramQuery": [{"output":false, "name":"idEsecuzioneAttivita","type":"int"}, {"output":false,"name":"idUtente","type":"int"}, {"output":false,"name":"Utente","type":"nvarchar"}, {"output":false,"name":"Data","type":"date"}, {"output":false,"name":"OraInizio","type":"nvarchar"}, {"output":false,"name":"OraFine","type":"nvarchar"}, {"output":false,"name":"Minuti","type":"int"}, {"output":false,"name":"CodiceProgetto","type":"nvarchar"}, {"output":false,"name":"idCliente","type":"int"}, {"output":false,"name":"RagioneSociale","type":"nvarchar"}, {"output":false,"name":"idProgetto","type":"int"}, {"output":false,"name":"Progetto","type":"nvarchar"}, {"output":false,"name":"idAttivita","type":"int"}, {"output":false,"name":"Attivita","type":"nvarchar"}, {"output":false,"name":"idTipoAttivita","type":"int"}, {"output":false,"name":"TipoAttivita","type":"nvarchar"}, {"output":false,"name":"Numero","type":"float"}, {"output":false,"name":"TariffaAttivita","type":"float"}, {"output":false,"name":"idLuogoEsecuzioneAttivita","type":"int"}, {"output":false,"name":"LuogoEsecuzioneAttivita","type":"nvarchar"}, {"output":false,"name":"CostoAttivita","type":"float"}, {"output":false,"name":"CostoOrario","type":"float"}, {"output":false,"name":"Note","type":"nvarchar"}, {"output":false,"name":"idSettore","type":"int"}, {"output":false,"name":"Settore","type":"nvarchar"}, {"output":false,"name":"idAreaBusiness","type":"int"}, {"output":false,"name":"AreaBusiness","type":"nvarchar"}, {"output":false,"name":"idTipoProgetto","type":"int"}, {"output":false,"name":"TipoProgetto","type":"nvarchar"}, {"output":false,"name":"processato","type":"tinyint"}], "formattedData":[ ["1779763","14280","Pablo Picasso","23/02/2021","09:00","13:00","240","0110018","43682","XXX S.P.A.","60367","XXX AM","80925","XXXXX AM","17803","Sviluppo","","150,00","3727","Ufficio","","","x","","","668","XXX","","",0] ],"_msgid":"71a8d293.c9817c"}

Could you please help me if it is an issue or what i'm doing wrong?

continuously getting ECONNCLOSED error

Hi,
I'm trying to parse a large CSV file (approx 10k records) and store it in SQL Server. As the SQL Server is remote or sometimes docker initialized, don't want to use bulk-insert.

Node-red is running in docker. data file is shared with it via volume mount.

Node-red can see the file (as I'm printing the entire payload as http response while debugging).

CSV is able to parse the whole file and share it per record basis.

My function before the MS-SQL invokes has below code:

exchCode = msg.payload["ExchangeCode"].trim()
exchange = msg.payload["Exchange"].trim()
indusry = msg.payload["Industry"].trim()
comp =  msg.payload["CompanyName"].trim()

pld =       "INSERT INTO [OCP_LOCAL].[ext].[Profile]"
pld = pld + "(CompanyName,ExchangeCode,Industry,Exchange,_seen) "
pld = pld + "VALUES ('" + comp + "','" + exchCode + "' ,'" + indusry + "','" + exchange + "',0)"

msg.payload = pld
return msg;

It is able to insert roughly 4 or 5 records in the database and for rest provides ECONNCLOSED. I believe the time taken to write one record is probably long hence provided below properties:
ConnectionTimeout: 60000
requesttimeout: 0
canceltimeout: 0

Also added delay of 1 second upto 5 seconds after CSV parser but still no luck. And last idea tried was Max Pool Size to 100 though still no difference on the number of records being inserted in DB.

[{"id":"1bf07b4d6f4c42cf","type":"tab","label":"csvreader","disabled":false,"info":""},{"id":"40fe383c56a1a128","type":"http in","z":"1bf07b4d6f4c42cf","name":"http-in","url":"/invokecsv","method":"get","upload":false,"swaggerDoc":"","x":170,"y":160,"wires":[["b0b04a9607b7444b"]]},{"id":"b0b04a9607b7444b","type":"file in","z":"1bf07b4d6f4c42cf","name":"ProfileMaster","filename":"/data/Profile_202108121438.csv","format":"utf8","chunk":false,"sendError":false,"encoding":"utf8","allProps":false,"x":350,"y":160,"wires":[["8cf03da220982022","712817f5220aab38"]]},{"id":"830d889c02bf783e","type":"debug","z":"1bf07b4d6f4c42cf","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1310,"y":180,"wires":[]},{"id":"8cf03da220982022","type":"http response","z":"1bf07b4d6f4c42cf","name":"","statusCode":"200","headers":{},"x":540,"y":240,"wires":[]},{"id":"9752a07b71f0349e","type":"switch","z":"1bf07b4d6f4c42cf","name":"","property":"payload[\"ExchangeCode\"]","propertyType":"msg","rules":[{"t":"nnull"}],"checkall":"true","repair":false,"outputs":1,"x":890,"y":60,"wires":[["129d3222e252ea8f"]]},{"id":"aa88f3f752cb3e99","type":"MSSQL","z":"1bf07b4d6f4c42cf","mssqlCN":"df8c0b88.91b0a8","name":"MSSQL","outField":"payload","returnType":"0","throwErrors":"0","query":"","modeOpt":"","modeOptType":"query","queryOpt":"payload","queryOptType":"msg","paramsOpt":"","paramsOptType":"none","rows":"","rowsType":"msg","params":[],"x":1140,"y":120,"wires":[["830d889c02bf783e","b1a99a1f10fd2aec"]]},{"id":"129d3222e252ea8f","type":"function","z":"1bf07b4d6f4c42cf","name":"update-sql-profile","func":"exchCode = msg.payload[\"ExchangeCode\"]\nexchange = msg.payload[\"Exchange\"]\nindusry = msg.payload[\"Industry\"]\ncomp = msg.payload[\"CompanyName\"]\n\npld = \"INSERT INTO [OCP_LOCAL].[ext].[Profile]\"\npld = pld + \"(CompanyName,ExchangeCode,Industry,Exchange) \"\npld = pld + \"VALUES ('\" + comp + \"','\" + exchCode + \"' ,'\" + indusry + \"','\" + exchange + \"')\"\n\n//pld = \"BULK INSERT [OCP_LOCAL].[ext].[Profile]\"\n//pld = pld + \" FROM '/data/ProfileMaster.csv'\"\n//pld = pld + \" WITH (\"\n//pld = pld + \"FIRSTROW=2,\"\n//pld = pld + \"LASTROW=5,\"\n//pld = pld + \"FIELDTERMINATOR=',',\"\n//pld = pld + \"TABLOCK\"\n//pld = pld + \")\"\nmsg.payload = pld\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":950,"y":200,"wires":[["aa88f3f752cb3e99"]]},{"id":"b1a99a1f10fd2aec","type":"debug","z":"1bf07b4d6f4c42cf","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"error","targetType":"msg","statusVal":"","statusType":"auto","x":1310,"y":260,"wires":[]},{"id":"461303971aa45e84","type":"delay","z":"1bf07b4d6f4c42cf","name":"","pauseType":"delay","timeout":"30","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"x":740,"y":60,"wires":[["9752a07b71f0349e"]]},{"id":"712817f5220aab38","type":"csv","z":"1bf07b4d6f4c42cf","name":"csv-parse","sep":",","hdrin":true,"hdrout":"all","multi":"one","ret":"\\r\\n","temp":"CompanyName,ExchangeCode,Industry,Exchange,_seen","skip":"0","strings":true,"include_empty_strings":true,"include_null_values":true,"x":580,"y":100,"wires":[["461303971aa45e84"]]},{"id":"df8c0b88.91b0a8","type":"MSSQL-CN","tdsVersion":"7_4","name":"aws","server":"${SQL_SERVER_URL}","port":"${SQL_SERVER_PORT}","encyption":false,"trustServerCertificate":true,"database":"OCP_LOCAL","useUTC":false,"connectTimeout":"60000","requestTimeout":"0","cancelTimeout":"0","pool":"100","parseJSON":false,"enableArithAbort":true,"credentials":{}}]

Catch errors

Hi bestlong!

Sorry for bothering you again, but you are the only one who is still out there on mssql for node-red. - So your work is most appreciated!

I've just noticed, that a catch-node does not catch erros of a mssql node, maybe you could implement to throw an exception by any chance?

BR
pate

Credentials via Environment-Variable

In Node documentation you write "You can use environment variables in any connection field, using mustache format."

This works fine for connection information stored in flow.json but not for user and password.

My unencrypted flows_cred.json looks like
{"7cd6f15f.369f7":{"username":"{{{DB_USER}}}","password":"{{{DB_PASSWORD}}}"}}

When trying to connect, I get the following errormessage:
Error connecting to server : 172.16.3.164, database : ASP, port : 1433, user : {{{DB_USER}}}

So it looks like there is no mustache token replacement for credentials. Values for server, database and port are set correctly via env.

Bulk Insert fails on table with primary key

When bulk inserting into a predefined table with a primary key, the insert fails and the following error is returned:

Invalid column type from bcp client for colid 1.

Column types used were NVARCHAR(255)

The whole thing worked fine if I didn't have a pre-existing table, or the pre-existing table didn't have a primary key.

Feature Request - Allow JSONATA expressions for sensitive data in Config SQL Server Node

Hi Steve,

As you may have guessed from some from my posts on forum we are extensively into DB work.
For SQL Server we use your node (MSSQL Plus) and it fits our purpose perfectly (Thank You for this node.)

Recently as part of some work - we are now checking how to deploy flows across environments where the DB names etc. may differ.

  1. We already allow go to flow post deployment to new env - and configure the credentials username/password which then gets used within flows.
  2. In other cases - we need to handle everything as external properties and not touch the flows themselves post deployment. This is good. And even here we plan to use ENV variable approach supported by the node.

Challenge - sending Password as ENV variable may run into security risk. We need to set it in settings.js which is in readable format.
Is it possible to add some functionality to password field where we can call some JSONATA function aka - base64decode.
Or maybe some extensible in addition to JSONATA - so that even if we set it in settings.js in encoded form - then using some "callable" mechanism set in password field - the password gets decoded and used by the node.

Decimal truncated

Consider adding support for decimal precision and scale

example...
request.addParameter('value', Decimal, 18728.4, { precision: 16, scale: 2 });

Currently, it looks like tedious is defaulting to decimal(18,2) meaning loss of precision.

mssql-plus issue with mustache format for server name and user/pwd

Hi,
I'm using mssql-plus 0.6.4 version.
i could not connect to sql using env variable. This is helpful for me to connect to different db located in the different location. I tried to create a sub flow for sql that would help me reuse the same flow across different locations with different servers.
It throws me an errr message
code: "ESOCKET"
message: "Failed to connect to :1433 - Could not connect (sequence)

you help would be really appreciable
I've attached the flow with subflow

[{"id":"ebc85c42.463a8","type":"subflow","name":"SQLPlus064","info":"","category":"","in":[{"x":100,"y":80,"wires":[{"id":"bf1289cd.07fcb8"}]}],"out":[{"x":300,"y":80,"wires":[{"id":"bf1289cd.07fcb8","port":0}]}],"env":[{"name":"ServerName","type":"str","value":"10.1.2.111"},{"name":"UserName","type":"str","value":"appliUser"},{"name":"Password","type":"str","value":"1234556"},{"name":"DBName","type":"str","value":"DB_DEV"}],"color":"#DDAA99"},{"id":"bf1289cd.07fcb8","type":"MSSQL","z":"ebc85c42.463a8","mssqlCN":"c29068d1.97ef38","name":"","outField":"payload","returnType":0,"throwErrors":1,"query":"Select TOP 10 * FROM TestBenchResult","modeOpt":"","modeOptType":"query","queryOpt":"","queryOptType":"editor","paramsOpt":"","paramsOptType":"none","params":[],"x":200,"y":80,"wires":[[]]},{"id":"c29068d1.97ef38","type":"MSSQL-CN","z":"ebc85c42.463a8","tdsVersion":"7_4","name":"DB-Con","server":"\"ServerName\"","port":"","encyption":true,"trustServerCertificate":true,"database":"ShopfloorDB_DEV","useUTC":false,"connectTimeout":"50000","requestTimeout":"50000","cancelTimeout":"50000","pool":"5","parseJSON":false,"enableArithAbort":true},{"id":"c81636eb.e7d248","type":"tab","label":"MSSQLPlus-CheckSub","disabled":false,"info":""},{"id":"7c1a36cd.c54a88","type":"inject","z":"c81636eb.e7d248","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":100,"wires":[["30366732.3efba8"]]},{"id":"de6a1325.7c759","type":"debug","z":"c81636eb.e7d248","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":440,"y":100,"wires":[]},{"id":"30366732.3efba8","type":"subflow:ebc85c42.463a8","z":"c81636eb.e7d248","name":"","env":[{"name":"ServerName","value":"10.1.2.111","type":"env"},{"name":"Password","value":"123456789","type":"str"}],"x":300,"y":80,"wires":[["de6a1325.7c759"]]}]

Thanks
Ramesh R

Invalid data type on Bulk Insert crashes Node-Red

If a error occurs on a bulk insert (e.g. if you try to insert a invalid date into a DateTime Column) then the Node's error handling will not be triggered and the Node-Red Runtime will crash.

Here is the error message

16 Aug 08:56:38 - [red] Uncaught Exception:
16 Aug 08:56:38 - TypeError: Invalid date value passed to bulk rows
    at Table._makeBulk (/data/node_modules/mssql/lib/table.js:69:21)
    at Immediate.<anonymous> (/data/node_modules/mssql/lib/tedious/request.js:216:13)
    at processImmediate (internal/timers.js:464:21)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /data/.npm/_logs/2021-08-16T17_56_38_410Z-debug.log

Port and Instance Name

Hi,

How can specify port number and instance name in the connection node? I see there was a port field in previous version, some one asked to make it optional, but it's no longer available in the latest version!

note that I connect to database using this server name ([server-ip][instance name],[port number]) but that format not supported in this node!

Thank You

Ameen

On Error return the msg_id

Is there a way to get the msg_id returned when an error happened?

I would like to valitade the msg_id specially when using RAISERROR on SP for error handling on node-red

Thanks!

Timeout

Hi,

I'm feeling like a complete noob and I'm sure it's me missing something silly but for the life on me,I can't connect to a SQL server with this node.

I have a "remote" SQL server I can access with Navicat with sql.mydomain.com\instance,50002 but failing to translate this into node config.

I've tried with

server: sql.mydomain.com\instance
port: 50002

and

server: sql.mydomain.com\instance,50002
port: 1433

but all I get is timeout errors...

Can someone give me a hand please?

Security Vulnerability in jsonwebtoken Library

Hello,

the version of the jsonwebtoken library which is used by your node has a security vulnerability.
It is recommended to use the newest version of the library.

Can you please fix this vulnerability?

Links:

[https://nvd.nist.gov/vuln/detail/CVE-2022-23529]

Thank you in advance.

Beta V0.5.0-beta - support "execute" and parameters

Hi @bestlong , contributers and anyone passing...

I have added a branch called params

The purpose of these modification is 2 fold...

  1. Improve protection against sql injection (through use of parameters)
  2. permit sending TVP (Table Values Parameters) to stored procedures.

See this thread for some additional insight

The UI has numerous changes to permit static or dynamic entry of parameters.

I have tested the changes in node-red of several machines and the user beta testing has (almost) got it working on his machine.


If any one could test and feedback that would be great.


Install -

  • stop node-red
  • open a terminal window
  • cd your .node-red directory e.g...
  • npm uninstall node-red-contrib-mssql-plus
  • npm install node-red-contrib-mssql-plus@beta
  • start node-red

This is a demo flow...

[{"id":"3eb71b1f.47fdb4","type":"MSSQL","z":"595a5dd5.a963a4","mssqlCN":"a51e405c.10f64","name":"","query":"MyCustomStoredProcedure","outField":"payload","returnType":"1","throwErrors":"0","queryMode":"execute","params":[{"output":false,"name":"tvp","type":"TVP(TestType)","valueType":"json","value":"{\"columns\":[{\"name\":\"a\",\"type\":\"VarChar(50)\"},{\"name\":\"b\",\"type\":\"Int\"}],\"rows\":[[\"hello tvp\",777],[\"bye tvp\",888],[\"call the cops\",999]]}"}],"x":500,"y":520,"wires":[["bfde3540.a91018"]]},{"id":"614325ad.afd22c","type":"inject","z":"595a5dd5.a963a4","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":500,"y":480,"wires":[["3eb71b1f.47fdb4"]]},{"id":"bfde3540.a91018","type":"debug","z":"595a5dd5.a963a4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":510,"y":580,"wires":[]},{"id":"a51e405c.10f64","type":"MSSQL-CN","z":"","tdsVersion":"7_4","name":"Local SQL","server":"192.168.99.99","port":"1433","encyption":false,"database":"testdb","useUTC":false,"connectTimeout":"15000","requestTimeout":"15000","cancelTimeout":"5000","pool":"5","parseJSON":false}]

Using Environment Variables

I set an Environment Variable on my flow

image

I am trying to pass this value to a Query, but no luck. Not sure if bug or I am screwing something up:

image

But in SQL, I just get blanks:

image

Port should be optional

Hi,

in the latest version, the port attribute is mandatory.
I guess 99% of all SQL server use named instances, where ports are not being used.
Could you please set this attribute optional again, as it throws configuration errors during deployment?

Thank you and best regards
Pate

Memory leak is still happen in both of newest version and prior version.

Memory Leak Case

There is Memory leak problem in both of your newest version 0.7.3 and prior version 0.3.3.

The testing flow was that you mentioned in the previous issue,
but I changed the input more quick, For example, 7200pcs / 5 seconds, and change the delay node to 10msg/s
(My work space need the input more large and more fast)

In this case, after running node-red around 1.5hours, the memory usage in node-red from 4% to 40%, and then node-red crashed.
Crashed again no matter how much times node-red restart.


Problem Solved after I modified your code, and very pleasure to improve your code.

However, I changed lots of your version 0.3.3 code, and actually there is no memory leak again, the problem is solving.
I am pleasure to help you to improve your code or maybe you can merge my code, but first I have couples of questions for you:

  1. First of all, why did you use the module mustache and others for the MS-SQL configuration input? After I saw the page of module mustache page, I found that was major use for HTML template, am I correct? Therefore, I tried to delete it all in the red frame of picture (as shown below), I found the output that was no difference to your origin code. Here is your version 0.3.3 code:
    image
    Could you simply explain me what you were doing here?

  2. Second, I noticed that you create new promise object for every node-mssql input, I am not sure whether the node-red use the cahch for every node. If yes, I guess that was very big problem. Why did you create a new promise object to include node-mssql promise object?

After I deleted these two point, and modified the code, the memory leak was never happen again even thought I set the input 9700 msg / 2 seconds, and total input 70 million messages.

Please help me, thanks.

New connection per node?

I want to know if a new connection is created for each node or if a single connection is shared between many nodes.

Plans to support both Native driver and Tedious?

Hello,

Do you have any plans to provide support for both the Native V8 driver for MSSQL alongside with the current one (Tedious)?

Im interested in using the Native version, so if you are not currently work on this, can we fork and add this functionality, or would you consider a PR?

Thanks.

SQL Editor area not filling available space.

I've recently updated from 0.3.2. to 0.5.0. and noticed the query editor portion of the node settings has shrunk significantly. This appears to be because the sidebar leaves space for the parameter 'editor' mode in 0.5.0, even when it isn't visible. It seems like the oneditresize function doesn't take into account whether the second editor is visible and leaves space for it regardless. This makes it hard to deal with more complex queries.

Below are screenshots of 0.5.0 and 0.3.2

image
image

Node Red Memory Leak When Using node-red-contrib-mssql-plus

I am using the mssql-plus node to put about 500,000 records into a sql table, 1000 records at a time. When I do this, the program will eventually crash my controller after about 3 times of running the program. It seems like the query strings are being held in the memory after the flow is completed. Is this a known issue or am I using the node incorrectly?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.