deliverr / data-knex-snowflake-dialect Goto Github PK
View Code? Open in Web Editor NEWknex.js dialect for the Snowflake data warehouse
License: MIT License
knex.js dialect for the Snowflake data warehouse
License: MIT License
Hi,
I get a 'not found' error when I try to install your package
PS C:\Repositories\xxdssfg> npm install knex-snowflake-client
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/knex-snowflake-client - Not found
Snowflake supports a column type called variant
, which supports JSON and other JSON-like semi-structured data. One syntax to be able to index a variant column is by using the colon character, what the Snowflake documentation calls dot notation, per docs here.
As far as I can tell, there is no current support for this notation. If, in a where
, I try to specify a field as this dot notation, e.g. foo:bar
, then the query compiler mistakenly tries to quote this as "foo:bar"
, when the correct quotation is supposed to be "foo":"bar"
. This results in an error.
I've found that Knex supports a custom wrapIdentifier
, per docs here. This is definitely one possible work-around. As far as I can tell, knex-snowflake-dialect already supplies its own wrapIdentifier
implementation, but it doesn't support dot notation. Hence, the above issue.
I'm working on seeing if I can get a work-around to work locally. Perhaps, however, this should be integrated into the library.
Let me know your thoughts. Thanks.
Not sure how far back this is an issue, but newer version of knex have different names for mysql compiler (e.g. knex/lib/dialects/mysql/schema/mysql-querycompiler
). Downgrading to [email protected]
fixes the issue for me
My app is using the .first() method on a query and with snowflake I was getting back a single item in an array instead of the first element in that array, it appears it's because the code to handle _method === 'first' is commented out in the adapter (
data-knex-snowflake-dialect/src/index.ts
Line 186 in f4b6abe
My use case works with this line added back, and a tweak to downcase columns in the first row. Was this just never implemented or was it commented out at some point for some reason?
Connection is getting staled after some inactivity period and knex stops working
Snowflake is case-insensitive if you use identifier without double quotes, But its case sensitive with double quotes.
(https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html)
But data-knex-snowflake-dialect always emits query with columns that are in upper-case.
For example:
const connection = knex({
client: SnowflakeDialect,
})
const builder = connection('TABLE1')
.withSchema('SCHEMA1')
.select({
'Id': 'id'
})
Expected: select "TABLE1"."id" as "Id" from "SCHEMA1"."TABLE1"
Actual: select "TABLE1"."ID" as "ID" from "SCHEMA1"."TABLE1"
Note: alias name and column name - both have to be case sensitive.
Workaround:
const builder = connection('TABLE1')
.withSchema('SCHEMA1')
.select([
knex.ref(
knex.raw('id')
).as(
knex.raw('Id')
)])
With this, I am able to make column name and alias however i want
Hi! This lib is leaking Snowflake Connection instances due to a cleanup code error here. In that code, we call connection.end()
when, in reality, the method defined by the Snowflake SDK is connection.destroy
(defined here).
This leads to never cleaning up the keepalive interval SF connections create by default, which, in turn, keeps all Connection instances around even when they were already removed from the pool managed by Knex. I've created a pull request to fix this.
Expected Result
Actual Result
Steps to reproduce
// eslint-disable-next-line import/no-unresolved
import { AzureFunction, Context, HttpRequest } from '@azure/functions';
import { SnowflakeDialect } from 'knex-snowflake-dialect';
const knex = require('knex')({
client: SnowflakeDialect as any,
connection: process.env.SNOWFLAKE_CONNECTION_STRING,
pool: {
min: 1,
max: 1,
},
});
const httpTrigger: AzureFunction = async function (
context: Context,
req: HttpRequest,
): Promise<void> {
try {
if (!req.body) throw new Error('no request body');
const db = knex;
console.log({ db });
const result = {};
context.res = {
status: 200,
body: result,
};
} catch (err) {
context.log.error(err);
context.res = {
status: 500,
body: err,
};
}
};
export default httpTrigger;
Visual Proof (screenshots, videos, text)
node_modules/knex/types/index.d.ts:1130:16 - error TS1110: Type expected.
1130 column1: `${TTable1}.${TKey1}`,
~~~
node_modules/knex/types/index.d.ts:1131:14 - error TS1005: ';' expected.
1131 column2: `${TTable2}.${TKey2}`
~
node_modules/knex/types/index.d.ts:1132:5 - error TS1128: Declaration or statement expected.
1132 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1132:6 - error TS1128: Declaration or statement expected.
1132 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1132:40 - error TS1005: '(' expected.
1132 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1134:33 - error TS1005: '?' expected.
1134 TTable1 extends TableNames,
~
node_modules/knex/types/index.d.ts:1135:15 - error TS1005: ';' expected.
1135 TTable2 extends TableNames,
~~~~~~~
node_modules/knex/types/index.d.ts:1136:13 - error TS1005: ';' expected.
1136 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~~~~~~~
node_modules/knex/types/index.d.ts:1136:66 - error TS1005: '(' expected.
1136 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1136:84 - error TS1109: Expression expected.
1136 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1137:13 - error TS1005: ',' expected.
1137 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~~~~~~~
node_modules/knex/types/index.d.ts:1137:65 - error TS1109: Expression expected.
1137 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~
node_modules/knex/types/index.d.ts:1138:43 - error TS1109: Expression expected.
1138 TRecord1 = ResolveTableType<TRecord>,
~
node_modules/knex/types/index.d.ts:1139:65 - error TS1109: Expression expected.
1139 TRecord2 = TRecord1 & ResolveTableType<TableType<TTable2>>,
~
node_modules/knex/types/index.d.ts:1141:5 - error TS1109: Expression expected.
1141 >(
~
node_modules/knex/types/index.d.ts:1142:16 - error TS1005: ')' expected.
1142 tableName: TTable2,
~
node_modules/knex/types/index.d.ts:1143:14 - error TS1005: ',' expected.
1143 column1: `${TTable2}.${TKey2}`,
~
node_modules/knex/types/index.d.ts:1144:14 - error TS1005: ',' expected.
1144 column2: `${TTable1}.${TKey1}`
~
node_modules/knex/types/index.d.ts:1145:6 - error TS1005: ';' expected.
1145 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1145:40 - error TS1005: '(' expected.
1145 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1147:36 - error TS1005: '?' expected.
1147 TJoinTargetRecord extends {} = any,
~
node_modules/knex/types/index.d.ts:1148:16 - error TS1005: ';' expected.
1148 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~~~~~~~
node_modules/knex/types/index.d.ts:1148:27 - error TS1128: Declaration or statement expected.
1148 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~
node_modules/knex/types/index.d.ts:1150:5 - error TS1109: Expression expected.
1150 >(
~
node_modules/knex/types/index.d.ts:1151:16 - error TS1005: ')' expected.
1151 tableName: TableDescriptor | AliasDict | QueryCallback,
~
node_modules/knex/types/index.d.ts:1152:14 - error TS1005: ';' expected.
1152 column1: string,
~
node_modules/knex/types/index.d.ts:1153:14 - error TS1005: ';' expected.
1153 column2: string
~
node_modules/knex/types/index.d.ts:1154:5 - error TS1128: Declaration or statement expected.
1154 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1154:6 - error TS1128: Declaration or statement expected.
1154 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1154:40 - error TS1005: '(' expected.
1154 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1156:36 - error TS1005: '?' expected.
1156 TJoinTargetRecord extends {} = any,
~
node_modules/knex/types/index.d.ts:1157:16 - error TS1005: ';' expected.
1157 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~~~~~~~
node_modules/knex/types/index.d.ts:1157:27 - error TS1128: Declaration or statement expected.
1157 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~
node_modules/knex/types/index.d.ts:1159:5 - error TS1109: Expression expected.
1159 >(
~
node_modules/knex/types/index.d.ts:1160:16 - error TS1005: ')' expected.
1160 tableName: TableDescriptor | AliasDict | QueryCallback,
~
node_modules/knex/types/index.d.ts:1161:14 - error TS1005: ';' expected.
1161 column1: string,
~
node_modules/knex/types/index.d.ts:1162:10 - error TS1005: ';' expected.
1162 raw: Raw
~
node_modules/knex/types/index.d.ts:1163:5 - error TS1128: Declaration or statement expected.
1163 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1163:6 - error TS1128: Declaration or statement expected.
1163 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1163:40 - error TS1005: '(' expected.
1163 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1165:33 - error TS1005: '?' expected.
1165 TTable1 extends TableNames,
~
node_modules/knex/types/index.d.ts:1166:15 - error TS1005: ';' expected.
1166 TTable2 extends TableNames,
~~~~~~~
node_modules/knex/types/index.d.ts:1167:13 - error TS1005: ';' expected.
1167 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~~~~~~~
node_modules/knex/types/index.d.ts:1167:66 - error TS1005: '(' expected.
1167 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1167:84 - error TS1109: Expression expected.
1167 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1168:13 - error TS1005: ',' expected.
1168 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~~~~~~~
node_modules/knex/types/index.d.ts:1168:65 - error TS1109: Expression expected.
1168 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~
node_modules/knex/types/index.d.ts:1169:43 - error TS1109: Expression expected.
1169 TRecord1 = ResolveTableType<TRecord>,
~
node_modules/knex/types/index.d.ts:1170:65 - error TS1109: Expression expected.
1170 TRecord2 = TRecord1 & ResolveTableType<TableType<TTable2>>,
~
node_modules/knex/types/index.d.ts:1172:5 - error TS1109: Expression expected.
1172 >(
~
node_modules/knex/types/index.d.ts:1173:16 - error TS1005: ')' expected.
1173 tableName: TTable2,
~
node_modules/knex/types/index.d.ts:1174:14 - error TS1005: ',' expected.
1174 column1: `${TTable1}.${TKey1}`,
~
node_modules/knex/types/index.d.ts:1175:15 - error TS1005: ',' expected.
1175 operator: string,
~
node_modules/knex/types/index.d.ts:1176:14 - error TS1005: ',' expected.
1176 column2: `${TTable2}.${TKey2}`
~
node_modules/knex/types/index.d.ts:1177:6 - error TS1005: ';' expected.
1177 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1177:40 - error TS1005: '(' expected.
1177 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1179:33 - error TS1005: '?' expected.
1179 TTable1 extends TableNames,
~
node_modules/knex/types/index.d.ts:1180:15 - error TS1005: ';' expected.
1180 TTable2 extends TableNames,
~~~~~~~
node_modules/knex/types/index.d.ts:1181:13 - error TS1005: ';' expected.
1181 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~~~~~~~
node_modules/knex/types/index.d.ts:1181:66 - error TS1005: '(' expected.
1181 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1181:84 - error TS1109: Expression expected.
1181 TKey1 extends StrKey<ResolveTableType<TableType<TTable1>>> & StrKey<TRecord1>,
~
node_modules/knex/types/index.d.ts:1182:13 - error TS1005: ',' expected.
1182 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~~~~~~~
node_modules/knex/types/index.d.ts:1182:65 - error TS1109: Expression expected.
1182 TKey2 extends StrKey<ResolveTableType<TableType<TTable2>>>,
~
node_modules/knex/types/index.d.ts:1183:43 - error TS1109: Expression expected.
1183 TRecord1 = ResolveTableType<TRecord>,
~
node_modules/knex/types/index.d.ts:1184:65 - error TS1109: Expression expected.
1184 TRecord2 = TRecord1 & ResolveTableType<TableType<TTable2>>,
~
node_modules/knex/types/index.d.ts:1186:5 - error TS1109: Expression expected.
1186 >(
~
node_modules/knex/types/index.d.ts:1187:16 - error TS1005: ')' expected.
1187 tableName: TTable2,
~
node_modules/knex/types/index.d.ts:1188:14 - error TS1005: ',' expected.
1188 column1: `${TTable2}.${TKey2}`,
~
node_modules/knex/types/index.d.ts:1189:15 - error TS1005: ',' expected.
1189 operator: string,
~
node_modules/knex/types/index.d.ts:1190:14 - error TS1005: ',' expected.
1190 column2: `${TTable1}.${TKey1}`,
~
node_modules/knex/types/index.d.ts:1191:6 - error TS1005: ';' expected.
1191 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1191:40 - error TS1005: '(' expected.
1191 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1193:36 - error TS1005: '?' expected.
1193 TJoinTargetRecord extends {} = any,
~
node_modules/knex/types/index.d.ts:1194:16 - error TS1005: ';' expected.
1194 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~~~~~~~
node_modules/knex/types/index.d.ts:1194:27 - error TS1128: Declaration or statement expected.
1194 TRecord2 extends {} = TRecord & TJoinTargetRecord,
~
node_modules/knex/types/index.d.ts:1196:5 - error TS1109: Expression expected.
1196 >(
~
node_modules/knex/types/index.d.ts:1197:16 - error TS1005: ')' expected.
1197 tableName: TableDescriptor | AliasDict | QueryCallback,
~
node_modules/knex/types/index.d.ts:1198:14 - error TS1005: ';' expected.
1198 column1: string,
~
node_modules/knex/types/index.d.ts:1199:15 - error TS1005: ';' expected.
1199 operator: string,
~
node_modules/knex/types/index.d.ts:1200:14 - error TS1005: ';' expected.
1200 column2: string
~
node_modules/knex/types/index.d.ts:1201:5 - error TS1128: Declaration or statement expected.
1201 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1201:6 - error TS1128: Declaration or statement expected.
1201 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:1201:40 - error TS1005: '(' expected.
1201 ): QueryBuilder<TRecord2, TResult2>;
~
node_modules/knex/types/index.d.ts:2421:1 - error TS1128: Declaration or statement expected.
2421 }
There are changes in the recent upgrade of v0.95 which now changes the names of files such as dialects/mysql/schema/tablecompiler into dialects/mysql/schema/mysql-tablecompiler.js.
Also unique() function builds query statement with {indexname}(columns) which needs a function similar to index and dropIndex in TableCompiler something like below to remove indexing?
unique(columns, indexName) {
this.pushQuery(
alter table ${this.tableName()} add unique (${this.formatter.columnize( columns )})
);
}
;
I'm trying to use the latest version (1.0.1), and in my minimal app that tries to get a connection I'm getting a generic error "Unable to acquire a connection". Currently, driverName
is a property on the config object passed to the constructor and knex is not doing anything with that key so driverName is undefined when the super() call happens to Knex.Client.
I can get it to work if I do this:
const db = Knex(config);
db.client.driverName = "snowflake-sdk";
db.client.initializeDriver();
db.client.initializePool(config);
Am I missing something? This was changed in #36 which was recently merged. I have version 1.0.1 of knex and of the dialect.
Hi there, I can't get this to work with either the connection URL nor the breakdown. It works fine using vanilla snowflake driver though, so I have no idea whether this is a bug or I'm doing something wrong. Any idea how to debug?
Connection URL and breakdown shown below:
export const snowflake = knex({
client: SnowflakeDialect,
debug: config.logLevel === 'debug',
connection: `snowflake://${config.snowflakeUser}:${config.snowflakePassword}@${config.snowflakeAccount}.snowflakecomputing.com/${config.snowflakeDatabase}?warehouse=${config.snowflakeWarehouse}`,
pool: {
min: 1,
max: 1,
},
// connection: {
// host: `${config.snowflakeAccount}.snowflakecomputing.com`,
// user: config.snowflakeUser,
// password: config.snowflakePassword,
// database: config.snowflakeDatabase,
// connectAttributes: {
// warehouse: config.snowflakeWarehouse,
// },
// },
// pool: {
// min: config.snowflakePoolMinConnections,
// max: config.snowflakePoolMaxConnections,
// },
});
Working snippet from the snowflake documentation:
const connection = snowflakeSdk.createConnection({
account: config.snowflakeAccount,
username: config.snowflakeUser,
password: config.snowflakePassword,
warehouse: config.snowflakeWarehouse,
database: config.snowflakeDatabase,
application: 'api',
});
connection.connect(function (err, conn) {
if (err) {
logger.error('Unable to connect: ' + err.message);
} else {
logger.info('Successfully connected to Snowflake.');
}
});
Hi! This dialect relies on SF connection error
event emissions. However, the default SF SDK Connection
class does not emit them (I created an issue on their repo as well related to this confusing behavior). This means bad connections will still be kept in the pool since the validateConnection
implementation is rather simple.
A better validateConnection
implementation could make use of the isValidAsync
SDK method which will verify if the connection is up and do a heartbeat to ensure it's ready to receive queries. I'm willing to submit a PR to fix it.
Expected Result
use of snowflake connection should connect to snowflake database
Actual Result
a bunch of type error
Steps to reproduce
"dependencies": {
"@azure/cosmos": "^3.11.5",
"@types/node": "^14.17.4",
"adal-node": "^0.2.2",
"aws-sdk": "^2.936.0",
"date-fns": "^2.22.1",
"date-fns-tz": "^1.1.4",
"express-jwt": "^6.0.0",
"express-jwt-authz": "^2.4.1",
"guid": "0.0.12",
"jwks-rsa": "^2.0.3",
"knex": "^0.95.6",
"knex-snowflake-dialect": "^0.2.5",
"mongodb": "^3.6.9",
"mssql": "^6.3.2",
"node-fetch": "^2.6.1",
"pg": "^8.6.0",
"quick-pivot": "^2.7.0",
"tedious": "^11.0.9",
"ts-jest": "^26.5.6"
},
"devDependencies": {
"@azure/functions": "^1.2.3",
"@types/jest": "^26.0.23",
"@typescript-eslint/eslint-plugin": "^4.28.1",
"@typescript-eslint/parser": "^4.28.1",
"cz-conventional-changelog": "^3.3.0",
"eslint": "^7.29.0",
"eslint-config-airbnb-base": "^14.2.1",
"eslint-config-prettier": "^6.15.0",
"eslint-plugin-import": "^2.23.4",
"eslint-plugin-prettier": "^3.4.0",
"husky": "^4.3.8",
"jest": "^26.6.3",
"lint-staged": "^10.5.4",
"nodemon": "^2.0.7",
"prettier": "^2.3.2",
"pretty-quick": "^3.1.1",
"standard-version": "^9.3.0",
"stub-azure-function-context": "^2.0.0-alpha.6",
"typescript": "^4.3.4"
},
// eslint-disable-next-line import/no-unresolved
import { AzureFunction, Context, HttpRequest } from '@azure/functions';
import { SnowflakeDialect } from 'knex-snowflake-dialect';
const knex = require('knex')({
client: SnowflakeDialect as any,
connection: process.env.SNOWFLAKE_CONNECTION_STRING,
pool: {
min: 1,
max: 1,
},
});
const httpTrigger: AzureFunction = async function (
context: Context,
req: HttpRequest,
): Promise<void> {
try {
if (!req.body) throw new Error('no request body');
const db = knex;
console.log({ db });
const result = {};
context.res = {
status: 200,
body: result,
};
} catch (err) {
context.log.error(err);
context.res = {
status: 500,
body: err,
};
}
};
export default httpTrigger;
Visual Proof (screenshots, videos, text)
node_modules/knex-snowflake-dialect/lib/index.d.ts:5:52 - error TS2339: Property 'Client' does not exist on type 'typeof import("/home/xiemicha/projects/portal-api/node_modules/knex/types/index")'.
5 export declare class SnowflakeDialect extends Knex.Client {
~~~~~~
node_modules/knex-snowflake-dialect/lib/index.d.ts:11:66 - error TS2694: Namespace '"/home/xiemicha/projects/portal-api/node_modules/knex/types/index"' has no exported member 'Transaction'.
11 transaction(container: any, config: any, outerTx: any): Knex.Transaction;
The readme in the repo says to install by the following package name.
npm install knex-snowflake-client
This one doesn't work.
I was able to install by the following package name.
npm install knex-snowflake-dialect
This leads me to believe that the readme has a typoed or outdated name.
This may or may not be the same as #9.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.