Problem injecting into query about sqlite-worker

[WebReflection]

You can't create raw parts of the query unless you use the raw utility, see:
https://github.com/WebReflection/sqlite-tag#api

the raw utility is also not possible via Worker, you need a direct handler, as described in here:
https://github.com/WebReflection/sqlite-worker#after-initialization-helpers

Alternatively, you can use static-params to implement your raw helper anywhere you like, including the worker.

In few words, you have discovered that this module is SQL Injection safe by default.

[WebReflection]

Also, to be honest, this would just work:

await query`CREATE TABLE IF NOT EXISTS TEST54 (Property text not null,Value text not null)`;

so you have all options in here.

[javieranton-zz]

So I debugged and found that all doesn't parse arguments. I'm not a js wizard so can't say what is wrong. What I know is that if I replace index.js's code with this it works:

//for query `SELECT * FROM ${'todos'} WHERE value = '${'a'}'`
                       all(t){ 
                            console.log(t);
                            console.log(arguments);
                            //0: (3) ["SELECT * FROM ", " WHERE value = ", "", raw: Array(3)]
                            //1: "todos"
                            //2: "a"
                            let finalQuery = arguments[0][0];
                            for(let iter = 1; iter< arguments[0].length; iter++){
                                finalQuery += arguments[iter];
                                finalQuery += arguments[0][iter];
                            }
                            console.log("final query: "+finalQuery);
                            return u([finalQuery]);
                        }

Hope this helps, feel free to close

[javieranton-zz]

Am I right in thinking that sqlite-tag is only meant to be used server-side?

[WebReflection]

There is a live demo and Code Pen demo in the README, so you're wrong?

[javieranton-zz]

I managed to get it working. It's pretty disappointing to see how IndexedDB isn't really reliable, at least not yet. Hope it will improve with time. Thanks for the help

[WebReflection]

Glad you managed! It made me realize the raw utility somehow should work via worker too … I’ll think about it, and find a way to make it happen.

[javieranton-zz]

Awesome :-D

[WebReflection]

@javieranton-zz the raw utility is now available via Worker instances too. This should simplify your code, providing 1:1 API between initializations.

[javieranton-zz]

@WebReflection Wow, so quick, thank you!

Just tested and getting the error plain is not defined coming from
https://github.com/WebReflection/sqlite-worker/blob/main/dist/index.js

Seems plain is not defined before it's used? The test folder works, but could it be that it ins't using the new worker's raw utility?

[WebReflection]

The test folder has been updated to use the raw utility ... that's very weird ...

[WebReflection]

@javieranton-zz gotcha, fixed!

[javieranton-zz]

So cool!

Here is one last idea from me: Currently the templates still need to have their beginning hard-coded.

For example, the following is ok:

let tableName = "TEST54";
let fields = "Property text not null,Value text not null";
await query`CREATE TABLE IF NOT EXISTS ${raw`${tableName}`} (${raw`${fields}`})`;

But the following isn't:

let createQuery = "CREATE TABLE IF NOT EXISTS TEST54 (Property text not null,Value text not null)";
await query`${createQuery}`;

Your tool is extremely valuable to me because I am migrating an app to the web that currently uses sqlite on Android, iOS and Windows. As such I already have all the transactions ready to be fed into the DB.

But because the beginning of each sentence needs to be hard-coded I will have to manually parse each transaction and call its respective js function that calls CREATE/INSERT/SELECT etc

It would be awesome if the entire query could be injected, not just the last part

But I am ever so happy with this as it is, so thanks! I will make sure to mention your repository during release

[WebReflection]

That's a SQL injection hazard, and no way I'll make it default here. Just use raw.

let createQuery = raw`CREATE TABLE IF NOT EXISTS TEST54 (Property text not null,Value text not null)`;
await query`${createQuery}`;

[WebReflection]

Alternatively, just use the template literal tag signature:

await query([
  "CREATE TABLE IF NOT EXISTS TEST54 (Property text not null,Value text not null)"
]);

[WebReflection]

A tag is a function with the following signature:

function tag(template:string[] , ...values:any[]) {
}

If you pass an array with one single entry that's a template with no values.

I hope this helps, yet watch out SQL injections, and never pass values directly, or it'll fail/don't work as expected.

[javieranton-zz]

Oh, I hadn't realized I still was missing raw ':-)

[javieranton-zz]

Hey, I was just thinking...

I can't stop thinking about how much of a game-changer this library is. We finally are able to do all sorts of SQL magic on web apps, with persistence!

One of the problems I've encountered with this module is that when you delete all the records in one table, a CREATE query must precede any further INSERTs. It's like the DB forgets all about any table that ever existed as soon as all of the records within it are gone

To get around this, I am hard coding individual CREATE scripts for each and every table I ever expect to use, and when I get a "no such table" error while attempting an INSERT, I then detect which table the INSERT was targeting and run its CREATE script again before re-attempting the INSERT

Hope I haven't lost you, I'm not very good with words

I am wondering about whether it is possible to integrate some measure into this library so that this work-around isn't necessary. Just sharing this thought

Edit: Another thing I've noticed is that this module suffers of the same problem many sqlite3 ports suffer from. For example, the c sqlite3 port on iOS is very sensitive to consecutive queries and requires abstracting a manual method for separating consecutive DB transactions some good 40 to 200 ms away

This particular implementation requires spacing individual transactions 500ms to be completely sure that they all go through (as tested on a PC and a mobile device).

If this spacing isn't respected individual queries are lost and never processed. Needless to say, this erratic behavior only occurs when firing the transactions in separate workers. When they are all bundled together in a single worker call all is well

I am guessing that this is due to sql.js itself and that nothing can be done from this end.

Android's and general Java sqlite3 implementations also have this issue, but it happens so rarely that it doesn't matter (like 0.0001% of the time). I haven't tested other sqlite3 implementations

[WebReflection]

Hope I haven't lost you, I'm not very good with words

You did ... I understand code better than words ... any example?

P.S. sqlite-worker/table might be the answer ... try that, it creates/modify on the go

[WebReflection]

P.S. multiple workers cannot share the same DB ... don't ever do that, as each DB is stored in memory.

[javieranton-zz]

OK, I think I found some problem.

The following works only via worker, (it doesn't work via init):

let insertQuery = "INSERT OR IGNORE INTO someTable (Property, Value) VALUES ('l','s')";
await query`${raw`${insertQuery}`}`;//init doesn't like this (but worker does)

In order to make it work via init, the query needs to be passed the whole insert, not just a string via raw:

let insertQuery = "INSERT OR IGNORE INTO someTable (Property, Value) VALUES ('l','s')";
await query`${raw`${INSERT OR IGNORE INTO someTable (Property, Value) VALUES ('l','s')}`}`;//both init and worker like this

Edit:
Other variation that also doesn't work using init

let insertQuery = raw`INSERT OR IGNORE INTO someTable (Property, Value) VALUES ('l','s')`;
await query`${insertQuery}`;//init doesn't like this (but worker does)

Edit2:
This works both using init and worker ("template literal tag signature")

let insertQuery = "INSERT OR IGNORE INTO someTable (Property, Value) VALUES ('l','s')";
await query([insertQuery]);//both init and worker like this

[javieranton-zz]

Also, the issue that requires spacing separate transactions apart 500ms remains even after only using init and not workers (this is ok, it likely has nothing to do with this module)

[WebReflection]

I am not sure what you mean, because I've used transaction heavily before and never had this issue.

With sql.js I'd go like:

await query`BEGIN TRANSACTION`;
await Promise.all([
  query`INSERT INTO table VALUES (${Math.random()})`;,
  // all other operations
]);
await query`END TRANSACTION`;

It's true that my experience is with synchronous SQLite there though, but if you could give me an example that requires such delay, I might try to figure out a solution.

[WebReflection]

Btw, the whole purpose of this module and sqlite-tag syntax is to avoid VALUES ('l','s') stuff like that.

Why don't you just use query as it's meant to be used? It looks like you are abusing raw all over, and I am not sure I want to support this pattern at all, specially when values are hard coded in quotes ... that's not how anyone should write SQL, no matter the library used.

[javieranton-zz]

Btw, the whole purpose of this module and sqlite-tag syntax is to avoid VALUES ('l','s') stuff like that.

Why don't you just use query as it's meant to be used? It looks like you are abusing raw all over, and I am not sure I want to support this pattern at all, specially when values are hard coded in quotes ... that's not how anyone should write SQL, no matter the library used.

I only wrote the query like that so that it could be easier to understand. init' and workers's querys accept different params. You yourlsef said that

await query`${raw`${someVar}`}`;

should work, but this is only true for worker. I am satisfied because it appears that "template literal tag signatures" work with init i.e.:

await query([someVar]);

So this is enough for me. But I am just warning you that worker and init behave different.

Edit: the reason why I need to pass whole queries is that the app I am migrating is a repository manager that stores entire transaction histories. Each repository consists of a big file of transactions since the repository was initialized. Each client connecting to the repository needs to be able to "synchronize" their local DB (sqlite) with the repository's transaction history, so the client has to process entire arrays of sql statements. It's mostly an organigram repository, but it also has other functionalities (groupsapp.online)

[javieranton-zz]

I am not sure what you mean, because I've used transaction heavily before and never had this issue.

With sql.js I'd go like:

await query`BEGIN TRANSACTION`;
await Promise.all([
  query`INSERT INTO table VALUES (${Math.random()})`;,
  // all other operations
]);
await query`END TRANSACTION`;

It's true that my experience is with synchronous SQLite there though, but if you could give me an example that requires such delay, I might try to figure out a solution.

I know, I know. Multiple INSERTS should be bundled and executed within the same transaction. But the "issue" I am referring to affects the case where multiple INSERTS are executed consecutively in different transactions. It's not a big deal, please don't worry about this. I don't think there is anything we can do anyway since this affects sqlite3 in multiple platforms (except Android and Java and perhaps others where sqlite3 seems to have some sort of first class citizen status)

Edit: An example that requires such delay would be running the very same code you wrote in a loop for 100 times. If you check how many times values have actually been inserted you will see that it is less than 100. Then if you try again but this time insert a delay between each iteration, you will get 100 times