Bug: Non-UTF-8 strings return error code 400 about surrealdb

[bladepoint6969]

It doesn't look like any escape character is valid other than quotes. This makes it impossible to send valid JSON

For example, posting the following request body returns an error:

{
	"poster": "user:k29aio2dilsyqvd6scpg",
	"title": "Hello quotes",
	"body": "do \" work?\nDo newlines?"
}
{
	"code": 400,
	"details": "Request problems detected",
	"description": "There is a problem with your request. Refer to the documentation for further information.",
	"information": "The request body contains invalid data"
}

Meanwhile, deliberately posting invalid JSON "works":

{
	"poster": "user:k29aio2dilsyqvd6scpg",
	"title": "Hello quotes",
	"body": "do \" work?
Do newlines?"
}
[
	{
		"time": "72.864µs",
		"status": "OK",
		"result": [
			{
				"body": "do \" work?\nDo newlines?",
				"id": "post:2phtv4rag4k5jf51yths",
				"poster": "user:k29aio2dilsyqvd6scpg",
				"title": "Hello quotes"
			}
		]
	}
]

[LeoDog896]

This occurs in both surrealdb.deno and surrealdb.js

[tobiemh]

Hi @finnbear , technically that's escaping the second " no? So the string doesn't ever complete.

The following queries do work however.

SELECT * FROM '\'';
SELECT * FROM "\"";

However, if there is an attempt to escape a character in a string, and the string is not defined with that character, then it will fail.

So strings defined using ' marks will allow \' escaping, and strings defined using " marks will allow \" escaping.

@finnbear, @rushmorem the issue with allowing non-UTF-8 characters in strings, comes when we serialise these strings as a database key (either as a record id, or as a value in an index).

When serialising we end the string with a 0 u8, and when deserialising, we look for a '\u{0}' char...
https://github.com/surrealdb/storekey/blob/626e02f5940503a68bb665e730a80086d111effe/src/encode.rs#L418-L422
https://github.com/surrealdb/storekey/blob/626e02f5940503a68bb665e730a80086d111effe/src/decode.rs#L241-L265

Therefore if we allowed these characters in a string, we would have problems. This issue definitely needs more thought!

[finnbear]

I think the test case can be reduced to

> select * from "\";
{"code":400,"details":"Request problems detected","description":"There is a problem with your request. Refer to the documentation for further information.","information":"There was a problem with the database: Parse error on line 1 at character 0 when parsing 'select * from \"\\\";'"}

Not sure what's going on, but seems like SurrealDB doesn't like \ in the query string.

[finnbear]

Hi @finnbear , technically that's escaping the second " no? So the string doesn't ever complete.

You might be right.

However, the following (which, by your logic, should escape to a literal \, right?) also fails to parse.

> select * from "\\"
{"code":400,"details":"Request problems detected","description":"There is a problem with your request. Refer to the documentation for further information.","information":"There was a problem with the database: Parse error on line 1 at character 0 when parsing 'select * from \"\\\\\"'"}

[tobiemh]

That falls under this part...

So strings defined using ' marks will allow \' escaping, and strings defined using " marks will allow \" escaping.

That's a good point, you currently can't put a \ character in a string. This logic happens here...

surrealdb/lib/src/sql/strand.rs

Lines 90 to 102 in 04831b1

	fn strand_single(i: &str) -> IResult<&str, String> {
	let (i, _) = tag(SINGLE)(i)?;
	let (i, v) = alt((escaped(is_not(SINGLE_ESC), '\\', one_of(SINGLE)), tag("")))(i)?;
	let (i, _) = tag(SINGLE)(i)?;
	Ok((i, String::from(v).replace(SINGLE_ESC, SINGLE)))
	}
	fn strand_double(i: &str) -> IResult<&str, String> {
	let (i, _) = tag(DOUBLE)(i)?;
	let (i, v) = alt((escaped(is_not(DOUBLE_ESC), '\\', one_of(DOUBLE)), tag("")))(i)?;
	let (i, _) = tag(DOUBLE)(i)?;
	Ok((i, String::from(v).replace(DOUBLE_ESC, DOUBLE)))
	}