McJsonEscapeString (overzealous) about mcjson

[totyaxy]

Hi, new information, I searched and found these two functions:

https://www.freepascal.org/docs-html/fcl/fpjson/stringtojsonstring.html
https://www.freepascal.org/docs-html/fcl/fpjson/jsonstringtostring.html

These work exactly as I want and only convert the characters you really need, leaving the json completely human readable and editable (for the translators, in my case). It doesn't even convert the "/" character, or only if you specifically ask it to, see: strict parameter.

But since your function library supports several programming environments, you obviously cannot solve it with fpc functions. But this is a good example of the fact that only really necessary conversions are needed, the official freepascal function shows this.

Thank you!

[hydrobyte]

I think I can improve McJsonEscapeString and McJsonUnEscapeString with this Strict parameter.

[totyaxy]

Thank you!

For me, it would be best if it worked exactly like the two functions from the fpjson unit.
So there should be a (easiest) level (json human read/write level) where you only work with escape as:

\ " #8 #9 #10 #12 #13

If this is the Strict:= false; then for me its okay.

Just theory (preserve the capabilities of your code):

type TMCJsonEscapeLevels=(elNormal, elStrict, elForce)

Normal level escape see above.
Strict level escape (same as with above fpc unit), include it: /
Force level escape, when escape almost all (as works now your library).

[hydrobyte]

Hi,

Just checking:

• elNormal: \ " #8 #9 #10 #12 #13 <-> \\ \" \b \t \n \f \r
• elStrict: \ " / #8 #9 #10 #12 #13 <-> \\ \" \/ \b \t \n \f \r
• elForce: \ " / #8 #9 #10 #12 #13 #u000X <-> \\ \" \/ \b \t \n \f \r \u000X

I think I can do this using a new constructor as a way to pass a TMCJsonEscapeLevels parameter as RAII.

Also, I think I can automatically use McJsonEscapeString and McJsonUnEscapeString with AsString get/set in order to improve this functionality, as you requested before.

I'll try.

[totyaxy]

Thanks, that sounds good, just a note, these levels refer to the escape function, unescape can always be completely (force) like fpjson.JSONStringToString works, or it can be controlled, for example:

type TMCJsonEscapeLevels=(elNormal, elStrict, elForce);

function McJsonEscapeString   (const aStr: string; const aStrict: TMCJsonEscapeLevels = elNormal): string;
function McJsonUnEscapeString (const aStr: string; const aStrict: TMCJsonEscapeLevels = elForce): string;

Naturally, if it will be automatic, then for example:

type TMCJsonEscapeLevels=(elNone, elNormal, elStrict, elForce);

property
  AutoEscape: TMCJsonEscapeLevels read FAutoEscape write FAutoEscape default elNormal;
  AutoUnEscape: TMCJsonEscapeLevels read FAutoUnEscape write FAutoUnEscape default elForce;

elNone for the speed tests... or for the custom usage (like as works now).

[hydrobyte]

Hi,

Right, I need this elNone level type as a backward compatibility (the user is responsible to escape strings).

Also, since UnEscape always do a complete verification, I'll leave McJsonUnEscapeString as it is now.

The AutoEscape property idea will be at first a new constructor, in order to permit these kind of examples (this make sense for you?):

var
  JNone, JNormal, JJson, JForce: TMcJsonItem;
begin
  JNone := TMcJsonItem.Create;              //elNone is the default
  JNone.AsJSON := '{"path":"C:\\Windows"}'; //or use McJsonEscapeString
  IsEqual := ( JNone['path'].AsString = 'C:\\Windows' );

  JNormal := TMcJsonItem.Create(elNormal);
  JNormal.AsJSON := '{"path":"C:\Windows"}'; //internally = '{"path":"C:\\Windows"}'
  IsEqual := ( JNormal['path'].AsString = 'C:\Windows' );

  JJson:= TMcJsonItem.Create(elNormal); 
  JJson.AsJSON := '{"json":"{"i": 123}"}'; //internally = '{"json":"{\"i\":123}"}'
  IsEqual := ( JJson['json'].AsString = '{"i":123}' );

  JForce:= TMcJsonItem.Create(elForce);
  JForce.AsJSON := '{"force":"a#13bç"}'; //internally = '{"force":"a\nb\u00E7"}' considering CP1252
  IsEqual := ( JForce['json'].AsString = 'a#13bç' );
end;

AsJSON and AsString Getters and Setters are the focus here.

One problem about "auto escaping" is that all the chars position might change when there are insertions, so the error messages about parsing might have wrong at pos "%s" compared to the original string.

What I don't get yet is how see escaped strings within levels {elNormal, elStrict and elForce};

Revising all this subject I've realized that McJsonUnEscapeString is ignoring "invisible" chars like #13. I think the right behavior might be:

IsEqual := ( McJsonUnEscapeString('\b\t\r\f\n') = #8#9#10#12#13 );

[totyaxy]

Thank you for your work, I read it about 3 times, but I still don't fully understand it. :)

I wrote earlier that you need escape/unescape ONLY for storage (file). I understand that immediately storing the escape values in the structure can be problematic, but you see this better.

As I wrote before, what if escape/unescape was activated only when saving/loading?
Then, during saving, strings of type string would be escaped. When loading, the reverse would happen. In this way, the internal structure would basically not change(!), and later modifications could be simpler. (I don't know if your library currently knows whether a value is a string or not. Maybe that's why you linked it to AsString as getters/setters?)

But if you think that what you wrote can be solved completely safely, for example: "JForce:= TMcJsonItem.Create(elNormal);" with AsString getter/setter, that makes perfect sense to me. However you solve it, I think I'll be able to use it. If you release a test version, I will try it. Thank you!

[hydrobyte]

Hi,
It is confusing, indeed (I was looking fpjson.pp).

I was thinking beyond just escape/unescape when saving/loading.

I'm trying to improve AsJSON and AsString getters and setters, what will cover saving/loading tasks too. Internally, all values are stored as strings (to keep the class simpler).

To sum up, I'm trying to make users' experience easier, like in the path example

Also with:

  JJson:= TMcJsonItem.Create(elNormal); 
  JJson.AsJSON := '{"json":"{"i": 123}"}'; //internally = '{"json":"{\"i\":123}"}'
  IsEqual := ( JJson['json'].AsString = '{"i":123}' );

This example will also work if you are reading an unescaped fle:

  JJson:= TMcJsonItem.Create(elNormal); 
  JJson.LoadFromFile('unescaped.json'); // UTF-8 content like {"json":"{"i": 123}"}
  IsEqual := ( JJson['json'].AsString = '{"i":123}' );

But, my main concern is that an unescaped JSON file with content like {"json":"{"i": 123}"} or {"path":"C:\Windows"} is also a INVALID JSON file. McJson will not write invalid files, so why it need to read them?

I think this leads me to focus efforts only on AsString like this:

  JNormal := TMcJsonItem.Create(elNormal);
  JNormal.Add('path1').AsString := 'C:\Windows'; //auto-escaped to '{"path1":"C:\\Windows"}'
  // or
  JNormal.S['path2'] := 'C:\Users'; //auto-escaped to '{"path2":"C:\\Users"}'
  IsEqual := ( JNormal['path1'].AsString = 'C:\Windows' );
  IsEqual := ( JNormal.S['path2'] = 'C:\Users' );

Could you share again an example of what file are you dealing with and how do you want to use McJson to read it?

[totyaxy]

Since your json library is suitable for handling any json file, I don't really understand the request, but I'll show you an example naturally. By giving suggestions, your library develops and learns more and more.

In the active project, I generate a json file completely from scratch to make it transparent and modifiable. But this is only file-level data (write/read) because I basically store (and works) the data in an internal structure:

type
   TCustomLanguageItemList = specialize TFPGMap<integer, TLanguageItem>;

The reason for this is, for example, that it is necessary to sort them in ascending order by key (I didn't see a sort option in your library).

Simplified (!) examples as you ask.

About 20000 keys:

{
  "200": {
    "OrgSrc": "Value",
    "ModSrc": "Value",
    "OrgDst": "Value",
    "ModDst": "Value",
    "Info": ""}
}

  Src := JSONStringToString(aJsonItem.Values[ltSrc.NameStr].AsString);
  Dst := JSONStringToString(aJsonItem.Values[ltDst.NameStr].AsString);

  aJsonItem.Values[ltSrc.NameStr].AsString := StringToJSONString(Src);
  aJsonItem.Values[ltDst.NameStr].AsString := StringToJSONString(Dst);

Value := JSONStringToString(JsonItem.Values[lt.NameStr].AsString);

JsonID.Add(lt.NameStr).AsString := StringToJSONString(Self.Data[i].GetItem(lt));

[hydrobyte]

Please, edit your previous post, focus into the JSON tiny example (related to a 20,000 keys file), in order to see the necessity of using JSONStringToString.

[totyaxy]

As you requested.

[hydrobyte]

Hi,
Your example doesn't show how JSONStringToString is necessary.

{
  "200": {
    "OrgSrc": "Value",
    "ModSrc": "Value",
    "OrgDst": "Value",
    "ModDst": "Value",
    "Info": ""}
}

But it is OK. I'm finishing a version that MsJsonEscapeString has a new parameter like this: TJEscapeType = (jetNormal, jetStrict, jetUnicode, jetNone).
A second commit will permit code like this:

  JNormal := TMcJsonItem.Create(jetNormal);
  JNormal.Add('path1').AsString := 'C:\Windows'; //auto-escaped to '{"path1":"C:\\Windows"}'
  JNormal.S['path2'] := '"C:\New Users"'; //auto-escaped to '{"path2":"\"C:\\New Users\""}'

  IsEqual := ( JNormal['path1'].AsString = 'C:\Windows' );
  IsEqual := ( JNormal.S['path2']        = '"C:\New Users"' );

[totyaxy]

Hi, about 20,000 keys with texts, a lot of json forbidden characters are included, another project has many more keys and texts. I don't think I knew what you were asking exactly. If I don't use text-json conversion, the text will surely be wrong somewhere, which is why I used the json text conversion procedure.
I replaced all the StringToJSONString/JSONStringToString functions with yours (McJsonEscapeString (jetNormal) / McJsonUnEscapeString) and it works fine, thank you!

ps.: Just a note, I've been getting a message from your library when compiling for quite some time:
pascal McJSON.pas(183,14) Warning: An inherited method is hidden by "ToString:AnsiString;"

[totyaxy]

Just a note again, as I know you pay attention to the speed, as I measure the time of the entire processing, it quickly became apparent that your json-text conversion functions are a little slower than those of the fpsjon unit, the total processing time (nothing changed apart from these functions) fpjson : 0.58 secs, with MCJson 0.87 secs.

[hydrobyte]

Hi,

Thanks for the reply.

In order to fix the warning related to ToString in Lazarus you can change it to override. I can't do this because it will fire an error with older versions of Delphi. But thanks anyway. I'm also a "no warning" guy. ;)

About the performance, I'll take a second look at fpjson.pp functions and try to figure it out some point of optimization, but McJson*EscapeString functions are quite simple indeed compared to fpjson.pp.

Regards,

[totyaxy]

Thank you, its works for me:

function ToString: string; {$IFNDEF FPC} overload; {$ELSE} override; {$ENDIF} // Supress fpc compiler warning...

Thanks for everything!

[hydrobyte]

Hi!

I was avoiding IFNDEF but this is a special case.

I'll revise my goals here.

Thanks.