Converting object to an encodable object failed: _LinkedHashMap len:7 about cbor

[shamblett]

Sorry I'm not clear on what you are doing, so getDecodedData() returns the json above which is correct, then you say 'to which I apply utf8Encoder ', which Utf8Encoder? Do you mean you call strToByteString?Could you post your code?

[mohitpubnub]

getDecodedData() is not returning String! Those are List
In order to convert these lists I am manually decoding the values.
like

import 'dart:convert' as convert;
var decoder = convert.Utf8Decoder(allowMalformed: true);
data.map(
            (key, value) => MapEntry( decoder.convert(key), parse(value)));
parse(dynamic){
// checking structure of data and applying decode accordingly in nested loop
return decodedvalue;
}

So The decoder is accepting List which is valid for all other data except those two string values u-3352055 and s-1707983

[mohitpubnub]

i.e
If I convert very first key [118]. using decoder decoder.convert([118]) it returns 'v' which is my key

So such manual work I am doing because decodedToJSON() is failing with this data.

My poc is:

import 'dart:convert' as convert;
import 'dart:convert';
import 'package:cbor/cbor.dart' as cbor;
import 'package:typed_data/typed_data.dart' as typed;

dynamic decode(String token) {
  var padding = '';
  if (token.length % 4 == 3) {
    padding = '=';
  } else if (token.length % 4 == 2) {
    padding = '==';
  }
  String clean = token.replaceAll('_', '/').replaceAll('-', '+') + padding;
  var payload = convert.base64Decode(clean);
  final inst = cbor.Cbor();
  inst.decodeFromList(payload);
  //  var json = inst.decodedToJSON();
  var decoder = convert.Utf8Decoder(allowMalformed: true);
  var res = inst.getDecodedData();
  var actual = res[0];
  var newMap = actual.map(
            (key, value) => MapEntry( decoder.convert(key), parse(value)));
  print(inst.decodedPrettyPrint());
  // print(json);
}

parse(dynamic data) {
  var decoder = convert.Utf8Decoder(allowMalformed: true);
  var asciEncoder = AsciiEncoder();
  if (data is Map) {
    data.map((key, value) => MapEntry(decoder.convert(key), parse(value)));
    return data;
  } 
  else {
    if(data is typed.Uint8Buffer) return data;
    return decoder.convert(data);
  }
}

You can use the token, i shared in first post. To reproduce the issue exactly

[shamblett]

OK I see what you mean now, I'll have look at this.

[shamblett]

If you dump the base64 decoded payload you get this sub-sequence

... 67, 117, 115, 114, 161, 105, 117, 45, 51, 51, 53, 50, 48, 53, 53, 15, 67, 115, 112, 99 ...

This corresponds to the decoded sequence

[117, 115, 114]: {u-3352055: 15}, [115, 112, 99]

As you can see the sequence 161,105,117,45,51,51,53,50,48,53,53,15 is invalidly coded CBOR. If you enter the whole of the base64decoded payload into cbor.me you get

16 # unsigned(22)


##### 132 unused bytes after the end of the data item:

65 11 65 11 26 93 12 95 22 67 11 11 10 10 67 11 10 11 16 68 99 10 97 11 16 67 10 11 11 16 67 11 11 11 16 10 11 45 51 51 53 50 48 53 53 15 67 11 11 99 16 10 11 45 49 55 48 55 57 56 51 24 31 67 11 97 11 16 68 99 10 97 11 16 67 10 11 11 16 67 11 11 11 16 67 11 11 99 16 68 10 10 11 97 16 67 11 10 10 88 32 21 13 17 97 50 16 19 20 10 18 12 20 11 10 76 28 21 52 16 25 12 97 80 12 20 18 68 15 19 96 58

Which is saying its an CBOR sequence invalid, your base64 decoded sequence seems to be

167, 65, 118, 2, 65, 116, 26, 93, 128, 95, 226, 67, 116, 116, 108, 10, 67, 114, 101, 115, 164, 68, 99, 104, 97, 110, 160, 67, 103, 114, 112, 160, 67, 117, 115, 114, 161, 105, 117, 45, 51, 51, 53, 50, 48, 53, 53, 15, 67, 115, 112, 99, 161, 105, 115, 45, 49, 55, 48, 55, 57, 56, 51, 24, 31, 67, 112, 97, 116, 164, 68, 99, 104, 97, 110, 160, 67, 103, 114, 112, 160, 67, 117, 115, 114, 160, 67, 115, 112, 99, 160, 68, 109, 101, 116, 97, 160, 67, 115, 105, 103, 88, 32, 218, 134, 179, 97, 50, 16, 193, 207, 102, 186, 122, 206, 117, 106, 76, 28, 215, 52, 16, 3, 255, 125, 97, 80, 125, 201, 185, 68, 150, 19, 96, 58

The reason the client seems to parse most of this is because it makes sense out of what it can as it goes along.

[mohitpubnub]

@shamblett Thank You for your efforts!!!
I am just wondering how it's invalidly coded CBOR? Because the same token is being used by other colleagues who are decoding it in different languages (javascript, C# - The code which I have gone through) and cbor libraries are decoding it without any error/exception.
This token is the one which is used in unit tests to check decoding mechanism.. It's working well since long time!!!

[shamblett]

Ok I think what you need to do is decode the token on your other test platforms and see if that decodes on cbor.me also compare it with the decoded bas64 above

[shamblett]

Putting together a simple c# console app on windows to decode the supplied token

using System;

namespace CborTokenDecode
{
    static class Program
    {
        static void Main(string[] args)
        {
            string base64Encoded = "p0F2AkF0Gl2AX-JDdHRsCkNyZXOkRGNoYW6gQ2dycKBDdXNyoWl1LTMzNTIwNTUPQ3NwY6Fpcy0xNzA3OTgzGB9DcGF0pERjaGFuoENncnCgQ3VzcqBDc3BjoERtZXRhoENzaWdYINqGs2EyEMHPZrp6znVqTBzXNBAD_31hUH3JuUSWE2A6";
            string base64Decoded = "";
            byte[] data = null;
            try
            {
                data = Convert.FromBase64String(base64Encoded);
            }
            catch (System.FormatException e)
            {
                Console.WriteLine(e);
                return;
            }

            base64Decoded = System.Text.Encoding.ASCII.GetString(data);
            Console.Write(base64Decoded);
        }
    }
}

gives

System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64String(String s)
   at CborTokenDecode.Program.Main(String[] args) in C:\dev\c#\CborTokenDecode\Program.cs:line 14

C:\dev\c#\CborTokenDecode\bin\Debug\netcoreapp3.1\CborTokenDecode.exe (process 19536) exited with code 0.
Press any key to close this window . . .

[mohitpubnub]

@shamblett It needs some padding things just like I did in my POC code:

  var padding = '';
  if (token.length % 4 == 3) {
    padding = '=';
  } else if (token.length % 4 == 2) {
    padding = '==';
  }
  String clean = token.replaceAll('_', '/').replaceAll('-', '+') + padding;

Still not sure it's only reason!
Right now, I am trying with different tokens(which works in various plateforms)... getting valid toke is eating up my time so expect delay till I come up with final conclusion.

[shamblett]

Missed this thanks. Converting your cleaned token in VS C# now gives the following base64 decoded byte sequence

A7,41,76,02,41,74,1A,5D,80,5F,E2,43,74,74,6C,0A,43,72,65,73,A4,44,63,68,61,6E,A0,43,67,72,70,A0,43,75,73,72,A1,69,75,2D,33,33,35,32,30,35,35,0F,43,73,70,63,A1,69,73,2D,31,37,30,37,39,38,33,18,1F,43,70,61,74,A4,44,63,68,61,6E,A0,43,67,72,70,A0,43,75,73,72,A0,43,73,70,63,A0,44,6D,65,74,61,A0,43,73,69,67,58,20,DA,86,B3,61,32,10,C1,CF,66,BA,7A,CE,75,6A,4C,1C,D7,34,10,03,FF,7D,61,50,7D,C9,B9,44,96,13,60,3A

which pasting into cbor.me gives

{h'76': 2, h'74': 1568694242, h'74746C': 10, h'726573': {h'6368616E': {}, h'677270': {}, h'757372': {"u-3352055": 15}, h'737063': {"s-1707983": 31}}, h'706174': {h'6368616E': {}, h'677270': {}, h'757372': {}, h'737063': {}}, h'6D657461': {}, h'736967': h'DA86B3613210C1CF66BA7ACE756A4C1CD7341003FF7D61507DC9B9449613603A'}

as you can see the string faults("u-3352055") are still present.

It looks maybe this sequence

A1                                # map(1)
         69                             # text(9)
            752D33333532303535          # "u-3352055"
         0F 

from cbor.me the 9 text characters are not UTF-8.

[mohitpubnub]

@shamblett : Thank You for your help. I'm getting valid results from other sample tokens.
Though decodedToJSON() is not helpful as it encounters exception (may be due to complexity of object.) So I am manually iterating through map returned from getDecodedData(). And decoding all 'Key' of Maps (even in nested maps also). using Utf8Decoder()

I suspect json conversion failure is due to nature of data which is encoded in this token.
So may be we can close this issue. thinking that cbor library is doing it's job correctly here.

Thanks again for prompt response and help