ohler55 / oj Goto Github PK
View Code? Open in Web Editor NEWOptimized JSON
Home Page: http://www.ohler.com/oj
License: MIT License
Optimized JSON
Home Page: http://www.ohler.com/oj
License: MIT License
Most other JSON decoders strip out //...\n
and /*...*/
comments before processing. oj is one of the few that does not. The JSON spec does not require decoders to do so, but it does offer the option. It would be nice to have consistency across all decoders.
The standard 'json' gem supports the :symbolize_names => true, option to symbolize all hash keys to ruby symbols upon parsing. When there is large number of deserialized JSON objects with similar keys, this could could potentially save considerable amount of memory and is faster for key comparisons.
As such I would suggest Oj to add the support for this option under the :compat mode, both for the reasons above and making :compat mode closer to the 'json' gem.
The following was witnessed on an old Fedora 10 box:
$ cat gem_make.out
/opt/ruby-enterprise-1.8.7-2011.03/bin/ruby extconf.rb
>>>>> Creating Makefile for ree version 1.8.7 <<<<<
creating Makefile
make
gcc -I. -I/opt/local/include -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64 -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ree -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DREE_RUBY -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fPIC -g -O2 -c dump.c
dump.c: In function ‘dump_float’:
dump.c:410: warning: floating constant exceeds range of ‘double’
dump.c:413: warning: floating constant exceeds range of ‘double’
gcc -I. -I/opt/local/include -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/i686-linux -I. -D_FILE_OFFSET_BITS=64 -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ree -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DREE_RUBY -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fPIC -g -O2 -c load.c
load.c: In function ‘read_num’:
load.c:669: error: ‘INFINITY’ undeclared (first use in this function)
load.c:669: error: (Each undeclared identifier is reported only once
load.c:669: error: for each function it appears in.)
make: *** [load.o] Error 1
I added the following to the top of load.c
, and was able to get things make and almost all of the tests to pass:
//Workaround:
#ifndef INFINITY
#define INFINITY (1.0/0.0)
#endif
The only failing test:
$ ruby -wW1 tests.rb
Loaded suite tests
Started
.......................................................F.........
Finished in 0.019552 seconds.
1) Failure:
test_time_compat(Juice) [tests.rb:249]:
<"1325775487.000000000"> expected but was
<"1325825887.000000000">.
65 tests, 100 assertions, 1 failures, 0 errors
I'm not sure if that is safe to use everywhere, but it's better than not compiling, right? I'll send a pull request shortly.
According to RubyDoc, SyntaxError
is "raised when encountering Ruby code with an invalid syntax."
In my opinion, it should not be raised when attempting to parse JSON. SyntaxError
(and other descendants of ScriptError
) are typically only raised by the Ruby interpreter. Specifically, SyntaxError
doesn't descend from StandardError
, making it difficult to catch JSON parsing error while still raising Ruby syntax errors that may occur within the same code block.
I'd encourage you to define a custom Oj::ParseError
class that inherits from StandardError
. I believe this is what the other JSON libraries do.
I haven't had a chance to really dig through to see if UTF-8 characters are always \uXXXX
escaped or not, but it would be nice to have the option to allow them to go right into the output stream unescaped (except for characters which must be escaped per the JSON spec).
The :ascii_only
option could maybe default to whatever the behavior is today.
What do you think?
The full details are on Travis:
http://travis-ci.org/#!/simeonwillbanks/nerd_quiz/jobs/1525709
Thanks for Oj!
Simeon
In my production application, I ran some benchmarks to find the fastest JSON rendering engine. I'm using RABL to render.
I ran the same test, for 6 different templates (our most used ones), with each record from our DB, 10 times each.
I ran once for yajl and once for oj. I did not perform any configuration for either.
I rant the tests on a staging machine (same config as production), from a machine in the same EC2 zone.
Results were consistent when run multiple times.
Rails was running 3.2.9, Ruby 1.9.3, Apache.
I found that yajl was faster. I wasn't expecting it to be. Is this possible/reasonable in some circumstances (every app is different)? Is there anything I else I might have missed or might want to consider before moving ahead?
Thanks.
example program:
require 'oj'
require 'yajl'
require 'json'
require 'bigdecimal'
stuff = [
BigDecimal.new('10'),
Date.today,
Time.now,
DateTime.now
]
puts Yajl.dump stuff
puts JSON.dump stuff
puts Oj.dump stuff, :mode => :compat
output:
["0.1E2","2012-06-05","2012-06-05 14:56:29 -0700","2012-06-05T14:56:29-07:00"]
["0.1E2","2012-06-05","2012-06-05 14:56:29 -0700","2012-06-05T14:56:29-07:00"]
[null,null,1338933389.857938566,null]
why not?
>> Oj.load("149235374128953450.0")
SyntaxError: invalid format, extra characters at line 1, column 19 [load.c:935]
from (irb):8:in `load'
from (irb):8
from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
>> Oj.load("[1492353741289534500000000.0]")
SyntaxError: invalid format, expected , or ] while in an array at line 1, column 27 [load.c:570]
from (irb):9:in `load'
from (irb):9
from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
Is that valide behavior? Do you need more input?
I have an issue loading JSON object with Oj in a set of machines. Some examples:
1.9.3p0 :016 > res=Oj.dump({:a=>'b'})
=> "{\":a\":\"b\"}"
1.9.3p0 :015 > Oj.load(res)
SystemStackError: JSON is too deeply nested
1.9.3p0 :015 > Oj.load('{}')
SystemStackError: JSON is too deeply nested
I'm have the same Ruby version 1.9.3p0 in both machines. The working machines are 32 bits (Ubuntu 10.04) and the failing machines are 64 bits (12.10 and 12.04).
Some suggestion?
Hey There,
Was trying to use the T gem which pulls in oj as a dependency. Trying to install OJ with sudo gem install oj
fails in this manner (running OS X 10.7.3 with system ruby 1.8.7):
└(~/src/puppetlabs-training)▷ sudo gem install oj
Building native extensions. This could take a while...
ERROR: Error installing oj:
ERROR: Failed to build gem native extension.
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
>>>>> Creating Makefile for ruby version 1.8.7 <<<<<
creating Makefile
make
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE -fno-common -pipe -fno-common -c cache.c
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE -fno-common -pipe -fno-common -c cache8.c
gcc -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin11.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -DHAS_TOP_LEVEL_ST_H=1 -DRUBY_TYPE=ruby -DHAS_ENCODING_SUPPORT=0 -DRUBY_VERSION=1.8.7 -DHAS_IVAR_HELPERS=0 -DRUBY_VERSION_MICRO=7 -DRUBY_RUBY -DRUBY_VERSION_MAJOR=1 -DHAS_PROC_WITH_BLOCK=0 -DHAS_RSTRUCT=1 -DHAS_RB_TIME_TIMESPEC=0 -DRUBY_VERSION_MINOR=8 -DHAS_NANO_TIME=0 -Wall -fno-common -arch i386 -arch x86_64 -g -Os -pipe -fno-common -DENABLE_DTRACE -fno-common -pipe -fno-common -c dump.c
In file included from dump.c:38:
oj.h:84: error: expected specifier-qualifier-list before ‘uint8_t’
oj.h:119: error: expected specifier-qualifier-list before ‘uint8_t’
dump.c:58: error: expected specifier-qualifier-list before ‘uint32_t’
dump.c:166: error: expected ‘)’ before ‘num’
dump.c: In function ‘dump_unicode’:
dump.c:228: error: ‘uint32_t’ undeclared (first use in this function)
dump.c:228: error: (Each undeclared identifier is reported only once
dump.c:228: error: for each function it appears in.)
dump.c:228: error: expected ‘;’ before ‘code’
dump.c:229: error: ‘uint8_t’ undeclared (first use in this function)
dump.c:229: error: expected ‘;’ before ‘b’
dump.c:232: error: ‘b’ undeclared (first use in this function)
dump.c:234: error: ‘code’ undeclared (first use in this function)
dump.c:252: error: expected expression before ‘)’ token
dump.c:259: error: expected ‘;’ before ‘c1’
dump.c:262: error: ‘c1’ undeclared (first use in this function)
dump.c: In function ‘dump_array’:
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:607: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:609: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:611: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:615: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:632: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:634: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:636: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:640: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘hash_cb_strict’:
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:670: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:672: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:674: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:678: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:686: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:688: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:691: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:693: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_compat’:
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:719: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:721: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:723: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:727: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:749: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:751: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:754: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:756: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_object’:
dump.c:796: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c:798: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_hash’:
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:881: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:883: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:885: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:889: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘dump_obj_to_json’:
dump.c:1301: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_leaf_str’:
dump.c:1355: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1364: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_fixnum’:
dump.c:1371: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1384: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_float’:
dump.c:1391: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1400: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf’:
dump.c:1485: error: ‘struct _Leaf’ has no member named ‘type’
dump.c:1511: error: ‘struct _Leaf’ has no member named ‘type’
dump.c: In function ‘dump_leaf_to_json’:
dump.c:1523: error: ‘struct _Out’ has no member named ‘hash_cnt’
In file included from dump.c:38:
oj.h:84: error: expected specifier-qualifier-list before ‘uint8_t’
oj.h:119: error: expected specifier-qualifier-list before ‘uint8_t’
dump.c:58: error: expected specifier-qualifier-list before ‘uint32_t’
dump.c:166: error: expected ‘)’ before ‘num’
dump.c: In function ‘dump_unicode’:
dump.c:228: error: ‘uint32_t’ undeclared (first use in this function)
dump.c:228: error: (Each undeclared identifier is reported only once
dump.c:228: error: for each function it appears in.)
dump.c:228: error: expected ‘;’ before ‘code’
dump.c:229: error: ‘uint8_t’ undeclared (first use in this function)
dump.c:229: error: expected ‘;’ before ‘b’
dump.c:232: error: ‘b’ undeclared (first use in this function)
dump.c:234: error: ‘code’ undeclared (first use in this function)
dump.c:252: error: expected expression before ‘)’ token
dump.c:259: error: expected ‘;’ before ‘c1’
dump.c:262: error: ‘c1’ undeclared (first use in this function)
dump.c: In function ‘dump_array’:
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:598: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:607: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:609: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:611: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:615: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:632: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:634: error: ‘struct _DumpOpts’ has no member named ‘array_size’
dump.c:636: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:640: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘hash_cb_strict’:
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:666: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:670: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:672: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:674: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:678: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:682: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:686: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:688: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:691: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:693: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_compat’:
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:715: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:719: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:721: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:723: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:727: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:745: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:749: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:751: error: ‘struct _DumpOpts’ has no member named ‘before_size’
dump.c:754: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c:756: error: ‘struct _DumpOpts’ has no member named ‘after_size’
dump.c: In function ‘hash_cb_object’:
dump.c:796: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c:798: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_hash’:
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:877: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:881: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:883: error: ‘struct _DumpOpts’ has no member named ‘hash_size’
dump.c:885: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c:889: error: ‘struct _DumpOpts’ has no member named ‘indent_size’
dump.c: In function ‘dump_obj_to_json’:
dump.c:1301: error: ‘struct _Out’ has no member named ‘hash_cnt’
dump.c: In function ‘dump_leaf_str’:
dump.c:1355: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1364: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_fixnum’:
dump.c:1371: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1384: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf_float’:
dump.c:1391: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c:1400: error: ‘struct _Leaf’ has no member named ‘value_type’
dump.c: In function ‘dump_leaf’:
dump.c:1485: error: ‘struct _Leaf’ has no member named ‘type’
dump.c:1511: error: ‘struct _Leaf’ has no member named ‘type’
dump.c: In function ‘dump_leaf_to_json’:
dump.c:1523: error: ‘struct _Out’ has no member named ‘hash_cnt’
lipo: can't open input file: /var/tmp//cckBhLRR.out (No such file or directory)
make: *** [dump.o] Error 1
Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/oj-1.2.5 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/oj-1.2.5/ext/oj/gem_make.out
I presume I'll probably need a couple of extra flags to compile on OS X, but I thought I'd file a ticket on it.
Jarmo and I've discussed on this thread at the tcs-ruby group and I toyed with a extconf.rb quickfix for tcs-ruby.
The main issue is that my trickery with RUBY_DESCRIPTION
is messing up your defines/CPPFLAGS from your 'ruby' == type
checks. I also DRY'd up your windows platform checks by using RbConfig::CONFIG['host_os']
.
I don't like the type.delete('tcs-')
nonsense, but I don't have time to look into alternatives like RbConfig::CONFIG['RUBY_BASE_NAME']
which is 1.9-only, not available in JRuby 1.6.8, and I don't know about rubinius.
C:\Users\Jon\Documents\CDev\oj-git>git log -1 --oneline
82e33d5 BigDecimals dumped and loaded in all modes
C:\Users\Jon\Documents\CDev\oj-git>gem build oj.gemspec
Successfully built RubyGem
Name: oj
Version: 1.4.2
File: oj-1.4.2.gem
C:\Users\Jon\Documents\CDev\oj-git>gem i oj-1.4.2.gem
Building native extensions. This could take a while...
Successfully installed oj-1.4.2
1 gem installed
C:\Users\Jon\Documents\CDev\oj-git>gem li oj
*** LOCAL GEMS ***
oj (1.4.2, 1.4.1.a1)
C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:
# Running tests:
....................................F........................E......................
Finished tests in 0.031200s, 2692.3077 tests/s, 3974.3590 assertions/s.
1) Failure:
test_encode(Juice) [test/tests.rb:881]:
<"\u3074\u30FC\u305F\u30FC"> expected but was
<"\xE3\x81\xB4\xE3\x83\xBC\xE3\x81\x9F\xE3\x83\xBC">.
2) Error:
test_range_object(Juice):
ArgumentError: bad value for range
test/tests.rb:880:in `initialize'
test/tests.rb:880:in `load'
test/tests.rb:880:in `dump_and_load'
test/tests.rb:622:in `test_range_object'
84 tests, 124 assertions, 1 failures, 1 errors, 0 skips
C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\test_mimic.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:
# Running tests:
.........E.......
Finished tests in 0.015600s, 1089.7436 tests/s, 1346.1538 assertions/s.
1) Error:
test_load_proc(Mimic):
NotImplementedError: Calling a Proc with a block not supported in this version. Use func() {|x| } syntax instead.
test/test_mimic.rb:79:in `load'
test/test_mimic.rb:79:in `test_load_proc'
17 tests, 21 assertions, 0 failures, 1 errors, 0 skips
C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\test_fast.rb
tcs-ruby 1.9.3p231 (2012-05-25, TCS patched 2012-05-27) [i386-mingw32]
Run options:
# Running tests:
.................................
Finished tests in 0.015600s, 2115.3846 tests/s, 6730.7692 assertions/s.
33 tests, 105 assertions, 0 failures, 0 errors, 0 skips
Oj.default_options[:mode]
# => :object
Oj.load(Oj.dump(Exception.new("oops!"))).message
# => "Exception"
This occurs in ree-1.8.7-2012.02, ruby-1.8.7-p358, and rbx-1.2.4, and and rbx-head. I couldn't get oj to build on jruby so I couldn't check. It doesn't affect ruby-1.9.2-p320, ruby-1.9.3-p194.
Syck had/has a similar bug: ruby/ruby@eb1eee0
Hi,
nothing like an example to describe this issue:
╰─○ irb
>> require 'json'
=> true
>> require 'oj'
=> true
>> Oj.mimic_JSON
TypeError: Generator is not a class
from (irb):3:in `mimic_JSON'
from (irb):3
from /home/fbernier/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `<main>'
>> Oj.mimic_JSON
=> JSON
╰─○ irb
>> require 'oj'
=> true
>> Oj.mimic_JSON
=> JSON
As you can see, if json is already included, it Oj,mimic_JSON thorws an error, but if I call it again, it works. Any clue on this?
Hi!
This code produces segfault:
Oj.load('{":s": 1}', :symbol_keys => true)
My Ruby is ruby-1.9.3-p362
For better compatibility with other JSON libraries, it'd be nice if Oj could be configured to use the a date format option that acted like xmlschema but always treated nsec as zero.
It would be nice if you could implement the possibility do use an other key for creation of custom objects. I think the proper way would be to use the value of JSON.create_id.
https://github.com/ohler55/oj/blob/master/ext/oj/load.c#L479
I'm currently experiencing strange aborts with the latest Oj (1.2.11). i've tracked it down with gdb to fast.c:1121 where xfree(json) is called. the abort trace is as follows:
#0 0x00007ffff6b52b45 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007ffff6b53fc6 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007ffff6b8dfe7 in __libc_message () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007ffff6b93516 in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
#4 0x00007ffff6b9839c in free () from /lib64/libc.so.6
No symbol table info available.
#5 0x00007fffe07a43f2 in doc_open (clas=578516760, str=703305680) at fast.c:1121
json = 0x2a327b10 "\240w2*"
len = 201005
obj = 707934560
given = 1
allocate = 1
#6 0x00007ffff7b162c1 in vm_call_cfunc (me=0x227e3820, blockptr=0x2a1f85f0, recv=<optimized out>, num=1, reg_cfp=0x2a1f85c8, th=<optimized out>) at vm_insnhelper.c:404
val = 0
def = 0x227e3850
cfp = 0x6ae4
#7 vm_call_method (th=<optimized out>, cfp=0x2a1f85c8, num=<optimized out>, blockptr=0x2a1f85f0, flag=<optimized out>, id=<optimized out>, me=0x227e3820, recv=578516760)
at vm_insnhelper.c:534
val = <optimized out>
i'm not sure where the double free is coming from ... i've also tried to disable GC and got the same results.
if you need more information please let me know.
Hi,
I think Oj should respect the default encoding on Ruby 1.9. For example, I'd expect an UTF-8 string when dumping a hash in this context:
[1] pry(main)> puts Encoding.default_external
UTF-8
=> nil
[2] pry(main)> require "oj"
=> true
[3] pry(main)> puts Oj.dump(foo: "éà").encoding
ASCII-8BIT
For some reason, the MultiJSON's continusous integration broke 7 months ago and didn't get fixed until today. At some point over the past 7 month, a new error related to Oj emerged. It may very well be a bug in MultiJSON, not in Oj. I'm having trouble reproducing it locally. I thought you might have an idea about what the issue is. It does seem to be related to Oj, since the JRuby specs, which skip Oj, are passing. Arguably, we should not skip the Oj specs on JRuby, but that's a different issue.
Anyway, let me know if you have any ideas.
I recently wanted to see what code changed in versions 1.3.6 and 1.3.7 using GitHub's Compare View but couldn't easily diff those versions because the releases were not tagged.
It appears that you tagged v1.0.6 but haven't tagged any releases before or since.
When I release gems, I use a rake
task that automatically:
I believe oj
could benefit from a similar automation of the release process that includes tagging.
For example, here's the Compare View for the 3.4.1 release of the twitter
gem. If you want to see a wider range of changes, you can simply change the tags referenced in the URL, for example, here's the Compare View for all the changes between 3.4.0 and 3.5.0.
Should an empty string really cause a ParseError?
I'm currently migrating from Yajl which didn't throw an exception if parsing an empty string and wondered if there is a reason behind not catching empty strings and simply returning?
I'm sure the reason is that ""
is infact invalid json, however from a library perspective it would be nice to see that case caught instead of throwing an exception imo.
Thanks 👍
Attempting install through bundler. Debug info here: https://gist.github.com/4126905
Are you able to reproduce? Please let me know if/how I can help fix this.
Thanks!
I'm trying to replace my use of the json gem with oj, but I need to ensure backwards compatibility in timestamp formats. I'm using the equivalent of Time#xmlschema (e.g. "2012-07-06T17:44:33-04:00"). Oj insists on a Unix timestamp format for Time objects.
I've tried monkey patching a Time#to_json method, with no success. Do I have any other options?
I'm using oj
, multi_json
, and rabl
to spit out json from my rails-api
. The problem I'm having is that in production mode the options are set to this:
{:indent=>0, :circular=>false, :auto_define=>true, :ascii_only=>false, :symbol_keys=>false, :mode=>:compat, :create_id=>"json_class"}
And yet I get this error:
TypeError (In :strict mode all Hash keys must be Strings.)
Which sucks, because it's not actually in strict mode, but compat mode.
I think the problem is here:
https://github.com/ohler55/oj/blob/master/ext/oj/dump.c#L739
I can make Oj.load segfault with the json from https://gist.github.com/2937493. I've also added the terminal output and the osx crash reporter output to that gist.
Anything I can do to help fix this? Would really like some v. fast json serialisation for my project.
i am upgrading my box from 10.04 to 12.04 and in 12.04 i get the following error. 10.04 works as expected. I am bundling these versions:
Installing oj (1.4.5) with native extensions
Installing oj_mimic_json (1.0.0)
rake aborted!
JSON::Ext::Class is not a module
(See full trace by running task with --trace)
require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:14:in
module:Ext'<module:JSON>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json/ext.rb:9:in
<top (required)>'require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/json_pure-1.7.3/lib/json.rb:58:in
module:JSON'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/core_ext/object/to_json.rb:3:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json/encoding.rb:1:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/activesupport-3.0.14/lib/active_support/json.rb:2:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/prototype_helper.rb:2:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:38:in
module:JavaScriptHelper'<module:Helpers>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/javascript_helper.rb:5:in
module:ActionView'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/url_helper.rb:1:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:16:in
module:FormTagHelper'<module:Helpers>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_tag_helper.rb:7:in
module:ActionView'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/form_helper.rb:4:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers/active_model_helper.rb:1:in
require'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers.rb:37:in
module:Helpers'<module:ActionView>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/helpers.rb:3:in
<top (required)>'<class:Base>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/actionpack-3.0.14/lib/action_view/base.rb:160:in
module:ActionView'<top (required)>' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bundler/gems/s3-swf-upload-plugin-f669a2322b7b/lib/s3_swf_upload/view_helpers.rb:171:in
<top (required)>'require' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bundler/gems/s3-swf-upload-plugin-f669a2322b7b/lib/s3_swf_upload.rb:4:in
<top (required)>'require' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:68:in
block (2 levels) in require'each' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:66:in
block in require'each' /usr/lib/ruby/gems/1.9.1/gems/bundler-1.2.1/lib/bundler/runtime.rb:55:in
require'require' /tmp/jenkins/workspace/monorail-1204/config/application.rb:4:in
<top (required)>'require' /tmp/jenkins/workspace/monorail-1204/Rakefile:6:in
<top (required)>'load' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in
load_rakefile'raw_load_rakefile' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:82:in
block in load_rakefile'standard_exception_handling' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:81:in
load_rakefile'block in run' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in
standard_exception_handling'run' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in
<top (required)>'load' /tmp/jenkins/workspace/monorail-1204/vendor/bundle/ruby/1.9.1/bin/rake:23:in
'require 'json'
require 'oj' # json
h = {:url => "http://google.com" }
puts "to_json: " << h.to_json
puts "Oj.dump: " << Oj.dump(h)
outputs:
to_json: {"url":"http://google.com"}
Oj.dump: {":url":"http://google.com"}
How can I get it to work the same as to_json (remove ":" from symbol, not escape \ )
After calling Oj.mimic_JSON, I noticed that the unparse method is not supported.
Should that module function also be redefined by Oj?
Based on this #5 when I try the mimic option I get the following error -
1.9.3-p194 :001 > Oj.mimic_JSON
TypeError: JSON module already exists. Can not mimic. Do not require 'json' before calling mimic_JSON.
This is a Rails 3.2.8 and is using oj 1.3.2
Thanks for your help.
Docs state that the :compat mode:
... will serialize any Object but will check to see if the Object implements a to_hash() or to_json() method. If either exists that method is used for serializing the Object
Then in dump.c:977 there is a test for as_json:
} else if (rb_respond_to(obj, oj_as_json_id)) {
dump_val(rb_funcall(obj, oj_as_json_id, 0), depth, out);
} else if (rb_respond_to(obj, oj_to_json_id)) {
which is undocumented (so not sure if this is cruft or an omission in the docs).
If you then combine this with ActiveSupport which mixes as_json into Object, then to_json will never get called.
Times are encoded incorrectly (invalid JSON) when their integer representation is zero or negative:
require 'oj'
require 'yajl'
require 'json'
require 'bigdecimal'
stuff = [
BigDecimal.new('10'),
Date.new(1970,1,1),
Time.utc(1970,1,1),
DateTime.new(1970,1,1)
]
puts Yajl.dump stuff
puts JSON.dump stuff
puts Oj.dump stuff, :mode => :compat
#=> ["0.1E2","1970-01-01","1970-01-01 00:00:00 UTC","1970-01-01T00:00:00+00:00"]
#=> ["0.1E2","1970-01-01","1970-01-01 00:00:00 UTC","1970-01-01T00:00:00+00:00"]
#=> [0.1E2,"1970-01-01",.000000000,"1970-01-01T00:00:00+00:00"]
Oj shouldn't produce symbols when parsing a JSON message:
res = Oj.dump {:a=>'b'} #=> "{\":a\":\"b\"}"
Oj.load(res) #=> {:a => 'b'}
It's wrong because the format never specifies the symbol type which opens unexpected behaviors but more importantly, symbols are never garbage collected which means that an attacker could easily flood a VM with unused objects.
I'm getting failures on both Win7 32bit and Arch 32bit on different versions of MRI trunk. Last example is from a passing run on 1.9.3p297 on Win7.
~bt
is null in both of the trunk failing cases.
[jon@archee ~]$ uname -a
Linux archee 3.6.2-1-ARCH #1 SMP PREEMPT Sat Oct 13 07:19:40 CEST 2012 i686 GNU/Linux
[jon@archee oj-git]$ gem i oj-1.4.2.a1.gem --user-install
Building native extensions. This could take a while...
Successfully installed oj-1.4.2.a1
1 gem installed
[jon@archee oj-git]$ ruby -v test/tests.rb
ruby 2.0.0dev (2012-09-28 trunk 37049) [i686-linux]
Run options:
# Running tests:
[38/84] Juice#test_exception*** {
"^o":"StandardError",
"~mesg":"A Message",
"~bt":null}
= 0.00 s
1) Failure:
test_exception(Juice) [test/tests.rb:580]:
<["test/tests.rb:570:in `test_exception'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:844:in `run_test'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:1211:in `run'",
"/usr/local/lib/ruby/2.0.0/test/unit/testcase.rb:17:in `run'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:837:in `block in _run_suite'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:829:in `map'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:829:in `_run_suite'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:650:in `block in _run_suites'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:648:in `each'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:648:in `_run_suites'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:789:in `_run_anything'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:978:in `run_tests'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:965:in `block in _run'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:964:in `each'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:964:in `_run'",
"/usr/local/lib/ruby/2.0.0/minitest/unit.rb:953:in `run'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:21:in `run'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:767:in `run'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:364:in `block (2 levels) in autorun'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:27:in `run_once'",
"/usr/local/lib/ruby/2.0.0/test/unit.rb:363:in `block in autorun'"]> expected but was
<nil>.
Finished tests in 0.063030s, 1332.7092 tests/s, 2030.7949 assertions/s.
84 tests, 128 assertions, 1 failures, 0 errors, 0 skips
C:\Users\Jon\Documents\CDev\oj-git>gem i oj-1.4.2.a1.gem
Temporarily enhancing PATH to include DevKit...
Building native extensions. This could take a while...
Successfully installed oj-1.4.2.a1
1 gem installed
C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
ruby 2.0.0dev (2012-10-19 trunk 37268) [i386-mingw32]
Run options:
# Running tests:
[38/84] Juice#test_exception*** {
"^o":"StandardError",
"~mesg":"A Message",
"~bt":null}
= 0.00 s
1) Failure:
test_exception(Juice) [test/tests.rb:580]:
<["test/tests.rb:570:in `test_exception'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:844:in `run_test'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:1211:in `run'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit/testcase.rb:17:in `run'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:837:in `block in _run_suite'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:829:in `map'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:829:in `_run_suite'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:650:in `block in _run_suites'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:648:in `each'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:648:in `_run_suites'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:789:in `_run_anything'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:978:in `run_tests'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:965:in `block in _run'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:964:in `each'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:964:in `_run'",
"C:/rubytrunk/lib/ruby/2.0.0/minitest/unit.rb:953:in `run'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:21:in `run'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:767:in `run'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:364:in `block (2 levels) in autorun'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:27:in `run_once'",
"C:/rubytrunk/lib/ruby/2.0.0/test/unit.rb:363:in `block in autorun'"]> expected but was
<nil>.
Finished tests in 0.123515s, 680.0793 tests/s, 1036.3114 assertions/s.
84 tests, 128 assertions, 1 failures, 0 errors, 0 skips
C:\Users\Jon\Documents\CDev\oj-git>ruby -v test\tests.rb
ruby 1.9.3p297 (2012-10-19 revision 37266) [i386-mingw32]
Run options:
# Running tests:
.....................................*** {
"^o":"StandardError",
"~mesg":"A Message",
"~bt":[
"test/tests.rb:570:in `test_exception'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'",
"C:/ruby193/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:563:in `each'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'",
"C:/ruby193/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:21:in `run'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'",
"C:/ruby193/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'"
]}
...............................................
Finished tests in 0.038004s, 2210.2937 tests/s, 3394.3795 assertions/s.
84 tests, 129 assertions, 0 failures, 0 errors, 0 skips
When I switched from using yajl to oj I ran into this issue and when I switch back to yajl it works fine again. It's not sidekiqs fault. For some reason oj interprets dates as float or vice versa.
See this related Sidekiq issue for more details sidekiq/sidekiq#354
Hi, I'm in desperate need of JSON libraries in Ruby AND Javascript to transfer complex objects that have a lot of internal references. In this regards oj
seems to solve the problem at Ruby's end. However, how do I de-serialize (in the browser) the JSON generated by OJ so that object references are sanely maintained? Are there any Javascript equivalents of OJ?
Also, I noticed that you have come up with your own convention wrt internal object references, i.e. ^i1
and ^r1
. Wouldn't it be better to follow either one of the conventions that some people are already recommending:
This was brought to my attention by @rwz in a pull request submitted to MultiJSON: intridea/multi_json#65
Judging from the error message, it appears this behavior is intentional:
TypeError: In :compat mode all Hash keys must be Strings or Symbols, not Fixnum.
I thought the idea behind :compat
mode was to be fully compatible with the standard library JSON parser (which converts Fixnum
keys to strings)?
What do you think?
Defining RSTRING_NOT_MODIFIED
before you include ruby.h
will tell Rubinius to shut off it's pointer caching compatibility API. By doing so, you're making a promise to rbx that you won't modify any pointers you get back from the Ruby VM.
I haven't had a chance to look over the codebase here to see if you're doing that anywhere, but this could offer a pretty significant speedup on Rubinius (I saw something like a 25x speedup with yajl-ruby).
The Rubinius guys can offer a much more detailed explanation of how it works if you're interested. They hang out in the #rubinius channel on chat.freenode.net.
Here's the example from yajl-ruby: https://github.com/brianmario/yajl-ruby/blob/master/ext/yajl/yajl_ext.h#L27-30
Do I need to configure anything to get JSON Gem compatibility in Rails?
In Rails: puts JSON.dump([Time.now]) # -> ["2012-12-05T16:52:54+01:00"]
In Plain Ruby: puts JSON.dump([Time.now]) # -> ["2012-12-05 16:54:23 +0100"]
Oj: puts Oj.dump([Time.now], mode: :compat) # -> [1354722798.022583000]
The rails version makes more sense than the plain ruby one, since it can be used by JS' Date constructor (new Date("2012-12-05T16:52:54+01:00")
is valid and correct).
I'm not sure about the proper solution, though.
If an untrusted source returns a JSON document that is deeply nested, it can cause oj
to crash hard.
Reproduction steps:
> ruby -r oj -e 'Oj.load(-> n { ?[ * n + ?] * n }.(9999999))'
-e:1: [BUG] Bus Error
(you may need to add nines, depending on how much free memory you have)
See intridea/multi_json#59 for an in-depth discussion of the issue.
>> Oj.load("\"i \"")
=> "i "
>> Oj.load("\"^i\"")
=> "^i"
>> Oj.load("\"^i \"")
SyntaxError: Not a valid ID number at line 1, column 6 [load.c:212]
from (irb):9:in `load'
from (irb):9
from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
>> Oj.load("\"^ia\"")
SyntaxError: Not a valid ID number at line 1, column 6 [load.c:212]
from (irb):5:in `load'
from (irb):5
from /Users/alx/.rvm/rubies/ruby-1.9.2-p290/bin/irb:16:in `<main>'
ruby: 1.9.2p290
oj: 1.2.9
Please let me know, if you need more info.
Hello, Oj looks quite impressive. First time using it. But I have a question:
The regular JSON string outputs something like this:
{"json":[{"id":1,"title":"Casablanca"}]}
While the Oj.dump puts backslashes before the double quotes:
{"json":"[{\"id\":1,\"title\":\"Casablanca\"}]"}
I was wondering if there is a way to avoid them, short of replacing them of course which would make things much slower.
Thanks.
When dumping a hash, the :symbol_keys
option does not seem to have any effect when passed as false
, example:
require 'oj'
# notice the ':' prefix of they key
puts Oj.dump({:abc => 123}, :symbol_keys => false)
#=> {":abc":123}
# same result
puts Oj.dump({:abc => 123}, :symbol_keys => true)
#=> {":abc":123}
Is this expected behavior?
The readme states compatibility with 1.8.7 but when I tried to use it locally it appears that there are 1.9-only method signatures and defines being used.
So far from what I can see, DBL2NUM
and rb_hash_lookup2
are the two offenders. There could be more but I haven't had a chance to dig down further. DBL2NUM
appears to just use rb_float_new
(which exists in 1.8.7 as well) but the change from rb_hash_lookup2
will require some small refactoring I think?
Trying to install under Windows (ruby 1.9.3p231
), but failing:
C:\Users\jarmo>gem install oj
Temporarily enhancing PATH to include DevKit...
Building native extensions. This could take a while...
ERROR: Error installing oj:
ERROR: Failed to build gem native extension.
C:/tcs-ruby193_require_fenix_gc_hash_20120527/bin/ruby.exe extconf.rb
>>>>> Creating Makefile for tcs-ruby version 1.9.3 <<<<<
creating Makefile
make
generating oj-i386-mingw32.def
compiling cache.c
<command-line>:0:4: warning: missing whitespace after the macro name
cache.c: In function 'oj_cache_new':
cache.c:54:5: warning: implicit declaration of function 'bzero'
cache.c:54:5: warning: incompatible implicit declaration of built-in function 'bzero'
compiling cache8.c
<command-line>:0:4: warning: missing whitespace after the macro name
compiling dump.c
<command-line>:0:4: warning: missing whitespace after the macro name
In file included from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/defines.h:223:0,
from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/ruby.h:67,
from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby.h:32,
from oj.h:43,
from dump.c:39:
c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/win32.h:282:12: error: conflicting types for 'gettimeofday'
c:\bin\devkit\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/time.h:39:29: note: previous declaration of 'gettimeofday' was here
In file included from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/ruby.h:1381:0,
from c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby.h:32,
from oj.h:43,
from dump.c:39:
c:/tcs-ruby193_require_fenix_gc_hash_20120527/include/ruby-1.9.1/ruby/missing.h:48:8: error: redefinition of 'struct timezone'
c:\bin\devkit\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/sys/time.h:26:8: note: originally defined here
In file included from dump.c:39:0:
oj.h:50:21: fatal error: pthread.h: No such file or directory
compilation terminated.
make: *** [dump.o] Error 1
Gem files will remain installed in C:/tcs-ruby193_require_fenix_gc_hash_20120527/lib/ruby/gems/1.9.1/gems/oj-1.3.4 for inspection.
Results logged to C:/tcs-ruby193_require_fenix_gc_hash_20120527/lib/ruby/gems/1.9.1/gems/oj-1.3.4/ext/oj/gem_make.out
So this is a tough one as it is/was incredibly difficult to debug. I used oj
with Sidekiq, which is using the Celluloid actor framework. I ran into the issue that workers were dying silently. After almost a week of debugging, it turned out that Celluloid Fibers exceeded the static 4k limit and died (without leaving much of a backtrace).
I traced to cause of the SystemStackError: stack level too deep
down to oj
. Once I switched to yajl
the errors were gone - no more dead actors. I was wondering, is there an incompatibility between oj
and Fibers? Is oj
using the stack too aggressively? Anything I can do to help debug this further?
Here is the full story: sidekiq/sidekiq#233
Hey Petr,
can you please add support for rails native time format (ISO 8061
)? It can be called rails :time_format => :rails
Here is a small sample, so you can see a difference:
t = [Time.now]
# rails native
puts JSON.dump t
# => ["2013-01-25T14:55:10+01:00"]
# oj modes
puts Oj.dump t, {:mode => :compat, :time_format => :unix}
# => [1359122110.187338000]
puts Oj.dump t, {:mode => :compat, :time_format => :ruby}
# => ["2013-01-25 14:55:10 +0100"]
puts Oj.dump t, {:mode => :compat, :time_format => :xmlschema}
# => ["2013-01-25T14:55:10.187338000+01:00"]
Basically its almost exactly the same like :xmlschema
format just without milliseconds.
I notice Oj has a serialization format, Object JSON. That's cool. But what I'd really like to see is support for JSYNC (http://jsync.org/).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.