kgiszczak / shale Goto Github PK

Is there a Shale wrapper or command line utility that converts between types?
E.g. YAML <=> JSON <=> CSV <=> TOML <=> XML

If such one exist it will be awesome.

For some reasons we need to render empty xml nodes even if actual value absence.
For now the only available solution to add || '' for every attributes in decorator.
Similar feature we have in representable:
https://trailblazer.to/2.1/docs/representable.html#representable-function-api-render-nil
Do you have plans to implement such feature?
I understand that it can make deserialization process more complicated.

Sometimes we need to render only partion of available attributes.
For now we use representable for such things:
https://trailblazer.to/2.1/docs/representable.html#include-and-exclude
https://trailblazer.to/2.1/docs/representable.html#representable-function-api-skip-render
Do you have plans to implement this or maybe there is some hacky solution?

Some times we need to define an xml tag without content inside like this:

  <Tag> </Tag>

Currently, shale is closing xml tags without content like this:

  <Tag/>

Im fixing this mapping content to empty string. Like this

  attribute :content, Shale::Type::String, default: -> { " " }

and this

  xml do
    root 'Tag'
    map_content to: :content

It could be a nice improvement to provide an option to choose how to close tags without content!.

I have a big CSV file that I'd like to extract several columns.

Previously, I would resolve those:

  valid_columns = %w[header1 header5 header24]
  filtered_rows = []
  CSV.parse(csv_content, headers: true) do |row|
    filtered_row = row.to_h.slice(*valid_columns)
    filtered_rows << filtered_row
  end
  filtered_rows

I tried Person.from_csv(context, headers: true, only: valid_columns) but it ignores the option only.

If I explicitly map each column as an attribute to be ignored, that works. Is there a more efficient way to do this?

Big fan of this gem. The new render_nil stuff would be extremely useful for my team. Any chance you could cut a new release?

Hey!

First of all, thanks a lot for this gem, having a serializer and deserializer for JSON, YAML and XML at the same time is absolutely wonderful!

I came across an issue that currently hinders its usefulness for me though. Maybe it's such a special case that it's not worth investigating or maybe there is already a solution that I simply didn't see.

In my case, the issue arose when using Rails models as custom models:

class Person < ApplicationRecord
  has_one :pet, polymorphic: true
end

class Dog < ApplicationRecord
  belongs_to :person, as: :pet
end

class Cat < ApplicationRecord
  belongs_to :person, as: :pet
end

# Mapper

class PersonMapper < Shale::Mapper
  model Person

  attribute :id, Shale::Type::Integer
  attribute :pet, ???
end

class DogMapper < Shale::Mapper
  model Dog
  ...
end

class CatMapper < Shale::Mapper
  model Cat
  ...
end

As you can see, I didn't find a way to describe the polymorphic type for pet.
How I would handle this if I'd (de)serialize it by hand:

Serialize (JSON as example)

{
  "id": 1,
  "pet": {
    "type": "Dog",
    ...
  }
}

Take the actual type of the object to be serialized and add it as type field.

Deserialize

Expect a type field and use the corresponding mapper.

Is this something that's simply not possible or did I miss it?
As far as I can see it, I can manipulate the values when (de)serializing, but not the type.

Thanks a lot in advance!

https://www.shalerb.org/#compiling-json-and-xml-schema uses a simple example for generation that doesn't include $refs. Given a file that uses a simple relative path ref that exists in the same directory, the shaleb -c -i user.json command fails with the following message:

/Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:184:in `resolve_ref': can't resolve reference 'http://example.com/schemas/common.json#/definitions/address' (Shale::SchemaError)
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:310:in `compile'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:336:in `block in compile'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:335:in `each'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:335:in `compile'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:89:in `as_models'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema/json_compiler.rb:124:in `to_models'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/lib/shale/schema.rb:51:in `from_json'
	from /Library/Ruby/Gems/2.6.0/gems/shale-1.0.0/exe/shaleb:88:in `<top (required)>'
	from /usr/local/bin/shaleb:23:in `load'
	from /usr/local/bin/shaleb:23:in `<main>'

cat user.json| jq

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "http://example.com/schemas/user.json",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string"
    },
    "lastName": {
      "type": "string"
    },
    "address": {
      "$ref": "./common.json#/definitions/address"
    }
  }
}

cat common.json| jq

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "http://example.com/schemas/common.json",
  "definitions": {
    "address": {
      "type": "object",
      "properties": {
        "street": {
          "type": "string"
        },
        "city": {
          "type": "string"
        }
      }
    }
  }
}

Example XML:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<PACKAGE>
  <version>2</version>
  <empty_offers>1</empty_offers>
  <definictions>definictions.xml</definictions>
  <FILES>
    <file>13467_20230926_100629_001.xml</file>
  </FILES>
</PACKAGE>

Shale mapper:

require 'shale/adapter/ox'
Shale.xml_adapter = Shale::Adapter::Ox

module Asari::XmlParser
  class Cfg < Shale::Mapper
    attribute :version, Shale::Type::Integer
    attribute :empty_offers, Shale::Type::Integer
    attribute :definictions, Shale::Type::String

    xml do
      map_element 'version', to: :version
      map_element 'empty_offers', to: :empty_offers
      map_element 'definictions', to: :definictions
    end
  end
end

When the XML header <?xml version="1.0" encoding="utf-8" standalone="yes"?> is included in XML file, attributes are not mapped correctly:

# with XML header included
Asari::XmlParser::Cfg.from_xml(xml))
=> #<Asari::XmlParser::Cfg:0x00007f82971aa038 @version=nil, @empty_offers=nil, @definictions=nil>

# without XML header included
Asari::XmlParser::Cfg.from_xml(xml))
=> #<Asari::XmlParser::Cfg:0x00007f8299c22d10 @version=2, @empty_offers=1, @definictions="definictions.xml">

When I try for example Nokogiri adapter, everything works as expected with or without XML header. I've tried to specify root element, but It's not working either.

Am I doing something wrong? All examples are without XML header tag, but It's possible to generate XML document with XML header included, so I think It should work.

Sometimes we need to add the encoding to declaration. like this:

  <?xml version="1.0" encoding="UTF-8"?>

It could be a nice improvement!.

Hi! It's me again!

As already mentioned in the other issue, I am very interested in the CSV serialization capabilities of this (cool!) library.

However, it seems that the CSV serialization is not working in a very useful manner when data is nested.

As an example, given this setup:

  address = Address.new(city: "San Francisco")
  person = Person.new(first_name: "John", last_name: "Doe", address: Address.new(city: "San Francisco"))
  person_presenter = PersonPresenter.new(person)

These are the results:

person_presenter.to_csv == <<~CSV.chomp
    John,Doe,,false,[],"{""city""=>""San Francisco"", ""street""=>nil, ""zip""=>nil}"
CSV

person_presenter.to_json == <<~JSON.chomp
    {\"first_name\":\"John\",\"last_name\":\"Doe\",\"married\":false,\"hobbies\":[],\"address\":{\"city\":\"San Francisco\"}}
JSON

person_presenter.to_xml == <<~XML.chomp
    <person>
        <first_name>John</first_name>
        <last_name>Doe</last_name>
        <married>false</married>
        <address>
            <city>San Francisco</city>
        </address>
   </person>
XML

Note how the JSON and XML can sensible handle the nested data structure, but the CSV now actually contains a JSON string in a field. My guess is that this will not be acceptable for most scenarios where CSV is actually used. Also note that in this JSON string, the nil values are rendered, while they are not rendered anywhere else. No render_nil was used anywhere. This lets me guess that this is somewhat of an unhandled or unintended behaviour.

My suggestion would be to flatten the structure so that it will be compatible with CSV. Something like this:

Or with render_nil: true:

<<~CSV.chomp
    John,Doe,false,[],San Francisco,,
CSV

Note that it render an empty string into each field, so that the table which is represented in CSV does maintain is coherence for each row.

With the following headers:

<<~CSV.chomp
    first_name,last_name,married,hobbies,address.city,address.street,address.zip
CSV

What do you think? Or did I overlook some feature of the library?

Hi! I just discovered this cool library some days ago, and it seems to me like the most versatile and powerful serializer around in the Ruby space. Especially the capability to have JSON, XML, and CSV as a target from the same serializer could prove very useful in a recent project of mine.

However, I realized that the CSV capability is only available on the master branch at the moment. Is there any time schedule to do a release with it?

Hi,

I'm evaluating shale for a project, and was wondering if there's an easier way to get from flattened JSON to nested models.

Example:

class Person < ApplicationRecord
  belongs_to :address

  # attribute :id
  # attribute :name
end

class Address < ApplicationRecord
  # attribute :street
  # attribute :city
end

class PersonMapper < Shale::Mapper
  model Person

  attribute :id,     Shale::Type::Integer
  attribute :name,   Shale::Type::String
  attribute :street, Shale::Type::String
  attribute :city,   Shale::Type::String

  json do
    map :street, to: { address: :street }
    map :city,   to: { address: :city }
  end

  def address_from_json(model, key, value)
    model.build_address unless model.address
  end
end

… something like this. I could create a method for every single attribute that's "delegated" to address, but it gets verbose and tedious.

• The "to: with hash" api is just an example, there's probably a better way
• Could there also be a way to initialize the nested relation properly (model.build_address)
• Is there an existing way that I overlooked?

Even in icognito tab:

Hi. I think 18ebdce broke serializations of Time and Date classes

E.g. Date

irb(main):006:0> require 'shale'
=> true
irb(main):007:1* class Person < Shale::Mapper
irb(main):008:1*   attribute :dob, Shale::Type::Date
irb(main):009:0> end
=> :dob=
irb(main):010:0> p = Person.new(dob: '2022-08-12')
=> #<Person:0x00007fba055250f0 @dob=#<Date: 2022-08-12 ((2459804j,0s,0n),+0s,2299161j)>>
irb(main):011:0> p.to_json
Traceback (most recent call last):
        25: from /Users/shan/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'                    
        24: from /Users/shan/.rbenv/versions/2.7.1/bin/irb:23:in `load'                      
        23: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
         6: from (irb):11:in `<main>'                                                               
         5: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/complex.rb:705:in `to_json'
         4: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/complex.rb:210:in `to_json'
         3: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/complex.rb:118:in `as_json'
         2: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/complex.rb:118:in `each_value'
         1: from /Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/complex.rb:159:in `block in as_json'
/Users/shan/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/shale-0.7.0/lib/shale/type/date.rb:35:in `as_json': wrong number of arguments (given 2, expected 1) (ArgumentError)

Hello,

In v0.5.0 you dropped Ruby 2.6 support, which implicitly also dropped JRuby support: as of JRuby 9.3.6.0 they're still only 2.6 compatible. Would it be possible to restore this easily? Are you currently using 2.7-only features?

JRuby 9.4.0.0 will Ruby 3.1 compatible, but it might take a while still to be released and for codebases to be able to make a smooth transition to it.

Thanks for your consideration!

I've been trying to see if shale would be a good choice for a rails app I'm working on. I saw in #16 that you pointed to the Custom Model documentation which seems helpful, but the downside is that in order to use that developers have to redefine the attributes that are already defined in schema.rb.

If I set up a very basic rails app with a schema that looks like this:

(Created a migration with rails generate migration CreatePerson first_name:string last_name:string address:string)

ActiveRecord::Schema[7.0].define(version: 2023_10_05_210646) do
  create_table "people", force: :cascade do |t|
    t.string "first_name"
    t.string "last_name"
    t.string "address"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end
end

Then I'm able to call Person.attribute_types and get something that has the type for each attribute key along with other schema-adjacent data:

irb(main):001> Person.attribute_types
=> 
{"id"=>
  #<ActiveRecord::ConnectionAdapters::SQLite3Adapter::SQLite3Integer:0x00007f455dd81cc0
   @limit=nil,
   @precision=nil,
   @range=-9223372036854775808...9223372036854775808,
   @scale=nil>,
 "first_name"=>
  #<ActiveModel::Type::String:0x00007f455dd81310
   @false="f",
   @limit=nil,
   @precision=nil,
   @scale=nil,
   @true="t">,
 "last_name"=>
  #<ActiveModel::Type::String:0x00007f455dd81310
   @false="f",
   @limit=nil,
   @precision=nil,
   @scale=nil,
   @true="t">,
 "address"=>
  #<ActiveModel::Type::String:0x00007f455dd81310
   @false="f",
   @limit=nil,
   @precision=nil,
   @scale=nil,
   @true="t">,
 "created_at"=>#<ActiveRecord::Type::DateTime:0x00007f455dd413f0 @limit=nil, @precision=6, @scale=nil>,
 "updated_at"=>#<ActiveRecord::Type::DateTime:0x00007f455dd413f0 @limit=nil, @precision=6, @scale=nil>}

So I'm wondering if it would be feasible for shale to use that data so that instead of having to redefine attributes in our mapper like this:

class PersonMapper < Shale::Mapper
  model Person

  attribute :first_name, Shale::Type::String
  attribute :last_name, Shale::Type::String
  attribute :address, AddressMapper
end

we could do something like this and let shale create the attributes based on what's already defined on the model:

class PersonMapper < Shale::Mapper
  model Person

  attributes Shale::Schema.from_active_model(Person)
end

Hi! Thank you for this great gem. Would you consider making render_nil a global setting, like Shale.render_nil = true ?

There two mappers that by my mind should work identically:

require 'shale'

class OnePerson < Shale::Mapper
  xml do
    root 'Person'
  end
  
  attribute :name, Shale::Type::String
end

class AnotherPerson < Shale::Mapper  
  attribute :name, Shale::Type::String

  xml do
    root 'Person'
  end
end

puts OnePerson.new(name: 'John Doe').to_xml(:pretty)
# <Person>
#   <name>John Doe</name>
# </Person>

puts AnotherPerson.new(name: 'John Doe').to_xml(:pretty)
# <Person/>

After little research I recognized that xml method fully override config setted by attribute method calls.
It's not critical but I think that it should be documented as minimum.

Thanks for your hard work on this, this looks awesome!

I was curious of union types are supported (e.g. anyOf in JSON Schema or xsd:choice in XSD).

Cool gem! Have you considered switching shale to be included within a class rather than inheriting from Shale::Mapper? I think shale would have better interoperability with ActiveRecord models, for example, if that were the case.

I've been working on trying to use more keywords from the JSON Schema dialect, such as minimum, maximum, and required. I wasn't able to get shale's JSON mapping to work which makes sense if it's just for serializing/deserializing, so as of now I've thought of three possible approaches

1. Custom classes for every possibility

I think this is what developers would have to do right now to use more of the dialect. To have an integer with a minimum value of 1 might look like this:

  class IntegerMinimumOneType < Shale::Type::Integer
  end

  class IntegerMinimumOneJSONType < Shale::Schema::JSONGenerator::Base
    def as_type
      { "type" => "integer", "minimum" => 1 }
    end
  end

  Shale::Schema::JSONGenerator.register_json_type(IntegerMinimumOneType, IntegerMinimumOneJSONType)

  class PersonMapper < Shale::Mapper
    model Person
    attribute :age, IntegerMinimumOneType
  end

This would be fine for common things, like PositiveInteger, but this would get increasingly difficult to manage if the schema needed multiple validations for properties, such as a minimum, maximum, and required all at once.

2. Instantiated types for more flexibility

This isn't possible with shale right now, but I have a local branch where I've gotten it to work and would be happy to make a PR if this is a direction you'd like to go in. To achieve the same schema as above the API could look like this:

  class BoundedIntegerType < Shale::Type::Integer
    attr_reader :min, :max

    def initialize(min = nil, max = nil)
      @min = min
      @max = max
    end
  end

  class BoundedIntegerJSONType < Shale::Schema::JSONGenerator::Base
    def as_type
      { "type" => "integer", "minimum" => instance.min, "maximum": instance.max }.compact
    end
  end

  Shale::Schema::JSONGenerator.register_json_type(BoundedIntegerType, BoundedIntegerJSONType)

  class PersonMapper < Shale::Mapper
    model Person
    attribute :age, BoundedIntegerType.new(1) # minimum of 1, no maximum
  end

The responsibility is still on the consumer of the gem to create the types needed for their schema, but having a generator that can receive an instance of a class can allow for a lot of flexibility.

3. Modify the API to include more options, similar to collection

Another option to allow more of the dialect to be used is to allow the attribute API to receive all the possible keywords, either individually or through some sort of hash.

  class PersonMapper < Shale::Mapper
    model Person
    attribute :age, Shale::Type::Integer, minimum: 1
  end

This might involve some more work on the shale side of things to ensure that the provided keywords are compatible with the type. For example this should be considered invalid since maxItems is for arrays:

    attribute :age, Shale::Type::Integer, max_items: 5

Apologies if I've overlooked something, but based on reading the docs and trying things out I think right now Option 1 is all I can do, so I wanted to explore Option 2 since it seemed less cumbersome. It also seemed like implementing Option 3 would add a lot more complexity to the gem. Should I go ahead and make a PR for this, or do you have guidance on how to achieve this another way?

Hi there! First of all, what a nice gem you've given us here. Amazed by how elegant it's API turned out to be. Excited to use it in a new API integration.

Also, 300+ stars, all these features, and no issues? Not even closed ones? What kind of sorcery is that? 😃

Anyways, I'll be the first. I was facing this exception when trying to use it in a Rails project with Ruby 3.1:

/Users/xxx/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:15:in `require': cannot load such file -- rexml/document (LoadError)

According to this blog post, rexml is now a bundled gem, so I had to add it manually in my gemfile as well, which fixed it.

In our case, since performance isn't a huge concern, I'd like to avoid the rexml dependency and use Nokogiri that we already use. Tried changing the adapter in an initiailzer, as per the gem's README instructions, but it raises the exception before reaching the initializer.

Maybe the requirement of rexml could be relaxed, or else instructions for adding rexml in the Gemfile could be added in the readme.