manfred / ensure-encoding Goto Github PK

I added this to one of my gem to handle encoding better, i added it and just changed one line and added require and bundler became crazy when i added my gem (that has dependency to yours) to my app.

bundle stderr: /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/ui/shell.rb:92:in `[]': invalid byte sequence in US-ASCII (ArgumentError)
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/ui/shell.rb:92:in `strip_leading_spaces'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/ui/shell.rb:97:in `word_wrap'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/ui/shell.rb:83:in `tell_me'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/ui/shell.rb:29:in `error'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/friendly_errors.rb:9:in `rescue in with_friendly_errors'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/lib/bundler/friendly_errors.rb:7:in `with_friendly_errors'
    from /home/deploy/.rvm/gems/ruby-2.1.4/gems/bundler-1.9.6/bin/bundle:18:in `<top (required)>'
    from /home/deploy/.rvm/gems/ruby-2.1.4/bin/bundle:23:in `load'
    from /home/deploy/.rvm/gems/ruby-2.1.4/bin/bundle:23:in `<main>'
    from /home/deploy/.rvm/gems/ruby-2.1.4/bin/ruby_executable_hooks:15:in `eval'
    from /home/deploy/.rvm/gems/ruby-2.1.4/bin/ruby_executable_hooks:15:in `<main>'

I use Ruby 2.1.4 and bundle 1.9.6

What is the problem?

C:\Users\dblock\Source\GitHub\Gravity\forks\dblock>rake db:import
(in C:/Users/dblock/Source/GitHub/Gravity/forks/dblock)
The system cannot find the path specified.
The system cannot find the path specified.
C:/Ruby/lib/ruby/1.9.1/rake.rb:30: warning: already initialized constant RAKEVERSION
WARNING: Possible conflict with Rake extension: String#ext already exists
WARNING: Possible conflict with Rake extension: String#pathmap already exists
C:/Ruby/lib/ruby/1.9.1/rake.rb:402: warning: already initialized constant EMPTY_TASK_ARGS
C:/Ruby/lib/ruby/1.9.1/rake.rb:450: warning: already initialized constant EMPTY
C:/Ruby/lib/ruby/1.9.1/rake.rb:958: warning: already initialized constant RUBY_EXT
C:/Ruby/lib/ruby/1.9.1/rake.rb:962: warning: already initialized constant RUBY
C:/Ruby/lib/ruby/1.9.1/rake.rb:1031: warning: already initialized constant LN_SUPPORTED
C:/Ruby/lib/ruby/1.9.1/rake.rb:1240: warning: already initialized constant ARRAY_METHODS
C:/Ruby/lib/ruby/1.9.1/rake.rb:1243: warning: already initialized constant MUST_DEFINE
C:/Ruby/lib/ruby/1.9.1/rake.rb:1247: warning: already initialized constant MUST_NOT_DEFINE
C:/Ruby/lib/ruby/1.9.1/rake.rb:1251: warning: already initialized constant SPECIAL_RETURN
C:/Ruby/lib/ruby/1.9.1/rake.rb:1257: warning: already initialized constant DELEGATING_METHODS
C:/Ruby/lib/ruby/1.9.1/rake.rb:1561: warning: already initialized constant DEFAULT_IGNORE_PATTERNS
C:/Ruby/lib/ruby/1.9.1/rake.rb:1567: warning: already initialized constant DEFAULT_IGNORE_PROCS
C:/Ruby/lib/ruby/1.9.1/rake.rb:1604: warning: already initialized constant FileList
C:/Ruby/lib/ruby/1.9.1/rake.rb:1630: warning: already initialized constant EARLY
C:/Ruby/lib/ruby/1.9.1/rake.rb:1960: warning: already initialized constant DEFAULT_RAKEFILES
rake aborted!
stack level too deep
C:/Ruby/lib/ruby/gems/1.9.1/gems/activesupport-3.0.5/lib/active_support/dependencies.rb:231

rails new EnsureTest
edit Gemfile, add
gem "ensure-encoding", :git => "https://github.com/Manfred/Ensure-encoding.git"
bundle install
rake

Error:

rake aborted!
stack level too deep

This is being thrown from the gem in rails 3.0.7, ruby 1.9.2, and the mysql2 gem instead of sqlite3

#Gemfile
  gem 'ensure-encoding'

#Controller
  require 'ensure/encoding'

  def clean_data
    if (params[:data])
      data = params[:data].ensure_encoding('UTF-8', :external_encoding => :sniff, :invalid_characters => :transcode) 
      @data = CGI.unescape(data)
    end
  end

I grabbing some text from mechanize, and text from html website is always a mess (ISO ? UTF-8 ?)

I decide to test your gem, but if you invoque a second method on a string, i have an error :

test = "Le boîtier étanche permet de le mettre en \x9Cuvre dans les conditions les plus extrêmes." 
test.valid_encoding?
=> false
test.ensure_encoding('UTF-8').valid_encoding?
=> true
test.ensure_encoding('UTF-8').gsub("t","a")
 => "Le boîaier éaanche permea de le meaare en \x9Cuvre dans les condiaions les plus exarêmes." 
test.ensure_encoding('UTF-8').gsub("t","a").gsub("p","e")
   ArgumentError: invalid byte sequence in UTF-8
    from (irb):14:in `gsub'
    from (irb):14
    from /home/pierre/.rvm/gems/ruby-1.9.2-p136/gems/railties-3.0.3/lib/rails/commands/console.rb:44:in `start'
    from /home/pierre/.rvm/gems/ruby-1.9.2-p136/gems/railties-3.0.3/lib/rails/commands/console.rb:8:in `start'
    from /home/pierre/.rvm/gems/ruby-1.9.2-p136/gems/railties-3.0.3/lib/rails/commands.rb:23:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'

Classic encoding doesn't work :
test.encode('UTF-8').valid_encoding?
=> false
test.force_encoding('UTF-8').valid_encoding?
=> false
test.encode("UTF-8",:invalid=>:replace, :replace => "").valid_encoding?
=> false

The only solution i find is this :
test.force_encoding('binary').gsub(156.chr,"Oe").valid_encoding?
=> true

test.force_encoding('binary').gsub(156.chr,"Oe").valid_encoding?
Is there a way to convert binary to utf-8 special character ?