etsy / cascading.jruby Goto Github PK
View Code? Open in Web Editor NEWThis project forked from mrwalker/cascading.jruby
A JRuby DSL for Cascading
Home Page: https://github.com/etsy/cascading.jruby/wiki
This project forked from mrwalker/cascading.jruby
A JRuby DSL for Cascading
Home Page: https://github.com/etsy/cascading.jruby/wiki
coerce_to_java currently converts all Fixnum to java.lang.Integer, but it should instead use java.lang.Long (just as we use java.lang.Double instead of java.lang.Float for Float).
Here's an example fail:
too big for int: 32503701600000 /opt/jruby/lib/ruby/gems/1.8/gems/cascading.jruby-0.0.9/lib/cascading/operations.rb:117:in `coerce_to_java'
In implementing our own custom Buffers, we've had to unique field names, then rename to the desired existing data tuple field name to work around this issue. There should be no name conflict between the data tuple and the grouping tuple, so something must be going wrong using Scope to propagate names through the every block.
where cannot simply wrap a Janino expression in negation because expressions can be preceded by import statements. Proper implementation will require parsing of the imports so that only the expression is negated.
Currently, there is no requirement that nodes in the composition hierarchy have unique names. This can lead to 2 kinds of problems:
This problem could be resolved in several ways:
The current workaround is simply to use globally unique names, which the programmer must ensure themselves.
assembly "joined" do
left_join *["visit_lvl"] + joined_tables + [{:on => ['group', 'subgroup', 'ab_test', 'ab_test_group']}]
debug_scope # this dies a horrible death
insert "blah" => "blah"
debug_scope # this is cool
end
Currently sources and sinks must have unique names because they are stored as name -> tap maps. This results in a branch/pass anti-pattern when people want to write the output of an assembly to different taps. It can also be surprising when earlier sinks are compiled away by c.j.
We should exception on duplicate source/sink names, but maybe also provide an optional parameter to distinguish between the name of the tap and the assembly from which it reads.
Keys in the hash form of the :on argument to join are sorted lexicographically and the first of these is the grouping key field name that propagates. This should either produce a warning or be documented somewhere visible.
insert "a" => nil should equal null and not ""
This issue is a test of github's issue tracking. This minor issue causes the rake tasks in the cascading.jruby gem to fail unless you include the Cascading jar on the classpath. The fix is to simply issue a warning if the TextLine class cannot be loaded rather than throwing an exception.
A group_by with no every block or an empty every block seems to not set the primary key to the grouping keys. Fixing this bug should be held off until we refactor the code such that Every can -only- occur in the group_by/join block.
There's no reason to be able to produce an arbitrary number of fields containing the group count. Further, count with no arguments should produce a single 'count' field. Currently, passing no parameters results in a no-op.
If you list sinks in the order in which assemblies/branches are defined, this is not an issue. However, if you list them in a different order, sometimes the cascade will pass composition-time checking but yield a runtime Planner exception. In the cases I've seen so far, it appears that a branch will inherit the fields of the tail pipe of its enclosing assembly (for some unknown reason, only at runtime).
The workaround is to list the sinks in the same order as the assemblies, but this is an annoying limitation and needs to be fixed.
min and max are the only composite aggregators for which SQL mode has been verified. Should cover this with tests and see if it extends to the other aggregators.
TL;DR - I wasn't able to get the exact cascading 2.0 WIP build that cascading.jruby is written against and c.j doesn't work against the latest as of yesterday (281)
I made some tweaks to fit the API refactors, which you can see here - https://github.com/blake-education/cascading.jruby
I got past the initial errors, only to stumble into "NativeException: java.lang.IllegalArgumentException: resultGroupFields and cogroup resulting joined fields must be same size"
I'm an absolute cascading newb, but it seems that the thrown exceptions were added in this commit to cascading from April 3 cwensel/cascading@689e3a8#diff-6
So at that point I gave up :)
I know cascading 2.0 is a moving target, so do you reckon you could state and archive the build against which c.j is tested? The cascading project doesn't seem to keep old 2.0-wip builds around.
cast should accept a hash or "field:type" arguments and not discard fields that aren't explicitly cast.
This bug needs to be researched and documented better. The general idea is that a group_by that follows a branch at the end of an assembly will fail to be included in the dataflow. The workaround is simply to wrap the group_by in another branch.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.