etsy / cascading.jruby Goto Github PK

coerce_to_java currently converts all Fixnum to java.lang.Integer, but it should instead use java.lang.Long (just as we use java.lang.Double instead of java.lang.Float for Float).

Here's an example fail:

too big for int: 32503701600000
/opt/jruby/lib/ruby/gems/1.8/gems/cascading.jruby-0.0.9/lib/cascading/operations.rb:117:in `coerce_to_java'

In implementing our own custom Buffers, we've had to unique field names, then rename to the desired existing data tuple field name to work around this issue. There should be no name conflict between the data tuple and the grouping tuple, so something must be going wrong using Scope to propagate names through the every block.

where cannot simply wrap a Janino expression in negation because expressions can be preceded by import statements. Proper implementation will require parsing of the imports so that only the expression is negated.

Currently, there is no requirement that nodes in the composition hierarchy have unique names. This can lead to 2 kinds of problems:

• If peers in the hierarchy are named the same thing, the last defined "wins." This means nodes created earlier are not available for sinking, joins, unions, or lookups (because they've been overwritten in the list of children).
• If non-peers are named the same thing, then a lookup (like in sinks, joins, and union) becomes ambiguous. Whichever node is located via depth-first search "wins" (see Node#find_child).

This problem could be resolved in several ways:

1. Enforce unique names in Node#add_child so collisions cannot happen
2. Require node names to be unambiguously qualified when there is a potential collision (e.g. "cascade_name.flow_name.assembly_name.branch_name" as opposed to just "branch_name")

The current workaround is simply to use globally unique names, which the programmer must ensure themselves.

assembly "joined" do
left_join *["visit_lvl"] + joined_tables + [{:on => ['group', 'subgroup', 'ab_test', 'ab_test_group']}]

  debug_scope # this dies a horrible death

  insert "blah" => "blah"

 debug_scope # this is cool

end

Currently sources and sinks must have unique names because they are stored as name -> tap maps. This results in a branch/pass anti-pattern when people want to write the output of an assembly to different taps. It can also be surprising when earlier sinks are compiled away by c.j.

We should exception on duplicate source/sink names, but maybe also provide an optional parameter to distinguish between the name of the tap and the assembly from which it reads.

Keys in the hash form of the :on argument to join are sorted lexicographically and the first of these is the grouping key field name that propagates. This should either produce a warning or be documented somewhere visible.

insert "a" => nil should equal null and not ""

This issue is a test of github's issue tracking. This minor issue causes the rake tasks in the cascading.jruby gem to fail unless you include the Cascading jar on the classpath. The fix is to simply issue a warning if the TextLine class cannot be loaded rather than throwing an exception.

A group_by with no every block or an empty every block seems to not set the primary key to the grouping keys. Fixing this bug should be held off until we refactor the code such that Every can -only- occur in the group_by/join block.

There's no reason to be able to produce an arbitrary number of fields containing the group count. Further, count with no arguments should produce a single 'count' field. Currently, passing no parameters results in a no-op.

If you list sinks in the order in which assemblies/branches are defined, this is not an issue. However, if you list them in a different order, sometimes the cascade will pass composition-time checking but yield a runtime Planner exception. In the cases I've seen so far, it appears that a branch will inherit the fields of the tail pipe of its enclosing assembly (for some unknown reason, only at runtime).

The workaround is to list the sinks in the same order as the assemblies, but this is an annoying limitation and needs to be fixed.

min and max are the only composite aggregators for which SQL mode has been verified. Should cover this with tests and see if it extends to the other aggregators.

TL;DR - I wasn't able to get the exact cascading 2.0 WIP build that cascading.jruby is written against and c.j doesn't work against the latest as of yesterday (281)

I made some tweaks to fit the API refactors, which you can see here - https://github.com/blake-education/cascading.jruby

I got past the initial errors, only to stumble into "NativeException: java.lang.IllegalArgumentException: resultGroupFields and cogroup resulting joined fields must be same size"

I'm an absolute cascading newb, but it seems that the thrown exceptions were added in this commit to cascading from April 3 cwensel/cascading@689e3a8#diff-6

So at that point I gave up :)

I know cascading 2.0 is a moving target, so do you reckon you could state and archive the build against which c.j is tested? The cascading project doesn't seem to keep old 2.0-wip builds around.

cast should accept a hash or "field:type" arguments and not discard fields that aren't explicitly cast.

This bug needs to be researched and documented better. The general idea is that a group_by that follows a branch at the end of an assembly will fail to be included in the dataflow. The workaround is simply to wrap the group_by in another branch.