Make escaping work differently in `@byrow!` about dataframesmeta.jl

[tshort]

You can do the following:

@byrow! df begin
    :now_works = mean(Array(_I_(vars)[row,:]))
end

That's clumsy, not obvious, and it's likely to be inefficient. So, there are opportunities to make escaping work better here.

[pdeffebach]

Looking at the implementation of @byrow!, I would have expected it to be a wrapper for eachrow(df). Instead, its vectorized replace operations. Is there a reason for this? I think I read somewhere that DataFrameRows aren't that performant.

[tshort]

@byrow! produces a for loop over the rows of the DataFrame. You can see this using macroexpand as follows (I stripped the comments in the output):

julia> macroexpand(:(@byrow! df begin
           :works = mean([:x1, :x2, :x3])
       end))

quote  
    _N = length(df[1]) 
    _DF = begin 
            (DataFramesMeta.transform)(df)
        end 
    begin  
        function ##666(##669, ##668, ##667, ##670) 
            for row = 1:_N
                begin  
                    ##667[row] = mean([##668[row], ##669[row], ##670[row]])
                end
            end
        end 
        ##666(_DF[:x2], _DF[:x1], _DF[:works], _DF[:x3])
    end 
    _DF
end

I don't think it's possible to make an efficient mean operation on columns specified by a variable vars. With the current DataFrames/DataFramesMeta design, you can only have type-stable operations if you specify the columns ahead of time.

[pdeffebach]

I hadn't used macroexpand! that is a very elegant solution.

I don't think it's possible to make an efficient mean operation on columns specified by a variable vars.

Is this just for row-wise operations? Or are you referring in general to _I_(vars) ^[foot] use in any @with macro.

I am kind of playing with a DataFrameRow to behave more like a vector, and can see that type stability is an issue for that as well.

Generic row-wise operations on vectors are something I use a lot in Stata, so they are an important part of my transition away from stata, so I would be interested in spending more effort to try and get this working.

foot: Which we should rename to cols() imo cause its prettier.

[nalimilan]

I think we could get good performance by making eachrow return a special iterator type parameterized on the types of the columns. It would be guaranteed to yield TypedDataFrameRow{C} rows. That would be interesting to experiment. One issue is that it wouldn't make things faster if you just call eachrow inside a function, so I'm not sure it makes sense to change the behavior of eachrow in DataFrames. Maybe if the compiler becomes smarter to specialize loops automatically.

Regarding calling mean on DataFrameRow, JuliaData/DataFrames.jl#1449 deprecates the current behavior which prevents this, so that we can implement it later.

[pdeffebach]

Closed in favor of #229