Pass default value to clean_copy about pyvtreat

[jtanman]

Wow thanks for the quick reply! Really appreciate all the work on this library!

[jtanman]

Thanks again!

[JohnMount]

That is a great idea, and very clear description. I'll probably add some variation of this this weekend. Thanks!

…

On Nov 8, 2019, at 5:42 PM, James Tan ***@***.***> wrote: Hi, I'm using the unsupervised treatment module and noticed that the clean_copy fills in any missing/bad values with the mean for the distribution. I was wondering if I could specify a default value based on column name through something like a dictionary? The reason is I might have a variable that has a long tail that few people ever reach. This might be something like money spent after reaching level 100 in a game or something. Now most people won't reach level 100, so their value will be missing/NA. However, of those that do, most people will spend 0, and a few might spend a large amount, maybe 1000. vtreat will default fill in the clean copy of revenue (which is hugely influential indicator) with maybe like 150 which kind of ruins any signal from this long tail distribution. Could I pass a dictionary with like 0 as a default value in this case? Thank you! This could be something like transform = vtreat.UnsupervisedTreatment( cols_to_copy=[], # columns to "carry along" but not treat as input variables cols_fill_values={ 'col1': 0, 'col2': mean, 'col3': 1 } ) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=ABATZEBL3IJL6H27VGEGD63QSYIQBA5CNFSM4JLBX5L2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HYDKSOA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABATZEG6HIX2ADHCM7LTUZ3QSYIQBANCNFSM4JLBX5LQ>.

--------------- John Mount http://www.win-vector.com/ <http://www.win-vector.com/> Our book: Practical Data Science with R http://practicaldatascience.com <http://practicaldatascience.com/>

[JohnMount]

I have an implementation that accepts the default value or simple imputation method as a key in the parameters structure, and also specific column plans (again values or simple imputation methods) as a dictionary argument. Examples can be found here. The API may change when we go to document it (a lot more design critique happens there), so probably won't close the issue until we get some documentation done.

[JohnMount]

We have some documentation for this feature here.