dmlc / cub

Distributed Machine Learning Common Codebase

DMLC-Core is the backbone library to support all DMLC projects, offers the bricks to build efficient and scalable distributed machine learning libraries.

Developer Channel

What's New

• Note on Parameter Module for Machine Learning

Contents

• Documentation and Tutorials
• Contributing

Known Issues

• RecordIO format is not portable across different processor endians. So it is not possible to save RecordIO file on a x86 machine and then load it on a SPARC machine, because x86 is little endian while SPARC is big endian.

Contributing

Contributing to dmlc-core is welcomed! dmlc-core follows google's C style guide. If you are interested in contributing, take a look at feature wishlist and open a new issue if you like to add something.

• DMLC-Core uses C++11 standard. Ensure that your C++ compiler supports C++11.
• Try to introduce minimum dependency when possible

CheckList before submit code

• Type make lint and fix all the style problems.
• Type make doc and fix all the warnings.

NOTE

deps:

libcurl4-openssl-dev

cub's Issues

[Clarification] Provision for AMD HIP backend for MXNet

In the process of merging HIP Port of MXNet, we are creating a provision for CUDA and HIP backend for CUB as well. CUB being sub-module for MXNet.
Please refer ticket apache/mxnet#6257 for disucssion with MXNet team
We are ready with the changes, can we raise a Pull Request, so that the changes can be reviewed and merged to master.

Development model

While I agree that squashing all commits to avoid a huge .git folder makes sense to speed up cloning of mxnet, I believe you should not delete old commits from dmlc/cub as it breaks checking out older versions of mxnet that relied on those.

Checking out old versions of mxnet can be important when debugging changes introduced between two revisions (git bisect). Another usecase is when having some code that was not touched for a while and used to work but does not work with current mxnet master ;)

@mli, what do you think? Of course you can't change this retrospectively, but perhaps you agree that it makes sense not to change the history of this repo in the future?

digits "specified from least significant to most significant"

cub/cub/device/device_radix_sort.cuh

Line 59 in 05eb57f

* characters, etc.) specified from least-significant to most-significant. For a

I am having a hard time interpreting the documentation where it says the keys may be an ordered sequence of characters, digits etc., "specified from least significant to most significant". This phrase occurs in >1 place I think.

The issue is that the examples only seem to let the key value be a builtin type like int, float, double and so on. Those types do have less and more significant bits, but it would not be meaningful to say that we specify the digits "from least significant to most significant" unless we were making a statement about the system endian-ness. I can only see that this would be meaningful if the templated type were allowed to be, say, "int32_t[4]", and if that's the case it would be nice for it to be clarified.