Diagnosing/tuning mapper memory blowout about scalding

[johnynek]

How many items are you keeping?

What are the types you are sorting?

[johnynek]

By the way, if you just want to get it done, it's only a couple times slower to do:

.groupBy('key) {
_.sortBy('field).reverse.take(10000)
}

to get the top 10k. This will not do any map-side aggregation however.

We may need to make a special case optimization for medium sized sortWithTake situations. Currently, we are serializing every key into one row, and if the number of items is large, or the objects are large, this could present memory problems.

lastly, we have some fixes coming to improve memory utilization for non-primitive types within the tuples.

[benlee]

I was doing a _.sortedReverseTake to try and keep the most recent 1000 items, sorting 24-byte string ids. Just so I understand the slower solution -- I understand that the mapper than emit each value with no map-side aggregation. How would the sort be accomplished, though? For each 'key, would the reducer receive values in a secondary sort order on 'field, or would the reducer actually have to sort an unordered stream of values.

[benlee]

Per Argyris' request -- here's a snippet (maybe I'm doing something wrong?)

.groupBy('venueId) {                                                                                             
   _.sortedReverseTake[String]('checkinId -> 'checkinId, 1000).toList[String]('shout -> 'shouts)
}

[johnynek]

Can you try without the toList and without the sortedReverseTake?

I wonder which one is really causing the problem.

[johnynek]

I guess the issue here is the:

.toList[String]('shout -> 'shouts)

You are putting all the shouts at a venue into one list in one row of a reducer, and this does not spill to disk. If there are a few venues with a lot shouts, you could have an issue.

Could you try without that part just to verify? Java strings take like 45-50 bytes even for the shortest strings. When they are living in memory, that could potentially add up. If you can avoid using strings and use Longs or Ints, you will almost certainly see some big savings. For instance, I don't know if your checkinId can fit in a Long or not, but that might help.

[johnynek]

I know I'm totally bombarding this issue, but Chris pointed out that the default spill threshold (an internal cascading variable) is rather small. I bumped it to his recommended value (which could cause OOM if people have really low memory clusters). It hasn't caused any issues for us, and it may help your issue.

Let me know if the latest version works better for you (PS: you can also turn up or down the number of reducers for a particular part of the job and that kind of tuning could help).

[benlee]

Thanks, I'll give it a shot!

[azymnis]

Ben, did Oscar's suggestion solve your issue? Are you still getting OOM errors?

[johnynek]

Assuming our suggestions or cascading improvements helped here.