upmem / dpu_kmeans Goto Github PK

It looks like part of all of the CPU benchmarks were done with Elkan.

To do:

• Check which
• If necessary rerun them with Lloyd

The script benchmarks/strong_scaling/CPU+DPU.py crashes on 2048 DPU.
The script 2048_only doesn't.
There's probably a memory leak. Check the C arrays.

We can accelerate centroid initialization and inertia calculation by using the daal4py backend.

This Email is mainly about the K-means application https://github.com/upmem/dpu_kmeans. I have attached the code that gives "task will not fit in WRAM error" using the K-means code on 2546 DPUs. The code computes K-means with 1,000,000 data points of dimension 10. However, the same code works for 100,000 data points.

import numpy as np
import time
from dpu_kmeans import KMeans as DPUKMeans
from dpu_kmeans import _dimm

#error: tasks will not fit in WRAM(base)

if __name__ == "__main__":
    k, dim, num_elements, iter = 10, 10, 1000000, 1
    input = np.random.randint(0, 100, size=(num_elements, dim))
    init_centroids = np.zeros((k, dim), dtype=np.int32)
    #_dimm.set_n_dpu(num_dpus)

    for i in range(k):
        init_centroids[i] = input[i]
    
    input = _dimm.DIMM_data(input)
    dpu_kmeans = DPUKMeans(k, n_init=1, verbose=False, max_iter=iter, tol=0)
    dpu_kmeans.n_iter_ = iter

    start = time.time()
    result, iterations, duration = dpu_kmeans.fit(input)
    end = time.time()
    t = (end-start)*1000.


    print("the time consumed is "+str(t)+" ms")
    print("the kernel time consumed is "+str(duration))
    print("the number of iterations "+str(iterations))
    print("centroids of kmeans: ")
    print(np.rint(result))

To do:

• use templating to set the name of the experiment, get it from params
• no need to differentiate params.yaml, stage names and output names (doesn't do anything anyway)
• add a top level key on metrics to differentiate between different kind of experiments
• consider using branches (probably not) if it's the only way to select what to show, check with DVC studio
• use dvc exp show --param-deps to avoid showing everything

Right now cluster relocation is done on the host side. It essentially calls one Lloyd iteration to get the labels and then computes all points-clusters distances. This is likely slow.

To do:

• Profile the application to see if this needs to be optimized.
• If it's slow, either
  • Move it to the DPU side, finding the points furthest from all centroids can be done as part of the main loop.
  • Do it with daal4py

Sklearnex uses daal4py to accelerate k-means. We need to patch and rebuild it to measure init times and times per iteration.