intel-analytics / bigdl-trainings Goto Github PK

Hi, thank you for your content! It's very confusing to follow from these videos.

https://software.intel.com/en-us/ai-academy/frameworks/bigdl/tech-overview

Is there a better resource?

Working function

def array2rdd(ds):
#build Sample from ndarrays
def build_sample(balance,sex,education,marriage,age,default):
feature = np.array([balance,sex,education,marriage,age]).flatten()
label = np.array(default)
return Sample.from_ndarray(feature, label)

rdd = ds.map(lambda X: build_sample(X[0],X[1],X[2],X[3],X[4],X[5]))
return rdd

cc_rdd_train = array2rdd(trainingData.rdd.map(list))
cc_rdd_train.cache()
cc_rdd_train.count()

cc_rdd_test = array2rdd(validData.rdd.map(list))
cc_rdd_test.cache()
cc_rdd_test.count()

Did you mean to pass t and decompose t's members like this?

#convert ndarray data into RDD[Sample]
def array2rdd(ds):
    #build Sample from ndarrays
    def build_sample(c0,c1,c2,c3,prediction):
        feature = np.array([c0, c1, c2, c3]).flatten()
        label = np.array(prediction)
        #print("what ami", Sample.from_ndarray(feature, label))
        return Sample.from_ndarray(feature, label)
    rdd = ds.map(lambda t: build_sample(t[0],t[1],t[2],t[3],t[4]))
    return rdd

iris_rdd_train = array2rdd(iris_k_train)
iris_rdd_train.cache()
print("Training Count: " + str(iris_rdd_train.count()))

iris_rdd_test = array2rdd(iris_k_test)
iris_rdd_test.cache()
print ("Test Count: " + str(iris_rdd_test.count()))

see feedforward-iris.ipynb

There is a broken link between the /notebooks/feedforward-iris-pipeline.ipynb and the code which tries to load spark.read.csv("../data/iris/iris_training.csv",

Could you kindly point me to the csv?