xiaogp / recsys_spark Goto Github PK

Spark SQL 实现 ItemCF，UserCF，Swing，推荐系统，推荐算法，协同过滤

Scala 100.00%

spark-sql collaborative-filtering recommender-system

recsys_spark's Issues

UserCF

    // 用户共现矩阵
    val item_user2 = item_user.flatMap { row =>
      val userlist = row.getAs[scala.collection.mutable.WrappedArray[Long]](1).toArray.sorted
      val result = new ArrayBuffer[(Long, Long, Double)]()
      for (i <- 0 to userlist.length - 2) {
        for (j <- i + 1 to userlist.length - 1) {
          result += ((userlist(i), userlist(j), 1.0 / math.log(1 + userlist.length))) // 热门商品惩罚
        }
      }
      result
    }.withColumnRenamed("_1", "useridI").withColumnRenamed("_2", "useridJ").withColumnRenamed("_3", "score")

这段代码对于大数据量会oom 每个item都会有上百万个user对他产生行为，大佬该怎么优化呢

在u2u2i的过程中，没有看到找相似user的过程。或者说，
val df_user_prefer2 = df_user_prefer1.withColumn("score", col("pref") * col("similar") * (lit(1) / log(col("sum_item") * hot_item_regular + math.E))).select("useridJ", "itemid", "score")
这一行，为什么可以直接去掉useridI这一项呢？

swing to use

it is obvious that the implementation of swing has joined many times,the effectiveness can be ensured ? And flatmap also bring about serious data skew.

Recommend Projects