我正在尝试在 Scala/Spark 中进行替换采样，定义每个类的概率。
这就是我在 R 中的做法。
# Vector to sample from
x <- c("User1","User2","User3","User4","User5")
# Occurenciens from which to obtain sampling probabilities
y <- c(2,4,4,3,2)
# Calculate sampling probabilities
p <- y / sum(y)
# Draw sample with replacement of size 10
s <- sample(x, 10, replace = TRUE, prom = p)
# Which yields (for example):
[1] "User5" "User1" "User1" "User5" "User2" "User4" "User4" "User2" "User1" "User3"
我怎样才能在 Scala / Spark 中做同样的事情？
斯卡拉：
def weightedSampleWithReplacement[T](data: Array[T],
weights: Array[Double],
n: Int,
random: Random): Array[T] = {
val cumWeights = weights.scanLeft(0.0)(_ + _)
val cumProbs = cumWeights.map(_ / cumWeights.last)
Array.fill(n) {
val r = random.nextDouble()
data(cumProbs.indexWhere(r < _) - 1)
}
}
Spark 有一种RDD.sample()无需替换即可采样的方法，但不能使用权重。但是，您可能可以按照上述方法调整该方法来执行此操作。