资源分类
39
0
0
This project Implement the article of :
Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest."Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. IEEE, 2008.
IForest On Spark use spark to sampling data, and separate each partitoin to a spark worker. Each partition train n isolate trees. The train process is runing on paralle mode.
The prediction uses all isolation trees trained by spark, to predict the outlier factors.
SKLearn Iforest:http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html
Comparation: SKLearn Iforest:http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html
SVM OneClass:
SVM OneClass Result:
IForest On Spark:
IForest On Spark Test Result:
Project rely on spark-2.1.0-bin-hadoop2.7.Download at :http://spark.apache.org/downloads.html
How To Use:
var prop = new IForestProperty
prop.max_sample = 5000
prop.n_estimators = 1500
prop.max_depth_limit = (math.log(prop.max_sample) / math.log(2)).toInt
prop.bootstrap = true
prop.partition = 10
var ift = new IForestOnSpark(prop) var data_mtx:DenseMatrix[Double] = ... (train data in matrix)
ift.fit(data_mtx, spark)
x: DenseVector[Double] = ... (test data) var output = ift.predict(x)
//Serialize model to HDFS
var if_seralizer = new IForestSerializer
if_seralizer.serialize("hdfs://127.0.0.1/ifserialized", ift) //Load model from HDFS
var if_loader = new IForestSerializer
var localmodel = if_loader.deserialize("hdfs://172.16.22.14:9000/ifserialized")
