| URL: | http://markahall.blogspot.co.nz/2017/07/integrating-spark-mllib-into-weka.html | |
| Author: | Mark Hall <mhall{[at]}pentaho.com> | |
| Maintainer: | Mark Hall <mhall{[at]}pentaho.com> |
Development continuation of distributedWekaSpark that is compatible with Spark 2.x libraries. For Spark 1.6 comaptibility use distributedWekaSparkDev instead. Adds access to source files (CSV, Parquet and Avro) via Spark DataFrames, and adds support for both desktop and distributed execution of MLlib algorithms. Provides Spark wrappers for the classes in distributedWekaBase. Includes generic Spark 1.4.0 libraries - To run against Hadoop/HDFS, it is necessary to delete all the libraries, except those listed in the wekaSpark.props file, in ${user.home}/wekafiles/distributedWekaSpark2Dev/lib and copy in the libraries from the 'jars' directory of your Spark 2.x distribution.