Sunday, 19 June 2016

2 - Setup Spark Development Environment on windows with Eclipse  & SBT

Pre-Requisite
1. Java 8 ( At least , appropriate bit version)
2. Scala ( Scala and IDE)
3. Maven ( Download and Install)
4. SBT ( Simple Building Tool)

Download Pages
www.spark.apache.org
www.scala-lang.org
www.scala-sbt.org

Setup Environment Variables:




Apache Spark

               Downloading and Running Spark on windows

Spark is 3rd generation distributed technologies,

Download build using below URL.

1) www.spark.apache.org -> Choose Spark-build with pre-build for hadoop 2.x version
2) Extract Download and bin folder. 
3) Open prompt and navigate Saprk/bin location.
4) Type Spark-Shell and Opens Scala> prompt
5) Open Spark Admin UI and type below URL

http://localhost:4040 ( Spark Web UI)

Sample Program to Test Spark Environment:
> val  v1 = 1 to 100
> val dv1 = sc.parallelize(v1)
> dv1.filter(_<10).collect()
> val result = dv1.filter(_<10).collect()
> result

Output: array of numbers ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9)