Step 1: Preparation

  • Install Java JDK (1.8 or 11)
  • Install Python 3.9.x

Step 2: Download Apache Spark

  1. Navigate to https://spark.apache.org/downloads.html.
  2. Choose the Spark release and package type as shown below:
Choose Spark release and package type

You can download this version: spark-3.5.4-bin-hadoop3.tgz

After downloading, open a new terminal and run the following command to verify the checksum:

certutil -hashfile spark-3.5.4-bin-hadoop3.tgz SHA512

Step 3: Configure environment variables

  1. Create the empty folders C:\Hadoop\bin and C:\Spark.
  2. Download winutils.exe and copy it to C:\Hadoop\bin.
  3. Copy the extracted spark-3.5.4-bin-hadoop3 folder to C:\Spark.
  4. Create the following environment variables: 
    SPARK_HOME=C:\Spark\spark-3.5.4-bin-hadoop3
HADOOP_HOME=C:\Hadoop
  5. Add %SPARK_HOME%\bin and %HADOOP_HOME%\bin to the Path variable.

Step 4: Launch Spark and test

Close the old terminal, open a new one and type:

spark-shell
spark-shell

Open a web browser and go to http://localhost:4040/.

Spark UI executors

Test

Create a text file named testhadoop.txt with the following content:

12345678910
HelloWorld

Then run in spark-shell:

scala> val x = sc.textFile("testhadoop.txt")
x: org.apache.spark.rdd.RDD[String] = testhadoop.txt MapPartitionsRDD[1] at textFile at <console>:24

scala> x.take(11).foreach(println)
12345678910
HelloWorld

scala> val y = x.map(_.reverse)
y: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at map at <console>:25

scala> y.take(11).foreach(println)
01987654321
dlroWolleH

That's all for getting started with Apache Spark.

Reference: https://phoenixnap.com/kb/category/devops-and-development