Spark Tutorial - Web UI on WSL

$ pwd
/mnt/c/hadoop/pyskt_tutorial

evan@evan:/mnt/c/hadoop/pyskt_tutorial$ virtualenv venv

evan@evan:/mnt/c/hadoop/pyskt_tutorial$ source venv/bin/activate
(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ pip install pyspark
Requirement already satisfied: pyspark in ./venv/lib/python3.8/site-packages (3.2.1)
Requirement already satisfied: py4j==0.10.9.3 in ./venv/lib/python3.8/site-packages (from pyspark) (0.10.9.3)

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ mkdir data && cd data
(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial/data$ vi README.md

This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. As with the Scala and Java examples, we use a SparkSession to create Datasets. For applications that use custom classes or third-party libraries, we can also add code dependencies to spark-submit through its --py-files argument by packaging them into a .zip file (see spark-submit --help for details). SimpleApp is simple enough that we do not need to specify any code dependencies.

We can run this application using the bin/spark-submit script:

from pyspark.sql import SparkSession

logFile = "data/README.md"  # Should be some file on your system
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
logData = spark.read.text(logFile).cache()

numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))

input("Typing....")

spark.stop()

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ $SPARK_HOME/bin/spark-submit --master local[4] /mnt/c/hadoop/pyskt_tutorial/SimpleApp.py