Spark

(base) ~$ sudo apt-get update -y
(base) ~$ sudo apt-get upgrade -y
(base) ~$ sudo apt install git
(base) ~$ git --version

$ pwd
/mnt/c/hadoop/pyskt_tutorial

evan@evan:/mnt/c/hadoop/pyskt_tutorial$ virtualenv venv

evan@evan:/mnt/c/hadoop/pyskt_tutorial$ source venv/bin/activate
(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ pip install pyspark
Requirement already satisfied: pyspark in ./venv/lib/python3.8/site-packages (3.2.1)
Requirement already satisfied: py4j==0.10.9.3 in ./venv/lib/python3.8/site-packages (from pyspark) (0.10.9.3)

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ mkdir data && cd data
(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial/data$ vi README.md

This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. As with the Scala and Java examples, we use a SparkSession to create Datasets. For applications that use custom classes or third-party libraries, we can also add code dependencies to spark-submit through its --py-files argument by packaging them into a .zip file (see spark-submit --help for details). SimpleApp is simple enough that we do not need to specify any code dependencies.

We can run this application using the bin/spark-submit script:

from pyspark.sql import SparkSession

logFile = "data/README.md"  # Should be some file on your system
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
logData = spark.read.text(logFile).cache()

numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))

input("Typing....")

spark.stop()

(venv) evan@evan:/mnt/c/hadoop/pyskt_tutorial$ $SPARK_HOME/bin/spark-submit --master local[4] /mnt/c/hadoop/pyskt_tutorial/SimpleApp.py

$ sudo apt-get install openjdk-8-jdk
$ sudo wget https://archive.apache.org/dist/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz
$ sudo tar -xvzf spark-3.2.0-bin-hadoop3.2.tgz

evan@evan:/mnt/c/hadoop$ pwd
/mnt/c/hadoop

evan@evan:/mnt/c/hadoop$ ls
spark-3.2.0-bin-hadoop3.2  spark-3.2.0-bin-hadoop3.2.tgz

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export SPARK_HOME=/mnt/c/hadoop/spark-3.2.0-bin-hadoop3.2
export PATH=$JAVA_HOME/bin:$PATH
export PATH=$SPARK_HOME/sbin:$PATH
export PYSPARK_PYTHON=/usr/bin/python3

evan@evan:/mnt/c/hadoop/spark-3.2.0-bin-hadoop3.2$ pwd
/mnt/c/hadoop/spark-3.2.0-bin-hadoop3.2
evan@evan:/mnt/c/hadoop/spark-3.2.0-bin-hadoop3.2$ pysaprk
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/04/19 19:39:57 WARN Utils: Your hostname, evan resolves to a loopback address: 127.0.1.1; using 172.17.233.202 instead (on interface eth0)
22/04/19 19:39:57 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/04/19 19:39:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.2.0
      /_/

Using Python version 3.8.10 (default, Mar 15 2022 12:22:08)
Spark context Web UI available at http://172.17.233.202:4040
Spark context available as 'sc' (master = local[*], app id = local-1650364801421).
SparkSession available as 'spark'.
>>> rd = sc.textFile("README.md")
>>> rd.count()
109

$ python --version
Python 3.8.7

$ java --showversion

$ java --showversion
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.