GitHub Page : exemple-pyspark-read-and-write
from pyspark.sql import SparkSession |
sparkSession = SparkSession.builder.appName("example-pyspark-read-and-write").getOrCreate() data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)] df = sparkSession.createDataFrame(data) |
# Write into HDFS df.write.csv("hdfs://cluster/user/hdfs/test/example.csv") |
This Code only shows the first 20 records of the file.
# Read from HDFS df_load = sparkSession.read.csv('hdfs://cluster/user/hdfs/test/example.csv') df_load.show() |
In order to run any PySpark job on Data Fabric, you must package your python source file into a zip file. Pay attention that the file name must be __main__.py
Prior to spark session creation, you must add the following snippet:
import os os.environ["HADOOP_USER_NAME"] = "hdfs" os.environ["PYTHON_VERSION"] = "3.5.2" |
At time of writing only 2 pythons versions are available: 3.5.2 and 2.7.13