Pyspark - Read & Write files from Hive

GitHub Page : exemple-pyspark-read-and-write

Common parts

Libraries dependency

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession, HiveContext

Set Hive metastore uri

Initialise Hive metastore
SparkContext.setSystemProperty("hive.metastore.uris", "thrift://nn1:9083")

Creating Spark Context & Connection

Create Spark Session
sparkSession = (SparkSession
data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)]
df = sparkSession.createDataFrame(data)

How to write a table into Hive?

Code example

Use SQL queries to create a table

Create an Hive table
# Write into Hive

How to read a table from Hive?

Code example

Use SQL queries to read a table

Create an Hive table
# Read from Hive
df_load = sparkSession.sql('SELECT * FROM example')

How to use on Data Fabric?

In order to run any PySpark job on Data Fabric, you must package your python source file into a zip file. Pay attention that the file name must be

How to use on Data Fabric's Jupyter Notebooks?

Prior to spark session creation, you must add the following snippet:

Notebook configuration
import os

os.environ["HADOOP_USER_NAME"] = "hdfs"
os.environ["PYTHON_VERSION"] = "3.5.2"

At time of writing only 2 pythons versions are available: 3.5.2 and 2.7.13