...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
# Read from Hive
df_load = sparkSession.sql('SELECT * FROM example')
df_load.show() |
How to use on Data Fabric?
In order to run any PySpark job on Data Fabric, you must package your python source file into a zip file. Pay attention that the file name must be __main__.py
How to use on Data Fabric's Jupyter Notebooks?
Prior to spark session creation, you must add the following snippet:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
import os
os.environ["HADOOP_USER_NAME"] = "hdfs"
os.environ["PYTHON_VERSION"] = "3.5.2" |
As time of writing only 2 pythons versions are available: 3.5.2 and 2.7.13