Github Project : example-spark-scala-read-and-write-from-hive
Common part
sbt Dependencies
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.6.1" % "provided" libraryDependencies += "com.databricks" %% "spark-csv" % "1.3.0"
assembly Dependency
// In build.sbt import sbt.Keys._ assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
// In project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1")
HDFS URI
HDFS URI are like that : hdfs://namenodedns:port
Default port is 8020.
Init SparkContext and HiveContext
// Configuration of SparkContext
val conf = new SparkConf().setAppName("example-spark-scala-read-and-write-from-hive")
// Creation of SparContext and HiveContext
val sc = new SparkContext(conf)
val hiveContext = new HiveContext(sc)
// Configuring hiveContext to find hive metastore
hiveContext.setConf("hive.metastore.warehouse.dir", params.hiveHost + "user/hive/warehouse")
How to write to a Hive table with Spark Scala?
Code example
// ====== Creating a dataframe with 1 partition
import hiveContext.implicits._
val df = Seq(HelloWorld("helloworld")).toDF().coalesce(1)
// ======= Writing files
// Writing Dataframe as a Hive table
hiveContext.sql("DROP TABLE IF EXISTS helloworld")
hiveContext.sql("CREATE TABLE helloworld (message STRING)")
df.write.mode("overwrite").saveAsTable("helloworld");
logger.info("Writing hive table : OK")
How to read from a Hive table with Spark Scala ?
Code example
// ======= Reading files
// Reading hive table into a Spark Dataframe
val dfHive = hiveContext.sql("SELECT * from helloworld")
logger.info("Reading hive table : OK")
logger.info(dfHive.show())