Subject

When you run a SparkSession on a notebook like Jupyter or Zeppelin you don't have access to the SparkUI. Here the solution to have access.

How to

• From the Saagie Manager launch a Jupyter Spark Python notebook or a Zeppelin notebook

• Get or create a SparkSession (

• Get the APPLICATION ID ("spark" is a SparkSession) :
  Example in Jupyter Python :

  Open

  

  • Scala : spark.sparkContext.applicationId

  • Python : spark.sparkContext.applicationId

  • Java : spark.sparkContext().applicationId(

• Get the PLATFORM ID. You can have this id in the URL of your notebook.
  Example with a Jupyter Notebook https://6-13755-sparkjupyter.prod.saagie.io/tree , the PLATFORM ID is "6"

• Get the DOMAIN. The domain is accessible in the URL.
  Example with a Jupyter Notebook https://6-13755-sparkjupyter.prod.saagie.io/tree , the DOMAIN is "prod.saagie.io"
  
  

• To finish the Spark UI is accessible by the combinaison https://platformId-applicationId.domain
  Example with previous values :
  https://6-1d43d1d3-aedc-42c4-92e1-4fe0f8d953e3-0022.prod.saagie.io