Python - HDFS with high availability

Preamble

This article is for using Python on an HDFS with high availability option. The particularity of high availability is to have two namenodes for one HDFS, in case of failure.

Libraries dependency

from hdfs import InsecureClient

Downloading

try:
	client_hdfs = InsecureClient('http://nn1:50070','hdfs')
	ret = client_hdfs.download('my_hdfs_file','my_local_path')
except:
	try:
			client_hdfs = InsecureClient('http://nn2:50070','hdfs')
			ret = client_hdfs.download('my_hdfs_file','my_local_path')
	except:
			print("no namenode available")

Saagie User Group Wiki

Python - HDFS with high availability

Analytics

Preamble

Libraries dependency

Downloading

Related content