Python - HDFS with high availability

Owned by Nicolas (Unlicensed)
Dec 07, 2018
1 min read

Preamble

This article is for using Python on an HDFS with high availability option. The particularity of high availability is to have two namenodes for one HDFS, in case of failure.

Libraries dependency

from hdfs import InsecureClient

Downloading

try:
	client_hdfs = InsecureClient('http://nn1:50070','hdfs')
	ret = client_hdfs.download('my_hdfs_file','my_local_path')
except:
	try:
			client_hdfs = InsecureClient('http://nn2:50070','hdfs')
			ret = client_hdfs.download('my_hdfs_file','my_local_path')
	except:
			print("no namenode available")