Manual upload of big files to HDFS

Owned by Aurèle Morvan (Unlicensed)
Last updated: Jul 12, 2018


Your browser might prevent you from manually uploading files over a few gigabytes to HDFS via HUE.


A workaround is:

  1. Split file as a zip archive on your computer using 7-Zip (Windows) or the split utility (Linux) into multiples parts under 1 gigabyte (resulting in files named myfile.zip.001, myfile.zip.002, myfile.zip.003 ...)
  2. Upload the parts to HDFS via HUE

  3. Create and run a SQOOP job as follows (replace the values for the mypath and myzip variables accordingly):

    mypath="/hdfspath/to/data/"
myzip="name of myfile" # without the .zip extension

hadoop fs -chmod 777 "$mypath"

hadoop fs -ls "$mypath$myzip.zip".*

hadoop fs -cat "$mypath$myzip.zip".* > file.zip
ls -la
unzip file.zip -d "$myzip"
ls -la "$myzip/"

hadoop fs -put -f $( echo "$myzip/" | sed s/\ /\%20/g ) "$mypath"