Upgrade to Debian 10, Java 11, Hadoop 3.3.1, build/run on ARM

[sebastian-nagel]

• upgrade to Debian 10, Java 11, Hadoop 3.3.1

• allow to build/run on ARM architecture (aarch64 / arm64)

• note: this PR includes Makefile build: all Docker builds to be based on newly built hadoop-base image #103, Avoid hard-coded Hadoop version in file paths #104, Docker-compose: network name and port fixup #105

• tested both on x86_64 / amd64 and aarch64 / arm64

  • build (make)
  • launch cluster (docker-compose up)
  • UI (namenode, resourcemanager)
  • wordcount (make wordcount)

• status of native library support (here ARM):

$> hadoop checknative
2021-07-02 15:22:55,211 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-07-02 15:22:55,214 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-07-02 15:22:55,218 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2021-07-02 15:22:55,253 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /opt/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /usr/lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/aarch64-linux-gnu/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

$> hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T10:51Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /opt/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar

[jacobmstein]

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

[OneCricketeer]

Hadoop 3.3.3 officially supports Java 11 - https://hadoop.apache.org/docs/current/

[lemonGGGit]

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

@jacobmstein, Hello, Can you work it on 「Raspberry Pi 4 with Ubuntu Server」 now?