Dataflow fails launching on M1: FAILED_TO_LOAD_NATIVE_LIBRARY

[briandealwis]

• Cloud Tools for Eclipse version: 1.8.4
• Google Cloud SDK version: 334.0.0
• Eclipse version: 2021‑03
• OS: macOS 11.2.3 for Apple M1
• Java version: 15.0.2

What did you do?

1. Created new Dataflow app
2. Tried to launch locally.

Execution failed:

Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:342)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:176)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
	at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:97)
	at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:89)
	at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
	at org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:52)
	at org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:107)
	at org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:86)
	at org.apache.beam.sdk.io.Read$Bounded.<init>(Read.java:93)
	at org.apache.beam.sdk.io.Read$Bounded.<init>(Read.java:88)
	at org.apache.beam.sdk.io.Read.from(Read.java:47)
	at org.apache.beam.sdk.transforms.Create$Values.expand(Create.java:328)
	at org.apache.beam.sdk.transforms.Create$Values.expand(Create.java:273)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
	at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
	at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:173)
	at com.example.dflow.StarterPipeline.main(StarterPipeline.java:53)

[briandealwis]

@suztomo notes that Beam 2.29 should have an updated Snappy version with an mac/aarch64 support.

[meltsufin]

@briandealwis is this still an issue?

[briandealwis]

It looks like Apache Beam 2.36.0 included support for Mac M1 (https://issues.apache.org/jira/browse/BEAM-11703). I have not tested.

[elefeint]

We'll have to retest this scenario.