"illegal hardware instruction" error when running MUSCLE v5.2 and above on Apple M series with subdir osx_64

[ori-scala]

After installing the latest MUSCLE version (5.2), I'm getting the following error when trying to run MUSCLE.

illegal hardware instruction

I assume it's related to the Apple M series chipset, I'm using Apple M1 Pro.

To reproduce:

conda install bioconda::muscle==5.2.0 -y

Then run muscle --help and get:

muscle --help
[1]    14859 illegal hardware instruction  muscle --help

Then, downgrade using:

conda install bioconda::muscle==5.1.0 -y

and muscle --help works again with the following output:

muscle --help

muscle 5.1.osx64 []  34.4Gb RAM, 10 cores
Built May 15 2023 21:29:02
(C) Copyright 2004-2021 Robert C. Edgar.
https://drive5.com

This is help.txt

[ori-scala]

@apcamargo Thank you for the 5.3 build. Now both --version and --help are working well but running an actual alignment ends up with the same error:

[1]    42192 illegal hardware instruction  muscle -align  -output  -log

Maybe you can add that to the tests as well? Something like

muscle -align some_fasta_file.fasta -output output.fasta -log muscle_log.txt

[apcamargo]

Does -super5 work? It is included in the tests.

[ori-scala]

Nope, same error

[1]    42600 illegal hardware instruction  muscle -super5  -output  -log

[apcamargo]

Ok, so the problem might be on your end. Is your Mac a M series? Because the output of your muscle --help shows osx64, which is Intel. Maybe the binary of the wrong architecture is being installed.

How are you installing muscle?

[ori-scala]

The mac indeed has an M series chipset (M1 Pro) but I'm setting the conda env to osx64 using conda config --env --set subdir osx-64 as other packages I'm using aren't built for M series and I'm forced to do that.

I'm installing using an env.yml file or by conda install -c bioconda muscle==5.3.0 -y

[apcamargo]

I guess the other binaries are being executed through Rosetta? If so, I don't know why muscle isn't (assuming the issue is that the issue is that the binary is not compatible with the processor's architecture).

Do you have the same issue if you try to install other packages with ARM builds (e.g. diamond, mmseqs2, or hhsuite)?

I don't think I know enough about this to help you. If you're using Pixi, you can try to ask for help in their Discord (people there are very helpful). Otherwise, maybe @bioconda/osx can help?

[ori-scala]

You're correct, the other binaries are being executed through Rosetta. It doesn't happen with other packages nor with any muscle version prior to 5.2

[apcamargo]

In this case, I don't know what is going on. It could be something with the macOS ARM CI (as the build runs correctly there), but the core team would know better than me.

[ori-scala]

Ok, thank you anyway Antônio!

[apcamargo]

If you create an environment only for muscle and install the ARM version, does it work?

[ori-scala]

Yep, that works as expected.

[apcamargo]

This explains why the tests didn't catch this. From my understanding, the only major difference between the Bioconda build and Robert's build is that he is using GCC and Bioconda uses clang. As far as I know, there's no way for us to use GCC to build for macOS, but I don't really know our building system that well.

As you said, version 5.1 worked in Rosetta and it seems he used Clang back then (I'm inferring this from his CI). But the code changed a lot since then...

[apcamargo]

Can you update the issue title to something that reflects better the info we gathered?

[ori-scala]

@apcamargo updated the title as requested

[milot-mirdita]

Seems muscle is using AVX(2/512/...) in version 5.2. Rosetta does not emulate AVX.
Here is likely the issue:
https://github.com/bioconda/bioconda-recipes/blob/b310cef61eebed800970d596e79e059c8d9b91d5/recipes/muscle/vcxproj_make.py#L74C1-L75C43

Its probably not a good idea to use march native for conda packages.

For MMseqs2 I was explicitly setting to use up to SSE4_1 in comda, with the latest release 16 I made MMseqs2 use AVX2 in conda, so the same issue would likely happen now if someone were to use it in a Rosetta emulated environment.

[apcamargo]

Thanks, @milot-mirdita! I had no idea Rosetta doesn't emulate AVX.

I guess this is. Matter of how much MUSCLE benefits from AVX. If the performance increase is substantial, it might be worth it to leave as is. Perhaps @rcedgar has a thought about this

[milot-mirdita]

No matter what, conda recipes shouldn't be compiled with march native, since this will result in binaries that use the instruction set of whatever GitHub runner happens to compile the binary. Most likely recent intel machines with avx512.

Conda's default is I think nocona, which is ancient. A good compromise might be using the new x86-64-v3 psABI specifier.

[rcedgar]

I don't think muscle has any explicit SIMD code, if I'm not forgetting something then any AVX instructions are compiler optimizations.