Compiling Python using pyenv against Homebrew on Ubuntu 22.04 linux/amd64 container fails with "internal compiler error: Illegal instruction"

[colindean]

Additional Info

• I am using Windows Subsystem for Linux

Output of brew config

HOMEBREW_VERSION: 4.2.0
ORIGIN: https://github.com/Homebrew/brew
HEAD: ff899e5b3ff6731208564ef502db237e2627e8ba
Last commit: 3 days ago
Core tap JSON: 21 Dec 23:45 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 3.1.4 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: quad-core 64-bit amd_k8
Clang: N/A
Git: 2.34.1 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.5.0-14-generic x86_64 GNU/Linux
OS: Ubuntu 22.04.3 LTS
Host glibc: 2.35
/usr/bin/gcc: 11.4.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: 11.4.0
gcc: 13.2.0
xorg: N/A

Output of brew doctor

Your system is ready to brew.

Description of issue

Nearly all compilations fail with

Parser/parser.c:39076:1: internal compiler error: Illegal instruction

The file, line, and column are changing, of course.

The farthest the build has gotten is with using python-build directly with GCC 11:

MAKE_OPTS="-j" CC="$(brew --prefix gcc@11)/bin/gcc-11" python-build --keep --verbose 3.11.7 /usr/local

after setting some envvars:

export brew_prefix="$(brew --prefix 2>/dev/null || true)"
export CPPFLAGS="${CPPFLAGS:+$CPPFLAGS }-I${brew_prefix}/include"
export LIBS="${LIBS:+${LIBS% } }-L${brew_prefix}/lib -Wl,-rpath,${brew_prefix}/lib"

I also tried some others with python-build:

MAKE_OPTS="-j" CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" CC="$(brew --prefix gcc@11)/bin/gcc-11" python-build --verbose 3.11.7 /usr/local

And tried against GCC 9, 10, 11, 12, and 13 and pyenv with simply:

CC="$(brew --prefix gcc@13)/bin/gcc-13" pyenv install --verbose 3.11.7

And of course, simply:

pyenv install --verbose 3.11.7

is what I was really going for all along.

Using the container

docker run --platform linux/amd64 -it --rm -v "$(pwd):/src" ubuntu:latest bash

running in colima

$ colima status
INFO[0000] colima is running using QEMU                 
INFO[0000] arch: x86_64                                 
INFO[0000] runtime: docker                              
INFO[0000] mountType: sshfs                             
INFO[0001] socket: unix:///Users/colin/.colima/default/docker.sock

on an M1 Pro MBP, the setup steps to reproduce this:

curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh | NONINTERACTIVE=1 bash -

(
	echo
	echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"'
) >>~/.bashrc

eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

echo 'export PYENV_ROOT="$HOME/.pyenv"' >>~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >>~/.bashrc
echo 'eval "$(pyenv init -)"' >>~/.bashrc

echo 'export PYENV_ROOT="$HOME/.pyenv"' >>~/.profile
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >>~/.profile
echo 'eval "$(pyenv init -)"' >>~/.profile

export PYENV_ROOT="${HOME}/.pyenv"
[[ -d ${PYENV_ROOT}/bin ]] && export PATH="${PYENV_ROOT}/bin:${PATH}"
eval "$(pyenv init -)"

(
	echo
	echo 'if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi'
) >>~/.bashrc
if which pyenv-virtualenv-init >/dev/null; then eval "$(pyenv virtualenv-init -)"; fi

# allow commands installed here, such as poetry
export PATH="$HOME/.local/bin:$PATH"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.profile

# things below here, I've copied in from a Makefile for ease of troubleshooting
# 

brew bundle install --no-lock --verbose --file=Brewfile
# this installs pyenv and python3's runtime and development deps
pyenv install --verbose --skip-existing 3.11.7
pyenv use 3.11.7
curl -sSL https://install.python-poetry.org | python3 -
poetry install

(plus some stuff that sets up certificates for inside my corporate firewall, irrelevant to this process beyond establishing connectivity)

Some of these steps may be redundant; this is a test container for a Python project's build system. The actual users of this are on macOS, but testing the setup has to be done in Linux, if possible, since we have no macOS CI runners available to us. The setup script will fail when it tries to install Python with pyenv; the Python build will fail with the error, or the same error on a different file or line.

Parser/parser.c:39076:1: internal compiler error: Illegal instruction

I found https://github.com/Homebrew/linuxbrew-core/issues/1683. It's very long and seems related largely to the O.P. locating their Linuxbrew installation outside of the blessed /home/linuxbrew/.linuxbrew, where mine definitely is located. This simple g++ test fails:

$ echo 'int main() { float f = 1.1F; }' > test.cpp
$ $(brew --prefix gcc@11)/bin/g++-11 test.cpp                                        
test.cpp:1:1: internal compiler error: Illegal instruction
    1 | int main() { float f = 1.1F; }
      | ^~~
0x172e4f7 internal_error(char const*, ...)
	???:0
0x7f0127248f1d __gmpn_popcount
	???:0
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://github.com/Homebrew/homebrew-core/issues> for instructions.

So, I think something's bugged in my GCC installations. I'm able to reproduce this reliably, though. Compiler bugs are rare, esp. in GCC so I'm at a loss. Help?

It seems like from this article that compiling specifying Homebrew's CC should be adequate, e.g.

CC="$(brew --prefix gcc)/bin/gcc-11" pyenv install --verbose 3.10.0

which was the first thing I tried when the builds failed.

Some other things I'm reviewing:

• Cannot find dependencies if they are installed with Linuxbrew pyenv/pyenv#2426
• python-build fails on Linux when homebrew (brew) on Linux is installed pyenv/pyenv#2159
• readline extension was not compiled, installed pyenv by Linuxbrew on Ubuntu 16 pyenv/pyenv#1479

[Bo98]

I think what to do here is to start narrowing this down. You mention Python a lot, but the GCC crash seems to indicate that Python is irrelevant. It could very easily be an issue with QEMU.

Here's an example I ran with a OrbStack + Rosetta. Check and see if this works or not under your Colima setup:

$ docker run --platform linux/amd64 -it --rm homebrew/ubuntu22.04:master
linuxbrew@a302fb021ebe:~$ brew install gcc@11
Running `brew update --auto-update`...
==> Homebrew collects anonymous analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).

==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations

==> Downloading https://ghcr.io/v2/homebrew/core/gcc/11/manifests/11.4.0-1
######################################################################### 100.0%
==> Fetching dependencies for gcc@11: gmp, isl, mpfr, libmpc, lz4, xz, zlib, zstd and binutils
==> Downloading https://ghcr.io/v2/homebrew/core/gmp/manifests/6.3.0
######################################################################### 100.0%
==> Fetching gmp
==> Downloading https://ghcr.io/v2/homebrew/core/gmp/blobs/sha256:3dca3544faca88
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/isl/manifests/0.26
######################################################################### 100.0%
==> Fetching isl
==> Downloading https://ghcr.io/v2/homebrew/core/isl/blobs/sha256:db14ba1e4ea23a
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/mpfr/manifests/4.2.1
######################################################################### 100.0%
==> Fetching mpfr
==> Downloading https://ghcr.io/v2/homebrew/core/mpfr/blobs/sha256:18857bac44d9f
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libmpc/manifests/1.3.1
######################################################################### 100.0%
==> Fetching libmpc
==> Downloading https://ghcr.io/v2/homebrew/core/libmpc/blobs/sha256:f6542ae5bcf
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/lz4/manifests/1.9.4
######################################################################### 100.0%
==> Fetching lz4
==> Downloading https://ghcr.io/v2/homebrew/core/lz4/blobs/sha256:1757fefc3840e1
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/xz/manifests/5.4.5
######################################################################### 100.0%
==> Fetching xz
==> Downloading https://ghcr.io/v2/homebrew/core/xz/blobs/sha256:50675435a60d119
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/zlib/manifests/1.3
######################################################################### 100.0%
==> Fetching zlib
==> Downloading https://ghcr.io/v2/homebrew/core/zlib/blobs/sha256:a47a3b5c8ee68
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/zstd/manifests/1.5.5-1
######################################################################### 100.0%
==> Fetching zstd
==> Downloading https://ghcr.io/v2/homebrew/core/zstd/blobs/sha256:96afcbd191c99
######################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/binutils/manifests/2.41_1
######################################################################### 100.0%
==> Fetching binutils
==> Downloading https://ghcr.io/v2/homebrew/core/binutils/blobs/sha256:ff5d14a01
######################################################################### 100.0%
==> Fetching gcc@11
==> Downloading https://ghcr.io/v2/homebrew/core/gcc/11/blobs/sha256:ec2b9b1a17f
######################################################################### 100.0%
==> Installing dependencies for gcc@11: gmp, isl, mpfr, libmpc, lz4, xz, zlib, zstd and binutils
==> Installing gcc@11 dependency: gmp
==> Downloading https://ghcr.io/v2/homebrew/core/gmp/manifests/6.3.0
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/70a72a71216843d66a953c06ff6337445ce9bc94fae9f0e301e2f59005274a8e--gmp-6.3.0.bottle_manifest.json
==> Pouring gmp--6.3.0.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/gmp/6.3.0: 23 files, 3.9MB
==> Installing gcc@11 dependency: isl
==> Downloading https://ghcr.io/v2/homebrew/core/isl/manifests/0.26
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/ec0bbec77171645273dd59a9ccfdf290deccd999b3ca2082e0478d282b777ec0--isl-0.26.bottle_manifest.json
==> Pouring isl--0.26.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/isl/0.26: 74 files, 9.8MB
==> Installing gcc@11 dependency: mpfr
==> Downloading https://ghcr.io/v2/homebrew/core/mpfr/manifests/4.2.1
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/a2a3424f4974f6febfa0334a93f35f508eaef3f4ad04320f73d9498302295635--mpfr-4.2.1.bottle_manifest.json
==> Pouring mpfr--4.2.1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/mpfr/4.2.1: 31 files, 3.9MB
==> Installing gcc@11 dependency: libmpc
==> Downloading https://ghcr.io/v2/homebrew/core/libmpc/manifests/1.3.1
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/fdfa98e0f8bb3ce075cb32776ac2345aa2f89252706c162aecfc841085fa76be--libmpc-1.3.1.bottle_manifest.json
==> Pouring libmpc--1.3.1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/libmpc/1.3.1: 13 files, 614.8KB
==> Installing gcc@11 dependency: lz4
==> Downloading https://ghcr.io/v2/homebrew/core/lz4/manifests/1.9.4
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/379e59b981667f9585b33a2ff318769d8edca3ce6fd2e9a67ed291ae3e0cc872--lz4-1.9.4.bottle_manifest.json
==> Pouring lz4--1.9.4.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/lz4/1.9.4: 22 files, 659.8KB
==> Installing gcc@11 dependency: xz
==> Downloading https://ghcr.io/v2/homebrew/core/xz/manifests/5.4.5
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/4e81fda476fb634a7e1ac650019bfe768a65d6c387015992df4cd75adf9b3fce--xz-5.4.5.bottle_manifest.json
==> Pouring xz--5.4.5.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/xz/5.4.5: 313 files, 3.8MB
==> Installing gcc@11 dependency: zlib
==> Downloading https://ghcr.io/v2/homebrew/core/zlib/manifests/1.3
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/b4c20c0489e59e6d8c5f8e4abdd47f07d52fe38699b6f0fea76781631025eecd--zlib-1.3.bottle_manifest.json
==> Pouring zlib--1.3.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/zlib/1.3: 13 files, 441.3KB
==> Installing gcc@11 dependency: zstd
==> Downloading https://ghcr.io/v2/homebrew/core/zstd/manifests/1.5.5-1
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/fc13698599720a53800064b40cddd854399651cf8760d9b0e46092f2e8da701a--zstd-1.5.5-1.bottle_manifest.json
==> Pouring zstd--1.5.5.x86_64_linux.bottle.1.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/zstd/1.5.5: 31 files, 2.7MB
==> Installing gcc@11 dependency: binutils
==> Downloading https://ghcr.io/v2/homebrew/core/binutils/manifests/2.41_1
Already downloaded: /home/linuxbrew/.cache/Homebrew/downloads/5719c1ab7c1a761142ecbf7e99d196b1a5e6c1cae43737424dae6237609d4b86--binutils-2.41_1.bottle_manifest.json
==> Pouring binutils--2.41_1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/binutils/2.41_1: 4,729 files, 469.0MB
==> Installing gcc@11
==> Pouring [email protected]_64_linux.bottle.1.tar.gz
==> Creating the GCC specs file: /home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.4.0
🍺  /home/linuxbrew/.linuxbrew/Cellar/gcc@11/11.4.0: 2,192 files, 347.8MB
==> Running `brew cleanup gcc@11`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
linuxbrew@a302fb021ebe:~$ echo 'int main() { float f = 1.1F; }' > test.cpp
linuxbrew@a302fb021ebe:~$ $(brew --prefix gcc@11)/bin/g++-11 test.cpp 

If you don't have the issue with the above, try slim down your setup as much as possible, e.g. run basic GCC 11 tests and remove setup steps to see what might trigger the issue.

[colindean]

Well, guess what failed ;-)

linuxbrew@56127b3f6fc5:~$ echo 'int main() { float f = 1.1F; }' > test.cpp
linuxbrew@56127b3f6fc5:~$ brew install gcc@11
…
linuxbrew@56127b3f6fc5:~$ <snip struggles with setting up certs>
linuxbrew@56127b3f6fc5:~$ $(brew --prefix gcc@11)/bin/g++-11 test.cpp
test.cpp:1:1: internal compiler error: Illegal instruction
…

Qemu bug?

[Bo98]

Either that or you need to configure Colima to emulate CPUs with popcnt (Nehalem or later). I imagine that should be possible. QEMU's default CPU type I believe is quite conservatively old.

I'm surprised GCC is using popcnt since we target Core2 on Linux - I will investigate that eventually.

If all fails, a Colima Rosetta profile might work.

[colindean]

That was it; I needed to specify to QEMU to use Nehalem in the Colima configuration in ~/.colima/default/colima.yaml:

cpuType: "Nehalem,check"

Edit: I ended up needing to go even newer and found that Skylake-Client-v3 instead of Nehalem worked best for me to silence warnings about Nehalem lacking AVX.

[carlocab]

I'm surprised GCC is using popcnt since we target Core2 on Linux - I will investigate that eventually.

Pretty sure that the GCC build includes a bootstrap stage which is used to build the installed compiler. So our compiler shims are used only for the bootstrap compiler and not the installed one.

[Bo98]

Pretty sure that the GCC build includes a bootstrap stage which is used to build the installed compiler. So our compiler shims are used only for the bootstrap compiler and not the installed one.

It seems to crash in gmp though which we have flags to use core2 on, and where __gmpn_popcount should be from asm:

config.status: linking mpn/x86_64/core2/popcount.asm to mpn/popcount.asm

But I agree in general that avoiding the shim for the target compiler is also not ideal.

[carlocab]

IIRC gmp does runtime detection of CPU features, so I'm not surprised it contains a popcnt instruction. It's also now less clear where the bug is exactly though...

[Bo98]

There is --enable-fat that does the runtime detection you describe, but we don't seem to be using it (sounds like we should?).

Though mpn_popcount doesn't appear to be in the fat function list so it might not solve the specific issue here.