libelf deprecation on macOS

[iMichka]

Hello

I am a maintainer of the Homebrew project and wanted to give a small heads up: we would like to deprecate libelf on macOS.

The upstream libelf website is gone, the remaining source code can only be through archive.org ... (since at least 2019, which was 5 years ago ...). avrdude is one of the last projects that is using libelf in macOS.

Are there any plans for a replacement solution on your side?

[mcuee]

@iMichka

What is the replacement library recommended by Homebrew project?

[mcuee]

As of now, I do not know of any plans to use an alternative library for the avrdude project.

BTW, for Windows MSVC build, we have a local version of libelf.
https://github.com/avrdudes/libelf

[mcuee]

Google says a potential "replacement" of libelf is elfutils. But then Homebrew says it is Linux only.
https://formulae.brew.sh/formula/elfutils
https://github.com/Homebrew/homebrew-core/blob/1cde8401d8be5f28d74a402afe67fd52ac0575ed/Formula/e/elfutils.rb

[mcuee]

Windows MSVC build: local fork version of the legacy libelf.
https://github.com/avrdudes/libelf

Windows: MSYS2 mingw uses legacy libelf.
https://packages.msys2.org/package/mingw-w64-x86_64-libelf

macOS: Macports uses legacy libelf
https://ports.macports.org/port/libelf/details/

macOS: Homebrew still uses legacy libelf now but mark as deprecated.
https://formulae.brew.sh/formula/libelf

Debian Linux -- using elfutils to provide libelf
https://packages.debian.org/search?keywords=libelf

Arch Linux -- using elfutils to provide libelf
https://archlinux.org/packages/core/x86_64/libelf/

FreeBSD/OpenBSD -- using elftoolchain to provide libelf
https://wiki.freebsd.org/LibElf
https://man.openbsd.org/elf
https://sourceforge.net/p/elftoolchain/wiki/Home/

NetBSD seems to be still using legacy libelf.
https://ftp.netbsd.org/pub/NetBSD/NetBSD-current/pkgsrc/devel/libelf/index.html
https://pkgsrc.se/devel/libelf

[mcuee]

Looks like there is no easy replacement of libelf for macOS now. So I think this may not be an easy task and we may not be able to deal with it anytime soon.

[dl8dtl]

Macports have a port of elftoolchain.
FreeBSD has a port of elfutils, so it's not quite clear why it is said to be "Linux only".
Other than this: the usual reason for deprecating something is that there's a better replacement available, rather than something not being developed anymore. There are many things that are no longer being developed since they are fairly complete, and just don't need further development. Take TaylorUUCP for an example: every MacOS these days still comes with it. ;-)

[mcuee]

FreeBSD has a port of elfutils, so it's not quite clear why it is said to be "Linux only".

I think it is from Homebrew project point of view. Homebrew supports macOS (normal Homebrew) and Linux. As of now, it seems Homebrew only supports elfutils under Linux and not macOS.

[mcuee]

Macports have a port of elftoolchain.

We need to check if it works for the purpose of avrdude though. If it works, then the solution is to ask Homebrew to provide elftoolchain.

[dl8dtl]

Macports have a port of elftoolchain.

We need to check if it works for the purpose of avrdude though.

Under FreeBSD, the (OS-provided) elftoolchain does the job. I no longer reference any 3rd-party ELF library in the devel/avrdude port.

Either way, my main point was, don't deprecate something unless there's a replacement which is at least as good as the deprecated item. FreeBSD still also offers the old libelf port (rather than kicking it out), but of course, installing it would conflict with elfutils then. You can only install one of those, because they install files into the same place.

[stefanrueger]

Thanks for the advance warning @iMichka. I agree with @dl8dtl that it seems unfortunate to deprecate elflib without an obvious replacement. AVR elf files won't have changed in the last decade or so, so that a library reading them doesn't need to be "new".

@mcuee @MCUdude Is is possible to drop the reliance of MacOS on libelf before the new AVRDUDE release next week?

[mcuee]

@stefanrueger

I think we should stick to libelf on macOS with avrdude 7.3 release.

Then we can explore the alternative for avrdude 7.4 release.

[mcuee]

Macports have a port of elftoolchain.

We need to check if it works for the purpose of avrdude though. If it works, then the solution is to ask Homebrew to provide elftoolchain.

I just took a look at elftoolchain macports and it is not in a good shape -- failed under Apple Silicon and almost no users (2 downloads in 30 days).
https://ports.macports.org/port/elftoolchain/details/

macports libelf is quite healthy (no issues with Apple Silicon, 641 downloads in 30 days)
https://ports.macports.org/port/libelf/details/

I think we can rule out the use of elftoolchain provided libelf as a candidate to replace libelf.

[mcuee]

As for elfutils, I do not see much efforts to port it to macOS. Google only finds one with the last commit date on 2019-Dec-24.
https://github.com/kdab-hotspot/elfutils

Unfortunately it does not seem to build well under my Mac Mini M1 (Apple macOS 14.3 with fully updated Homebrew). Good thing is that the sub libelf source codes seem to produce a static link library libelf.a.

mcuee@mcuees-Mac-mini build % pwd
/Users/mcuee/build/avr/avrdude_test/elfutils_macos/build

mcuee@mcuees-Mac-mini build %
CC=gcc-13 CFLAGS="-I/opt/homebrew/include -I/opt/homebrew/opt/gettext/include/" LDFLAGS="-L/opt/homebrew/opt/argp-standalone/lib/" PKG_CONFIG_PATH=/opt/homebrew/opt/libarchive/lib/pkgconfig ../configure --enable-maintainer-mode --disable-symbol-versioning

mcuee@mcuees-Mac-mini build % make
...
...
Making all in libcpu
  CC       i386_gendis.o
../../libcpu/i386_gendis.c: In function 'main':
../../libcpu/i386_gendis.c:70:10: error: 'error_message_count' undeclared (first use in this function)
   70 |   return error_message_count != 0;
      |          ^~~~~~~~~~~~~~~~~~~
../../libcpu/i386_gendis.c:70:10: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [i386_gendis.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

mcuee@mcuees-Mac-mini build % make -i
...
...
all kinds of errors
...
...
mcuee@mcuees-Mac-mini build % cd libelf 

mcuee@mcuees-Mac-mini libelf % ls -la libelf.a
-rw-r--r--  1 mcuee  staff  270096 Feb  3 13:05 libelf.a

mcuee@mcuees-Mac-mini libelf % file libelf.a
libelf.a: current ar archive

[iMichka]

What is the replacement library recommended by Homebrew project?

It's a complex situation. I have read all your comments, thanks for investigating.
One solution is maybe binutils that might bring equivalent tooling. It depends on what you were doing with libel.

I think we should stick to libelf on macOS with avrdude 7.3 release.
Then we can explore the alternative for avrdude 7.4 release.

That's fine, don't worry, there is no rush here.

elfutils

elfutils needs massive patching to be able to build it on macOS, so that's probably not the first solution I would consider.

[mcuee]

There is of course another option to bundle libelf with avrdude.

For example, we may use the following libelf for Windows and make it compatible with macOS.
https://github.com/avrdudes/libelf

[MCUdude]

@iMichka I have brew/avrdude related question. Avrdude 7.3 has a new dependency; libserialport. It's being used for automatic serial port discovery. However, Brew doesn't build Avrdude with libserialport. How can we change this?

% avrdude -curclock -patmega328p -Pch340
avrdude error: avrdude built without libserialport support; please compile again with libserialport installed
avrdude warning: serial adapter ch340 not connected to host
avrdude OS error: cannot open port ch340: No such file or directory
avrdude error: unable to open port ch340 for programmer urclock
avrdude error: avrdude built without libserialport support; please compile again with libserialport installed

avrdude done.  Thank you.

[mcuee]

@MCUdude

The forumla needs to be updated -- it is missing libserialport dependancy.

  #need to add the following line
  depends_on "libserialport"

Reference:
https://github.com/Homebrew/homebrew-core/blob/ca4f6490a40087a4789c8f8e934a21ed3a3bc233/Formula/a/avrdude.rb

  depends_on "cmake" => :build
  depends_on "hidapi"
  depends_on "libftdi"
  depends_on "libusb"
  depends_on "libusb-compat"
  #need to add the following line
  depends_on "libserialport"

  uses_from_macos "bison" => :build
  uses_from_macos "flex" => :build

  on_macos do
    # https://github.com/avrdudes/avrdude/issues/1653
    depends_on "libelf" => :build
  end

  on_linux do
    depends_on "elfutils"
    depends_on "readline"
  end

You may want to raise an issue here.
https://github.com/Homebrew/homebrew-core/issues

BTW, I usually use HEAD (to build from latest git main source codes) so I did not notice this issue.

brew install avrdude
brew unlink avrdude
brew install avrdude --HEAD         (to following avrdude git main)
brew reinstall avrdude           (to catch up with latest git main)

[mcuee]

Link to Homebrew issue here.

• Deprecated formulae with dependents Homebrew/homebrew-core#106902

libelf - has two dependents avrdude and dynamips. We've already migrated both of these to elfutils on Linux but we can't do so on macOS because elfutils is Linux-only.

One comment in the above issue discussion.

My libelf migration plan was elftoolchain (Homebrew/homebrew-core#104303) though not sure of actual compatibility.

But apparently the above efforts to get elftoolchain into Homebrew failed.

• elftoolchain 0.7.1 (new formula) Homebrew/homebrew-core#104303

[mcuee]

From the following discussion.

• libelf deprecation on macOS GNS3/dynamips#142 (comment)

Dynamips already points users to elfutils-libelf-devel on Redhat based systems, so it's already a compatible replacement.
Why not have a formula for elfutils-libelf in Homebrew?

The whole of elfutils might be hard due to linux-only stuff, but the libelf in elfutils by itself should be doable.
Then Homebrew can deprecate libelf in favor of elfutils-libelf.

[mcuee]

@MCUdude

Do you have the right for the following repo? If yes, please add me there as well. Thanks.
https://github.com/avrdudes/libelf

[dl8dtl]

@MCUdude

Do you have the right for the following repo? If yes, please add me there as well. Thanks. https://github.com/avrdudes/libelf

I added you all.

[mcuee]

@mariusgreuel has done a good job that avrdude-libelf actually builds under macOS. In fact it builds on Windows, Linux and macOS with CMake.

• To make this repo compatible with MSYS2 mingw and other OS (Linux, macOS, BSDs) libelf#1 (comment)

[mcuee]

Remove this from avrdude 7.4 milestone since there is no active development on this topic.

[flaviojs]

If the only issue is having source code available, then this page looks like a good new reference:
https://fossies.org/linux/misc/old/libelf-0.8.13.tar.gz/

[stefanrueger]

Sooo, we are planning to release AVRDUDE 8.0 sometime soon-ish. Does anyone (ideally with a Mac!) have a plan how to deal with the libelf deprecation on macOS? @MCUdude @mcuee is it a case of installing libelf source for the Mac and linking to it statically? Just to be clear I won't be able to do that but is anyone else?

[mcuee]

Sooo, we are planning to release AVRDUDE 8.0 sometime soon-ish. Does anyone (ideally with a Mac!) have a plan how to deal with the libelf deprecation on macOS? @MCUdude @mcuee is it a case of installing libelf source for the Mac and linking to it statically? Just to be clear I won't be able to do that but is anyone else?

My take is to close this issue as unplanned.

Worst case is just to change our CMake script to use our own libelf repo.
avrdudes/libelf#1