How to install rawpy on Mac M1？

[clearlon]

When I installed on MacOS with M1(arm) according to Installation from source on Linux/macOS , it showed that Successfully installed rawpy, but rawpy was not available.

Building wheels for collected packages: rawpy
  Building wheel for rawpy (setup.py) ... done
  WARNING: Legacy build of wheel for 'rawpy' created no files.
  Command arguments: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -u -c '
  exec(compile('"'"''"'"''"'"'
  #This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #- It imports setuptools before invoking setup.py, to enable projects that directly
  #import from `distutils.core` to work with newer packaging standards.
  #- It provides a clear error message when setuptools is not installed.
  #- It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #manifest_maker: standard file '"'"'-c'"'"' not found".
  #- It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/Users/clearlon/Downloads/rawpy/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /private/var/folders/3s/j3nhh0n94gn18m3ry7rt0zc80000gn/T/pip-wheel-g6enwm30
  Command output: [use --verbose to show]
  Running setup.py clean for rawpy
Failed to build rawpy
Installing collected packages: rawpy
  Running setup.py install for rawpy ... done
Successfully installed rawpy

[kmilos]

Duplicate of #167 and #170?

[JQuezada0]

@clearlon Not sure if you're still having this issue, and to be clear, I have no idea what I'm doing. But I ran into the same exact issue while trying to install something else that listed rawpy as a requirement in a requirements.txt and ultimately managed to get it installed.

My machine info:

macOS Moneterey - 12.2.1
MacBook Pro (14-inch, 2021)
Apple M1 Pro

I'm going to be intentionally super verbose for anyone else scouring google for answers.

I got this error first

ERROR: Could not find a version that satisfies the requirement rawpy (from versions: none)
ERROR: No matching distribution found for rawpy

Which then led to this issue I believe #167. And then that led to these instructions: https://github.com/letmaik/rawpy#installation-from-source-on-linuxmacos, where I ended up getting some behavior similar to you, but also different

  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from rawpy==0.17.1) (1.22.4)
Using legacy 'setup.py install' for rawpy, since package 'wheel' is not installed.
Installing collected packages: rawpy
  Running setup.py install for rawpy ... done
Successfully installed rawpy

It says successful, but in reality rawpy could not be found or used. But there's a peculiar line in there about wheel

Using legacy 'setup.py install' for rawpy, since package 'wheel' is not installed.

So I went ahead and installed wheel via pip3 install wheel. That went well, so I went ahead and tried installing from the rawpy repo again pip3 install ., and that's how I got to the same exact error as you

Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from rawpy==0.17.1) (1.22.4)
Building wheels for collected packages: rawpy
  Building wheel for rawpy (setup.py) ... done
  WARNING: Legacy build of wheel for 'rawpy' created no files.
  Command arguments: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize

  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)

  __file__ = %r
  sys.argv[0] = __file__

  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"

  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/Users/johnilquezada/Development/rawpy/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /private/var/folders/pp/h4fhyjq566v4gqvcfxdd2yrw0000gn/T/pip-wheel-_5q21bkb
  Command output: [use --verbose to show]
  Running setup.py clean for rawpy
Failed to build rawpy
Installing collected packages: rawpy
  Running setup.py install for rawpy ... done
Successfully installed rawpy

Says successful but it actually failed.

Then I happened to find this issue microsoft/vscode-python#15057. I followed the repro steps in there (but I changed the pip command to pip3):

1. pip3 install numpy cython
2. sudo TMPDIR=. pip3 install --verbose --no-clean .

This one ended in the same error again, but the output was slightly different. Now it mentioned sh: cmake: command not found, so in a different terminal I ran brew install cmake

Once that finished I went back and re-ran sudo TMPDIR=. pip3 install --verbose --no-clean.

Finally that actually truly installed rawpy, and going back to what I was originally trying to install made it past that step.

I'm now stuck on a different part, but hopefully this at least helps with getting rawpy installed on an M1 Mac.

[letmaik]

Keep in mind that there is always the option to use the amd64 version of Python on M1. It may be a little bit slower than the native one but I doubt it's a practical issue in most cases.

[JQuezada0]

@letmaik Thanks for the heads up. I just tried this via Anaconda and it appears to suffer from the same issue. I added the correct channel, and whether it's via the GUI or CLI, it seems to think rawpy doesn't exist.

Edit: I just went with making a docker image. Everything works fine in there.

[moebaid]

I've managed to successfully install from source on Mac M1. Initially, I had the same errors as @JQuezada0 and followed their solutions, up until running brew install cmake. After this I still had errors running sudo make install (the final step of libraw installation) due to issues with OpenMP. It seems others have had issues installing and using OpenMP on Mac M1 -- I ran the following in a new terminal:

brew install llvm
cd /opt/homebrew/opt/libomp/lib

cd /usr/local/lib
sudo ln -s /opt/homebrew/opt/libomp/lib/libomp.dylib libomp.dylib

I also ran brew install libomp, though this might have been redundant. After that, I was able to successfully finish installing libraw then rawpy.

[weaming]

I installed universal2 format python3 from https://www.python.org/downloads/release/python-3108, then use /usr/bin/arch -x86_64 /Library/Frameworks/Python.framework/Versions/3.10/bin/pip to install rawpy from pypi.org

[kamikaze]

any news on being able to install via pip install?

[letmaik]

For anyone waiting, please follow this upstream issue: actions/runner-images#2187

[avengersassemble]

I was able to install rawpy following instructions mentioned by @moebaid
My machine info:

macOS Moneterey - 12.6.2
MacBook Pro (13-inch, M1, 2020)

brew install llvm
cd /opt/homebrew/opt/libomp/lib

cd /usr/local/lib
sudo ln -s /opt/homebrew/opt/libomp/lib/libomp.dylib libomp.dylib


python3 -m venv venv
source venv/bin/activate/
git clone https://github.com/letmaik/rawpy
cd rawpy
pip install wheel numpy cython
pip install .

It successfully installed rawpy in my site-packages. Later, I deleted rawpy directory.

[rafalstapinski]

I was ultimately able to install rawpy with

• (pyenv installed) python==3.10
• numpy==1.23.2
• M1 Pro
• macOS Monterey

Adding my experience to the above solutions for installing on M1 with another issue I ran into with compiling the Cython code due to what seems like a change in the numpy C-API.

The Cython compilation issue was in the two places where _rawpy.pyx sets ndarr.base = <PyObject*> self raising the following error Assignment to a read-only property. After looking through this issue, I replaced the two instances of setting ndarr.base with the following call to numpy's C-API:

# ndarr.base = <PyObject*> self
np.PyArray_SetBaseObject(ndarr, self)

After making that change, I was able to build/install rawpy with pip install .

[letmaik]

@rafalstapinski Can you open a pull request with the change for numpy? Thanks!

[EmersonDove]

Hey I built some wheels using my m2 mini local runner you can install with

pip install newrawpy

happy to add my runner to this repo if the owners are interested. This was just a quick patch fix on my end.

https://github.com/open-shade/newrawpy/

[letmaik]

@EmersonDove Thanks for trying this out, it's good to know that it works out of the box just by adding another CI matrix entry with arm64 as python arch. I won't be able to add external runners to this repo though as I couldn't provide any guarantees to users on the provenance of the machine and this would become an attack vector to inject malicious code into the macOS wheel. By using standard GitHub Actions runners there is at least some provenance from the fact that the machines are operated by Microsoft directly. Hopefully it won't take too long for M1 runners to arrive, see also github/roadmap#528 which currently states Q4 2023 – Oct-Dec.

[EmersonDove]

True, I think I would have the same reaction if someone offered that to me

[donB092]

need rawpy on m1 air, new to py - how to?

[nullus]

Sort of shoehorning my own solution in here in the form of #209

I had a similar problem awhile ago, and decided to build against the system libraw, installed using Homebrew:

brew install libraw
env RAWPY_USE_SYSTEM_LIBRAW=1 python setup.py install

Glad to hear the M1 runners will be available soon!

[wing-kit]

Sort of shoehorning my own solution in here in the form of #209

I had a similar problem awhile ago, and decided to build against the system libraw, installed using Homebrew:

brew install libraw
env RAWPY_USE_SYSTEM_LIBRAW=1 python setup.py install

Glad to hear the M1 runners will be available soon!

    In file included from rawpy/_rawpy.cpp:1137:
    rawpy/def_helper.h:1:10: fatal error: 'libraw_version.h' file not found
    #include "libraw_version.h"
             ^~~~~~~~~~~~~~~~~~
    1 warning and 1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    [end of output]

just tried to follow it on a M1 Mac. Did you experience this? @nullus

[nullus]

No, looks like it can't find the header path though. If you run pkg-config --cflags libraw do you get any output? It's possible that brew install pkg-config will fix it.

Edit: the standard install procedure will correctly build libraw as part of the rawpy installation too! From memory there may be some other build dependencies required: brew install cmake for one.

[wing-kit]

@nullus thanks. brew install pkg-config did the job.

[occamz]

Looked into this a bit.

There are already beta runners (paid): https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/running-jobs-on-larger-runners?platform=mac#available-macos-larger-runners

The next milestone is General Availability (GA). Expected to be ready: Q1 2024 – Jan-Mar: github/roadmap#819

"Once we reach the GA milestone, we will start working on additional plans to support open source, along with new macOS runners."
actions/runner-images#8439 (comment)

So might be a while still :)

[chbrown]

Here's a one-liner shot in the dark that happened to work for me in my env (M2, btw):

pip install 'git+https://github.com/letmaik/[email protected]#egg=rawpy'

You may want to update the tag.

[greiginsydney]

It fails when I try it @chbrown :

DEPRECATION: git+https://github.com/letmaik/[email protected]#egg=rawpy' contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at https://github.com/pypa/pip/issues/11617

(Environment is Raspberry Pi, 32-bit bookworm)

[lufrikas]

Hello, I'm trying to install rawpy on my M3 iMac. I am following the "Installation from source on Linux/macOS" as others have done. The installation itself works fine. However, when trying to use rawpy, I get the following error upon import rawpy:

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/rawpy/_rawpy.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6LibRaw11open_bufferEPKvm'

Any ideas?

[kgantchev]

Hopefully it won't take too long for M1 runners to arrive, see also github/roadmap#528 which currently states Q4 2023 – Oct-Dec.

You can use the FlyCI's M1 and M2 runners. Our runners are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).

Install Instructrions

Step 1: Install the FlyCI GitHub app.
Step 2: Change the runs-on flag to use the FlyCI label (see below).

Example with M1 runners:

jobs:
 ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m1
   steps:
   - name: 👀 Checkout repo
     uses: actions/checkout@v4

Example with M2 runners:

jobs:
  ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m2
    steps:
      - name: 👀 Checkout repo
        uses: actions/checkout@v4

500 mins/month Free for Public Repos

If your repo is public, then FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner.

Best Regards,
Kiril Gantchev
CEO and co-founder of FlyCI