Docker build is broken

[t6tv8e]

Expected Behavior

After running docker build --rm -t gpt-engineer -f docker/Dockerfile . I should have a runnable container.

Current Behavior

Unable to build the container due to PyArrow and CMake issues.

Failure Information

Failure Logs

  creating /tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241/build/temp.linux-x86_64-cpython-311
43.03       -- Running cmake for PyArrow
43.03       cmake -DCMAKE_INSTALL_PREFIX=/tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241/build/lib.linux-x86_64-cpython-311/pyarrow -DPYTHON_EXECUTABLE=/usr/local/bin/python -DPython3_EXECUTABLE=/usr/local/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_AZURE=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /tmp/pip-install-673hp2i9/pyarrow_d35c22e7f3e947faaa29331bd28d4241
43.03       error: command 'cmake' failed: No such file or directory
43.03       [end of output]
43.03
43.03   note: This error originates from a subprocess, and is likely not a problem with pip.
43.03   ERROR: Failed building wheel for pyarrow

[Thinking80s]

command 'cmake' failed: No such file or directory

[viborc]

Hi @t6tv8e - I think the issue is related to the fact that cmake utility is not installed or not available in the system's PATH where you are trying to install the pyarrow package. You might want to check their docs.

Or, depending on your OS, install it along these lines (this should work for Ubuntu or Debian-based stuff:

sudo apt-get update
sudo apt-get install cmake

[t6tv8e]

I'm on macos and I have cmake on my machine

[viborc]

@t6tv8e, can you try running this command in your Terminal and share the output/screenshot here:

cmake --version

[t6tv8e]

[viborc]

Let's check if it's configured properly.

mkdir ~/cmake-test
cd ~/cmake-test
echo "cmake_minimum_required(VERSION 3.10)" > CMakeLists.txt
echo "project(HelloCMake)" >> CMakeLists.txt
cmake .

This should check if it is functioning correctly by creating a temporary project directory and attempting to configure a simple project.

If that fails, you might want to consider reinstalling it using homebrew or whatever you usually use like this:

brew install cmake

[t6tv8e]

[t6tv8e]

[viborc]

Hm... what's the output of your PATH?

echo $PATH

[t6tv8e]

That is:
/Users/t6t/.pyenv/shims:/Users/t6t/.nvm/versions/node/v20.12.1/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/t6t/.dotnet/tools

[viborc]

Hm... can you do this please:

which cmake

[t6tv8e]

/opt/homebrew/bin/cmake

[viborc]

Okay, I'll need to check this later today with the team. I'm unsure what the issue might be. Maybe you can try reinstalling the cmake and see if that makes any difference.

[t6tv8e]

Okay, thanks for your time already in any case

[viborc]

Of course, we are always trying to help!

[ErikBjare]

The issue is with cmake missing in the container, not the host system.

Seems the Dockerfile needs:

RUN apk add --update build-base or similar.

[t6tv8e]

Yeah, I've been tinkering with the Dockerfile.

FROM python:3.10-alpine AS builder

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \
    build-base

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.10-alpine

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \ 
    build-base

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["sh", "/app/entrypoint.sh"]

[t6tv8e]

Didn't work

[ErikBjare]

Identical error?

[t6tv8e]

Yes it's the same error...

[viborc]

Hey @t6tv8e, we just had our technical planning meeting and discussed this. One of our colleagues tried adding cmake to a Docker image, but that didn't solve the problem. We have a couple of other options we'll try out, but we'll keep you updated!

@ErikBjare, thanks for helping with this; I'm super thankful for checking on this, especially while you are away! 🙏

[zigabrencic]

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:

FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

[viborc]

That sounds good. Let's see what @t6tv8e says, and we can then figure out the next steps. Thanks for a quick turnaround, @zigabrencic! 💪

[t6tv8e]

@zigabrencic @viborc

That worked 👌

[viborc]

Perfect! We are now looking into testing this on Linux, Mac, and Windows and we'll see how to fix it for everyone. I'll leave this issue open for now just as a reminder to @zigabrencic and me to follow-up on this for other platforms.

Glad it worked for you @t6tv8e! Thanks, @zigabrencic and @ErikBjare for helping out!

[k1lgor]

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:

FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

@zigabrencic The no-cache flag is not the source of the issue. While pyarrow is supported on Alpine, it lacks a wheel. In other words, you need to build it from source.

I notice that the Dockerfile in #1144 will be reverted; use this one instead:

# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

The image size would still be reduced

And on the other hand, Docker containers are designed to run with an unprivileged user by default, and running commands with sudo, for reference RUN sudo pip install -e . , or as the root user can lead to potential security issues and unexpected behavior.

[zigabrencic]

Hey

Thank you for your suggestions.

Yes you are correct @k1lgor the cmake dependency broke this. I misstated this before.

Tried your proposal and:

• 1.) Image on Mac OS is still 1.43 GB's of size.
• 2.) Fails with: FileNotFoundError: [Errno 2] No such file or directory: 'git' once gpte engineer is executed.` Despite the fact that git is installed.

As a result I propose we keep the simpler version as implemented here.

The cross platform size benefit in my opinion isn't large enough to add the proposed complexity.

I made some updates to #1144:

• change the RUN sudo pip part to RUN pip => per @k1lgor proposal.
• python:3-11-slim to python:3-12-slim
• Added the line: pip install -U langchain-community => We might have to add this one elsewhere.

If there are no objections I'll open the PR again.

[k1lgor]

Hey @zigabrencic ,

I believe I may be able to locate the issue. tk tcl curl git are installed in /usr/bin/ since they are dpkg packages. It is necessary to add one COPY.

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
+ COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

And in addition, I have added langchain-community = "0.2.0" into pyproject.toml.
Could you test it on your Mac?

[zigabrencic]

Hey @k1lgor

Sure I can test it just please let me know which branch?

Or if you can provide the full docker file so I test the right thing :)

[k1lgor]

Here are the files. In the pyproject.toml I only added langchain-community ="0.2.0" and in Dockerfile - COPY --from=builder /usr/bin /usr/bin

Dockerifle

```bash
# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

pyproject.toml

[tool.poetry]
name = "gpt-engineer"
version = "0.3.0"
description = "Specify what you want it to build, the AI asks for clarification, and then builds it."
authors = ["Anton Osika <[email protected]>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/gpt-engineer-org/gpt-engineer"
repository = "https://github.com/gpt-engineer-org/gpt-engineer"
documentation = "https://gpt-engineer.readthedocs.io/en/latest/"
classifiers = [
  "Development Status :: 4 - Beta",
  "Topic :: Scientific/Engineering :: Artificial Intelligence",
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
openai = "^1.0"
termcolor = "2.3.0"
typer = ">=0.3.2"
rudder-sdk-python = ">=2.0.2"
dataclasses-json = "0.5.7"
tiktoken = ">=0.0.4"
tabulate = "0.9.0"
python-dotenv = ">=0.21.0"
langchain = ">=0.1.2"
langchain_openai = "*"
toml = ">=0.10.2"
tomlkit = "^0.12.4"
pyperclip = "^1.8.2"
langchain-anthropic = "^0.1.1"
regex = "^2023.12.25"
pillow = "^10.2.0"
datasets = "^2.17.1"
black = "23.3.0"
langchain-community ="0.2.0"

[tool.poetry.group.dev.dependencies]
pytest = ">=7.3.1"
pytest-cov = "^4.1.0"
mypy = "1.3.0"
ruff = ">=0.0.272"
pre-commit = "3.3.3"
tox = ">=3.0.0"

[tool.poetry.group.docs.dependencies]
autodoc_pydantic = ">=1.8.0"
myst_parser = ">=0.18.1"
nbsphinx = ">=0.8.9"
sphinx = ">=5.0.0"
sphinx-autobuild = ">=2021.3.14"
sphinx_book_theme = ">=0.3.3"
sphinx_rtd_theme = ">=1.0.0"
sphinx-typlog-theme = ">=0.8.0"
toml = ">=0.10.2"
myst-nb = ">=0.17.1"
linkchecker = ">=10.2.1"
sphinx-copybutton = ">=0.5.1"
markdown-include = ">=0.6.0"
sphinx_copybutton = ">=0.5.2"

[tool.poetry.scripts]
gpt-engineer = 'gpt_engineer.applications.cli.main:app'
ge = 'gpt_engineer.applications.cli.main:app'
gpte = 'gpt_engineer.applications.cli.main:app'
bench = 'gpt_engineer.benchmark.__main__:app'
gpte_test_application = 'tests.caching_main:app'

[tool.poetry.extras]
test = ["pytest", "pytest-cov"]
doc = ["autodoc_pydantic", "myst_parser", "nbsphinx", "sphinx", "sphinx-autobuild", "sphinx_book_theme", "sphinx_rtd_theme", "sphinx-typlog-theme", "myst-nb", "linkchecker", "sphinx-copybutton", "markdown-include", "sphinx_copybutton"]

[tool.ruff]
select = ["F", "E", "W", "I001"]
show-fixes = false
target-version = "py310"
task-tags = ["TODO", "FIXME"]
extend-ignore = ["E501", "E722"]

[tool.black]
target-version = ["py310"]

[tool.ruff.isort]
known-first-party = []
known-third-party = []
section-order = [
    "future",
    "standard-library",
    "third-party",
    "first-party",
    "local-folder",
]
combine-as-imports = true
split-on-trailing-comma = false
lines-between-types = 1

[tool.pytest.ini_options]
markers = [
    "requires_key: marks tests as requiring access to a valid OPENAI_API_KEY (deselect with '-m \"not requires_key\"')",
]