Run the integration tests in the CI pipeline BUT without API keys (Si…

…gnificant-Gravitas#3359) * integration tests in ci pipeline * Update CONTRIBUTING.md Co-authored-by: Reinier van der Leer <[email protected]> --------- Co-authored-by: Reinier van der Leer <[email protected]>
jryals579 · Apr 27, 2023 · 02f546d · 02f546d
1 parent 3b56716
commit 02f546d
Show file tree

Hide file tree

Showing 13 changed files with 897 additions and 84 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -72,8 +72,6 @@ jobs:
       - name: Run unittest tests with coverage
         run: |
           pytest --cov=autogpt --cov-report term-missing --cov-branch --cov-report xml --cov-report term
-        env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 
       - name: Upload coverage reports to Codecov
         uses: codecov/codecov-action@v3
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -105,3 +105,18 @@ To increase coverage if necessary, [write tests using `pytest`].
 For more info on running tests, please refer to ["Running tests"](https://significant-gravitas.github.io/Auto-GPT/testing/).
 
 [write tests using `pytest`]: https://realpython.com/pytest-python-testing/
+
+
+In Pytest, we use VCRpy. It's a package that allows us to save OpenAI and other API providers' responses.
+When you run Pytest locally:
+
+- If no prompt change: you will not consume API tokens because there are no new OpenAI calls required.
+- If the prompt changes in a way that the cassettes are not reusable:
+    - If no API key, the test fails. It requires a new cassette. So, add an API key to .env.
+    - If the API key is present, the tests will make a real call to OpenAI.
+        - If the test ends up being successful, your prompt changes didn't introduce regressions. This is good. Commit your cassettes to your PR.
+        - If the test is unsuccessful:
+            - Either: Your change made Auto-GPT less capable, in that case, you have to change your code.
+            - Or: The test might be poorly written. In that case, you can make suggestions to change the test.
+
+In our CI pipeline, Pytest will use the cassettes and not call paid API providers, so we need your help to record the replays that you break.
diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py
diff --git a/tests/integration/cassettes/test_local_cache/test_get_relevant.yaml b/tests/integration/cassettes/test_local_cache/test_get_relevant.yaml
diff --git a/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_default.yaml b/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_default.yaml
@@ -0,0 +1,93 @@
+interactions:
+- request:
+    body: '{"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "\nYour
+      task is to devise up to 5 highly effective goals and an appropriate role-based
+      name (_GPT) for an autonomous agent, ensuring that the goals are optimally aligned
+      with the successful completion of its assigned task.\n\nThe user will provide
+      the task, you will provide only the output in the exact format specified below
+      with no explanation or conversation.\n\nExample input:\nHelp me with marketing
+      my business\n\nExample output:\nName: CMOGPT\nDescription: a professional digital
+      marketer AI that assists Solopreneurs in growing their businesses by providing
+      world-class expertise in solving marketing problems for SaaS, content products,
+      agencies, and more.\nGoals:\n- Engage in effective problem-solving, prioritization,
+      planning, and supporting execution to address your marketing needs as your virtual
+      Chief Marketing Officer.\n\n- Provide specific, actionable, and concise advice
+      to help you make informed decisions without the use of platitudes or overly
+      wordy explanations.\n\n- Identify and prioritize quick wins and cost-effective
+      campaigns that maximize results with minimal time and budget investment.\n\n-
+      Proactively take the lead in guiding you and offering suggestions when faced
+      with unclear information or uncertainty to ensure your marketing strategy remains
+      on track.\n"}, {"role": "user", "content": "Task: ''Write a wikipedia style
+      article about the project: https://github.com/significant-gravitas/Auto-GPT''\nRespond
+      only with the output in the exact format specified in the system prompt, with
+      no explanation or conversation.\n"}], "temperature": 0.0, "max_tokens": null}'
+    headers:
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '1671'
+      Content-Type:
+      - application/json
+    method: POST
+    uri: https://api.openai.com/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAA1xSS2/UMBC+8ytGvnBJVt3tgzZHoBQkXgdQKyiqHHuSTOvMpPZkt6uq/x3FWfrg
+        aHvme/rekDeVcZ1V1w+hfHPSffy0Hz6ff6sPls3p9cHtr4tv2y8/6e3FWWMKI/U1Ot1tLJz0Q0Al
+        YVMYF9EqelMtj45XhweHR0fHhenFYzCVaQct9xeHpY6xlnJvf29pCjMm26Kp7s0QpR/0SuUGOZlq
+        dbJfmCfsx/vl6qQwKmrD49XBavVQGNcJOUym+n1vekz/YKMENJWxKVFSyzqJFFbkycBX22MF53RD
+        Z99/XPJ7TC7SMPFVYBnsqMLSy5jAtsgKHhO1jB5UYEYEYtAOIRsnYZAm4w3oyZZJtwHBRiUXMEG9
+        hSHKmjxxCxuJwZcu2JQA7waMSgknuIgJbXRdAZtIStwWYNlDI7G3Op0Xl3wmNqTqkkt4J+xHp6Cd
+        RBnb7nEdZFbW0hoZVAZyk2zkNEYE69wYrdtm6F3MyJjSZIB4JiPhxcRxHkkROmq78na0gXQLuwxB
+        O6tACZBb22atj8trnIVvMIQyaRydjhF9AY2EIJspg6xvJI+BGFOenkryNvr0Isks40PGzUu7SCcb
+        EnOcKk/TrxPskmf/DH/S5sKYx4coA0ZwpNnmzB2xwYjsMC3maEOwtUSrCBvSLjOPCeNERuwkDvOj
+        dkgRGkRfW3eToXp7g8Dopo8YtxBxTSnzqDw3kHlO50pylM/N9YiaXrhiUVtTLuC/2KYKArVUB5x+
+        CgxjHcjNFZqHwjTElLqriDYJm8oklcEUhtjjnan2Hv48vPoLAAD//wMAbCMiNQcEAAA=
+    headers:
+      CF-Cache-Status:
+      - DYNAMIC
+      CF-RAY:
+      - 7be206beca7ccfbc-SJC
+      Cache-Control:
+      - no-cache, must-revalidate
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Wed, 26 Apr 2023 21:47:58 GMT
+      Server:
+      - cloudflare
+      access-control-allow-origin:
+      - '*'
+      alt-svc:
+      - h3=":443"; ma=86400, h3-29=":443"; ma=86400
+      openai-model:
+      - gpt-3.5-turbo-0301
+      openai-organization:
+      - user-adtx4fhfg1qsiyzdoaxciooj
+      openai-processing-ms:
+      - '9927'
+      openai-version:
+      - '2020-10-01'
+      strict-transport-security:
+      - max-age=15724800; includeSubDomains
+      x-ratelimit-limit-requests:
+      - '3500'
+      x-ratelimit-remaining-requests:
+      - '3499'
+      x-ratelimit-reset-requests:
+      - 17ms
+      x-request-id:
+      - 575539561e8025c9a37920ffb4d1b354
+    status:
+      code: 200
+      message: OK
+version: 1
diff --git a/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_fallback.yaml b/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_fallback.yaml
@@ -0,0 +1,87 @@
+interactions:
+- request:
+    body: '{"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "\nYour
+      task is to devise up to 5 highly effective goals and an appropriate role-based
+      name (_GPT) for an autonomous agent, ensuring that the goals are optimally aligned
+      with the successful completion of its assigned task.\n\nThe user will provide
+      the task, you will provide only the output in the exact format specified below
+      with no explanation or conversation.\n\nExample input:\nHelp me with marketing
+      my business\n\nExample output:\nName: CMOGPT\nDescription: a professional digital
+      marketer AI that assists Solopreneurs in growing their businesses by providing
+      world-class expertise in solving marketing problems for SaaS, content products,
+      agencies, and more.\nGoals:\n- Engage in effective problem-solving, prioritization,
+      planning, and supporting execution to address your marketing needs as your virtual
+      Chief Marketing Officer.\n\n- Provide specific, actionable, and concise advice
+      to help you make informed decisions without the use of platitudes or overly
+      wordy explanations.\n\n- Identify and prioritize quick wins and cost-effective
+      campaigns that maximize results with minimal time and budget investment.\n\n-
+      Proactively take the lead in guiding you and offering suggestions when faced
+      with unclear information or uncertainty to ensure your marketing strategy remains
+      on track.\n"}, {"role": "user", "content": "Task: ''T&GF\u00a3OIBECC()!*''\nRespond
+      only with the output in the exact format specified in the system prompt, with
+      no explanation or conversation.\n"}], "temperature": 0.0, "max_tokens": null}'
+    headers:
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '1592'
+      Content-Type:
+      - application/json
+    method: POST
+    uri: https://api.openai.com/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAA0yOTUsDMRCG7/6KYS5esqUf1tqcFdSLHoqCIjKbne6mzWZCMusHpf9dCmp7e3m/
+        eHboG7ToOlLXp1Atlt3t6unbtbebl5vVc/2wWmiYbbfu+n54QINSb9jp72LkpE+B1UtEgy4zKTdo
+        J5dX0/nF/HKxNNhLwwEttkmr2Whe6ZBrqcaz8QQNDoVaRrvDlKVP+q6y5VjQTq/GBo/f//5sYlBF
+        KZw4k71B14l3XNC+7rDn8veaJTBapFJ8UYp6YJSoHA/8d+c9FMn528AdOIpRFFKWD98wUAQZNA0K
+        n147GRQIPij4BpTKdgSPganwsX4Swloy9Awq0HLkTMpAKWVJ2R90KxQKUGyA4MBX1VS4gUg9j3Bv
+        cO2jL917ZioS0WJRSWjQx4a/0I73b/uzHwAAAP//AwBPrscAswEAAA==
+    headers:
+      CF-Cache-Status:
+      - DYNAMIC
+      CF-RAY:
+      - 7be20701ea0a9669-SJC
+      Cache-Control:
+      - no-cache, must-revalidate
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Wed, 26 Apr 2023 21:48:01 GMT
+      Server:
+      - cloudflare
+      access-control-allow-origin:
+      - '*'
+      alt-svc:
+      - h3=":443"; ma=86400, h3-29=":443"; ma=86400
+      openai-model:
+      - gpt-3.5-turbo-0301
+      openai-organization:
+      - user-adtx4fhfg1qsiyzdoaxciooj
+      openai-processing-ms:
+      - '1833'
+      openai-version:
+      - '2020-10-01'
+      strict-transport-security:
+      - max-age=15724800; includeSubDomains
+      x-ratelimit-limit-requests:
+      - '3500'
+      x-ratelimit-remaining-requests:
+      - '3499'
+      x-ratelimit-reset-requests:
+      - 17ms
+      x-request-id:
+      - c9bf165259547ec59a88bd16b5f691f2
+    status:
+      code: 200
+      message: OK
+version: 1
diff --git a/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_typical.yaml b/tests/integration/cassettes/test_setup/test_generate_aiconfig_automatic_typical.yaml
@@ -0,0 +1,97 @@
+interactions:
+- request:
+    body: '{"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "\nYour
+      task is to devise up to 5 highly effective goals and an appropriate role-based
+      name (_GPT) for an autonomous agent, ensuring that the goals are optimally aligned
+      with the successful completion of its assigned task.\n\nThe user will provide
+      the task, you will provide only the output in the exact format specified below
+      with no explanation or conversation.\n\nExample input:\nHelp me with marketing
+      my business\n\nExample output:\nName: CMOGPT\nDescription: a professional digital
+      marketer AI that assists Solopreneurs in growing their businesses by providing
+      world-class expertise in solving marketing problems for SaaS, content products,
+      agencies, and more.\nGoals:\n- Engage in effective problem-solving, prioritization,
+      planning, and supporting execution to address your marketing needs as your virtual
+      Chief Marketing Officer.\n\n- Provide specific, actionable, and concise advice
+      to help you make informed decisions without the use of platitudes or overly
+      wordy explanations.\n\n- Identify and prioritize quick wins and cost-effective
+      campaigns that maximize results with minimal time and budget investment.\n\n-
+      Proactively take the lead in guiding you and offering suggestions when faced
+      with unclear information or uncertainty to ensure your marketing strategy remains
+      on track.\n"}, {"role": "user", "content": "Task: ''Help me create a rock opera
+      about cybernetic giraffes''\nRespond only with the output in the exact format
+      specified in the system prompt, with no explanation or conversation.\n"}], "temperature":
+      0.0, "max_tokens": null}'
+    headers:
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '1625'
+      Content-Type:
+      - application/json
+    method: POST
+    uri: https://api.openai.com/v1/chat/completions
+  response:
+    body:
+      string: !!binary |
+        H4sIAAAAAAAAA4yU0W/bRgzG3/dXEPcsB04ad53ehg1riwFNthXYgGUo6BMlcT4dVZJy4gT534uT
+        ZWcr9rAXAZZ5Hz/++J2eAjehDrFHj8OYVt9+1//0y+XP797n6w/DbnvTdts/9reXwzr9Fj+GKsj2
+        b4q+nLiIMoyJnCWHKkQldGpCffn6zdXmerPZXFVhkIZSqEM3+urVxWblk25ltX61vgxVmAw7CvVT
+        GFWG0T+57ChbqK/eXFfhRfvl/eXrKrg4pvOrzXr9XIXYC0eyUP/5FAayk6xKolAHNGNzzF5MSnbK
+        ZYAPOFANv97cvr39eJd/JIvKY2lXA2b4/j3M8/CeIEq2KRUB8B4dbKTImPiRDDhDT2nk3AGqs7kB
+        5gaGyTgyZoOtlv+8J1aYMn+eCPZsLNnABRK3dLGYALajdMsRUzpAQ8ZdpqYUHmco7bynxZpkkBZU
+        4g5kJEWrYFTZc1M60sNI6tBN3GCONLuyaRxFyxQqU9fL5LMaZWelcjaS2cVdfiuYrL7LK/hBUsKt
+        KDrBPXsPB5mKnYb2lGQEhLImSqm0NBc9JM50xBRx9EnJjj3MqNiQFuJhS5rJOULHim1LR2YzKQP2
+        E5iTzyPNCtJBOVo1F+/ZJkzFbLF507akLwuzqevIfIZciluiZosF0xHff+jJnhRTKhCaKc5sXYCy
+        TbqMUw6+sC7Lotxhx7mrCkFSR87zr6JHgxSReY9KJhmzL2bfURpnjmhGwzYRIDimotGAEw4F0jlA
+        FRjnjrT4jC66+F2yoQb3vUDEvATtIJMuATtjPNYn3pMCQs9dv/o8YWI/wEjaig4lIYu72zlB9L+C
+        o9QTqmGai/6BbolSdb4bS2wK5ShDET2AbM0xpmX7X6NuOeN5HYU2gutEMKA56ch0dvy76O5f4Sxf
+        E3H6emNzxiZO5fn4CKgy5QbYDZQSoVEFiUoMujnMUu44DNQwlntFdq47RcZ7UhhQd+TlROwxZ0rz
+        vVbC2APCfUGJU8N0QtlRpvk60UNkp4GyX4TnKrSc2fpPSmiSQx3MZQxV4NzQQ6jXz389f/MFAAD/
+        /wMAE8xLs6wFAAA=
+    headers:
+      CF-Cache-Status:
+      - DYNAMIC
+      CF-RAY:
+      - 7be203e8793fcfa4-SJC
+      Cache-Control:
+      - no-cache, must-revalidate
+      Connection:
+      - keep-alive
+      Content-Encoding:
+      - gzip
+      Content-Type:
+      - application/json
+      Date:
+      - Wed, 26 Apr 2023 21:46:11 GMT
+      Server:
+      - cloudflare
+      access-control-allow-origin:
+      - '*'
+      alt-svc:
+      - h3=":443"; ma=86400, h3-29=":443"; ma=86400
+      openai-model:
+      - gpt-3.5-turbo-0301
+      openai-organization:
+      - user-adtx4fhfg1qsiyzdoaxciooj
+      openai-processing-ms:
+      - '19109'
+      openai-version:
+      - '2020-10-01'
+      strict-transport-security:
+      - max-age=15724800; includeSubDomains
+      x-ratelimit-limit-requests:
+      - '3500'
+      x-ratelimit-remaining-requests:
+      - '3499'
+      x-ratelimit-reset-requests:
+      - 17ms
+      x-request-id:
+      - 5fe22bc0f23ce6b48845f33187e1a19d
+    status:
+      code: 200
+      message: OK
+version: 1
diff --git a/tests/test_local_cache.py → tests/integration/test_local_cache.py b/tests/test_local_cache.py → tests/integration/test_local_cache.py
@@ -89,6 +89,7 @@ def test_get(LocalCache, config, mock_embed_with_ada):
     assert cache.get("test") == ["test"]
 
 
+@pytest.mark.vcr
 @requires_api_key("OPENAI_API_KEY")
 def test_get_relevant(LocalCache, config) -> None:
     cache = LocalCache(config)

diff --git a/tests/integration/test_setup.py b/tests/integration/test_setup.py
@@ -0,0 +1,78 @@
+from unittest.mock import patch
+
+import pytest
+
+from autogpt.config.ai_config import AIConfig
+from autogpt.setup import (
+    generate_aiconfig_automatic,
+    generate_aiconfig_manual,
+    prompt_user,
+)
+from tests.utils import requires_api_key
+
+
+@pytest.mark.vcr
+@requires_api_key("OPENAI_API_KEY")
+def test_generate_aiconfig_automatic_default():
+    user_inputs = [""]
+    with patch("builtins.input", side_effect=user_inputs):
+        ai_config = prompt_user()
+
+    assert isinstance(ai_config, AIConfig)
+    assert ai_config.ai_name is not None
+    assert ai_config.ai_role is not None
+    assert 1 <= len(ai_config.ai_goals) <= 5
+
+
+@pytest.mark.vcr
+@requires_api_key("OPENAI_API_KEY")
+def test_generate_aiconfig_automatic_typical():
+    user_prompt = "Help me create a rock opera about cybernetic giraffes"
+    ai_config = generate_aiconfig_automatic(user_prompt)
+
+    assert isinstance(ai_config, AIConfig)
+    assert ai_config.ai_name is not None
+    assert ai_config.ai_role is not None
+    assert 1 <= len(ai_config.ai_goals) <= 5
+
+
+@pytest.mark.vcr
+@requires_api_key("OPENAI_API_KEY")
+def test_generate_aiconfig_automatic_fallback():
+    user_inputs = [
+        "T&GF£OIBECC()!*",
+        "Chef-GPT",
+        "an AI designed to browse bake a cake.",
+        "Purchase ingredients",
+        "Bake a cake",
+        "",
+        "",
+    ]
+    with patch("builtins.input", side_effect=user_inputs):
+        ai_config = prompt_user()
+
+    assert isinstance(ai_config, AIConfig)
+    assert ai_config.ai_name == "Chef-GPT"
+    assert ai_config.ai_role == "an AI designed to browse bake a cake."
+    assert ai_config.ai_goals == ["Purchase ingredients", "Bake a cake"]
+
+
+@pytest.mark.vcr
+@requires_api_key("OPENAI_API_KEY")
+def test_prompt_user_manual_mode():
+    user_inputs = [
+        "--manual",
+        "Chef-GPT",
+        "an AI designed to browse bake a cake.",
+        "Purchase ingredients",
+        "Bake a cake",
+        "",
+        "",
+    ]
+    with patch("builtins.input", side_effect=user_inputs):
+        ai_config = prompt_user()
+
+    assert isinstance(ai_config, AIConfig)
+    assert ai_config.ai_name == "Chef-GPT"
+    assert ai_config.ai_role == "an AI designed to browse bake a cake."
+    assert ai_config.ai_goals == ["Purchase ingredients", "Bake a cake"]