[AGPT-432] Provide Request & Response checking on Function Generation #64

majdyz · 2024-02-29T10:26:28Z

Changes:

Testing:

Fix bug on ./run benchmark due to the latest API change.
Add global option to mock ai_block LLM output + Add function generation tests (gen_test.py, testing without LLM call).
Add 1 simple seeded application (TicTacToe game).

Code:

Add strict checking on argument and return types for function generation.
Store created FunctionArgs & FunctionReturns for each function.
Add global assignments produced by LLM as part of the generated main code.
Fix the first function for the API generation prompt (provide function signature).

Out of scope:

Auto-detecting a new class type

majdyz · 2024-02-29T10:26:56Z

codex/__main__.py

-                deployment_id = deploy_data["deployment"]["id"]
-                deployment_file_name = deploy_data["deployment"]["file_name"]
+                deployment_id = deploy_data["id"]
+                deployment_file_name = deploy_data["file_name"]


The API is changed, this to make the benchmark work again.

majdyz · 2024-02-29T10:27:11Z

codex/common/ai_block.py

@@ -44,6 +44,10 @@ class config:
        arbitrary_types_allowed = True


+# This can be used for synchronous testing without making an actual API call
+MOCK_RESPONSE = ""


I'm adding this for testing without LLM call

As discussed in discord, I think its a clear design to mock the openai class

majdyz · 2024-02-29T10:30:44Z

codex/develop/agent.py

+                data=model,
+                include={"FunctionArgs": True, "FunctionReturn": True},
+            )
+


Now we create the Function.Definition before recursing.

This simplifies a lot of stuff:

Develop AIBlock only do update, create_item and update_item can be merged.

Created the objectfields for args and return types before generation, so we can validate inside the AIblock

linear · 2024-02-29T10:32:34Z

AGPT-432

majdyz · 2024-02-29T10:32:59Z

codex/develop/agent.py

-        function_name (str): The name of the function to create.
-        function_id (str): The id of the function to create. If not provided, a new root-function will be created.
-        function_template (str): The docstring/template of the function to create.
+        function (Function): The function to develop.


It's easier to group this to a single object

majdyz · 2024-02-29T10:34:13Z

codex/develop/develop.py

        self.imports = []
-        self.pydantic_classes = []


We are tracking all classes, there is is_pydantic in self.objects

majdyz · 2024-02-29T10:36:00Z

codex/develop/develop.py

+            isinstance(node, ast.AnnAssign) or
+            isinstance(node, ast.AugAssign)
+        ) and node.col_offset == 0:
+            self.globals.append(ast.unparse(node))


I'm tracking globals for now so that at least the generated code is working.

majdyz · 2024-02-29T10:36:30Z

codex/develop/develop.py

@@ -226,94 +250,6 @@ def validate(
    async def create_item(
        self, ids: Identifiers, validated_response: ValidatedResponse
    ) -> Function:
-        """This is just a temporary that doesnt have a database model"""


merging this with update_item

majdyz · 2024-02-29T10:37:54Z

codex/develop/model.py

    function_code: str
+    function_template: str = None
+
+    def __generate_function_template(f) -> str:


This will be used only for the first function, we let llm output template for the rest to be used.

majdyz · 2024-02-29T10:38:40Z

codex/tests/ast_test.py

@@ -1,7 +1,7 @@
 import ast

-import pytest
-from numpy import VisibleDeprecationWarning


Boilerplate changes to make the test work

schema.prisma

…eration-is-applying-strict-check-on-args-and' into zamilmajdy/agpt-432-function-generation-is-applying-strict-check-on-args-and

…_object

…tegrate-complex-types-into-all-compiling-step [AGPT-250] Add compiler support for complex types

majdyz · 2024-03-01T13:26:04Z

codex/requirements/hardcoded.py

                            description="Identifier for the user who is creating the invoice.",
                        ),
                        Parameter(
                            name="items",
-                            param_type="Array<InvoiceItemInput>",
+                            param_type="List[InvoiceItemInput]",


list with no capital?

majdyz · 2024-03-01T13:26:19Z

codex/requirements/hardcoded.py

@@ -2229,22 +2229,22 @@ def invoice_payment_tracking() -> ApplicationRequirements:
                    params=[
                        Parameter(
                            name="invoices",
-                            param_type="Array<InvoiceSummary>",
+                            param_type="List[InvoiceSummary]",


Swiftyos

Nice work!

Swiftyos · 2024-02-29T13:50:54Z

codex/__main__.py

@@ -95,6 +96,8 @@ async def fetch_deliverable(session, user_id, app_id):
                return deploy_data
        except Exception as e:
            click.echo(f"Error fetching deliverable: {e}")
+            print("Problematic URL: ", url)


We should use click.echo or logger instead of print here

Swiftyos · 2024-02-29T13:51:31Z

codex/common/ai_block.py

@@ -44,6 +44,10 @@ class config:
        arbitrary_types_allowed = True


+# This can be used for synchronous testing without making an actual API call
+MOCK_RESPONSE = ""


As discussed in discord, I think its a clear design to mock the openai class

…unction-generation-is-applying-strict-check-on-args-and

[AGPT-432] Provide Request & Response checking on Function Generation

100639f

majdyz commented Feb 29, 2024

View reviewed changes

schema.prisma Show resolved Hide resolved

majdyz requested a review from Swiftyos February 29, 2024 10:42

majdyz and others added 17 commits February 29, 2024 14:50

Merge conflicts

447efb3

style fixes by ruff

25dd09e

Populate db fix

d251401

Merge remote-tracking branch 'origin/zamilmajdy/agpt-432-function-gen…

7fa67b1

…eration-is-applying-strict-check-on-args-and' into zamilmajdy/agpt-432-function-generation-is-applying-strict-check-on-args-and

Make test run in parallel

def007b

style fixes by ruff

34602a8

Add support for generating Pydantic objects

f551927

Fix indentation in generate_pydantic_object function

e936bd7

Refactor code to use process_object_type instead of generate_pydantic…

b195677

…_object

style fixes by ruff

8c4d27c

Add object_type_ids parameter to process_object_type function

39072ed

style fixes by ruff

32ee9dd

Added handling of subtypes

344d2e1

Add Pydantic models to compiled function

db79d0b

Add empty line in compile_test.py

ef375bb

Add objects for FunctionReturn

f77edff

move create server code to compile

1188674

Swiftyos added 2 commits February 29, 2024 15:49

Add create_server_route_code function

e29816f

Working through adding a dedicated logic for route definitions

7b753bd

majdyz enabled auto-merge (squash) March 1, 2024 05:04

Swiftyos added 17 commits March 1, 2024 09:40

Remove unique constraint on compiledRouteId field

99aef2f

Add extract_path_params function and test

d1e0bc5

Add route code generation logic

826d729

Refactor server route creation code

154dbb0

Refactor run_example command to example command

896760d

Add RootFunction to deployment creation

94935c0

Refactor code formatting and error handling in compile_route()

a9038d6

Refactor logging statements in compile.py

9e2b108

Refactor recursive_compile_route to include new_object_types parameter

228a027

Update logger message and include additional function details

710688a

Fix formatting in routes.py

c54449f

Merge pull request #63 from Significant-Gravitas/swiftyos/agpt-250-in…

fccd885

…tegrate-complex-types-into-all-compiling-step [AGPT-250] Add compiler support for complex types

Delete migrations

5d871d3

skip adding __init__ funcitons

72f942f

fix mis named return objects in hardcoded requirements

df33b55

Update logger message in compile.py

e62950c

Fixing issues in hardcoded requirements

a8226ed

majdyz commented Mar 1, 2024

View reviewed changes

Swiftyos added 3 commits March 1, 2024 14:31

more requirement issues

4350a58

fixed "list of X" param type to a valid type

4e78ed8

Update response model names and descriptions

1974e43

Swiftyos approved these changes Mar 1, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into zamilmajdy/agpt-432-f…

f79dec1

…unction-generation-is-applying-strict-check-on-args-and

Swiftyos disabled auto-merge March 1, 2024 13:47

Swiftyos merged commit e95f544 into main Mar 1, 2024
1 check passed

Swiftyos deleted the zamilmajdy/agpt-432-function-generation-is-applying-strict-check-on-args-and branch March 1, 2024 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AGPT-432] Provide Request & Response checking on Function Generation #64

[AGPT-432] Provide Request & Response checking on Function Generation #64

majdyz commented Feb 29, 2024 •

edited

Loading

majdyz Feb 29, 2024

majdyz Feb 29, 2024

Swiftyos Feb 29, 2024

majdyz Feb 29, 2024

linear bot commented Feb 29, 2024

majdyz Feb 29, 2024

majdyz Feb 29, 2024

majdyz Feb 29, 2024

majdyz Feb 29, 2024

majdyz Feb 29, 2024

majdyz Feb 29, 2024

majdyz Mar 1, 2024

majdyz Mar 1, 2024

Swiftyos left a comment

Swiftyos Feb 29, 2024

Swiftyos Feb 29, 2024

[AGPT-432] Provide Request & Response checking on Function Generation #64

[AGPT-432] Provide Request & Response checking on Function Generation #64

Conversation

majdyz commented Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linear bot commented Feb 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Swiftyos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

majdyz commented Feb 29, 2024 •

edited

Loading