To prevent frustrated tester, we should let buggy code to be generated instead of failing the project generation (if it failed on the linter) by appending a # TODO line on the code.
---------
Co-authored-by: majdyz <[email protected]>
* Introduce event logging, captured event: All non-Get HTTP operations.
* Refactor routes.py on each API: removed redundant try-catch for unknown exception, use JsonResponse
* Fix broken api_test.py due to DB seeding being removed.
---------
Co-authored-by: majdyz <[email protected]>
Apparently the generated DB schema is duplicated across every `CompiledRoute`.
Combining the schemas of N routes with M tables will result in N-fold duplication and `prisma generate` failure.
In turn, this breaks code fixing for server.py, resulting in broken deployments.
Tempfix, applied in this commit: use the schema from only one of the `CompiledRoutes`.
Long-term fix: clean up Codex' DB structure.
The hardcoded dependencies `fastapi`, `pydantic`, `uvicorn` and `prisma` need special treatment,
because if they are included in the `packages` of a generated function,
those requirements are not scrutinized during code validation, allowing invalid versions.
Subprocess execution implementation (for Ruff & PyRight) is currently a blocking call. This doesn't work well with the event-loop used by the fastapi.
Scope:
Make AiBlock.validate async
Make subprocess execution async.
---------
Co-authored-by: majdyz <[email protected]>
Integrating PyRight project-based code-linting. This will allow us to detect:
* Misuse of async functions.
* Misuse of optional type and other annotated types (currently disabled)
* Wrong usage of pydantic and prisma model.
* Wrong usage of third-party library.
* Use of undeclared field
* Use of uninstalled third-party library (which will prompt LLM to add the requirement list)
Known trade-off:
Setting-up PyRight requires building a full-project (generate the Prisma model code, installing 3rd-party library, compiling the function with the object dependency) this slows down the code-validation during the code-validation step.
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: SwiftyOS <[email protected]>
* Update Interview and LLMCallAttempt models
* Refactor LLMCallAttempt model
* Refactor code and update logging IDs
* Refactor AIBlock class and handle exception in start_interview
* Add template analytics
* Refactor analytics.py to use get_template_performance() instead of load_data()
* Add migration lock file and analytics functionality
* Pr coments
* Add model field to AIBlock and LLMCallTemplate
This is a large but a much needed change to allow more sophisticated static code analysis to be added.
PyRight (used by pylance) will allow us to detect:
Misuse of async functions.
Misuse of optional type and other annotated types.
Wrong usage of pydantic and prisma model.
Wrong usage of third-party library.
The overhaul is required to allow us retrying while fixing the code as a project (not only as a function) and allow us to keep adding non-LLM fixes to it so that we can avoid wasting the retry.
The scope of this PR is still a refactoring and the adding of the PyRight linter, so this PR try to keep the existing behaviour.
The enablement and more tuning on the linter will be done in the followup PR.
---------
Co-authored-by: majdyz <[email protected]>
Lookup on the existing database table is already implemented, but this bug sometimes can get in the way.
---------
Co-authored-by: majdyz <[email protected]>
* Fix error handling in exec_external_on_contents function
* Add new models and update existing model in database schema
* Delete User-related models and enums
* Update code block delimiter in DatabaseGenerationBlock
* Refactor code for DatabaseEnums and DatabaseTable classes, and exec_external_on_contents function
* fix test
* Fix exec_external_on_contents function calls
* formatting
* Add database schema implementation requirements
* Fix function template and add validation for required fields
* Add pytest hook to .pre-commit-config.yaml
* Update pytest configuration to pass filenames and include only .py files
* [AGPT-533] Fix Optional type handling, add default None value
* Add auto fix prisma models and enums
* Make test happy
* Make test happy
* Replace typing list with the original typing list
* Revert prevCallId
* Revert prevCallId
* Add more auto imports
* Fix gen_test
---------
Co-authored-by: majdyz <[email protected]>
* Add migration lock file and update compile_test.py and compile.py
* Fix issue with arg type replacement in compile.py
* Remove migration_lock.toml file
* Add full import path to custom types in compile.py
* Change logger.warning to logger.debug in compile.py
* Refactor normalize_type function to support renamed types
* Remove unnecessary code in get_object_field_deps function
* Refactor get_object_field_deps function to skip processing certain field types
* Change logger.info to logger.debug in get_object_field_deps function
* Route schema changes
* Delete ruff.config.json
* save current changes
* style fixes by ruff
* Add class parsing capability
* Fix bugs
* Address comments
* Fix benchmarks
* Fix benchmarks
* Fix benchmarks
* Add comments
* fixing types
* Add error handling and status messages for application creation
* Refactor fetch_deliverable function and update application creation status
* Refactor benchmark output formatting
* Remove unnecessary print statement in fetch_deliverable function
* Propagate DB schema changes: Function <-> CompiledRouteSpec (#65)
Propagate DB schema changes: Function <-> CompiledRouteSpec (#65)
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: Swifty <[email protected]>
Co-authored-by: majdyz <[email protected]>
* emoved broken hardcoded requirements
* Add benchmarking functionality
* Update branch restrictions in GitHub workflows
* Add pytest-cov to dependencies
* Update ruff formatter command to include --check option
* Update workflow names
* Refactor code formatting and fix indentation issues
* Update Python dependency caching and installation process
* Add Prisma client generation step to CI workflow
* Refactor test_process_object_type to use async/await
* Add pytest and ValidationError imports, update arg_types assertion, and add test for ValidationError
* Remove unused imports and commented code
* Add logger to OpenAIChatClient class
* Add async context manager for database connection and disconnection
* Add blank line for readability
* Refactor function_def.arg_types assignment in ast_test.py
* mark integration tests
* Update test workflow name
* Add OpenAI API key configuration
* Better Types for Requirements Request/Response (#76)
Better Types for Requirements Request/Response
---------
Co-authored-by: majdyz <[email protected]>
* [AGPT-474] End-to-end Benchmark (#77)
* Remove unnecessary fetch_deliverable calls and update invoice_generator_requirements
* Fix typo in hardcoded.py
* Add missing app IDs and interview IDs for calendar booking, inventory management, and invoicing systems
* Add options to benchmark and example commands
* Refactor benchmarking code and add new functions
* Add task description method to ExampleTask enum
* Added end-to-end running
* Add hardcoded flag to filter examples in benchmark.py
* Refactor run_benchmark_example function to handle exceptions
* Fix error handling in specification routes
* Fix Function generation bug on nested object type (#79)
Co-authored-by: majdyz <[email protected]>
* Speeding up requirements system
---------
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: majdyz <[email protected]>
* Refactor code: Remove unused imports and clean up formatting
* [AGPT-478] Add Support for Composite Type with multiple Pydantic/nont-primitive classes (#78)
* [AGPT-478] Add support for multiple composite Pydantic objects
* Ruff Ruff!
* Fix failing test
* formatting
* fix populate-db command
* fix run all benchmarks
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: Swifty <[email protected]>
* Refactor package handling in packager.py (#81)
* [Fix] User provided app name being replaced (#82)
* Only make databases when required (#83)
* feat: make the name match
* feat: database becomes optional
* Update agent.py
* Integrate related types into the requirements system (#84)
* feat: make the name match
* feat: database becomes optional
* feat: initial model changes
* feat: add prompts
* feat: more prompts
* fix: nits
* fix: wrong type 😢
* feat: improved prompts
* Cleanup for field type (#85)
Co-authored-by: majdyz <[email protected]>
* [AGPT-489] Improve type composite-type matching & extracting algorithm (#86)
Co-authored-by: majdyz <[email protected]>
* [AGPT-489] Add Autogeneration of Typing import (#87)
Co-authored-by: majdyz <[email protected]>
* [AGPT-489] Fix broken class compiling logic (#88)
Co-authored-by: majdyz <[email protected]>
* [AGPT-489] Fix function fetching to always include RelatedTypes (#89)
[AGPT-489] Fix function fetching to always include RelatedTypes
---------
Co-authored-by: majdyz <[email protected]>
* fix lock file
* feat: endpoint updates (#95)
* feat: endpoint updates
* fix: ambiguous param
* [AGPT-486] Fix codegen generate code with missing pydantic object (#92)
Co-authored-by: majdyz <[email protected]>
* Fix retry prompt (#96)
Co-authored-by: majdyz <[email protected]>
* Allow benchmark on $PORT (#98)
* Ntindle/agpt 506 add a bit of debug tooling for requirements gen (#100)
* [AGPT-485] Fix missing pydantic object during requirement-step; leading to LLM to guess wrong signature (#97)
[AGPT-485] Fix missing pydantic object during requirement-step; leading to LLM to guess wrong signature
---------
Co-authored-by: majdyz <[email protected]>
* [AGPT-157] Add Database Integration Generation (#90)
* Added prisma client to server code
* Add DatabaseSchema to INCLUDE_FUNC
* Update functions to be async and add database schema documentation
* Add DatabaseSchema and DatabaseTables to ApiRouteSpec
* Updated dev prompts for db
* Update profile management descriptions
* Update route function definition to be async
* Remove logging of database schema
* Add Prisma schema file generation
* Refactor benchmark and agent modules
* Refactor code and comment out debug statements
* formatting
* Add available_functions to develop_route
* Update prisma import and method calls
* loading all related tables
* Refactor create_prisma_scheama_file function to handle cases with no database schema
* Fix async function call in packager.py
* Fix database table query in packager.py
* Fix database access and unnecessary imports
* review comments
* [AGPT-508] Enum support for Codex - Generation Logic Layer (#102)
Co-authored-by: majdyz <[email protected]>
* Add Enum Generation to Requriements (#101)
* Added prisma client to server code
* Add DatabaseSchema to INCLUDE_FUNC
* Update functions to be async and add database schema documentation
* Add DatabaseSchema and DatabaseTables to ApiRouteSpec
* Updated dev prompts for db
* Update profile management descriptions
* Update route function definition to be async
* Remove logging of database schema
* Add Prisma schema file generation
* Refactor benchmark and agent modules
* Refactor code and comment out debug statements
* formatting
* Add available_functions to develop_route
* Update prisma import and method calls
* loading all related tables
* Refactor create_prisma_scheama_file function to handle cases with no database schema
* Fix async function call in packager.py
* Fix database table query in packager.py
* Fix database access and unnecessary imports
* feat: add enums to the database gen
* fix: formatting
* Updated to new schema
* update hardcoded requirements
* fix: objectType generation without type bool
* Add database enum import statement
* removed unused var as failing lint check
---------
Co-authored-by: SwiftyOS <[email protected]>
* More Linting Issues Fixes (#99)
* fix: more linting issues
* Remove error log for async functions
* Fix return_type default value in FunctionDef class
* Added another validation check
---------
Co-authored-by: Swifty <[email protected]>
* [AGPT-463] Add code generation validation using Ruff (#103)
---------
Co-authored-by: majdyz <[email protected]>
* feat: extract external command on in memory code to a function with tests
* feat: prisma model parser
* feat: database test updates
now requires prisma to run
* feat: test for prisma accessible
* feat: update to use format and validate
Also saves the formatted and validated models in place
* fix: lint
* fix: remove unused variable
* feat: allow multiple generators
* .gitignore and .env.example (#107)
* feat: .env.example
* feat: gitignore
* fix: replace all bad chars with NOTHING
also fix the format string
* Add Database Docker Compose File (#108)
* feat: .env.example
* feat: gitignore
* fix: replace all bad chars with NOTHING
also fix the format string
* feat: add docker compose with only db
---------
Co-authored-by: Swifty <[email protected]>
* feat: remove vector requirement (#106)
* Fix lifecycle and prisma models and enums missing (#105)
* fix: add prisma model imports fro root level
* fix: reorder start code and add lifecycle
* Validate Database Generation (#104)
* feat: extract external command on in memory code to a function with tests
* feat: prisma model parser
* feat: database test updates
now requires prisma to run
* feat: test for prisma accessible
* feat: update to use format and validate
Also saves the formatted and validated models in place
* fix: lint
* fix: remove unused variable
* feat: allow multiple generators
* formatting
* Change logger.info to logger.debug in exec_external_on_contents function
* Update error logging in exec_external_on_contents function
* Update response model and route annotation in compile.py
* Force
* Ruff
* Fix Api route spec
* formatting
* Dont generate envexample passwords with symbols in them (#117)
* fix: don't use special chars in db password
* fix: linting
* [AGPT-540] Add missing initialization of available object types (#113)
* [AGPT-540] Add missing initialization of available object types
* [AGPT-540] Add missing initialization of available object types
* [AGPT-540] Add missing initialization of available object types
* Ruff
* allow non-annotated type checking
* Include all available function for ruff
* Fix tests
* Make DB work
* Make stubbing work
* Make papa-lint happy
* Ruffs
* Add prisma model validation layer
* Improve prisma querying by avoiding conflict
* Move on during failure formatting
---------
Co-authored-by: majdyz <[email protected]>
* main into dev (#118)
* Update setup guide.
* [AGPT-544] Vague 500 errors in interview stage (#115)
* Add error handling for user not found in database during interview step.
* Add error handling for app not found in database during interview step.
* Add error handling for interview not found in database during interview step.
* Add error handling for nonexistent questions in interview step.
* Add error handling for non-"ask" tool in user answers in interview step.
* Clarify comment describing interview next_step.
---------
Co-authored-by: Toran Bruce Richards <[email protected]>
* [AGPT-511] Resume running from any point (#114)
* Refactor populate_db function and add get_resume_points command
* Fix app ID reference in benchmark.py
* Add ResumePoint model
* Added runner funcitons to running codex from the cli
* Add application description field to ApplicationBase model and update create_app method in CodexClient
* Refactor code to run tasks concurrently
* Add include parameter to find_many() method in get_resume_points()
* Add disconnect calls to asyncio event loop
* Update resume_point name to display only the first 30 characters
* Remove commented out code and unused imports
* Remove unused code
* formatting
* Fix validation error handling in develop.py
* formatting
* fixes
* Add validation error for unimplemented main function
* Validate and try-fix generated endpoint db and model usage (#116)
* feat: log status
* feat: trivial implementation of copy
works for request -> response and response-> request for unfilled object but not list[unfilled_object]
* fix: i cant rememember
* feat: more complex
* feat: mostly working
* fix: fixes from tests
* feat: tests
* fix: linting problems and test name
* feat: add more supported types
* fix: remove unused data
* fix: stop using print
* fix: arrays of arrays
* fix: lint
* fix: reduce mess
* feat: more complex type that I hope we never actually use
* fix: lint
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: majdyz <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Toran Bruce Richards <[email protected]>
