* feat: initial implementation of post endpoints, non working due to circular dependency
* fix: linting
* fix: recursive import
* feat: refactor to support module additions
* feat: fix up add route endpoint, move models to api layer
* feat: add deletes
## **User description**
Changes:
* Hint the UI page generation with its connection to another pages.
* Add validation on the page connection prompting
* Cleaner auto import on missing function and missing object (remove wildcard, import only the missing part). on missing object we try to find the related function and import the file that hosts that function (only support the object type referred by the ApiRouteSpec, it can still be improved).
Out of scope:
* Validation of ui.link and page connection (if LLM hallucinate the page on `ui.link`, we are not failing this yet!)
___
## **Type**
Enhancement, Bug fix
___
## **Description**
- Enhanced code validation to include checks for `ui.link` usage and ensure it references valid functions.
- Improved import handling in `__fix_missing_imports` by adding async fetching of functions related to object types and better error message integration.
- Added a new database function `get_object_type_referred_functions` to fetch functions related to an object type.
- Updated `PageDecompositionEntry` to use a dictionary for `used_functions` to include descriptions, and added `PageConnection` model for describing connections between pages.
- Enhanced validation in `PageDecompositionBlock` to include checks for main function naming, existence of a home page, and valid page connections.
- Updated Jinja2 template example to reflect new features in page decomposition, including detailed function usage and page connections.
___
## **Changes walkthrough**
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement
</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>code_validation.py</strong><dd><code>Enhance Code Validation and Import Handling</code> </dd></summary>
<hr>
codex/develop/code_validation.py
<li>Added validation for <code>ui.link</code> usage in function code to ensure it <br>references valid functions.<br> <li> Enhanced the <code>__fix_missing_imports</code> function to handle missing imports <br>more accurately, including async fetching of functions related to <br>object types.<br> <li> Improved error handling by appending problematic lines to error <br>messages.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-deae541000e158a1fc146c31004d52ed52a858802dd058803d920d97477071d6">+35/-10</a> </td>
</tr>
<tr>
<td>
<details>
<summary><strong>database.py</strong><dd><code>Add Database Function for Object Type Function Fetching</code> </dd></summary>
<hr>
codex/develop/database.py
<li>Added a new function <code>get_object_type_referred_functions</code> to fetch <br>functions related to an object type.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-61c5f5198f842fcc472b6b31ee2e5a53eac489a88152cbc792895f1f9d60f694">+23/-1</a> </td>
</tr>
<tr>
<td>
<details>
<summary><strong>ai_page_decompose.py</strong><dd><code>Enhance Page Decomposition with Detailed Function Usage and Page </code><br><code>Connections</code></dd></summary>
<hr>
codex/requirements/blocks/ai_page_decompose.py
<li>Updated <code>PageDecompositionEntry</code> to use a dictionary for <code>used_functions</code> <br>to include descriptions.<br> <li> Added <code>PageConnection</code> model to describe connections between pages.<br> <li> Enhanced validation in <code>PageDecompositionBlock</code> to include checks for <br>main function naming, existence of a home page, and valid page <br>connections.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-0539cf8ae85457e8d87308e181529144d8f564ad2453eabecc509b4f02c4abe2">+55/-9</a> </td>
</tr>
</table></td></tr><tr><td><strong>Documentation
</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>system.j2</strong><dd><code>Update Example to Demonstrate Enhanced Page Decomposition Features</code></dd></summary>
<hr>
codex/prompts/gpt-4-turbo/requirements/page_decompose/system.j2
<li>Modified the example to reflect changes in the <code>PageDecompositionEntry</code> <br>model using a dictionary for <code>used_functions</code>.<br> <li> Added <code>connections</code> field to demonstrate inter-page linking in the <br>example.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-c081d45d8aa12144bab35cca95d5b45f35e7a380a84e71632b1d425d9a712f6c">+26/-11</a> </td>
</tr>
</table></td></tr></tr></tbody></table>
___
> ✨ **PR-Agent usage**:
>Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions
---------
Co-authored-by: majdyz <[email protected]>
* temporary fix for unique constraints
* fix formatting
* use user id, app name and deliverable id
* fix name
* linting fix
* Apply suggestions from code review
Co-authored-by: codiumai-pr-agent-pro[bot] <151058649+codiumai-pr-agent-pro[bot]@users.noreply.github.com>
* fix formatting
---------
Co-authored-by: codiumai-pr-agent-pro[bot] <151058649+codiumai-pr-agent-pro[bot]@users.noreply.github.com>
* adding DB creation / migration steps
* set up codex to create a functional db for users
* adding env to dockerfile
* add variables to .env.example
* [AGPT-678] Ensure requirements systems creates what user requested (#216)
* Add InterviewBlock class and model definitions
* Updated the interview process
* new interview integrated with the requirements system
* updated prompt
* fix
* tweaking prompt
* delete simple agent
* change from my personal discord id to another
* fix prompts
* Add reference to the prompt in validation error message
* return feature functionality
* fix regerssion
* updated prompt
* Fix default values in InterviewStep table
* Pr comments
* Updating the requirements flow
* progressing the requirements system
* Updated requirements system
* Fix typo in variable name
* Propogating changes to develop
* Add print_interview and print_spec_summary functions***
***Implement explore_database function***
***Update module_routes/user.j2 template***
***Refactor generate_requirements and define_module_routes functions
* Add gpt-4-turbo model and new prompt templates
* Add unique constraint to repo field
* Update debug print statements and remove unnecessary migration files
* Add migrations
* isort
* Add note about allowed_access_roles in user.j2
* [AGPT-691] Clean up old unused requirements code (#218)
* delete old code
* [AGPT-693] Add back access level (#219)
* Added back access level
* Update migrations
* Delete comment
* Updated where access level is defnined.
* Add interactions
* updated migrations
* fixing deployment issue (#223)
* setting up two docker files (#222)
* Fix deployment error (#224)
* fixing deployment issue
* Update generate_readme function to include specification
* set up codex to create a functional db for users
* use instance connection name
* fix linting and formatting
---------
Co-authored-by: Swifty <[email protected]>
* Add InterviewBlock class and model definitions
* Updated the interview process
* new interview integrated with the requirements system
* updated prompt
* fix
* tweaking prompt
* delete simple agent
* change from my personal discord id to another
* fix prompts
* Add reference to the prompt in validation error message
* return feature functionality
* fix regerssion
* updated prompt
* Fix default values in InterviewStep table
* Pr comments
* Updating the requirements flow
* progressing the requirements system
* Updated requirements system
* Fix typo in variable name
* Propogating changes to develop
* Add print_interview and print_spec_summary functions***
***Implement explore_database function***
***Update module_routes/user.j2 template***
***Refactor generate_requirements and define_module_routes functions
* Add gpt-4-turbo model and new prompt templates
* Add unique constraint to repo field
* Update debug print statements and remove unnecessary migration files
* Add migrations
* isort
* Add note about allowed_access_roles in user.j2
* [AGPT-691] Clean up old unused requirements code (#218)
* delete old code
* [AGPT-693] Add back access level (#219)
* Added back access level
* Update migrations
* Delete comment
* Updated where access level is defnined.
* Add interactions
* updated migrations
* feat: first go of UploadFile
* fix: fixes to main and quietly publishing to gh for now
* fix(deploy): add note to create the repos on dev env in private
(because people were contacting us about the publicly created agpt-coder repos)
* fix(deploy): Do not log created GitHub repo URL with token
* feat(requirements, develop): Add fastapi.UploadFile as available built-in type
* fix(requirements): Allow list for authorization_roles while refining requirements
* feat(deploy): Use `gitpython` for Git operations in packager (#168)
- Rewrite `git_init` subroutine in `deploy.packager` to use `gitpython` rather than `subcommand.run(["git", ...])`
- Add `gitpython` dependency
* Avoid persisting request/response with no fields (#176)
Co-authored-by: majdyz <[email protected]>
* Update health check to use db user and db name. (#173)
* feat(deploy): Switch generated requirements.txt to Poetry (#164)
- Add `create_pyproject` and `poetry_lock` subroutines in packager.py
- Replace step to create requirements.txt with step to create pyproject.toml and poetry.lock
- Add default value `None` to `python_path` parameter on `execute_command` utility function
- Update generated Dockerfile and README.md for switch to Poetry
* [AGPT-510] Debugging Part 1: Auto Resume points, debug command start (#153)
* Add debug commands to codex main module
* Refactor debug command and add WhatToDebug model
* Refactor debug command and add explore_database function
* Add Interview field to Specification model
* Removed need for resume points table
* Formatting od debug messages
* poetry lock update
* Add migrations
* [AGPT-594] Introduce Event Logging (#169)
* Introduce event logging, captured event: All non-Get HTTP operations.
* Refactor routes.py on each API: removed redundant try-catch for unknown exception, use JsonResponse
* Fix broken api_test.py due to DB seeding being removed.
---------
Co-authored-by: majdyz <[email protected]>
* [AGPT-596] Create .github/workflows and related deployment file (#175)
* Create .github/workflows and related deployment file
* Update packager.py
* [AGPT-593] Avoid persisting request/response with no fields - Part 2 (#179)
Co-authored-by: majdyz <[email protected]>
* fix prisma query
* Add support for enhancing errors with context aware details (#180)
* fix: minor annoyance
* feat: v1 of error enhancement
* feat: refactor the initial implementation
* fix: linting 🙄
* feat: more loggin
* nit: typehinting
* fix: add migrations to docker image
* fix: run validate and formate and also fix the param used (#184)
* [AGPT-621] Moving on with code generation with a buggy code (Ruff - PyRight failure) (#181)
To prevent frustrated tester, we should let buggy code to be generated instead of failing the project generation (if it failed on the linter) by appending a # TODO line on the code.
---------
Co-authored-by: majdyz <[email protected]>
* feat: updates to our dockercompose for discussion (#177)
* [AGPT-631] Fix docker setup (#186)
* Fixing docker setup
* Added comment back in
* formatting
* create and push to github repo
* Fixing formatting and logging
* update to only create github repo, no zip file
* remove local changes
* fix formatting
* fix linting step
* fix formating
* use one single migration
---------
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: majdyz <[email protected]>
Co-authored-by: Swifty <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
* refactor(common, develop/validate): Use `pathlib` to clean up file writing & path processing
* refactor(deploy): Use `pathlib` to clean up file operations
* feat(deploy): Use `gitpython` for Git operations in packager (#168)
- Rewrite `git_init` subroutine in `deploy.packager` to use `gitpython` rather than `subcommand.run(["git", ...])`
- Add `gitpython` dependency
* Avoid persisting request/response with no fields (#176)
Co-authored-by: majdyz <[email protected]>
* Update health check to use db user and db name. (#173)
* feat(deploy): Switch generated requirements.txt to Poetry (#164)
- Add `create_pyproject` and `poetry_lock` subroutines in packager.py
- Replace step to create requirements.txt with step to create pyproject.toml and poetry.lock
- Add default value `None` to `python_path` parameter on `execute_command` utility function
- Update generated Dockerfile and README.md for switch to Poetry
* [AGPT-510] Debugging Part 1: Auto Resume points, debug command start (#153)
* Add debug commands to codex main module
* Refactor debug command and add WhatToDebug model
* Refactor debug command and add explore_database function
* Add Interview field to Specification model
* Removed need for resume points table
* Formatting od debug messages
* poetry lock update
* Add migrations
* [AGPT-594] Introduce Event Logging (#169)
* Introduce event logging, captured event: All non-Get HTTP operations.
* Refactor routes.py on each API: removed redundant try-catch for unknown exception, use JsonResponse
* Fix broken api_test.py due to DB seeding being removed.
---------
Co-authored-by: majdyz <[email protected]>
* [AGPT-596] Create .github/workflows and related deployment file (#175)
* Create .github/workflows and related deployment file
* Update packager.py
* [AGPT-593] Avoid persisting request/response with no fields - Part 2 (#179)
Co-authored-by: majdyz <[email protected]>
* fix prisma query
* Add support for enhancing errors with context aware details (#180)
* fix: minor annoyance
* feat: v1 of error enhancement
* feat: refactor the initial implementation
* fix: linting 🙄
* feat: more loggin
* nit: typehinting
* fix: add migrations to docker image
* fix: run validate and formate and also fix the param used (#184)
* [AGPT-621] Moving on with code generation with a buggy code (Ruff - PyRight failure) (#181)
To prevent frustrated tester, we should let buggy code to be generated instead of failing the project generation (if it failed on the linter) by appending a # TODO line on the code.
---------
Co-authored-by: majdyz <[email protected]>
* feat: updates to our dockercompose for discussion (#177)
* [AGPT-631] Fix docker setup (#186)
* Fixing docker setup
* Added comment back in
* formatting
* create and push to github repo
* Fixing formatting and logging
* adding types
---------
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: majdyz <[email protected]>
Co-authored-by: Swifty <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
Co-authored-by: Nicholas Tindle <[email protected]>
* Introduce event logging, captured event: All non-Get HTTP operations.
* Refactor routes.py on each API: removed redundant try-catch for unknown exception, use JsonResponse
* Fix broken api_test.py due to DB seeding being removed.
---------
Co-authored-by: majdyz <[email protected]>
- Add `create_pyproject` and `poetry_lock` subroutines in packager.py
- Replace step to create requirements.txt with step to create pyproject.toml and poetry.lock
- Add default value `None` to `python_path` parameter on `execute_command` utility function
- Update generated Dockerfile and README.md for switch to Poetry
* fix(deploy): Deprecate `pipreqs` as a secondary package guesser
The develop stage already outputs a list of packages required by the application,
and `pipreqs` is unreliable.
- Remove usage of `pipreqs` from deploy.packager
- Remove `pipreqs` dependency
* tweak incantations regarding imports & requirements
Subprocess execution implementation (for Ruff & PyRight) is currently a blocking call. This doesn't work well with the event-loop used by the fastapi.
Scope:
Make AiBlock.validate async
Make subprocess execution async.
---------
Co-authored-by: majdyz <[email protected]>
Integrating PyRight project-based code-linting. This will allow us to detect:
* Misuse of async functions.
* Misuse of optional type and other annotated types (currently disabled)
* Wrong usage of pydantic and prisma model.
* Wrong usage of third-party library.
* Use of undeclared field
* Use of uninstalled third-party library (which will prompt LLM to add the requirement list)
Known trade-off:
Setting-up PyRight requires building a full-project (generate the Prisma model code, installing 3rd-party library, compiling the function with the object dependency) this slows down the code-validation during the code-validation step.
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: SwiftyOS <[email protected]>
This is a large but a much needed change to allow more sophisticated static code analysis to be added.
PyRight (used by pylance) will allow us to detect:
Misuse of async functions.
Misuse of optional type and other annotated types.
Wrong usage of pydantic and prisma model.
Wrong usage of third-party library.
The overhaul is required to allow us retrying while fixing the code as a project (not only as a function) and allow us to keep adding non-LLM fixes to it so that we can avoid wasting the retry.
The scope of this PR is still a refactoring and the adding of the PyRight linter, so this PR try to keep the existing behaviour.
The enablement and more tuning on the linter will be done in the followup PR.
---------
Co-authored-by: majdyz <[email protected]>
* Re-apply improvements from bb43387, with additional fixes:
- Fix Prisma client generation
- Fix `DATABASE_URL` in docker-compose.yml
* Simplify mechanism to wait for database availability before starting app container (introduced in a1963c9 through #133)
- Remove deploy/wait-for-it.sh
* Add instructions to run app to generated README
- Add `generate_readme` function to packager.py
- Fix usage/formatting of block strings in packager.py
* Add docstring to generate_readme
* Fix instructions regarding .env
* Change author in generated README to "AutoGPT" (remove mention of Codex)
* Update README generated by packager.py
Remove `cd` steps to go into and out of `project` folder. This is no longer needed since requirements.txt is now at the root level of the package.
* Update instructions in packager.py
---------
Co-authored-by: SwiftyOS <[email protected]>
* Add Git user name and email to .env.example
* Add Git repository initialization and commit code in deploy step
* Extract git repository init to function.
* Update Git user configuration to only apply locally in newly initialized repository.
* Unset local Git user configuration after commiting
* Lint
- Remove `ffmpeg` from `apt-get install` step
- Fix and deduplicate `COPY` statements: exclude irrelevant files and optimize for caching
- Add `DATABASE_URL` to `app` service in `docker-compose.yml`
* Fix function template and add validation for required fields
* Add pytest hook to .pre-commit-config.yaml
* Update pytest configuration to pass filenames and include only .py files
Nicholas Tindle