## **User description**
Changes:
* Hint the UI page generation with its connection to another pages.
* Add validation on the page connection prompting
* Cleaner auto import on missing function and missing object (remove wildcard, import only the missing part). on missing object we try to find the related function and import the file that hosts that function (only support the object type referred by the ApiRouteSpec, it can still be improved).
Out of scope:
* Validation of ui.link and page connection (if LLM hallucinate the page on `ui.link`, we are not failing this yet!)
___
## **Type**
Enhancement, Bug fix
___
## **Description**
- Enhanced code validation to include checks for `ui.link` usage and ensure it references valid functions.
- Improved import handling in `__fix_missing_imports` by adding async fetching of functions related to object types and better error message integration.
- Added a new database function `get_object_type_referred_functions` to fetch functions related to an object type.
- Updated `PageDecompositionEntry` to use a dictionary for `used_functions` to include descriptions, and added `PageConnection` model for describing connections between pages.
- Enhanced validation in `PageDecompositionBlock` to include checks for main function naming, existence of a home page, and valid page connections.
- Updated Jinja2 template example to reflect new features in page decomposition, including detailed function usage and page connections.
___
## **Changes walkthrough**
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement
</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>code_validation.py</strong><dd><code>Enhance Code Validation and Import Handling</code> </dd></summary>
<hr>
codex/develop/code_validation.py
<li>Added validation for <code>ui.link</code> usage in function code to ensure it <br>references valid functions.<br> <li> Enhanced the <code>__fix_missing_imports</code> function to handle missing imports <br>more accurately, including async fetching of functions related to <br>object types.<br> <li> Improved error handling by appending problematic lines to error <br>messages.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-deae541000e158a1fc146c31004d52ed52a858802dd058803d920d97477071d6">+35/-10</a> </td>
</tr>
<tr>
<td>
<details>
<summary><strong>database.py</strong><dd><code>Add Database Function for Object Type Function Fetching</code> </dd></summary>
<hr>
codex/develop/database.py
<li>Added a new function <code>get_object_type_referred_functions</code> to fetch <br>functions related to an object type.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-61c5f5198f842fcc472b6b31ee2e5a53eac489a88152cbc792895f1f9d60f694">+23/-1</a> </td>
</tr>
<tr>
<td>
<details>
<summary><strong>ai_page_decompose.py</strong><dd><code>Enhance Page Decomposition with Detailed Function Usage and Page </code><br><code>Connections</code></dd></summary>
<hr>
codex/requirements/blocks/ai_page_decompose.py
<li>Updated <code>PageDecompositionEntry</code> to use a dictionary for <code>used_functions</code> <br>to include descriptions.<br> <li> Added <code>PageConnection</code> model to describe connections between pages.<br> <li> Enhanced validation in <code>PageDecompositionBlock</code> to include checks for <br>main function naming, existence of a home page, and valid page <br>connections.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-0539cf8ae85457e8d87308e181529144d8f564ad2453eabecc509b4f02c4abe2">+55/-9</a> </td>
</tr>
</table></td></tr><tr><td><strong>Documentation
</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>system.j2</strong><dd><code>Update Example to Demonstrate Enhanced Page Decomposition Features</code></dd></summary>
<hr>
codex/prompts/gpt-4-turbo/requirements/page_decompose/system.j2
<li>Modified the example to reflect changes in the <code>PageDecompositionEntry</code> <br>model using a dictionary for <code>used_functions</code>.<br> <li> Added <code>connections</code> field to demonstrate inter-page linking in the <br>example.<br>
</details>
</td>
<td><a href="https://github.com/Significant-Gravitas/codex/pull/263/files#diff-c081d45d8aa12144bab35cca95d5b45f35e7a380a84e71632b1d425d9a712f6c">+26/-11</a> </td>
</tr>
</table></td></tr></tr></tbody></table>
___
> ✨ **PR-Agent usage**:
>Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions
---------
Co-authored-by: majdyz <[email protected]>
* feat: raise the validations up
* feat: add error enhancements to the AI Block
* fix: extract the TODO_COMMENT to the constants
* fix: move ErrorEnhancements
* fix: move TODO_COMMENT
* feat: add interview ID to the log
* feat: enhancement is used in system prompt
* fix: imports in the gen_test
* feat: add decimal support
* fix(develop): don't error when the only issue is llm errors
* fix: regex
* feat: naive fuzzy match
* feat: naive fuzzy match
* feat(prompts): add context to the prompt
* fix(prompt): improve language
* fix(error_enhancements): copy paste typos and small missed logic bugs
* Add handling of "reportPrivateImportUsage" error to error enhancements
* refactor(error_enhancements): Improve code structure and readability
- Extract common error enhancement logic into separate functions
* Fix error in error_message extraction
I had previously extracted non-generally applicable functionality to a function. This undoes that.
* Run ruff linter
* Remove typo in enhance_error function declaration.
* Fix type hint in enhance_error function declaration to allow return dictionary values to be None.
* Fix type hint in enhance_error function declaration to allow return dictionary values to be None.
* Formats code_validation.py with Ruff
* feat: first go of UploadFile
* fix: fixes to main and quietly publishing to gh for now
* fix(deploy): add note to create the repos on dev env in private
(because people were contacting us about the publicly created agpt-coder repos)
* fix(deploy): Do not log created GitHub repo URL with token
* feat(requirements, develop): Add fastapi.UploadFile as available built-in type
* fix(requirements): Allow list for authorization_roles while refining requirements
Changes:
* Avoid using corrupted env files by moving the parent dependency folder for linting out of TMP.
* Add stricter env existence check (check all the required executables: pyright prisma virtualenv-clone).
* Fix execution bug due to migrating from str -> Path.
* Fix unrelated error on code-validation.
* Add `-I` on pip install for installing linting deps to fix corrupted installations.
---------
Co-authored-by: majdyz <[email protected]>
* refactor(common, develop/validate): Use `pathlib` to clean up file writing & path processing
* refactor(deploy): Use `pathlib` to clean up file operations
To prevent frustrated tester, we should let buggy code to be generated instead of failing the project generation (if it failed on the linter) by appending a # TODO line on the code.
---------
Co-authored-by: majdyz <[email protected]>
Subprocess execution implementation (for Ruff & PyRight) is currently a blocking call. This doesn't work well with the event-loop used by the fastapi.
Scope:
Make AiBlock.validate async
Make subprocess execution async.
---------
Co-authored-by: majdyz <[email protected]>
Integrating PyRight project-based code-linting. This will allow us to detect:
* Misuse of async functions.
* Misuse of optional type and other annotated types (currently disabled)
* Wrong usage of pydantic and prisma model.
* Wrong usage of third-party library.
* Use of undeclared field
* Use of uninstalled third-party library (which will prompt LLM to add the requirement list)
Known trade-off:
Setting-up PyRight requires building a full-project (generate the Prisma model code, installing 3rd-party library, compiling the function with the object dependency) this slows down the code-validation during the code-validation step.
---------
Co-authored-by: majdyz <[email protected]>
Co-authored-by: SwiftyOS <[email protected]>
This is a large but a much needed change to allow more sophisticated static code analysis to be added.
PyRight (used by pylance) will allow us to detect:
Misuse of async functions.
Misuse of optional type and other annotated types.
Wrong usage of pydantic and prisma model.
Wrong usage of third-party library.
The overhaul is required to allow us retrying while fixing the code as a project (not only as a function) and allow us to keep adding non-LLM fixes to it so that we can avoid wasting the retry.
The scope of this PR is still a refactoring and the adding of the PyRight linter, so this PR try to keep the existing behaviour.
The enablement and more tuning on the linter will be done in the followup PR.
---------
Co-authored-by: majdyz <[email protected]>
