2025-07-01 - 2026-07-01
Overview
There has been no commit activity in this period.
92 Issues closed from 1 user
Closed
#111 [PR #111] [CLOSED] Add support for Cohere's generative model and embeddings
Closed
#112 [PR #115] [MERGED] updates question-answer url
Closed
#105 [PR #92] [MERGED] Add Vicuna 13b along w/ Transformer test set
Closed
#106 [PR #98] [MERGED] Streamlit app for Pinecone
Closed
#107 [PR #99] [MERGED] Add Anthropic 100k
Closed
#108 [PR #100] [CLOSED] change wording in bias check prompt template
Closed
#109 [PR #103] [MERGED] Add MosaicML
Closed
#110 [PR #107] [CLOSED] updates to our scenario
Closed
#99 [PR #84] [MERGED] Remove Insight section
Closed
#100 [PR #80] [MERGED] Remove catch for embedding failure
Closed
#101 [PR #82] [MERGED] Restrict file size/quantity
Closed
#102 [PR #85] [MERGED] Fix expt summary refresh
Closed
#103 [PR #88] [MERGED] Update anthropic model
Closed
#104 [PR #90] [CLOSED] [Do Not Merge] Use compression in file transfer
Closed
#95 [PR #68] [MERGED] feat: move segment key to env file
Closed
#96 [PR #70] [CLOSED] Add back env vars
Closed
#97 [PR #71] [MERGED] fix env
Closed
#98 [PR #73] [MERGED] Error handling
Closed
#89 [PR #59] [MERGED] CSV fix
Closed
#90 [PR #58] [MERGED] Fix GPT-4 and expose QA prompt
Closed
#91 [PR #61] [MERGED] Reset "Summary" when # Eval Qs OR test set changes
Closed
#92 [PR #63] [MERGED] Update baseline results
Closed
#93 [PR #64] [MERGED] Add horizontal scroll to tables
Closed
#94 [PR #65] [MERGED] logo + favicon
Closed
#84 [PR #48] [MERGED] feat: add analytics and fix bug with demo env
Closed
#85 [PR #49] [MERGED] Tune prompts and make Descriptive the default
Closed
#86 [PR #50] [MERGED] Prompt improvements
Closed
#87 [PR #53] [MERGED] Minor docs cleanup on API
Closed
#88 [PR #56] [MERGED] Update prompt for better speed and loading message
Closed
#79 [PR #39] [MERGED] feat: improve UI
Closed
#80 [PR #42] [MERGED] Add OpenAI grading prompt
Closed
#81 [PR #43] [MERGED] a few things to get ready for release
Closed
#82 [PR #44] [MERGED] Improve retrieval prompt
Closed
#83 [PR #45] [MERGED] @benisgold/prerelease
Closed
#74 [PR #33] [MERGED] Finish about page
Closed
#75 [PR #22] [MERGED] Improve documentation
Closed
#76 [PR #24] [MERGED] feat: add about page and make things look better
Closed
#77 [PR #34] [MERGED] chore: lfg
Closed
#78 [PR #38] [MERGED] feat: underline current menu item
Closed
#70 [PR #14] [MERGED] Keep descriptive outputs
Closed
#71 [PR #16] [MERGED] Pull in newset SVM retriever
Closed
#72 [PR #18] [MERGED] feat: wrap metrics and results table in cards
Closed
#73 [PR #17] [MERGED] Re-try question generation
Closed
#69 [PR #8] [MERGED] feat: monoreop
Closed
#64 [PR #2] [MERGED] stream
Closed
#65 [PR #3] [MERGED] @benisgold/stream
Closed
#66 [PR #4] [MERGED] Add API
Closed
#67 [PR #1] [MERGED] feat: Add timeline
Closed
#68 [PR #5] [MERGED] Latency improvement for llama-index
Closed
#47 [GH-ISSUE #83] Remove the Insight panel (not sure it is correct and uses subjective weighting)
Closed
#40 [GH-ISSUE #75] Analytics
Closed
#41 [GH-ISSUE #77] Remove HF embeddings (slow) and HF fallback in the event of illegal token
Closed
#43 [GH-ISSUE #78] Add 50MB file limit and single file
Closed
#44 [GH-ISSUE #81] Experiment numbers are not refreshing correctly
Closed
#45 [GH-ISSUE #79] Catch server errors and return alert to the user
Closed
#34 [GH-ISSUE #62] Expt numbering is wrong after we start w/ 3 examples
Closed
#35 [GH-ISSUE #67] Fix SVM in prod
Closed
#36 [GH-ISSUE #66] Move API Key
Closed
#37 [GH-ISSUE #72] Prod (likely) hitting OOM
Closed
#38 [GH-ISSUE #69] Unbreak
Closed
#31 [GH-ISSUE #60] horizontal scroll is needed on Summary table
Closed
#32 [GH-ISSUE #57] Anthropic model is failing on prod
Closed
#33 [GH-ISSUE #55] Show intermediate states in the loading bar
Closed
#28 [GH-ISSUE #51] Remove Doppler dep
Closed
#29 [GH-ISSUE #54] Add Langchain logo to top left and make brain emoji the favicon
Closed
#30 [GH-ISSUE #52] [Nit] Change link in repo to new app (https://autoevaluator.langchain.com/)
Closed
#22 [GH-ISSUE #37] update labels in graph to match the axis labels
Closed
#23 [GH-ISSUE #40] Add instructions to the demo landing page
Closed
#24 [GH-ISSUE #36] current menu item should be underlines
Closed
#25 [GH-ISSUE #46] Past experimental results should be cleared when eval set changes
Closed
#26 [GH-ISSUE #47] Write-up on learnings / opportunities
Closed
#27 [GH-ISSUE #41] The Test Dataset does not match what is shown in Experiment Results when app is first opened
Closed
#19 [GH-ISSUE #31] Create "default" page w/ pre-populated data (from Karpathy pod) and "playground"
Closed
#20 [GH-ISSUE #32] Make all side panels drop-down lists
Closed
#21 [GH-ISSUE #35] add a file to the demo envioronment
Closed
#16 [GH-ISSUE #30] disable adding test data by default in the playground environment
Closed
#17 [GH-ISSUE #29] [mega request] Caching on back-end of index so it does not re-generate each expt
Closed
#18 [GH-ISSUE #28] Bug in the FAST scoring
Closed
#11 [GH-ISSUE #20] Fails to run w/ Llama-Ix
Closed
#12 [GH-ISSUE #21] Possible to hang during generate_eval( )
Closed
#13 [GH-ISSUE #27] [backlog] App re-generating index when it does not need to
Closed
#14 [GH-ISSUE #26] Add sidebar scrolling
Closed
#15 [GH-ISSUE #25] Failure on upload of some PDFs
Closed
#7 [GH-ISSUE #13] Add back descriptive scoring output
Closed
#8 [GH-ISSUE #15] bug: incorrect # of output rows
Closed
#9 [GH-ISSUE #19] add logrocket for error tracking and product analytics
Closed
#4 [GH-ISSUE #10] Center / enlarge the spinner (maybe show processing stages, as before)
Closed
#5 [GH-ISSUE #12] Change app name: "Evaluator AI - evaluate your QA chains." to "Auto Evaluator"
Closed
#6 [GH-ISSUE #11] Chart w/ answer score vs retrieval score and latency at size
Closed
#1 [GH-ISSUE #6] Add Mantine table (collapsible cols)
Closed
#2 [GH-ISSUE #9] Add support for modifying input prompt
Closed
#3 [GH-ISSUE #7] Add OpenAI closedQA grader prompt
113 Issues created by 1 user
Opened
#1 [GH-ISSUE #6] Add Mantine table (collapsible cols)
Opened
#2 [GH-ISSUE #9] Add support for modifying input prompt
Opened
#3 [GH-ISSUE #7] Add OpenAI closedQA grader prompt
Opened
#4 [GH-ISSUE #10] Center / enlarge the spinner (maybe show processing stages, as before)
Opened
#5 [GH-ISSUE #12] Change app name: "Evaluator AI - evaluate your QA chains." to "Auto Evaluator"
Opened
#6 [GH-ISSUE #11] Chart w/ answer score vs retrieval score and latency at size
Opened
#7 [GH-ISSUE #13] Add back descriptive scoring output
Opened
#8 [GH-ISSUE #15] bug: incorrect # of output rows
Opened
#9 [GH-ISSUE #19] add logrocket for error tracking and product analytics
Opened
#10 [GH-ISSUE #23] refactor Body.tsx into multiple sub react components
Opened
#11 [GH-ISSUE #20] Fails to run w/ Llama-Ix
Opened
#12 [GH-ISSUE #21] Possible to hang during generate_eval( )
Opened
#13 [GH-ISSUE #27] [backlog] App re-generating index when it does not need to
Opened
#14 [GH-ISSUE #26] Add sidebar scrolling
Opened
#15 [GH-ISSUE #25] Failure on upload of some PDFs
Opened
#16 [GH-ISSUE #30] disable adding test data by default in the playground environment
Opened
#17 [GH-ISSUE #29] [mega request] Caching on back-end of index so it does not re-generate each expt
Opened
#18 [GH-ISSUE #28] Bug in the FAST scoring
Opened
#19 [GH-ISSUE #31] Create "default" page w/ pre-populated data (from Karpathy pod) and "playground"
Opened
#20 [GH-ISSUE #32] Make all side panels drop-down lists
Opened
#21 [GH-ISSUE #35] add a file to the demo envioronment
Opened
#22 [GH-ISSUE #37] update labels in graph to match the axis labels
Opened
#23 [GH-ISSUE #40] Add instructions to the demo landing page
Opened
#24 [GH-ISSUE #36] current menu item should be underlines
Opened
#25 [GH-ISSUE #46] Past experimental results should be cleared when eval set changes
Opened
#26 [GH-ISSUE #47] Write-up on learnings / opportunities
Opened
#27 [GH-ISSUE #41] The Test Dataset does not match what is shown in Experiment Results when app is first opened
Opened
#28 [GH-ISSUE #51] Remove Doppler dep
Opened
#29 [GH-ISSUE #54] Add Langchain logo to top left and make brain emoji the favicon
Opened
#30 [GH-ISSUE #52] [Nit] Change link in repo to new app (https://autoevaluator.langchain.com/)
Opened
#31 [GH-ISSUE #60] horizontal scroll is needed on Summary table
Opened
#32 [GH-ISSUE #57] Anthropic model is failing on prod
Opened
#33 [GH-ISSUE #55] Show intermediate states in the loading bar
Opened
#34 [GH-ISSUE #62] Expt numbering is wrong after we start w/ 3 examples
Opened
#35 [GH-ISSUE #67] Fix SVM in prod
Opened
#36 [GH-ISSUE #66] Move API Key
Opened
#37 [GH-ISSUE #72] Prod (likely) hitting OOM
Opened
#38 [GH-ISSUE #69] Unbreak
Opened
#39 [GH-ISSUE #74] File transfer and reading time is slow in prod
Opened
#40 [GH-ISSUE #75] Analytics
Opened
#41 [GH-ISSUE #77] Remove HF embeddings (slow) and HF fallback in the event of illegal token
Opened
#42 [GH-ISSUE #76] API continues to run after connection to client is closed / refreshed
Opened
#43 [GH-ISSUE #78] Add 50MB file limit and single file
Opened
#44 [GH-ISSUE #81] Experiment numbers are not refreshing correctly
Opened
#45 [GH-ISSUE #79] Catch server errors and return alert to the user
Opened
#46 [GH-ISSUE #87] Anthropic model appears to be deprecated
Opened
#47 [GH-ISSUE #83] Remove the Insight panel (not sure it is correct and uses subjective weighting)
Opened
#48 [GH-ISSUE #86] Missing logging / alert when back-end crashes
Opened
#49 [GH-ISSUE #89] OpenAI invalid API key error
Opened
#50 [GH-ISSUE #91] Back-end crashing
Opened
#51 [GH-ISSUE #93] Chunk size is always larger than allowed, even put the chunk size to '500' which is smallest in parameter list
Opened
#52 [GH-ISSUE #95] look into hosting vicuna 13B on EC2
Opened
#53 [GH-ISSUE #96] disable sidebar when the experiment is running
Opened
#54 [GH-ISSUE #94] make demo environment mobile friendly
Opened
#55 [GH-ISSUE #101] Replicat Vicuna API returning empty response
Opened
#56 [GH-ISSUE #102] Server error when using online Playground
Opened
#57 [GH-ISSUE #97] How generalizable is this framework?
Opened
#58 [GH-ISSUE #106] Same questions are being generated
Opened
#59 [GH-ISSUE #105] Support for more document types (HTML, CSV, etc)
Opened
#60 [GH-ISSUE #104] Python + Node versions are missing
Opened
#61 [GH-ISSUE #108] Isolated evaluation of retrieval
Opened
#62 [GH-ISSUE #114] Can I add my local LLM model in Model list in application (https://autoevaluator.langchain.com/)?
Opened
#63 [GH-ISSUE #110] Labelstudio integration
Opened
#64 [PR #2] [MERGED] stream
Opened
#65 [PR #3] [MERGED] @benisgold/stream
Opened
#66 [PR #4] [MERGED] Add API
Opened
#67 [PR #1] [MERGED] feat: Add timeline
Opened
#68 [PR #5] [MERGED] Latency improvement for llama-index
Opened
#69 [PR #8] [MERGED] feat: monoreop
Opened
#70 [PR #14] [MERGED] Keep descriptive outputs
Opened
#71 [PR #16] [MERGED] Pull in newset SVM retriever
Opened
#72 [PR #18] [MERGED] feat: wrap metrics and results table in cards
Opened
#73 [PR #17] [MERGED] Re-try question generation
Opened
#74 [PR #33] [MERGED] Finish about page
Opened
#75 [PR #22] [MERGED] Improve documentation
Opened
#76 [PR #24] [MERGED] feat: add about page and make things look better
Opened
#77 [PR #34] [MERGED] chore: lfg
Opened
#78 [PR #38] [MERGED] feat: underline current menu item
Opened
#79 [PR #39] [MERGED] feat: improve UI
Opened
#80 [PR #42] [MERGED] Add OpenAI grading prompt
Opened
#81 [PR #43] [MERGED] a few things to get ready for release
Opened
#82 [PR #44] [MERGED] Improve retrieval prompt
Opened
#83 [PR #45] [MERGED] @benisgold/prerelease
Opened
#84 [PR #48] [MERGED] feat: add analytics and fix bug with demo env
Opened
#85 [PR #49] [MERGED] Tune prompts and make Descriptive the default
Opened
#86 [PR #50] [MERGED] Prompt improvements
Opened
#87 [PR #53] [MERGED] Minor docs cleanup on API
Opened
#88 [PR #56] [MERGED] Update prompt for better speed and loading message
Opened
#89 [PR #59] [MERGED] CSV fix
Opened
#90 [PR #58] [MERGED] Fix GPT-4 and expose QA prompt
Opened
#91 [PR #61] [MERGED] Reset "Summary" when # Eval Qs OR test set changes
Opened
#92 [PR #63] [MERGED] Update baseline results
Opened
#93 [PR #64] [MERGED] Add horizontal scroll to tables
Opened
#94 [PR #65] [MERGED] logo + favicon
Opened
#95 [PR #68] [MERGED] feat: move segment key to env file
Opened
#96 [PR #70] [CLOSED] Add back env vars
Opened
#97 [PR #71] [MERGED] fix env
Opened
#98 [PR #73] [MERGED] Error handling
Opened
#99 [PR #84] [MERGED] Remove Insight section
Opened
#100 [PR #80] [MERGED] Remove catch for embedding failure
Opened
#101 [PR #82] [MERGED] Restrict file size/quantity
Opened
#102 [PR #85] [MERGED] Fix expt summary refresh
Opened
#103 [PR #88] [MERGED] Update anthropic model
Opened
#104 [PR #90] [CLOSED] [Do Not Merge] Use compression in file transfer
Opened
#105 [PR #92] [MERGED] Add Vicuna 13b along w/ Transformer test set
Opened
#106 [PR #98] [MERGED] Streamlit app for Pinecone
Opened
#107 [PR #99] [MERGED] Add Anthropic 100k
Opened
#108 [PR #100] [CLOSED] change wording in bias check prompt template
Opened
#109 [PR #103] [MERGED] Add MosaicML
Opened
#110 [PR #107] [CLOSED] updates to our scenario
Opened
#111 [PR #111] [CLOSED] Add support for Cohere's generative model and embeddings
Opened
#112 [PR #115] [MERGED] updates question-answer url
Opened
#113 [PR #109] Change wrong parameter name at annotation