langchain-ai/ambient-agent-101
1
0
Fork 0
mirror of https://github.com/langchain-ai/ambient-agent-101.git synced 2026-07-01 21:14:02 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
ambient-agent-101/tests/memory_testing.ipynb
T
catherine-langchain ba64be1651 Upload files
2025-07-06 12:39:26 -07:00

2641 lines
154 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"id": "d047044f",
"metadata": {},
"source": [
"# Test Cases for Memory"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9e2c818a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/Users/rlm/Desktop/Code/interrupt_workshop\n"
]
}
],
"source": [
"%cd ..\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
"import uuid\n",
"from langgraph.store.memory import InMemoryStore\n",
"from langgraph.checkpoint.memory import MemorySaver\n",
"from email_assistant.email_assistant_hitl_memory import overall_workflow"
]
},
{
"cell_type": "markdown",
"id": "397114bf",
"metadata": {},
"source": [
"## Accept `write_email` and `schedule_meeting`\n",
"\n",
"Our first test examines what happens when a user accepts the agent's actions without modification. This baseline case helps us understand how the system behaves when no feedback is provided:\n",
"\n",
"1. We'll use the same tax planning email from our previous tests\n",
"2. The system will classify it as \"RESPOND\" and propose scheduling a meeting\n",
"3. We'll accept the meeting schedule without changes\n",
"4. The agent will generate an email confirming the meeting\n",
"5. We'll accept the email without changes\n",
"\n",
"This test demonstrates the default behavior of our memory-enabled system. When a user simply accepts proposed actions, we expect minimal or no memory updates since there's no explicit feedback to learn from. However, the system will still leverage existing memory (if any) when generating its responses."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8be43b59",
"metadata": {},
"outputs": [],
"source": [
"from langgraph.types import Command\n",
"\n",
"# Helper function to display memory content\n",
"def display_memory_content(store, namespace=None):\n",
" # Display current memory content for all namespaces\n",
" print(\"\\n======= CURRENT MEMORY CONTENT =======\")\n",
" if namespace:\n",
" memory = store.get(namespace, \"user_preferences\")\n",
" if memory:\n",
" print(f\"\\n--- {namespace[1]} ---\")\n",
" print({\"preferences\": memory.value})\n",
" else:\n",
" print(f\"\\n--- {namespace[1]} ---\")\n",
" print(\"No memory found\")\n",
" else:\n",
" for namespace in [\n",
" (\"email_assistant\", \"triage_preferences\"),\n",
" (\"email_assistant\", \"response_preferences\"),\n",
" (\"email_assistant\", \"cal_preferences\"),\n",
" (\"email_assistant\", \"background\")\n",
" ]:\n",
" memory = store.get(namespace, \"user_preferences\")\n",
" if memory:\n",
" print(f\"\\n--- {namespace[1]} ---\")\n",
" print({\"preferences\": memory.value})\n",
" else:\n",
" print(f\"\\n--- {namespace[1]} ---\")\n",
" print(\"No memory found\")\n",
" print(\"=======================================\\n\")"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "649cee4f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 45, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Project Manager <pm@client.com>\",\n",
" \"subject\": \"Tax season let's schedule call\",\n",
" \"email_thread\": \"Lance,\\n\\nIt's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\\n\\nAre you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\\n\\nRegards,\\nProject Manager\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_1 = uuid.uuid4()\n",
"thread_config_1 = {\"configurable\": {\"thread_id\": thread_id_1}}\n",
"\n",
"# Run the graph until the first interrupt \n",
"# Email will be classified as \"respond\" \n",
"# Agent will create a schedule_meeting and write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_1):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt\n",
"display_memory_content(store)"
]
},
{
"cell_type": "markdown",
"id": "878e199e",
"metadata": {},
"source": [
"Accept the schedule_meeting tool call\n",
"\n",
"As we examine the initial `schedule_meeting` proposal, note how the system uses existing memory to inform its decisions:\n",
"\n",
"1. The default calendar preferences show a preference for 30-minute meetings, though the email requests 45 minutes\n",
"2. The agent still proposes a 45-minute meeting, respecting the sender's specific request\n",
"3. We accept this proposal without modification to see if simple acceptance triggers any memory updates\n",
"\n",
"After running this step, we'll check the memory contents to confirm whether acceptance alone leads to memory updates. Simple acceptance represents the baseline user experience - the system works as intended without requiring adjustments."
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "9589423b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the schedule_meeting tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'pm@client.com', 'subject': \"Re: Tax season let's schedule call\", 'content': 'Hello,\\n\\nThank you for reaching out. I am available on Tuesday, April 22nd at 2:00 PM for a 45-minute call to discuss tax planning strategies. I have scheduled the meeting accordingly.\\n\\nLooking forward to your suggestions and our discussion.\\n\\nBest regards,\\nLance'}}\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_1):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")"
]
},
{
"cell_type": "markdown",
"id": "e6b80f99",
"metadata": {},
"source": [
"Accept the write_email tool call\n",
"\n",
"Now we'll accept the email draft that confirms the meeting scheduling:\n",
"\n",
"1. The email draft is generated with knowledge of our calendar preferences\n",
"2. It includes details about the meeting time, duration, and purpose\n",
"3. We'll accept it without changes to complete the baseline test case\n",
"\n",
"After accepting, we'll check all memory stores to see if any updates occurred. As expected, simply accepting the agent's proposals doesn't provide strong learning signals - there's no clear feedback about what the user likes or dislikes about the agent's approach.\n",
"\n",
"The trace link shows the complete workflow execution, where we can see that the memory is used in the LLM call for response generation, but no memory updates occur, which is the expected behavior for simple acceptances."
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "12035cf6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the write_email tool call...\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_02WPNb2qpXXVx43d3Z0VzWqv)\n",
" Call ID: call_02WPNb2qpXXVx43d3Z0VzWqv\n",
" Args:\n",
" done: True\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_1):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after accepting the write_email tool call\n",
"display_memory_content(store)"
]
},
{
"cell_type": "markdown",
"id": "fcbc178d",
"metadata": {},
"source": [
"We can look at the full messages, and the trace: \n",
"\n",
"https://smith.langchain.com/public/86ff6474-29fe-452e-8829-b05a91b458eb/r\n",
"\n",
"You'll notice that memory is used in the LLM call to respond. \n",
"\n",
"But the memory store *not* updated, because we haven't added any feedback via HITL."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "10ce8197",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Tax season let's schedule call\n",
"**From**: Project Manager <pm@client.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Lance,\n",
"\n",
"It's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\n",
"\n",
"Are you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\n",
"\n",
"Regards,\n",
"Project Manager\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_6ZbPjmSN8h1bfl3tDCx5ssoj)\n",
" Call ID: call_6ZbPjmSN8h1bfl3tDCx5ssoj\n",
" Args:\n",
" day: 2025-04-22\n",
" check_calendar_availability (call_8M30JVmN7Wj2VlIcDqBE1995)\n",
" Call ID: call_8M30JVmN7Wj2VlIcDqBE1995\n",
" Args:\n",
" day: 2025-04-24\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-22: 9:00 AM, 2:00 PM, 4:00 PM\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-24: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" schedule_meeting (call_TChCTjTsVim5Zj8lnTPUtHs6)\n",
" Call ID: call_TChCTjTsVim5Zj8lnTPUtHs6\n",
" Args:\n",
" attendees: ['pm@client.com', 'lance@company.com']\n",
" subject: Tax Planning Strategies Discussion\n",
" duration_minutes: 45\n",
" preferred_day: 2025-04-22\n",
" start_time: 14\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Meeting 'Tax Planning Strategies Discussion' scheduled on Tuesday, April 22, 2025 at 14 for 45 minutes with 2 attendees\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_KMR1rdut5FcbS82k3ckvCHm7)\n",
" Call ID: call_KMR1rdut5FcbS82k3ckvCHm7\n",
" Args:\n",
" to: pm@client.com\n",
" subject: Re: Tax season let's schedule call\n",
" content: Hello,\n",
"\n",
"Thank you for reaching out. I am available on Tuesday, April 22nd at 2:00 PM for a 45-minute call to discuss tax planning strategies. I have scheduled the meeting accordingly.\n",
"\n",
"Looking forward to your suggestions and our discussion.\n",
"\n",
"Best regards,\n",
"Lance\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Email sent to pm@client.com with subject 'Re: Tax season let's schedule call' and content: Hello,\n",
"\n",
"Thank you for reaching out. I am available on Tuesday, April 22nd at 2:00 PM for a 45-minute call to discuss tax planning strategies. I have scheduled the meeting accordingly.\n",
"\n",
"Looking forward to your suggestions and our discussion.\n",
"\n",
"Best regards,\n",
"Lance\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_02WPNb2qpXXVx43d3Z0VzWqv)\n",
" Call ID: call_02WPNb2qpXXVx43d3Z0VzWqv\n",
" Args:\n",
" done: True\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_1)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "58201a21",
"metadata": {},
"source": [
"## Edit `write_email` and `schedule_meeting`\n",
"\n",
"This test explores how the system learns from direct edits to its proposed actions. When users modify the agent's suggestions, it creates clear, specific learning signals about their preferences:\n",
"\n",
"1. We'll use the same tax planning email as before\n",
"2. When the agent proposes a 45-minute meeting, we'll edit it to:\n",
" - Change the duration to 30 minutes (matching our stored preference)\n",
" - Make the subject line more concise\n",
"3. When the agent drafts an email, we'll edit it to be:\n",
" - Shorter and less formal\n",
" - Structured differently\n",
"\n",
"Edits provide the most explicit feedback about user preferences, letting the system learn exactly what changes are desired. We expect to see specific, targeted updates to our memory stores that reflect these edits."
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "ac260423",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 45, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n"
]
}
],
"source": [
"# Same email as before\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Project Manager <pm@client.com>\",\n",
" \"subject\": \"Tax season let's schedule call\",\n",
" \"email_thread\": \"Lance,\\n\\nIt's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\\n\\nAre you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\\n\\nRegards,\\nProject Manager\"\n",
"}\n",
"\n",
"# Compile the graph with new thread\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_2 = uuid.uuid4()\n",
"thread_config_2 = {\"configurable\": {\"thread_id\": thread_id_2}}\n",
"\n",
"# Run the graph until the first interrupt - will be classified as \"respond\" and the agent will create a write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_2):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt\n",
"display_memory_content(store,(\"email_assistant\", \"cal_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "5d73ba71",
"metadata": {},
"source": [
"Edit the `schedule_meeting` tool call\n",
"\n",
"When we edit the meeting proposal, we're providing direct, explicit feedback about our preferences. This creates a significant learning opportunity for the system:\n",
"\n",
"1. The agent initially proposes a 45-minute meeting (the duration requested in the email)\n",
"2. We edit it to 30 minutes and simplify the subject from \"Tax Planning Strategies Discussion\" to \"Tax Planning Discussion\"\n",
"3. This creates clear, specific feedback about our time preferences and naming conventions\n",
"\n",
"After the edit, we'll check the calendar preferences memory store to see how it's updated. The memory update should capture both:\n",
"- Our preference for shorter 30-minute meetings\n",
"- Our preference for more concise meeting subjects\n",
"\n",
"The trace reveals the precise memory update logic, showing how the system analyzes the difference between its proposal and our edits to extract meaningful patterns and preferences. We can see the detailed justification for each memory update, ensuring transparency in the learning process."
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "af760977",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user editing the schedule_meeting tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'pm@client.com', 'subject': \"Re: Tax season let's schedule call\", 'content': 'Hello,\\n\\nThank you for reaching out regarding tax planning strategies for this year. I have scheduled a 30-minute call for us on Tuesday, April 22nd at 2:00 PM to discuss your suggestions and review potential savings opportunities. If you need a longer duration, please let me know and I can adjust the meeting accordingly.\\n\\nLooking forward to our discussion.\\n\\nBest regards,\\nLance'}}\n",
"\n",
"Checking memory after editing schedule_meeting:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': \"30 minute meetings are preferred, but 15 minute meetings are also acceptable. User prefers 30 minute meetings over longer durations such as 45 minutes when possible. User prefers concise meeting subjects (e.g., 'Tax Planning Discussion' instead of 'Tax Planning Strategies Discussion').\"}\n"
]
}
],
"source": [
"# Now simulate user editing the schedule_meeting tool call\n",
"print(\"\\nSimulating user editing the schedule_meeting tool call...\")\n",
"edited_schedule_args = {\n",
" \"attendees\": [\"pm@client.com\", \"lance@company.com\"],\n",
" \"subject\": \"Tax Planning Discussion\",\n",
" \"duration_minutes\": 30, # Changed from 45 to 30\n",
" \"preferred_day\": \"2025-04-22\",\n",
" \"start_time\": 14 # 2pm\n",
"}\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"edit\", \"args\": {\"args\": edited_schedule_args}}]), config=thread_config_2):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after editing schedule_meeting\n",
"print(\"\\nChecking memory after editing schedule_meeting:\")\n",
"display_memory_content(store,(\"email_assistant\", \"cal_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "0dfc585a",
"metadata": {},
"source": [
"Looking at the memory after editing the calendar invitation, we can see that it's been updated with remarkably specific preferences:\n",
"\n",
"1. The system has identified that we prefer 30-minute meetings over longer durations\n",
"2. It's also captured our preference for concise meeting subjects\n",
"\n",
"What's particularly impressive about this memory update is:\n",
"- It doesn't just record our specific edit, but generalizes to a broader preference pattern\n",
"- It preserves all existing memory content while adding the new information\n",
"- It extracts multiple preference signals from a single edit interaction\n",
"\n",
"Now, let's edit the email draft to see how the system captures different types of communication preferences:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "81a1fa37",
"metadata": {},
"outputs": [],
"source": [
"display_memory_content(store,(\"email_assistant\", \"response_preferences\"))\n",
"# Now simulate user editing the write_email tool call\n",
"print(\"\\nSimulating user editing the write_email tool call...\")\n",
"edited_email_args = {\n",
" \"to\": \"pm@client.com\",\n",
" \"subject\": \"Re: Tax season let's schedule call\",\n",
" \"content\": \"Thanks! I scheduled a 30-minute call next Thursday at 3:00 PM. Would that work for you?\\n\\nBest regards,\\nLance Martin\"\n",
"}\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"edit\", \"args\": {\"args\": edited_email_args}}]), config=thread_config_2):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after editing write_email\n",
"print(\"\\nChecking memory after editing write_email:\")\n",
"display_memory_content(store,(\"email_assistant\", \"response_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "3ffbd5f9",
"metadata": {},
"source": [
"Our email edit reveals even more sophisticated learning capabilities:\n",
"\n",
"1. We've dramatically shortened and simplified the email content\n",
"2. We've changed the tone to be more casual\n",
"3. We've added a question asking for confirmation rather than assuming the time works\n",
"4. We've slightly altered the meeting details (day and time)\n",
"\n",
"Looking at the updated memory, we can see that the system has extracted a key insight about our communication style:\n",
"\n",
"```\n",
"When scheduling a meeting, ask the recipient to confirm if the proposed time works for them, rather than assuming and stating the meeting is already scheduled.\n",
"```\n",
"\n",
"This demonstrates the system's ability to:\n",
"- Analyze our edit not just at a superficial level, but to understand intent\n",
"- Extract generalizable principles from specific examples\n",
"- Preserve all existing guidance while adding new insights\n",
"- Maintain the organization and structure of the memory\n",
"\n",
"These targeted, high-quality memory updates will improve all future interactions without requiring repeated corrections."
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "8ad818d6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Tax season let's schedule call\n",
"**From**: Project Manager <pm@client.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Lance,\n",
"\n",
"It's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\n",
"\n",
"Are you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\n",
"\n",
"Regards,\n",
"Project Manager\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_3lgcL8JNtxaakOR8NqZyVH2n)\n",
" Call ID: call_3lgcL8JNtxaakOR8NqZyVH2n\n",
" Args:\n",
" day: 2025-04-22\n",
" check_calendar_availability (call_Ru9AMMrCf7wlWPcS9gV7ohCq)\n",
" Call ID: call_Ru9AMMrCf7wlWPcS9gV7ohCq\n",
" Args:\n",
" day: 2025-04-24\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-22: 9:00 AM, 2:00 PM, 4:00 PM\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-24: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" schedule_meeting (call_C6OAozNqxP5JDw42Y0T0m4Fg)\n",
" Call ID: call_C6OAozNqxP5JDw42Y0T0m4Fg\n",
" Args:\n",
" attendees: ['pm@client.com', 'lance@company.com']\n",
" subject: Tax Planning Discussion\n",
" duration_minutes: 30\n",
" preferred_day: 2025-04-22\n",
" start_time: 14\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Meeting 'Tax Planning Discussion' scheduled on Tuesday, April 22, 2025 at 14 for 30 minutes with 2 attendees\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_mvJ8FX2j8spcrtJnpU4dgmLC)\n",
" Call ID: call_mvJ8FX2j8spcrtJnpU4dgmLC\n",
" Args:\n",
" to: pm@client.com\n",
" subject: Re: Tax season let's schedule call\n",
" content: Thanks! I scheduled a 30-minute call next Thursday at 3:00 PM. Would that work for you?\n",
"\n",
"Best regards,\n",
"Lance Martin\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Email sent to pm@client.com with subject 'Re: Tax season let's schedule call' and content: Thanks! I scheduled a 30-minute call next Thursday at 3:00 PM. Would that work for you?\n",
"\n",
"Best regards,\n",
"Lance Martin\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_JRSWVKubpzaxPrU9deWet6YL)\n",
" Call ID: call_JRSWVKubpzaxPrU9deWet6YL\n",
" Args:\n",
" done: True\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_2)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "0e14918e",
"metadata": {},
"source": [
"## Ignore `write_email`, `schedule_meeting`, and `question`\n",
"\n",
"This test explores how the system learns from rejection. When users ignore (reject) the agent's suggestions, it creates a strong signal about content they don't want to handle:\n",
"\n",
"1. We'll first test ignoring a `schedule_meeting` request entirely\n",
"2. Then we'll test accepting a meeting but ignoring the follow-up email\n",
"3. Finally, we'll test ignoring a `question` for a different email context\n",
"\n",
"These rejection signals help the system learn what types of emails and actions a user prefers not to deal with, leading to more appropriate triage decisions in the future. We expect significant updates to the triage preferences memory after each ignore action."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "0d015c3f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 45, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Project Manager <pm@client.com>\",\n",
" \"subject\": \"Tax season let's schedule call\",\n",
" \"email_thread\": \"Lance,\\n\\nIt's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\\n\\nAre you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\\n\\nRegards,\\nProject Manager\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_3 = uuid.uuid4()\n",
"thread_config_3 = {\"configurable\": {\"thread_id\": thread_id_3}}\n",
"\n",
"# Run the graph until the first interrupt \n",
"# Email will be classified as \"respond\" \n",
"# Agent will create a schedule_meeting and write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_3):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "c782e711",
"metadata": {},
"source": [
"Ignore the `schedule_meeting` tool call\n",
"\n",
"When we ignore a meeting scheduling request, we're signaling that we don't want to handle this type of email through the assistant. This creates a powerful learning opportunity about our triage preferences:\n",
"\n",
"1. The assistant initially classified the tax planning email as \"RESPOND\"\n",
"2. But by ignoring the scheduling request, we indicate we'd prefer not to handle this type of email\n",
"3. The system needs to update its triage classification preferences to reflect this rejection\n",
"\n",
"After ignoring the request, we'll check the triage preferences memory to see how the rejection affected the system's understanding. The memory update should show a new pattern added to the \"not worth responding to\" section, specifically about tax planning calls or similar recurring discussions.\n",
"\n",
"The trace shows how the system processes this rejection, identifies the pattern, and updates the memory with specific justification for why this type of email should be classified differently in the future."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "16c4d83b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user ignoring the schedule_meeting tool call...\n",
"\n",
"Checking memory after ignoring first tool call:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"Emails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n- Client requests to schedule tax planning calls (unless explicitly instructed otherwise)\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"print(f\"\\nSimulating user ignoring the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"ignore\"}]), config=thread_config_3):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after ignoring first tool call\n",
"print(\"\\nChecking memory after ignoring first tool call:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "67462024",
"metadata": {},
"source": [
"Looking at the memory update after ignoring the schedule_meeting tool call, we can see a remarkable triage preference update:\n",
"\n",
"1. The system has added \"Client requests to schedule tax planning calls\" to the \"emails not worth responding to\" section\n",
"2. It correctly identified the general pattern (scheduling routine calls) rather than overfitting to just this specific instance\n",
"3. It included the parenthetical note \"(unless explicitly instructed otherwise)\" to maintain flexibility\n",
"\n",
"This update demonstrates the system's ability to:\n",
"- Infer general patterns from specific instances of rejection\n",
"- Update the triage filters that determine initial email classification\n",
"- Preserve the organization and priority of existing preferences\n",
"- Include appropriate qualifiers to avoid overly rigid rules\n",
"\n",
"Next, let's see what happens when we accept the meeting but reject the email draft:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "b869485a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 45, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"Simulating user accepting the schedule_meeting tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'pm@client.com', 'subject': \"Re: Tax season let's schedule call\", 'content': 'Hello,\\n\\nThank you for reaching out. I am available on Tuesday, April 22nd at 2:00 PM for a 45-minute call to discuss tax planning strategies. I have scheduled the meeting accordingly.\\n\\nLooking forward to your suggestions and the discussion.\\n\\nBest regards,\\nLance'}}\n",
"\n",
"Simulating user ignoring the write_email tool call...\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User ignored this email draft. Ignore this email and end the workflow.\n",
"\n",
"Checking memory after ignoring second tool call:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n- Client requests to schedule routine calls (such as tax planning or similar recurring discussions)\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Project Manager <pm@client.com>\",\n",
" \"subject\": \"Tax season let's schedule call\",\n",
" \"email_thread\": \"Lance,\\n\\nIt's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\\n\\nAre you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\\n\\nRegards,\\nProject Manager\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_3 = uuid.uuid4()\n",
"thread_config_3 = {\"configurable\": {\"thread_id\": thread_id_3}}\n",
"\n",
"# Run the graph until the first interrupt \n",
"# Email will be classified as \"respond\" \n",
"# Agent will create a schedule_meeting and write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_3):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_3):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"print(f\"\\nSimulating user ignoring the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"ignore\"}]), config=thread_config_3):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after ignoring second tool call\n",
"print(\"\\nChecking memory after ignoring second tool call:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "694db9f5",
"metadata": {},
"source": [
"When we accept the meeting but ignore the email draft, we're sending a more nuanced signal about our preferences:\n",
"\n",
"1. We're willing to schedule the meeting (accepting the first tool call)\n",
"2. But we don't want to send a confirmation email about it (ignoring the second tool call)\n",
"\n",
"Looking at the memory update, we see another evolution of our triage preferences:\n",
"\n",
"```\n",
"\"Client requests to schedule routine calls (such as tax planning or similar recurring discussions)\"\n",
"```\n",
"\n",
"The system has:\n",
"- Broadened the pattern from just \"tax planning calls\" to \"routine calls\" generally\n",
"- Added examples in parentheses for clarity\n",
"- Positioned this in the \"not worth responding to\" section\n",
"- Maintained all other existing preferences\n",
"\n",
"This demonstrates how the memory evolves over multiple interactions, becoming increasingly accurate and generalizable with each additional data point. The system is continuously refining its understanding based on our feedback patterns."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "272bb9ed",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Tax season let's schedule call\n",
"**From**: Project Manager <pm@client.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Lance,\n",
"\n",
"It's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\n",
"\n",
"Are you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\n",
"\n",
"Regards,\n",
"Project Manager\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_l9Jn2ghL4T0aPuMhjpXf36Ga)\n",
" Call ID: call_l9Jn2ghL4T0aPuMhjpXf36Ga\n",
" Args:\n",
" day: 2025-04-22\n",
" check_calendar_availability (call_8eOXPjqVpaIRDkABD80XRYUU)\n",
" Call ID: call_8eOXPjqVpaIRDkABD80XRYUU\n",
" Args:\n",
" day: 2025-04-24\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-22: 9:00 AM, 2:00 PM, 4:00 PM\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-24: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" schedule_meeting (call_5L9UBPXPfnqhuhxlBvHSQncm)\n",
" Call ID: call_5L9UBPXPfnqhuhxlBvHSQncm\n",
" Args:\n",
" attendees: ['pm@client.com', 'lance@company.com']\n",
" subject: Tax Planning Strategies Discussion\n",
" duration_minutes: 45\n",
" preferred_day: 2025-04-22\n",
" start_time: 14\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Meeting 'Tax Planning Strategies Discussion' scheduled on Tuesday, April 22, 2025 at 14 for 45 minutes with 2 attendees\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_635xNQnAYz2BBX1Y5NiGtuE6)\n",
" Call ID: call_635xNQnAYz2BBX1Y5NiGtuE6\n",
" Args:\n",
" to: pm@client.com\n",
" subject: Re: Tax season let's schedule call\n",
" content: Hello,\n",
"\n",
"Thank you for reaching out. I am available on Tuesday, April 22nd at 2:00 PM for a 45-minute call to discuss tax planning strategies. I have scheduled the meeting accordingly.\n",
"\n",
"Looking forward to your suggestions and the discussion.\n",
"\n",
"Best regards,\n",
"Lance\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User ignored this email draft. Ignore this email and end the workflow.\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_3)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "597930b3",
"metadata": {},
"source": [
"Now let's try an email that calls the `Question` tool\n",
"\n",
"For our third rejection test, we'll use a different type of email - a casual social invitation about brunch. This gives us insight into how the system learns about personal vs. professional communication preferences:\n",
"\n",
"1. The system classifies this personal invitation as \"RESPOND\"\n",
"2. Rather than answering directly, it uses the Question tool to ask for clarification\n",
"3. We'll ignore this question, indicating we don't want to handle these types of emails through the assistant\n",
"\n",
"This test shows how ignoring questions (not just actions) can also update our triage preferences. By rejecting the clarification attempt, we signal that this entire category of email doesn't warrant response through the assistant."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "efb91337",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'partner@home.com', 'subject': 'Re: Meet Jim and Lisa for brunch in 3 weeks?', 'content': 'That sounds like a great idea! I checked my calendar for three weeks from now (Thursday, May 8th), and I’m available at 9:00 AM, 2:00 PM, or 4:00 PM. Do any of those times work for you, Jim, and Lisa? The new place on 17th sounds perfect.\\n\\nLet me know what everyone prefers and I’ll confirm the reservation.'}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Partner <partner@home.com>\",\n",
" \"subject\": \"Meet Jim and Lisa for brunch in 3 weeks?\",\n",
" \"email_thread\": \"Hey, should we invite Jim and Lisa to brunch in 3 weeks? We could go to the new place on 17th that everyone is talking about.\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_4 = uuid.uuid4()\n",
"thread_config_4 = {\"configurable\": {\"thread_id\": thread_id_4}}\n",
"\n",
"# Run the graph until the first interrupt \n",
"# Email will be classified as \"respond\" \n",
"# Agent will create a schedule_meeting and write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_4):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt for Question tool\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "2bb6581a",
"metadata": {},
"source": [
"Ignore the `question` tool call\n",
"\n",
"When we ignore a question from the assistant about a personal social invitation, we're providing yet another type of feedback:\n",
"\n",
"1. The system initially tries to get clarification before responding\n",
"2. By ignoring the question, we indicate we don't even want to engage with this type of email\n",
"3. This suggests the entire category of social invitations should be handled differently\n",
"\n",
"After ignoring, we'll check the triage preferences again to see how they've evolved. We expect to see a new entry about social invitations added to the \"not worth responding to\" section.\n",
"\n",
"The memory update justification analyzes our rejection of the question about an informal social invitation and extracts a general pattern about our preference not to handle social invitations through the assistant."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "f227a298",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user ignoring the write_email tool call...\n",
"\n",
"Checking memory after ignoring Question tool:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"Emails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n- Social invitations from partner (e.g., brunch plans, casual meetups)\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"print(f\"\\nSimulating user ignoring the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"ignore\"}]), config=thread_config_4):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after ignoring Question tool\n",
"print(\"\\nChecking memory after ignoring Question tool:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "20bea846",
"metadata": {},
"source": [
"Looking at the memory update after ignoring the question about brunch plans, we see another sophisticated triage preference update:\n",
"\n",
"```\n",
"\"Social invitations from partner (e.g., brunch plans, casual meetups)\"\n",
"```\n",
"\n",
"This demonstrates how the system:\n",
"1. Correctly identifies personal social invitations as a distinct category\n",
"2. Specifically notes they're from \"partner\" - showing it's learning to distinguish senders\n",
"3. Provides examples to clarify the pattern\n",
"4. Adds this to the \"not worth responding to\" section\n",
"\n",
"These three ignores have collectively taught the system quite a bit about what types of emails we prefer not to handle through the assistant:\n",
"- Tax planning calls and routine client scheduling\n",
"- Social invitations from partners\n",
"- Each with appropriate specificity and generalizability\n",
"\n",
"In a real-world scenario, these learned preferences would ensure that similar future emails would be classified differently, saving the user time by automatically filtering out categories they've previously rejected."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "2ca65c1b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Meet Jim and Lisa for brunch in 3 weeks?\n",
"**From**: Partner <partner@home.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Hey, should we invite Jim and Lisa to brunch in 3 weeks? We could go to the new place on 17th that everyone is talking about.\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_5jTHskfgPCFQKakHl2dFxvsD)\n",
" Call ID: call_5jTHskfgPCFQKakHl2dFxvsD\n",
" Args:\n",
" day: 2025-05-08\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-05-08: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_3tvSXyHFCWGk4yzbBYMCdEWz)\n",
" Call ID: call_3tvSXyHFCWGk4yzbBYMCdEWz\n",
" Args:\n",
" to: partner@home.com\n",
" subject: Re: Meet Jim and Lisa for brunch in 3 weeks?\n",
" content: That sounds like a great idea! I checked my calendar for three weeks from now (Thursday, May 8th), and I’m available at 9:00 AM, 2:00 PM, or 4:00 PM. Do any of those times work for you, Jim, and Lisa? The new place on 17th sounds perfect.\n",
"\n",
"Let me know what everyone prefers and I’ll confirm the reservation.\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User ignored this email draft. Ignore this email and end the workflow.\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_4)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "6d92a42b",
"metadata": {},
"source": [
"## Respond (with feedback) `write_email`, `schedule_meeting`, and `question`\n",
"\n",
"Our final test set explores the \"response\" feedback pattern - providing guidance without directly editing or accepting. This conversational feedback mechanism offers a middle ground between acceptance and editing:\n",
"\n",
"1. First, we'll test feedback for meeting scheduling by requesting:\n",
" - Shorter duration (30 minutes instead of 45)\n",
" - Afternoon meeting times (after 2pm)\n",
" \n",
"2. Next, we'll test feedback for email drafting by requesting:\n",
" - Shorter, less formal language\n",
" - A specific closing statement about looking forward to the meeting\n",
" \n",
"3. Finally, we'll test feedback for questions by providing:\n",
" - A direct answer with additional context\n",
" - Specific preferences (brunch location, time)\n",
"\n",
"This natural language feedback approach lets users guide the assistant without having to do the work themselves. We expect to see detailed memory updates that extract the general principles from our specific feedback."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "07676231",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 45, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Project Manager <pm@client.com>\",\n",
" \"subject\": \"Tax season let's schedule call\",\n",
" \"email_thread\": \"Lance,\\n\\nIt's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\\n\\nAre you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\\n\\nRegards,\\nProject Manager\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_5 = uuid.uuid4()\n",
"thread_config_5 = {\"configurable\": {\"thread_id\": thread_id_5}}\n",
"\n",
"# Run the graph until the first interrupt \n",
"# Email will be classified as \"respond\" \n",
"# Agent will create a schedule_meeting and write_email tool call\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_5):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt \n",
"display_memory_content(store, (\"email_assistant\", \"cal_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "b85fc45d",
"metadata": {},
"source": [
"Provide feedback for the `schedule_meeting` tool call\n",
"\n",
"Instead of directly editing the meeting proposal or simply accepting it, we'll provide natural language feedback:\n",
"\n",
"1. We request a 30-minute meeting instead of 45 minutes\n",
"2. We express a preference for afternoon meetings after 2pm\n",
"3. The system must interpret this feedback and generate a new proposal\n",
"\n",
"This conversational approach is often more natural and efficient than direct editing, especially for mobile users or those who prefer to give high-level direction rather than detailed edits.\n",
"\n",
"After providing feedback, we'll examine the calendar preferences memory to see how this natural language guidance is captured. We expect to see the system extract both the meeting duration and time-of-day preferences as general principles."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "30a151f1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user providing feedback for the schedule_meeting tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'schedule_meeting', 'args': {'attendees': ['pm@client.com', 'lance@company.com'], 'subject': 'Tax Planning Strategies Discussion', 'duration_minutes': 30, 'preferred_day': '2025-04-22', 'start_time': 14}}\n",
"\n",
"Checking memory after providing feedback for schedule_meeting:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\nAfternoon meetings after 2pm are preferred.'}\n"
]
}
],
"source": [
"print(f\"\\nSimulating user providing feedback for the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"response\", \"args\": \"Please schedule this for 30 minutes instead of 45 minutes, and I prefer afternoon meetings after 2pm.\"}]), config=thread_config_5):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after providing feedback for schedule_meeting\n",
"print(\"\\nChecking memory after providing feedback for schedule_meeting:\")\n",
"display_memory_content(store, (\"email_assistant\", \"cal_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "8088757c",
"metadata": {},
"source": [
"Our memory check after providing feedback shows an elegantly simple calendar preference update:\n",
"\n",
"```\n",
"30 minute meetings are preferred, but 15 minute meetings are also acceptable.\n",
"Afternoon meetings after 2pm are preferred.\n",
"```\n",
"\n",
"The system has:\n",
"1. Captured both aspects of our feedback (duration and time of day)\n",
"2. Preserved the existing preference about 15-minute meetings\n",
"3. Added our preference for afternoon meetings after 2pm as a new line\n",
"4. Kept the format clean and readable\n",
"\n",
"This natural language feedback mechanism creates the same quality of memory updates as direct editing but requires less effort from the user. The system is able to extract structured preferences from unstructured feedback, showing its ability to learn from conversational interactions.\n",
"\n",
"Let's accept this revised meeting proposal and move to the email draft:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "545063be",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the schedule_meeting tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'pm@client.com', 'subject': \"Re: Tax season let's schedule call\", 'content': 'Hello,\\n\\nThank you for reaching out regarding tax planning strategies. I am available on Tuesday, April 22nd at 2:00 PM, and have scheduled a 30-minute call for us to discuss your suggestions. If you need more time or would prefer a different slot, please let me know.\\n\\nLooking forward to our conversation.\\n\\nBest regards,\\nLance'}}\n",
"\n",
"Checking memory after accepting schedule_meeting after feedback:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\nAfternoon meetings after 2pm are preferred.'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_5):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after accepting schedule_meeting after feedback\n",
"print(\"\\nChecking memory after accepting schedule_meeting after feedback:\")\n",
"display_memory_content(store, (\"email_assistant\", \"response_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "e72ede94",
"metadata": {},
"source": [
"Now provide feedback for the `write_email` tool call\n",
"\n",
"Similar to our meeting feedback, we'll now provide natural language guidance for the email draft:\n",
"\n",
"1. We request \"shorter and less formal\" language - a style preference\n",
"2. We ask for a specific closing statement about looking forward to the meeting\n",
"3. The system must interpret this guidance and rewrite the email accordingly\n",
"\n",
"After providing this feedback, we'll check the response preferences memory to see how these style and structure preferences are captured. We expect to see generalizable guidelines about email brevity, formality, and closing statements added to our preference profile."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "9831ad2d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user providing feedback for the write_email tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'pm@client.com', 'subject': \"Re: Tax season let's schedule call\", 'content': 'Hi,\\n\\nThanks for reaching out. I’ve scheduled us for a 30-minute call on Tuesday, April 22nd at 2:00 PM to go over tax planning. Let me know if you need a different time.\\n\\nLooking forward to our chat!\\n\\nBest,\\nLance'}}\n",
"\n",
"Checking memory after providing feedback for write_email:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"Use professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n- When scheduling meetings, prefer afternoon times after 2pm when possible, and default to 30-minute durations unless otherwise specified.\\n\\nWhen writing email responses:\\n- Favor shorter and less formal language when possible, unless the context requires formality.\\n- Include a closing statement expressing that you look forward to the meeting or conversation when confirming appointments.\"}\n"
]
}
],
"source": [
"print(f\"\\nSimulating user providing feedback for the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"response\", \"args\": \"Shorter and less formal. Include a closing statement about looking forward to the meeting!\"}]), config=thread_config_5):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after providing feedback for write_email\n",
"print(\"\\nChecking memory after providing feedback for write_email:\")\n",
"display_memory_content(store, (\"email_assistant\", \"response_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "b5b360a2",
"metadata": {},
"source": [
"The memory update after our email feedback shows highly sophisticated learning about both meeting scheduling and email writing preferences:\n",
"\n",
"1. The system has added a complete new section to the response preferences entitled \"When writing email responses\" with two key preferences:\n",
" - \"Favor shorter and less formal language when possible, unless the context requires formality\"\n",
" - \"Include a closing statement expressing that you look forward to the meeting or conversation when confirming appointments\"\n",
"\n",
"2. It has also added a new bullet point to the \"When responding to meeting scheduling requests\" section:\n",
" - \"When scheduling meetings, prefer afternoon times after 2pm when possible, and default to 30-minute durations unless otherwise specified\"\n",
"\n",
"This demonstrates the system's ability to:\n",
"- Organize learned preferences into appropriate categories\n",
"- Extract multiple insights from a single feedback instance\n",
"- Apply meeting preferences to both calendar and email contexts\n",
"- Capture nuance with appropriate qualifiers (\"when possible,\" \"unless otherwise specified\")\n",
"- Maintain the hierarchical structure of the memory\n",
"\n",
"The resulting email shows all these preferences applied: it's shorter, less formal, includes a closing statement about looking forward to the chat, and correctly references the 30-minute meeting time."
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "8c64999e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the write_email tool call...\n",
"\n",
"Checking memory after accepting write_email after feedback:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"Use professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n- When scheduling meetings, prefer afternoon times after 2pm when possible, and default to 30-minute durations unless otherwise specified.\\n\\nWhen writing email responses:\\n- Favor shorter and less formal language when possible, unless the context requires formality.\\n- Include a closing statement expressing that you look forward to the meeting or conversation when confirming appointments.\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\nAfternoon meetings after 2pm are preferred.'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_5):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after accepting write_email after feedback\n",
"print(\"\\nChecking memory after accepting write_email after feedback:\")\n",
"display_memory_content(store, (\"email_assistant\", \"response_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "f85e63cb",
"metadata": {},
"source": [
"Look at the full message history."
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "fa9cf91d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Tax season let's schedule call\n",
"**From**: Project Manager <pm@client.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Lance,\n",
"\n",
"It's tax season again, and I wanted to schedule a call to discuss your tax planning strategies for this year. I have some suggestions that could potentially save you money.\n",
"\n",
"Are you available sometime next week? Tuesday or Thursday afternoon would work best for me, for about 45 minutes.\n",
"\n",
"Regards,\n",
"Project Manager\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_SwwMW5F9zEWmajYTDLjKOoFN)\n",
" Call ID: call_SwwMW5F9zEWmajYTDLjKOoFN\n",
" Args:\n",
" day: 2025-04-22\n",
" check_calendar_availability (call_nIBAphQD4FFOdW89Ikc1uNfV)\n",
" Call ID: call_nIBAphQD4FFOdW89Ikc1uNfV\n",
" Args:\n",
" day: 2025-04-24\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-22: 9:00 AM, 2:00 PM, 4:00 PM\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-04-24: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" schedule_meeting (call_uUvycUMHS86YjPSYFKV1dpzq)\n",
" Call ID: call_uUvycUMHS86YjPSYFKV1dpzq\n",
" Args:\n",
" attendees: ['pm@client.com', 'lance@company.com']\n",
" subject: Tax Planning Strategies Discussion\n",
" duration_minutes: 45\n",
" preferred_day: 2025-04-22\n",
" start_time: 14\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User gave feedback, which can we incorporate into the meeting request. Feedback: Please schedule this for 30 minutes instead of 45 minutes, and I prefer afternoon meetings after 2pm.\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" schedule_meeting (call_isFznpqEGfVb7pBmY2itGtgY)\n",
" Call ID: call_isFznpqEGfVb7pBmY2itGtgY\n",
" Args:\n",
" attendees: ['pm@client.com', 'lance@company.com']\n",
" subject: Tax Planning Strategies Discussion\n",
" duration_minutes: 30\n",
" preferred_day: 2025-04-22\n",
" start_time: 14\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Meeting 'Tax Planning Strategies Discussion' scheduled on Tuesday, April 22, 2025 at 14 for 30 minutes with 2 attendees\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_GIsMacl5jL7qsjSM47WdK738)\n",
" Call ID: call_GIsMacl5jL7qsjSM47WdK738\n",
" Args:\n",
" to: pm@client.com\n",
" subject: Re: Tax season let's schedule call\n",
" content: Hello,\n",
"\n",
"Thank you for reaching out regarding tax planning strategies. I am available on Tuesday, April 22nd at 2:00 PM, and have scheduled a 30-minute call for us to discuss your suggestions. If you need more time or would prefer a different slot, please let me know.\n",
"\n",
"Looking forward to our conversation.\n",
"\n",
"Best regards,\n",
"Lance\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User gave feedback, which can we incorporate into the email. Feedback: Shorter and less formal. Include a closing statement about looking forward to the meeting!\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_0vCEdmEMmvBYGh24gvd9pqvh)\n",
" Call ID: call_0vCEdmEMmvBYGh24gvd9pqvh\n",
" Args:\n",
" to: pm@client.com\n",
" subject: Re: Tax season let's schedule call\n",
" content: Hi,\n",
"\n",
"Thanks for reaching out. I’ve scheduled us for a 30-minute call on Tuesday, April 22nd at 2:00 PM to go over tax planning. Let me know if you need a different time.\n",
"\n",
"Looking forward to our chat!\n",
"\n",
"Best,\n",
"Lance\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Email sent to pm@client.com with subject 'Re: Tax season let's schedule call' and content: Hi,\n",
"\n",
"Thanks for reaching out. I’ve scheduled us for a 30-minute call on Tuesday, April 22nd at 2:00 PM to go over tax planning. Let me know if you need a different time.\n",
"\n",
"Looking forward to our chat!\n",
"\n",
"Best,\n",
"Lance\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_UXqUyEknR5jjpA39rKdlEcc9)\n",
" Call ID: call_UXqUyEknR5jjpA39rKdlEcc9\n",
" Args:\n",
" done: True\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_5)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "0bed8841",
"metadata": {},
"source": [
"Now let's try an email that calls the `Question` tool to provide feedback\n",
"\n",
"Our final test examines feedback for questions. When the assistant needs clarification before proceeding, users can provide detailed information beyond just answering the question:\n",
"\n",
"1. For the brunch invitation email, we'll provide feedback that includes:\n",
" - Confirmation that we want to invite the people mentioned\n",
" - A specific location preference (Jack's)\n",
" - A time preference (before 11am)\n",
" \n",
"2. This gives the system multiple pieces of information:\n",
" - A direct answer to the question (yes, let's invite them)\n",
" - Additional context and preferences not explicitly asked for\n",
"\n",
"This tests the system's ability to process compound feedback and extract multiple data points from a single response."
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "e111a459",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"📧 Classification: RESPOND - This email requires a response\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'Question', 'args': {'content': 'Do you want me to check your calendar and suggest specific dates and times for the brunch, or would you like to propose a particular day and time to Jim and Lisa first?'}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"# Respond - Meeting Request Email\n",
"email_input_respond = {\n",
" \"to\": \"Lance Martin <lance@company.com>\",\n",
" \"author\": \"Partner <partner@home.com>\",\n",
" \"subject\": \"Meet Jim and Lisa for brunch in 3 weeks?\",\n",
" \"email_thread\": \"Hey, should we invite Jim and Lisa to brunch in 3 weeks? We could go to the new place on 17th that everyone is talking about.\"\n",
"}\n",
"\n",
"# Compile the graph\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_6 = uuid.uuid4()\n",
"thread_config_6 = {\"configurable\": {\"thread_id\": thread_id_6}}\n",
"\n",
"# Run the graph until the first interrupt\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_respond}, config=thread_config_6):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt for Question tool\n",
"display_memory_content(store)"
]
},
{
"cell_type": "markdown",
"id": "c540ebff",
"metadata": {},
"source": [
"Provide feedback for the `Question` tool call\n",
"\n",
"When the assistant asks about our preferences for the brunch invitation, we'll respond with rich, multi-faceted feedback:\n",
"\n",
"1. We confirm we want to invite Jim and Lisa\n",
"2. We specify a location preference (Jack's, not the place on 17th)\n",
"3. We express a time preference (before 11am)\n",
"\n",
"This tests the system's ability to handle compound responses that both answer the direct question and provide additional context. Note that we're not just answering \"yes\" or \"no\" - we're providing a rich context that should influence the assistant's next actions.\n",
"\n",
"An ideal system would use this feedback to both respond to the immediate email and update background knowledge that could be relevant for future similar social invitations. In our current implementation, we don't update background knowledge from question responses, but this would be a straightforward enhancement."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "61d8bfef",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user providing feedback for the Question tool call...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'partner@home.com', 'subject': 'Brunch with Jim and Lisa in 3 Weeks', 'content': 'Great idea! I’ll reach out to Jim and Lisa to invite them to brunch at Jack’s in three weeks. I’ll suggest Saturday, May 8th, with a start time before 11am—9:00 AM is available on our calendar. Let me know if you’d like to suggest a different time or if this works for you before I send the invite.'}}\n",
"\n",
"Checking memory after providing feedback for Question:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user providing feedback for the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"response\", \"args\": \"Yes, let's invite them, I really like brunch at Jack's, ideally before 11am.\"}]), config=thread_config_6):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after providing feedback for Question\n",
"print(\"\\nChecking memory after providing feedback for Question:\")\n",
"display_memory_content(store)"
]
},
{
"cell_type": "markdown",
"id": "7b9dbba1",
"metadata": {},
"source": [
"Currently, we don't update `background_information` when the user provides feedback for the `Question` tool, but this would be a valuable enhancement.\n",
"\n",
"Looking at how the system handled our question response:\n",
"\n",
"1. It correctly incorporated all three key pieces of information:\n",
" - Our affirmative decision to invite Jim and Lisa\n",
" - Our location preference (Jack's, not the place on 17th)\n",
" - Our time preference (before 11am)\n",
"\n",
"2. It drafted a complete email that:\n",
" - References reaching out to Jim and Lisa\n",
" - Specifies Jack's as the location\n",
" - Suggests a 9:00 AM time (before 11am as requested)\n",
" - Asks for confirmation before sending an invite\n",
"\n",
"This demonstrates the system's ability to extract and use detailed information from natural language feedback, even when not explicitly updating memory. The email correctly incorporates all aspects of our feedback and presents a coherent plan based on our preferences.\n",
"\n",
"A future enhancement could store these preferences (location preferences, time preferences for social events) in the background information memory for use in future similar situations."
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "5b4c0f6a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the write_email tool call...\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_k6NQ6GqqEIE1uNwVQXmx6xU6)\n",
" Call ID: call_k6NQ6GqqEIE1uNwVQXmx6xU6\n",
" Args:\n",
" done: True\n",
"\n",
"Checking memory after accepting write_email after answering Question:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_6):\n",
" # Inspect response_agent most recent message\n",
" if 'response_agent' in chunk:\n",
" chunk['response_agent']['messages'][-1].pretty_print()\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after accepting write_email after answering Question\n",
"print(\"\\nChecking memory after accepting write_email after answering Question:\")\n",
"display_memory_content(store)"
]
},
{
"cell_type": "markdown",
"id": "c80ed860",
"metadata": {},
"source": [
"Look at the full message history"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "e61fab2f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Respond to the email: \n",
"\n",
"**Subject**: Meet Jim and Lisa for brunch in 3 weeks?\n",
"**From**: Partner <partner@home.com>\n",
"**To**: Lance Martin <lance@company.com>\n",
"\n",
"Hey, should we invite Jim and Lisa to brunch in 3 weeks? We could go to the new place on 17th that everyone is talking about.\n",
"\n",
"---\n",
"\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Question (call_dF0WQiJIDf5ovo5veMixbBm9)\n",
" Call ID: call_dF0WQiJIDf5ovo5veMixbBm9\n",
" Args:\n",
" content: Do you want me to check your calendar and suggest specific dates and times for the brunch, or would you like to propose a particular day and time to Jim and Lisa first?\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"User answered the question, which can we can use for any follow up actions. Feedback: Yes, let's invite them, I really like brunch at Jack's, ideally before 11am.\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" check_calendar_availability (call_07aoxyrzmikMZPDN9XQ9rLbp)\n",
" Call ID: call_07aoxyrzmikMZPDN9XQ9rLbp\n",
" Args:\n",
" day: 2025-05-08\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Available times on 2025-05-08: 9:00 AM, 2:00 PM, 4:00 PM\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_Vq5Ube1Y2Wbr9xzB7bh1aIqw)\n",
" Call ID: call_Vq5Ube1Y2Wbr9xzB7bh1aIqw\n",
" Args:\n",
" to: partner@home.com\n",
" subject: Brunch with Jim and Lisa in 3 Weeks\n",
" content: Great idea! I’ll reach out to Jim and Lisa to invite them to brunch at Jack’s in three weeks. I’ll suggest Saturday, May 8th, with a start time before 11am—9:00 AM is available on our calendar. Let me know if you’d like to suggest a different time or if this works for you before I send the invite.\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Email sent to partner@home.com with subject 'Brunch with Jim and Lisa in 3 Weeks' and content: Great idea! I’ll reach out to Jim and Lisa to invite them to brunch at Jack’s in three weeks. I’ll suggest Saturday, May 8th, with a start time before 11am—9:00 AM is available on our calendar. Let me know if you’d like to suggest a different time or if this works for you before I send the invite.\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_k6NQ6GqqEIE1uNwVQXmx6xU6)\n",
" Call ID: call_k6NQ6GqqEIE1uNwVQXmx6xU6\n",
" Args:\n",
" done: True\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_6)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "a3b11d0c",
"metadata": {},
"source": [
"## Test Case for Notify Classification\n",
"\n",
"This test explores how memory updates when an email is initially classified as \"NOTIFY\" but the user decides it needs a response:\n",
"\n",
"1. The triage system initially classifies IT security updates as notifications\n",
"2. But the user decides this particular notification warrants acknowledgment\n",
"3. This creates a learning opportunity about which notification types need responses\n",
"\n",
"The \"notify\" category is designed for important information that doesn't need immediate action. But user feedback can help the system learn which subset of notifications actually do warrant responses, refining the initial classification over time."
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "a6e8a62a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"🔔 Classification: NOTIFY - This email contains important information\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'Email Assistant: notify', 'args': {}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"# Notify - Important FYI Email\n",
"email_input_notify = {\n",
" \"to\": \"Team Members <team@company.com>\",\n",
" \"author\": \"IT Department <it@company.com>\",\n",
" \"subject\": \"Critical Security Update\",\n",
" \"email_thread\": \"Dear Team,\\n\\nThis is an important security notification. We will be updating our authentication system this weekend. During the update window (Saturday 2am-4am), you will not be able to access company resources.\\n\\nPlease ensure you log out of all systems before the maintenance window.\\n\\nRegards,\\nIT Department\"\n",
"}\n",
"\n",
"# Compile the graph with new thread\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_7 = uuid.uuid4()\n",
"thread_config_7 = {\"configurable\": {\"thread_id\": thread_id_7}}\n",
"\n",
"# Run the graph until the first interrupt - should be classified as \"notify\"\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_notify}, config=thread_config_7):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt for Notify\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "1b02f3be",
"metadata": {},
"source": [
"Now simulate user deciding to respond with feedback.\n",
"\n",
"This test explores the reclassification pathway from \"notify\" to \"respond\":\n",
"\n",
"1. The system initially classifies a security update as information-only (\"notify\")\n",
"2. We're presented with this notification without a suggested action\n",
"3. We decide this security update actually requires acknowledgment\n",
"4. We provide feedback indicating we want to respond and confirm our compliance\n",
"\n",
"This represents an important learning signal - that security notifications, particularly those requesting specific user actions, should be treated as items requiring response rather than just information.\n",
"\n",
"After providing this feedback, we'll check the triage preferences memory to see how this \"override\" affects future classifications. We expect to see security notifications with action requests moved into the \"worth responding to\" section."
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "510235cd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user deciding to respond with feedback...\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'write_email', 'args': {'to': 'it@company.com', 'subject': 'Re: Critical Security Update', 'content': 'Dear IT Department,\\n\\nThank you for the notification regarding the upcoming authentication system update. I acknowledge receipt of this important notice and will ensure that I am logged out of all systems before the maintenance window begins on Saturday from 2am to 4am.\\n\\nBest regards,\\nLance'}}\n",
"\n",
"Checking memory after responding with feedback to Notify:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"Emails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n- Important security notifications from IT Department requiring acknowledgment or confirmation of action\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\"}\n"
]
}
],
"source": [
"print(\"\\nSimulating user deciding to respond with feedback...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"response\", \"args\": \"We should acknowledge receipt of this important notice and confirm that we'll be logged out before the maintenance window.\"}]), config=thread_config_7):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after responding with feedback to Notify\n",
"print(\"\\nChecking memory after responding with feedback to Notify:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "0c4b1139",
"metadata": {},
"source": [
"The memory update after our decision to respond to a security notification shows a remarkable triage preference refinement:\n",
"\n",
"1. The system has added a new bullet to the \"Emails that are worth responding to\" section:\n",
" ```\n",
" \"Important security notifications from IT Department requiring acknowledgment or confirmation of action\"\n",
" ```\n",
"\n",
"2. This update demonstrates:\n",
" - Precision: It specifically identifies \"security notifications\" (not all IT emails)\n",
" - Source awareness: It's from the \"IT Department\" specifically\n",
" - Action trigger: It identifies notifications \"requiring acknowledgment or confirmation\"\n",
" - Category reorganization: It moves this from \"notify\" to \"respond\" without removing notifications entirely\n",
"\n",
"The system didn't just record our specific override - it analyzed the *reason* we might want to respond (acknowledging required action) and created a generalizable rule. This would ensure that future similar security notifications requesting specific user actions would be correctly classified as needing response."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "85fa053f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user accepting the write_email tool call...\n",
"\n",
"Checking memory after accepting write_email for Notify:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"Emails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n- Important security notifications from IT Department requiring acknowledgment or confirmation of action\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\"}\n",
"=======================================\n",
"\n",
"\n",
"--- response_preferences ---\n",
"{'preferences': \"\\nUse professional and concise language. If the e-mail mentions a deadline, make sure to explicitly acknowledge and reference the deadline in your response.\\n\\nWhen responding to technical questions that require investigation:\\n- Clearly state whether you will investigate or who you will ask\\n- Provide an estimated timeline for when you'll have more information or complete the task\\n\\nWhen responding to event or conference invitations:\\n- Always acknowledge any mentioned deadlines (particularly registration deadlines)\\n- If workshops or specific topics are mentioned, ask for more specific details about them\\n- If discounts (group or early bird) are mentioned, explicitly request information about them\\n- Don't commit \\n\\nWhen responding to collaboration or project-related requests:\\n- Acknowledge any existing work or materials mentioned (drafts, slides, documents, etc.)\\n- Explicitly mention reviewing these materials before or during the meeting\\n- When scheduling meetings, clearly state the specific day, date, and time proposed\\n\\nWhen responding to meeting scheduling requests:\\n- If times are proposed, verify calendar availability for all time slots mentioned in the original email and then commit to one of the proposed times based on your availability by scheduling the meeting. Or, say you can't make it at the time proposed.\\n- If no times are proposed, then check your calendar for availability and propose multiple time options when available instead of selecting just one.\\n- Mention the meeting duration in your response to confirm you've noted it correctly.\\n- Reference the meeting's purpose in your response.\\n\"}\n",
"=======================================\n",
"\n",
"\n",
"--- cal_preferences ---\n",
"{'preferences': '\\n30 minute meetings are preferred, but 15 minute meetings are also acceptable.\\n'}\n",
"=======================================\n",
"\n",
"\n",
"--- background ---\n",
"No memory found\n",
"=======================================\n",
"\n"
]
}
],
"source": [
"print(f\"\\nSimulating user accepting the {Interrupt_Object.value[0]['action_request']['action']} tool call...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"accept\"}]), config=thread_config_7):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after accepting write_email for Notify\n",
"print(\"\\nChecking memory after accepting write_email for Notify:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "4878373b",
"metadata": {},
"source": [
"Look at the full message history."
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "0501ff69",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Email to notify user about: \n",
"\n",
"**Subject**: Critical Security Update\n",
"**From**: IT Department <it@company.com>\n",
"**To**: Team Members <team@company.com>\n",
"\n",
"Dear Team,\n",
"\n",
"This is an important security notification. We will be updating our authentication system this weekend. During the update window (Saturday 2am-4am), you will not be able to access company resources.\n",
"\n",
"Please ensure you log out of all systems before the maintenance window.\n",
"\n",
"Regards,\n",
"IT Department\n",
"\n",
"---\n",
"\n",
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"User wants to reply to the email. Use this feedback to respond: We should acknowledge receipt of this important notice and confirm that we'll be logged out before the maintenance window.\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" write_email (call_8tZq4eSFWr2WE01sPgEfgFr0)\n",
" Call ID: call_8tZq4eSFWr2WE01sPgEfgFr0\n",
" Args:\n",
" to: it@company.com\n",
" subject: Re: Critical Security Update\n",
" content: Dear IT Department,\n",
"\n",
"Thank you for the notification regarding the upcoming authentication system update. I acknowledge receipt of this important notice and will ensure that I am logged out of all systems before the maintenance window begins on Saturday from 2am to 4am.\n",
"\n",
"Best regards,\n",
"Lance\n",
"=================================\u001b[1m Tool Message \u001b[0m=================================\n",
"\n",
"Email sent to it@company.com with subject 'Re: Critical Security Update' and content: Dear IT Department,\n",
"\n",
"Thank you for the notification regarding the upcoming authentication system update. I acknowledge receipt of this important notice and will ensure that I am logged out of all systems before the maintenance window begins on Saturday from 2am to 4am.\n",
"\n",
"Best regards,\n",
"Lance\n",
"==================================\u001b[1m Ai Message \u001b[0m==================================\n",
"Tool Calls:\n",
" Done (call_or3Ir137SK0RmLgs2qFrMcWB)\n",
" Call ID: call_or3Ir137SK0RmLgs2qFrMcWB\n",
" Args:\n",
" done: True\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_7)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "c8e48281",
"metadata": {},
"source": [
"## Test Case for Notify + Ignore\n",
"\n",
"Our final test explores the opposite pattern - when a user decides notifications don't even warrant being shown:\n",
"\n",
"1. The system classifies a company picnic announcement as a notification\n",
"2. The user decides this doesn't even warrant notification status\n",
"3. This creates a signal to further refine the classification boundaries\n",
"\n",
"By ignoring certain types of notifications, users can teach the system which information is truly important to them versus which can be filtered out entirely. This completes the full spectrum of classification refinement."
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "74e1fe6f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running the graph until the first interrupt...\n",
"🔔 Classification: NOTIFY - This email contains important information\n",
"\n",
"INTERRUPT OBJECT:\n",
"Action Request: {'action': 'Email Assistant: notify', 'args': {}}\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"# Notify - Important FYI Email\n",
"email_input_notify = {\n",
" \"to\": \"Team Members <team@company.com>\",\n",
" \"author\": \"HR Department <hr@company.com>\",\n",
" \"subject\": \"Company Picnic Next Month\",\n",
" \"email_thread\": \"Dear Team,\\n\\nWe're planning the annual company picnic for next month. The tentative date is Saturday, June 15th from noon to 4pm at Central Park. There will be food, games, and activities for families.\\n\\nMore details will follow in the coming weeks.\\n\\nRegards,\\nHR Department\"\n",
"}\n",
"\n",
"# Compile the graph with new thread\n",
"checkpointer = MemorySaver()\n",
"store = InMemoryStore()\n",
"graph = overall_workflow.compile(checkpointer=checkpointer, store=store)\n",
"thread_id_8 = uuid.uuid4()\n",
"thread_config_8 = {\"configurable\": {\"thread_id\": thread_id_8}}\n",
"\n",
"# Run the graph until the first interrupt - should be classified as \"notify\"\n",
"print(\"Running the graph until the first interrupt...\")\n",
"for chunk in graph.stream({\"email_input\": email_input_notify}, config=thread_config_8):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after first interrupt for Notify + Ignore\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "911a8c79",
"metadata": {},
"source": [
"Now simulate user deciding to ignore the notification.\n",
"\n",
"In this final test, we explore the downgrade pathway from \"notify\" to \"ignore\":\n",
"\n",
"1. The system initially classifies a company picnic announcement as \"notify\"\n",
"2. We decide we don't even want to be notified about these social events\n",
"3. By choosing \"ignore,\" we signal this entire category should be filtered out\n",
"\n",
"This represents another important learning signal - that certain types of company announcements (particularly social events) shouldn't even be surfaced as notifications, further refining our triage preferences.\n",
"\n",
"After ignoring, we'll check the triage preferences memory for the final time to see how this override affects future classifications. We expect to see company social events moved from the \"notify\" section to the \"not worth responding to\" section."
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "088e7bc7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Simulating user deciding to ignore the notification...\n",
"\n",
"Checking memory after ignoring Notify:\n",
"\n",
"======= CURRENT MEMORY CONTENT =======\n",
"\n",
"--- triage_preferences ---\n",
"{'preferences': \"\\nEmails that are not worth responding to:\\n- Marketing newsletters and promotional emails\\n- Spam or suspicious emails\\n- CC'd on FYI threads with no direct questions\\n- Company social event announcements (e.g., company picnic)\\n\\nThere are also other things that should be known about, but don't require an email response. For these, you should notify (using the `notify` response). Examples of this include:\\n- Team member out sick or on vacation\\n- Build system notifications or deployments\\n- Project status updates without action items\\n- Important company announcements\\n- FYI emails that contain relevant information for current projects\\n- HR Department deadline reminders\\n- Subscription status / renewal reminders\\n- GitHub notifications\\n\\nEmails that are worth responding to:\\n- Direct questions from team members requiring expertise\\n- Meeting requests requiring confirmation\\n- Critical bug reports related to team's projects\\n- Requests from management requiring acknowledgment\\n- Client inquiries about project status or features\\n- Technical questions about documentation, code, or APIs (especially questions about missing endpoints or features)\\n- Personal reminders related to family (wife / daughter)\\n- Personal reminder related to self-care (doctor appointments, etc)\\n\"}\n"
]
}
],
"source": [
"print(\"\\nSimulating user deciding to ignore the notification...\")\n",
"for chunk in graph.stream(Command(resume=[{\"type\": \"ignore\"}]), config=thread_config_8):\n",
" # Inspect interrupt object if present\n",
" if '__interrupt__' in chunk:\n",
" Interrupt_Object = chunk['__interrupt__'][0]\n",
" print(\"\\nINTERRUPT OBJECT:\")\n",
" print(f\"Action Request: {Interrupt_Object.value[0]['action_request']}\")\n",
"\n",
"# Check memory after ignoring Notify\n",
"print(\"\\nChecking memory after ignoring Notify:\")\n",
"display_memory_content(store, (\"email_assistant\", \"triage_preferences\"))"
]
},
{
"cell_type": "markdown",
"id": "05136d96",
"metadata": {},
"source": [
"The final memory update completes our triage preference refinement journey:\n",
"\n",
"1. The system has added \"Company social event announcements (e.g., company picnic)\" to the \"Emails not worth responding to\" section\n",
"\n",
"2. This demonstrates:\n",
" - Content-based categorization: It identifies \"social event announcements\" specifically\n",
" - Example inclusion: It provides an example \"(company picnic)\" for clarity\n",
" - Category downgrade: It moves this from \"notify\" to \"not worth responding to\"\n",
" - Structural preservation: It maintains the original memory organization\n",
"\n",
"Through our series of tests, we've now explored the full spectrum of triage learning:\n",
"- Upgrading notifications to responses (security updates)\n",
"- Downgrading notifications to ignores (company social events)\n",
"- Downgrading responses to ignores (personal social invitations, routine calls)\n",
"- Refining response preferences (meeting duration, timing, email style)\n",
"\n",
"Each interaction has produced targeted, intelligent memory updates that maintain the overall structure while adding new insights. This continuous refinement creates an increasingly personalized assistant that learns and adapts to user preferences over time."
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "027f3f4e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"Email to notify user about: \n",
"\n",
"**Subject**: Company Picnic Next Month\n",
"**From**: HR Department <hr@company.com>\n",
"**To**: Team Members <team@company.com>\n",
"\n",
"Dear Team,\n",
"\n",
"We're planning the annual company picnic for next month. The tentative date is Saturday, June 15th from noon to 4pm at Central Park. There will be food, games, and activities for families.\n",
"\n",
"More details will follow in the coming weeks.\n",
"\n",
"Regards,\n",
"HR Department\n",
"\n",
"---\n",
"\n",
"================================\u001b[1m Human Message \u001b[0m=================================\n",
"\n",
"The user decided to ignore the email even though it was classified as notify. Update triage preferences to capture this.\n"
]
}
],
"source": [
"state = graph.get_state(thread_config_8)\n",
"for m in state.values['messages']:\n",
" m.pretty_print()"
]
},
{
"cell_type": "markdown",
"id": "b075a3ea",
"metadata": {},
"source": [
"## Testing with Local Deployment\n",
"\n",
"You can find this graph with memory integration in the `src/email_assistant` directory:\n",
"\n",
"* `src/email_assistant/email_assistant_hitl_memory.py`\n",
"\n",
"Testing this locally gives you the full experience of a memory-enabled HITL system:\n",
"\n",
"1. **Start the local server**: Run `langgraph dev` to launch the agent locally\n",
"2. **Connect Agent Inbox**: Use the graph URL from the `langgraph.json` file\n",
"3. **Submit test emails**: Try different email types to see classification in action\n",
"4. **Provide various feedback types**: Try accepting, editing, ignoring, and responding\n",
"5. **Observe memory evolution**: Check the Memory tab in LangGraph Studio to see changes\n",
"\n",
"![inbox](img/agent-inbox-edit.png)\n",
"\n",
"The Memory tab in LangGraph Studio offers a real-time view of how your preferences are being captured and updated with each interaction:\n",
"\n",
"![studio-img](img/memory-studio.png)\n",
"\n",
"Through continued use, the system becomes increasingly personalized:\n",
"- It learns which emails you want to respond to, be notified about, or ignore\n",
"- It adapts to your communication style preferences\n",
"- It remembers your scheduling preferences\n",
"- It refines its understanding with each interaction\n",
"\n",
"This combination of HITL and memory creates a system that balances automation with control - handling routine tasks automatically while learning from your feedback to become more aligned with your preferences over time."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f60fa538",
"metadata": {},
"outputs": [],
"source": [
"! langgraph dev"
]
},
{
"cell_type": "markdown",
"id": "43b6319d",
"metadata": {},
"source": [
"![inbox](img/agent-inbox-edit.png)\n",
"\n",
"As you provide feedback or edit replies, you can see memories accumulate in the `memory` tab in LangGraph Studio.\n",
"\n",
"![studio-img](img/memory-studio.png)\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "f2ad7580",
"metadata": {},
"source": [
"### Use LangMem to add background memory! \n",
"\n",
"Add this to the `llm_call` node:\n",
"\n",
"```\n",
" # Search for existing background memory\n",
" # TODO: Here, semantic search over a facts collection of background information from emails could be added. \n",
" # background = get_memory(store, (\"email_assistant\", \"background\"), default_background)\n",
"```\n",
"\n",
"Add this to the interrupt handler node:\n",
"\n",
"```\n",
"elif tool_call[\"name\"] == \"Question\":\n",
" # Don't execute the tool, and add a message with the user feedback to incorporate into the email\n",
" result.append({\"role\": \"tool\", \"content\": f\"User answered the question, which can we can use for any follow up actions. Feedback: {user_feedback}\", \"tool_call_id\": tool_call[\"id\"]})\n",
" # TODO: Here, we could update the background information with the user's answer. \n",
" # update_memory(store, (\"email_assistant\", \"background\"), [{\n",
" # \"role\": \"user\",\n",
" # \"content\": f\"Update background information based upon these messages:\"\n",
" # }] + state[\"messages\"] + result)\n",
"```\n",
"\n",
"Consider using LangMem: \n",
"https://langchain-ai.github.io/langmem/"
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "-all",
"main_language": "python",
"notebook_metadata_filter": "-all"
},
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink