run-llama/create-llama
1
0
Fork 0
mirror of https://github.com/run-llama/create-llama.git synced 2026-07-15 05:08:15 -04:00
Code Issues 39 Packages Projects Releases 10 Wiki Activity

Results are not streaming in. #124

New Issue
Closed
opened 2026-02-15 19:16:07 -05:00 by yindo · 3 comments
yindo commented 2026-02-15 19:16:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @izakfilmalter on GitHub (Feb 26, 2025).

Originally assigned to: @thucpn on GitHub.

Looks like StreamData is deprecated in the ai sdk now. It's causing the results to no longer stream in, but appear once fully generated.

Originally created by @izakfilmalter on GitHub (Feb 26, 2025). Originally assigned to: @thucpn on GitHub. Looks like `StreamData` is deprecated in the `ai` sdk now. It's causing the results to no longer stream in, but appear once fully generated.
yindo closed this issue 2026-02-15 19:16:07 -05:00
yindo commented 2026-02-15 19:16:08 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Feb 27, 2025):

@izakfilmalter Thanks for reporting, we're working on updating our streaming code

@marcusschiesser commented on GitHub (Feb 27, 2025): @izakfilmalter Thanks for reporting, we're working on updating our streaming code
yindo commented 2026-02-15 19:16:08 -05:00
Author
Owner
Copy Link
Copy Source

@marcusschiesser commented on GitHub (Mar 2, 2025):

@izakfilmalter this is fixed in LlamaIndex 0.9.5 which is automatically pulled by the current create-llama code

@marcusschiesser commented on GitHub (Mar 2, 2025): @izakfilmalter this is fixed in LlamaIndex 0.9.5 which is automatically pulled by the current create-llama code
yindo commented 2026-02-15 19:16:08 -05:00
Author
Owner
Copy Link
Copy Source

@Gllidan commented on GitHub (Mar 20, 2025):

@marcusschiesser
Hello, I have upgraded to "ai": "4.0.3", "llamaindex": "0.9.11", but I still cannot support streaming output. Do I need to make adjustments to my code? Please help me with this.

Image

Image

`
import { initObservability } from "@/app/observability";
import { LlamaIndexAdapter, Message, StreamData } from "ai";
import { ChatMessage, Settings } from "llamaindex";
import { NextRequest, NextResponse } from "next/server";
import { createChatEngine } from "./engine/chat";
import { initSettings } from "./engine/settings";
import {
isValidMessages,
retrieveDocumentIds,
retrieveMessageContent,
} from "./llamaindex/streaming/annotations";
import { createCallbackManager } from "./llamaindex/streaming/events";
import { generateNextQuestions } from "./llamaindex/streaming/suggestion";

initObservability();
initSettings();

export const runtime = "nodejs";
export const dynamic = "force-dynamic";

export async function POST(request: NextRequest) {
// Init Vercel AI StreamData and timeout
const vercelStreamData = new StreamData();

try {
const body = await request.json();
const { messages, data }: { messages: Message[]; data?: any } = body;
if (!isValidMessages(messages)) {
return NextResponse.json(
{
error:
"messages are required in the request body and the last message must be from the user",
},
{ status: 400 },
);
}

// retrieve document ids from the annotations of all messages (if any)
const ids = retrieveDocumentIds(messages);
// create chat engine with index using the document ids
const chatEngine = await createChatEngine(ids, data);

// retrieve user message content from Vercel/AI format
const userMessageContent = retrieveMessageContent(messages);

// Setup callbacks
const callbackManager = createCallbackManager(vercelStreamData);
const chatHistory: ChatMessage[] = messages.slice(0, -1) as ChatMessage[];

// Calling LlamaIndex's ChatEngine to get a streamed response
const response = await Settings.withCallbackManager(callbackManager, () => {
  return chatEngine.chat({
    message: userMessageContent,
    chatHistory,
    stream: true,
  });
});

const onCompletion = (content: string) => {
  chatHistory.push({ role: "assistant", content: content });
  generateNextQuestions(chatHistory)
    .then((questions: string[]) => {
      if (questions.length > 0) {
        vercelStreamData.appendMessageAnnotation({
          type: "suggested_questions",
          data: questions,
        });
      }
    })
    .finally(() => {
      vercelStreamData.close();
    });
};

return LlamaIndexAdapter.toDataStreamResponse(response, {
  data: vercelStreamData,
  callbacks: { onCompletion },
});

} catch (error) {
console.error("[LlamaIndex]", error);
return NextResponse.json(
{
detail: (error as Error).message,
},
{
status: 500,
},
);
}
}
`

@Gllidan commented on GitHub (Mar 20, 2025): @marcusschiesser Hello, I have upgraded to "ai": "4.0.3", "llamaindex": "0.9.11", but I still cannot support streaming output. Do I need to make adjustments to my code? Please help me with this. ![Image](https://github.com/user-attachments/assets/7ecae03e-f28a-4cdf-a846-c8f24abfcc87) ![Image](https://github.com/user-attachments/assets/cdf689d1-dca6-4d25-b711-0cf23fd54318) ` import { initObservability } from "@/app/observability"; import { LlamaIndexAdapter, Message, StreamData } from "ai"; import { ChatMessage, Settings } from "llamaindex"; import { NextRequest, NextResponse } from "next/server"; import { createChatEngine } from "./engine/chat"; import { initSettings } from "./engine/settings"; import { isValidMessages, retrieveDocumentIds, retrieveMessageContent, } from "./llamaindex/streaming/annotations"; import { createCallbackManager } from "./llamaindex/streaming/events"; import { generateNextQuestions } from "./llamaindex/streaming/suggestion"; initObservability(); initSettings(); export const runtime = "nodejs"; export const dynamic = "force-dynamic"; export async function POST(request: NextRequest) { // Init Vercel AI StreamData and timeout const vercelStreamData = new StreamData(); try { const body = await request.json(); const { messages, data }: { messages: Message[]; data?: any } = body; if (!isValidMessages(messages)) { return NextResponse.json( { error: "messages are required in the request body and the last message must be from the user", }, { status: 400 }, ); } // retrieve document ids from the annotations of all messages (if any) const ids = retrieveDocumentIds(messages); // create chat engine with index using the document ids const chatEngine = await createChatEngine(ids, data); // retrieve user message content from Vercel/AI format const userMessageContent = retrieveMessageContent(messages); // Setup callbacks const callbackManager = createCallbackManager(vercelStreamData); const chatHistory: ChatMessage[] = messages.slice(0, -1) as ChatMessage[]; // Calling LlamaIndex's ChatEngine to get a streamed response const response = await Settings.withCallbackManager(callbackManager, () => { return chatEngine.chat({ message: userMessageContent, chatHistory, stream: true, }); }); const onCompletion = (content: string) => { chatHistory.push({ role: "assistant", content: content }); generateNextQuestions(chatHistory) .then((questions: string[]) => { if (questions.length > 0) { vercelStreamData.appendMessageAnnotation({ type: "suggested_questions", data: questions, }); } }) .finally(() => { vercelStreamData.close(); }); }; return LlamaIndexAdapter.toDataStreamResponse(response, { data: vercelStreamData, callbacks: { onCompletion }, }); } catch (error) { console.error("[LlamaIndex]", error); return NextResponse.json( { detail: (error as Error).message, }, { status: 500, }, ); } } `
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/create-llama#124
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?