Skip to content

Commit e63e413

Browse files
GeeTransitGeeTransit
authored
Merge pull request #4602 from OpenLiberty/gee-patch-1
LangChain4j streaming chat blog
2 parents 1cea3cf + 9af93fd commit e63e413

2 files changed

Lines changed: 212 additions & 0 deletions

File tree

‎img/blog/langchain4jStreamingDiagram.png‎

111 KB
Loading

‎posts/2025-11-13-langchain4j-jakartaee-streaming.adoc‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
---
2+
layout: post
3+
title: "LangChain4j streaming chat works with Jakarta WebSocket"
4+
# Do NOT change the categories section
5+
categories: blog
6+
author_picture: https://avatars3.githubusercontent.com/GeeTransit
7+
author_github: https://github.com/GeeTransit
8+
seo-title: LangChain4j streaming chat works with Jakarta WebSocket - OpenLiberty.io
9+
seo-description: DESCRIPTION
10+
blog_description: "DESCRIPTION"
11+
open-graph-image: https://openliberty.io/img/twitter_card.jpg
12+
open-graph-image-alt: Open Liberty Logo
13+
additional_authors:
14+
- name: Gilbert Kwan
15+
github: https://github.com/gkwan-ibm
16+
image: https://avatars3.githubusercontent.com/gkwan-ibm
17+
---
18+
= LangChain4j streaming chat works with Jakarta WebSocket
19+
George Zhang <https://github.com/GeeTransit>
20+
:imagesdir: /
21+
:url-prefix:
22+
:url-about: /
23+
24+
The introduction of large language models (LLMs) opens the possibility for users to interact in natural language with your application. However, as your application becomes more complex, your users might need to wait longer for large responses from the LLM. You can improve the user experience by streaming the response to the user as the LLM generates it, in doing so reducing delays and increasing perceived responsiveness.
25+
26+
In this post, you can explore how to stream responses from your LLM to your application client UI with Jakarta WebSocket. To learn more about how Jakarta WebSocket works, check out the https://openliberty.io/guides/jakarta-websocket.html[Bidirectional communication between services using Jakarta WebSocket^] guide.
27+
28+
== Try out the streaming chat sample application
29+
30+
You can check out a sample application that integrates LangChain4j's streaming chat with a Jakarta EE application. Clone the https://github.com/OpenLiberty/sample-langchain4j.git[sample-langchain4j] repository and go into the streaming-chat project by using the following commands:
31+
32+
[source,console]
33+
----
34+
$ git clone https://github.com/OpenLiberty/sample-langchain4j.git
35+
$ cd sample-langchain4j/streaming-chat
36+
----
37+
38+
Install Java 21 or newer if you haven’t already. You can use https://developer.ibm.com/languages/java/semeru-runtimes/downloads[IBM Semeru Runtimes] as your chosen Java runtime. Set the `JAVA_HOME` environment variable to your Java installation path.
39+
40+
The application supports the GitHub, Ollama, and Mistral AI providers. Set up an AI provider by following the https://github.com/OpenLiberty/sample-langchain4j/blob/main/streaming-chat/README.md[Prerequisites and Environment Set Up instructions].
41+
42+
Start the sample application by running `./mvnw liberty:dev` if you are on MacOS or Linux, and `mvnw.cmd liberty:dev` if you are on Windows. When the server starts, go to the http://localhost:9080 URL, send a question (for example "What are large language models?"), and check out how the response is streamed token by token.
43+
44+
== How does the application work?
45+
46+
From a high level, the user connects from their front end to a WebSocket endpoint. Each user message is passed to the AI service, which sends a request to the LLM provider. As the response is generated, the WebSocket endpoint sends each token to the user, rather than waiting until the entire response is generated.
47+
48+
image::/img/blog/langchain4jStreamingDiagram.png[Architectural diagram,width=70%,height=70%,align="center"]
49+
50+
The application builds a streaming chat model from a provider of your choice and sets a 200 token limit on the response. For example, the following code builds a streaming chat model that uses Ollama's Llama3.2 model.
51+
52+
[source,java]
53+
----
54+
StreamingChatModel streamingModel = OllamaStreamingChatModel.builder()
55+
.baseUrl("http://localhost:11434")
56+
.modelName("llama3.2")
57+
.numPredict(200)
58+
.build();
59+
----
60+
61+
See the https://github.com/OpenLiberty/sample-langchain4j/blob/7112a4f14bc7b63bd97f8f123f6ba3c16fdebba9/streaming-chat/src/main/java/io/openliberty/sample/langchain4j/util/ModelBuilder.java#L93-L99[ModelBuilder.java file] to learn more.
62+
63+
An assistant interface with a chat method is declared for LangChain4j to implement. To support multiple user conversations at the same time, the ChatMemoryAccess interface is extended to provide a method that deletes conversation history such as when a user exits. You can also specify a system message to influence the assistant's responses, such as being knowledgeable about the Open Liberty application server.
64+
65+
[source,java]
66+
----
67+
interface StreamingAssistant extends ChatMemoryAccess {
68+
@SystemMessage("You are a helpful chat bot knowledgeable about the Open Liberty application server runtime.")
69+
TokenStream streamingChat(@MemoryId String sessionId, @UserMessage String userMessage);
70+
}
71+
----
72+
73+
LangChain4j builds an AI service that implements the interface and uses the streaming chat model from earlier, with the 10 most recent messages of each conversation preserved.
74+
75+
[source,java]
76+
----
77+
StreamingAssistant assistant = AiServices.builder(StreamingAssistant.class)
78+
.streamingChatModel(streamingModel)
79+
.chatMemoryProvider(sessionId -> MessageWindowChatMemory.withMaxMessages(10))
80+
.build();
81+
----
82+
83+
See the https://github.com/OpenLiberty/sample-langchain4j/blob/7112a4f14bc7b63bd97f8f123f6ba3c16fdebba9/streaming-chat/src/main/java/io/openliberty/sample/langchain4j/StreamingChatAgent.java[StreamingChatAgent.java file] to learn more.
84+
85+
=== WebSocket endpoint
86+
87+
A JakartaEE WebSocket annotation declares a server endpoint that the front end can connect to.
88+
89+
[source,java]
90+
----
91+
@ServerEndpoint("/streamingchat")
92+
public class StreamingChatService {
93+
...
94+
}
95+
----
96+
97+
For each user message, the assistant generates a response. As the model generates the response, LangChain4j calls the callback that is given to `.onPartialResponse(...)` with each token, which gets sent through the WebSocket by `.sendText(token)`. If the response was cut short because of the token limit, the server ends the response with an ellipsis (`...`). The end of the response is signaled by sending an empty string.
98+
99+
[source,java]
100+
----
101+
@OnMessage
102+
public void onMessage(Session session, String message) throws Exception {
103+
CompletableFuture<ChatResponse> response = new CompletableFuture<>();
104+
assistant.streamingChat(session.getId(), message)
105+
.onPartialResponse(token -> {
106+
try {
107+
session.getBasicRemote().sendText(token);
108+
Thread.sleep(100);
109+
} catch (Exception error) {
110+
throw new RuntimeException(error);
111+
}
112+
})
113+
.onCompleteResponse(response::complete)
114+
.onError(response::completeExceptionally)
115+
.start();
116+
if (response.get().finishReason() == LENGTH)
117+
session.getBasicRemote().sendText(" ...");
118+
session.getBasicRemote().sendText("");
119+
}
120+
----
121+
122+
The error message is sent back to the user when errors occur.
123+
124+
[source,java]
125+
----
126+
@OnError
127+
public void onError(Session session, Throwable error) throws Exception {
128+
session.getBasicRemote().sendText("My failure reason is:\n\n" + error.getMessage());
129+
session.getBasicRemote().sendText("");
130+
}
131+
----
132+
133+
The chat history of the user is removed by calling `.evictChatMemory(...)` upon disconnect.
134+
135+
[source,java]
136+
----
137+
@OnClose
138+
public void onClose(Session session) {
139+
assistant.evictChatMemory(session.getId());
140+
}
141+
----
142+
143+
See the https://github.com/OpenLiberty/sample-langchain4j/blob/7112a4f14bc7b63bd97f8f123f6ba3c16fdebba9/streaming-chat/src/main/java/io/openliberty/sample/langchain4j/StreamingChatService.java[StreamingChatService.java file] to learn more.
144+
145+
=== Front end connection
146+
147+
On the front end, a WebSocket connection to the server endpoint is created.
148+
149+
[source,javascript]
150+
----
151+
// Connect to websocket
152+
var webSocket = new WebSocket('/streamingchat');
153+
----
154+
155+
When the user presses the enter key, the message is sent to the server through the WebSocket created earlier and sending messages is disabled temporarily.
156+
157+
[source,javascript]
158+
----
159+
function sendMessage() {
160+
var message = input.value;
161+
appendMessage('my-msg', message, new Date().toLocaleTimeString());
162+
appendMessage('thinking-msg', 'thinking...');
163+
webSocket.send(message);
164+
input.value = '';
165+
sendButton.disabled = true;
166+
};
167+
----
168+
169+
The following code adds each received token to the AI's last message element. If the front end receives an empty string, the response ends and the front end re-enables the user to send messages.
170+
171+
[source,javascript]
172+
----
173+
webSocket.onmessage = function (event) {
174+
if (event.data != '') { // stream ends with empty string
175+
if (messages.lastChild.querySelector('.thinking-msg')) { // if this is the first token
176+
messages.removeChild(messages.lastChild);
177+
appendMessage('agent-msg', '', new Date().toLocaleTimeString());
178+
}
179+
messages.lastChild.querySelector('.agent-msg').textContent += event.data;
180+
} else {
181+
sendButton.disabled = false;
182+
}
183+
};
184+
----
185+
186+
An error message is displayed if the WebSocket was closed for any reason.
187+
188+
[source,javascript]
189+
----
190+
webSocket.onclose = function (event) {
191+
...
192+
appendMessage('agent-msg', 'The connection to the streaming chat service was closed.');
193+
sendButton.disabled = true;
194+
};
195+
----
196+
197+
See the https://github.com/OpenLiberty/sample-langchain4j/blob/7112a4f14bc7b63bd97f8f123f6ba3c16fdebba9/streaming-chat/src/main/webapp/streamingChat.js[streamingChat.js file] to learn more.
198+
199+
== Conclusion
200+
201+
In this post, we explored how to use LangChain4j, set up a WebSocket endpoint, and stream responses to the front-end. By streaming the response, your users experience a shorter delay between sending a message and receiving a response, even if it is still being generated.
202+
203+
== What's next?
204+
205+
Check out the https://openliberty.io/guides/[Open Liberty guides^] for more information and interactive tutorials that walk you through other Jakarta EE and MicroProfile APIs with Open Liberty.
206+
207+
== Find out more
208+
209+
* https://docs.langchain4j.dev/tutorials/ai-services#streaming[LangChain4j streaming tutorial^]
210+
* https://github.com/marketplace?type=models[GitHub^], https://docs.mistral.ai/getting-started/models/models_overview/[Mistral AI^], and https://ollama.com/search[Ollama^] models
211+
* https://openliberty.io/guides/jakarta-websocket.html[Bidirectional communication between services using Jakarta WebSocket^]
212+
* https://openliberty.io/blog/2024/04/01/open-liberty-with-langchain4j-example.html[Run AI-enabled Jakarta EE and MicroProfile applications on Open Liberty^]

0 commit comments

Comments
 (0)