Streaming an LLM Response Into a Rails View With Turbo Streams
Most writing about LLMs in Rails apps stops at architecture: RAG, MCP, "AI as a multiplier." The part that actually bites is smaller and more concrete: a model takes eight seconds to answer, and HTTP gives you one response. You either stare at a spinner for eight seconds or you stream.
If you're already on Hotwire, you have the delivery mechanism. Here's the whole build.
Why Turbo Streams instead of raw SSE
The obvious answer is ActionController::Live and a Server-Sent Events endpoint: the Claude API already speaks SSE, so you're just proxying frames. It's less machinery, and it's the right call for a one-off "summarize this" widget.
Turbo Streams wins when the streaming response isn't the only thing that changes. A chat reply usually also flips a status badge, disables the composer, and appends to a sidebar - that's three targets, and SSE gives you one channel to hand roll DOM updates through. Turbo Streams gives you server rendered HTML for all of them over a subscription you already have, plus automatic reconnection you don't have to write. You also get out of the business of holding a Puma thread open for the duration of the model call.
The trade is that you're now fanning tokens through Action Cable, and naive implementations fall over there. That's the interesting part.
The pieces
Assume Rails 8 defaults (Solid Queue, Solid Cable) and the anthropic gem:
bundle add anthropic
Create the client once — it's threadsafe and carries its own connection pool (99 connections by default), so a per-request client wastes sockets:
# config/initializers/anthropic.rb
ANTHROPIC = Anthropic::Client.new(
api_key: Rails.application.credentials.anthropic_api_key
)
Two models, Chat has_many :messages, with role and content on the message plus a status enum (pending, streaming, complete, failed).
The controller does almost nothing
class MessagesController < ApplicationController
def create
chat = Chat.find(params[:chat_id])
chat.messages.create!(role: :user, content: params[:content])
reply = chat.messages.create!(role: :assistant, content: "", status: :pending)
StreamReplyJob.perform_later(reply)
redirect_to chat
end
end
The key move is creating the empty assistant message before the job runs. It gives you a persisted record with a stable dom_id, which means the page renders a real (empty) message bubble immediately and the job has a target to replace. No placeholder then swap dance.
The view subscribes to the chat:
<%= turbo_stream_from @chat %>
<div id="<%= dom_id(@chat, :messages) %>">
<%= render @chat.messages %>
</div>
And app/views/messages/_message.html.erb:
<%= tag.div id: dom_id(message), class: "message message--#{message.role}" do %>
<%= markdown(message.content) %>
<% if message.status == "streaming" %>
<span class="cursor" aria-hidden="true"></span>
<% end %>
<% end %>
The job, and the mistake everyone makes first
The naive version broadcasts on every chunk:
stream.text.each do |chunk|
message.update!(content: message.content + chunk) # don't
end
A 600-token response becomes hundreds of UPDATE statements, hundreds of partial renders, and hundreds of Action Cable frames — for a UI that can't render faster than the browser's paint loop anyway. With Solid Cable, every one of those is also a row insert that subscribers poll for. It works in development with one user and degrades sharply once real conversations overlap.
Buffer in memory and flush on a clock:
class StreamReplyJob < ApplicationJob
FLUSH_INTERVAL = 0.1 # seconds
def perform(message)
chat = message.chat
history = chat.messages.where.not(id: message.id).order(:created_at)
.map { |m| { role: m.role, content: m.content } }
message.update!(status: :streaming)
buffer = +""
last_flush = monotonic_now
stream = ANTHROPIC.messages.stream(
model: :"claude-sonnet-5",
max_tokens: 2048,
messages: history,
request_options: { max_retries: 0 }
)
stream.text.each do |text|
buffer << text
next if monotonic_now - last_flush < FLUSH_INTERVAL
last_flush = monotonic_now
broadcast(message, buffer, :streaming)
end
message.update!(content: buffer, status: :complete)
broadcast(message, buffer, :complete)
rescue Anthropic::Errors::APIError => e
Rails.logger.error("stream failed for message=#{message.id}: #{e.class}")
message.update!(status: :failed)
broadcast(message, buffer.presence || "Something went wrong.", :failed)
end
private
def broadcast(message, content, status)
message.content = content
message.status = status
Turbo::StreamsChannel.broadcast_replace_to(
message.chat,
target: ActionView::RecordIdentifier.dom_id(message),
partial: "messages/message",
locals: { message: message }
)
end
def monotonic_now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
end
Three details worth dwelling on.
Assign, don't save. In broadcast we mutate the in-memory record and render the partial against it. The database is touched exactly twice — once at the start, once at the end. The partial doesn't know or care that the object is dirty.
broadcast_replace_to, not broadcast_append_to. Replace re-renders the whole message on each flush, so the payload grows with the response. Append would send only the delta, which is cheaper — but it's not idempotent. A client that reconnects mid-stream (a tunnel, a sleeping phone) misses deltas and ends up with a message full of holes. Replace is self-healing: whatever the last frame said is the truth. For very long generations, append deltas into a trailing <span> and replace once at the end; for chat-length replies, just replace.
Not broadcast_replace_later_to. The _later_ variants enqueue a Turbo::Streams::ActionBroadcastJob to do the rendering, which is exactly right in a controller and exactly wrong here — you're already in a job, and you'd be enqueuing one Solid Queue job per flush.
Tuning the flush interval
Solid Cable polls its messages table every 0.1s by default, so flushing faster than that buys nothing — your chunks just queue up behind the poller. Match FLUSH_INTERVAL to your adapter: 0.1s on Solid Cable, and you can drop to ~0.05s on Redis, which pushes rather than polls, if you want a smoother crawl. Somewhere below that the effect stops reading as "typing" and starts reading as jitter.
What else breaks
Thread occupancy. A streaming job holds a worker thread for the full model call — seconds to minutes, not milliseconds. Size the Solid Queue pool for concurrent conversations, not requests per second, and consider a dedicated queue so a burst of chats can't starve your mailers.
Retries. The Ruby SDK retries certain failures — connection errors, 408, 409, 429, 5xx, and timeouts — twice by default, and its default timeout is ten minutes. Retrying a stream means the user watches the text restart from scratch, which is why the example passes max_retries: 0 and handles the failure in the UI — that's where you can offer a regenerate button.
The ensure you'll want later. If the job is killed (deploy, OOM) the message is stuck in streaming forever and the cursor blinks into eternity. A periodic sweep that fails any streaming message older than a few minutes is five lines and saves a support ticket.
Refresh mid-stream. Because you only persist at the end, a page reload during generation shows an empty bubble that then catches up on the next flush. Acceptable for most apps; if it isn't, persist the buffer every few seconds on a separate, slower clock than the broadcast.
None of this is much code — a job, a partial, and one subscription helper. The discipline is entirely in the buffering: the model produces tokens far faster than a browser needs them, and the whole job of the Rails layer is to throttle that firehose down to something a DOM can live with.
Sources: