Evaluating LLM Features in Rails: Regression Tests for Prompts with Minitest and Fixtures

Your AI feature works. You've added streaming, you've connected tools over MCP, and your RAG pipeline finds the right documents. Then someone adds "be concise" to the system prompt, or you move to a newer model because it's cheaper. A week later, support tells you the ticket classifier has been sending refund requests to the "general" queue.

Nothing raised an error and CI stayed green. The feature just got worse.

This is one of the most common ways LLM features fail in production. Normal tests don't catch it because they check that code runs, not that the output is still good. Rails already gives you most of what you need to fix this. Minitest, fixtures and a few conventions are enough to build regression tests for prompts that catch these problems before your users do.

The example: a ticket triage service

We'll use one feature throughout. A TicketTriage service takes a support ticket and returns a category, an urgency level and a one-sentence summary.

# app/services/ticket_triage.rb
class TicketTriage
  Result = Data.define(:category, :urgency, :summary)

  PROMPT = Rails.root.join("app/prompts/ticket_triage.md.erb")

  def initialize(client: LlmClient.new, model: Rails.configuration.x.llm.triage_model)
    @client = client
    @model = model
  end

  def call(ticket)
    raw = @client.complete(model: @model, system: system_prompt, user: render(ticket))
    data = JSON.parse(raw)
    Result.new(data["category"], data["urgency"], data["summary"])
  end

  def render(ticket)
    ERB.new(PROMPT.read, trim_mode: "-").result_with_hash(ticket:)
  end

  def system_prompt = "You triage support tickets. Respond only with JSON."
end

(Data.define needs Ruby 3.2 or later. That is already the minimum Ruby version for Rails 8.)

Two choices here make testing easier later. First, the prompt is stored in its own file under app/prompts, so any change to it shows up in the diff. Second, the client and model are injected, so tests can replace them.

Layer 1: deterministic tests for what you control

Most of the code around an LLM call is ordinary code: templating, parsing and error handling. Test it like ordinary code, quickly and on every push.

Snapshot the rendered prompt. A prompt is effectively source code that happens to be written in English. If an edit to a partial or a model method changes the text sent to the model, you want to see that change in review.

# test/services/ticket_triage_prompt_test.rb
require "test_helper"

class TicketTriagePromptTest < ActiveSupport::TestCase
  test "rendered prompt matches snapshot" do
    rendered = TicketTriage.new(client: nil).render(tickets(:refund_angry))
    path = Rails.root.join("test/snapshots/ticket_triage/refund_angry.txt")

    if ENV["UPDATE_SNAPSHOTS"] || !path.exist?
      path.dirname.mkpath
      path.write(rendered)
    end

    assert_equal path.read, rendered
  end
end

When you mean to change the prompt, run UPDATE_SNAPSHOTS=1 bin/rails test and commit the new snapshot. Reviewers then see the exact text the model will receive, not only the template.

Test parsing with stubbed responses. Use a fake client to check how your code handles valid JSON, JSON wrapped in a Markdown code fence, missing keys and refusals. The plain JSON.parse above will reject fenced JSON, and tests like these are how you find that out:

class FakeClient
  def initialize(response) = @response = response
  def complete(**) = @response
end

test "raises on non-JSON output" do
  triage = TicketTriage.new(client: FakeClient.new("Sorry, I can't help with that."))
  assert_raises(JSON::ParserError) { triage.call(tickets(:refund_angry)) }
end

These tests are fast and fully deterministic. They tell you nothing about whether the model still gives good answers. That's the next layer's job.

Layer 2: an eval suite built on fixtures

An eval is a test that calls the real model with known inputs and scores the outputs against expectations. Rails fixtures suit this well because your eval inputs are real domain records, with associations, that go through the same code path as production.

Start with the tickets themselves:

# test/fixtures/tickets.yml
refund_angry:
  customer: enterprise_acme
  subject: "Charged twice AGAIN"
  body: "This is the third month you've double-billed us. Refund it today or we cancel."

password_reset:
  customer: free_user
  subject: "Can't log in"
  body: "Reset email never arrives. Checked spam."

feature_question:
  customer: free_user
  subject: "Export to CSV?"
  body: "Is there a way to export my reports? Not urgent, just curious."

Keep the expected outputs in a separate file, next to the evals rather than inside the fixtures:

# test/evals/ticket_triage_cases.yml
- ticket: refund_angry
  category: billing
  urgency: high
  summary_must_mention: [refund, double]
- ticket: password_reset
  category: account_access
  urgency: medium
- ticket: feature_question
  category: product_question
  urgency: low

Now write the eval as a normal Minitest file. It skips itself unless you ask for it, so bin/rails test stays fast and doesn't use API credits:

# test/evals/ticket_triage_eval_test.rb
require "test_helper"

class TicketTriageEvalTest < ActiveSupport::TestCase
  CASES = YAML.load_file(Rails.root.join("test/evals/ticket_triage_cases.yml"))
  THRESHOLD = 0.9

  setup { skip "Set RUN_EVALS=1 to run evals" unless ENV["RUN_EVALS"] }

  test "triage accuracy stays above threshold" do
    triage = TicketTriage.new
    failures = []

    CASES.each do |c|
      result = triage.call(tickets(c["ticket"].to_sym))
      failures << "#{c["ticket"]}: category #{result.category}" if result.category != c["category"]
      failures << "#{c["ticket"]}: urgency #{result.urgency}" if result.urgency != c["urgency"]

      Array(c["summary_must_mention"]).each do |word|
        failures << "#{c["ticket"]}: summary missing '#{word}'" unless result.summary.downcase.include?(word)
      end
    end

    checks = CASES.sum { |c| 2 + Array(c["summary_must_mention"]).size }
    score = 1.0 - failures.size.fdiv(checks)
    assert score >= THRESHOLD, "Score #{score.round(3)} below #{THRESHOLD}:\n#{failures.join("\n")}"
  end
end

The test produces one aggregate score instead of one assertion per case, and that's deliberate. Models are non-deterministic. If you treat every case as pass/fail, a single odd answer fails CI and the team soon learns to ignore the suite. A threshold accepts some noise and still catches real regressions. When a regression happens, the failure message lists exactly which cases broke.

Run it with:

RUN_EVALS=1 bin/rails test test/evals

Scoring free text with a judge

Exact matching works for categories. It doesn't work for summaries, where many wordings are correct. The keyword check above is crude but useful. For more subtle quality checks, use a second model call as a judge with a narrow rubric:

# test/support/llm_judge.rb
module LlmJudge
  def assert_judged(output, criterion, client: LlmClient.new)
    verdict = client.complete(
      model: Rails.configuration.x.llm.judge_model,
      system: "You are a strict grader. Answer only PASS or FAIL.",
      user: "Criterion: #{criterion}\n\nOutput:\n#{output}"
    )
    assert verdict.strip.start_with?("PASS"), "Judge failed: #{criterion}\n---\n#{output}"
  end
end

Rails doesn't autoload test/support, so require this file from test_helper.rb and include LlmJudge in the test classes that use it.

Keep each criterion to one checkable claim, such as "The summary does not invent an order number" or "The summary is one sentence". Vague rubrics like "is this a good summary?" give vague, unstable verdicts. Pin the judge to a fixed model so that upgrading your production model doesn't also change the grader.

Making it stable enough to trust

A few habits separate a useful eval suite from a flaky one:

• Set temperature to 0 in eval runs if your provider and model support it. Some newer models don't accept a temperature setting at all. Even at 0, outputs can vary, so the threshold still matters.
• Run each case several times for important features and score the average. A few runs per case help you tell a real regression from a fluke. How many you need depends on how noisy the feature is.
• Store a baseline. Write each run's per-case results to a file such as tmp/evals/ticket_triage.json. Comparing against the last known good run shows which cases changed, not just that the score dropped.
• Add every production bug as a case. When a misrouted ticket gets reported, turn it into a fixture plus an expectation. This is the same discipline as writing a failing test before fixing a bug, and over time it makes the suite reflect your real traffic.
• Keep the set small and sharp. Thirty well-chosen cases covering edge cases (sarcasm, mixed languages, two issues in one ticket) beat three hundred near-duplicates. You'll pay for every case on every run.

Wiring it into CI

You don't want to pay for evals on every commit, and you don't want to skip them when they matter. A reasonable split:

1. Every push: snapshot and parsing tests (Layer 1). These are free and fast.
2. Pull requests that touch app/prompts/**, the LLM service classes, or the model config: run RUN_EVALS=1 bin/rails test test/evals. Most CI systems support path filters for this, such as paths in GitHub Actions or rules:changes in GitLab CI.
3. Nightly: run the full suite against the current model to catch provider-side drift. This matters most if you call a floating alias (for example a -latest name) instead of a pinned model version, because the model behind an alias can change without any change to your code.

Store the API key as a CI secret that's only available to these jobs. Post the score in the PR so reviewers see "triage: 0.94 → 0.88" next to the prompt diff.

The payoff

Once this is in place, a prompt change goes through the same process as any other code change: a readable diff, a snapshot showing the exact text the model receives, and a score showing whether the change helped or hurt. Moving to a new model becomes something you can measure. Change one config value, run the evals and compare the numbers.

None of this needs a dedicated eval platform. It's Minitest, YAML fixtures and some discipline about where prompts live. Start with ten cases for your most important AI feature this week. The first time the suite catches a regression that would otherwise have reached users, it will have been worth the effort.