Cost Controls for LLM Features in Rails: Per-Tenant Token Budgets, Caching and Model Routing

Shipping an LLM feature is the easy part. You've tested your prompts and your responses stream into the UI. A month later the invoice arrives, and one tenant who wired your "summarize" button into a nightly script has used more tokens than the rest of your customers combined.

LLM costs work differently from the rest of your infrastructure. You pay for every request. Each user controls how much a request costs, through how much text they paste in and how long they let the model answer. A single loop in someone else's code can multiply your bill. Rails gives you good building blocks for dealing with this, and you need three layers: metering and budgets, caching, and model routing. This article covers each one in that order.

Step zero: put every call behind one gateway

You can only control costs you can see. Before adding budgets, make sure no controller, job or model calls the provider SDK directly. Send everything through one service object:

class LlmGateway
  def initialize(tenant:, feature:)
    @tenant  = tenant
    @feature = feature
  end

  def complete(messages:, max_tokens: 1024, tier: :standard)
    model    = ModelRouter.pick(tier: tier, tenant: @tenant)
    budget   = TokenBudget.new(@tenant)
    reserved = estimate(messages, max_tokens)
    budget.reserve!(reserved)

    response = client.messages.create(model: model, max_tokens: max_tokens, messages: messages)
    record_usage(model, response.usage)
    budget.reconcile!(reserved: reserved, actual: response.usage.input_tokens + response.usage.output_tokens)
    response
  end
end

With one entry point, you have one place to enforce limits, one place to log usage and one place to swap models. Many teams skip this step and later find out they can't answer the question "which feature is costing us money?"

Layer 1: meter usage and enforce per-tenant budgets

Record what actually happened

Every provider response reports how many tokens it used. Anthropic's Messages API, for example, returns input_tokens and output_tokens, plus cache_creation_input_tokens and cache_read_input_tokens for tokens written to and read from the prompt cache. Store all of them, because they're billed at different rates:

create_table :llm_usages do |t|
  t.references :tenant, null: false, index: false
  t.string  :feature, null: false
  t.string  :model,   null: false
  t.integer :input_tokens,  null: false, default: 0
  t.integer :output_tokens, null: false, default: 0
  t.integer :cache_read_tokens,  null: false, default: 0
  t.integer :cache_write_tokens, null: false, default: 0
  t.bigint  :cost_microcents, null: false, default: 0
  t.timestamps
end
add_index :llm_usages, [:tenant_id, :created_at]

Work out the cost when you write the row, using a pricing table kept in config (config/llm_pricing.yml). Don't calculate it when someone views a report. Prices change, and you want each row to show what the request cost at the time it was made. Store integer microcents, not floats, and use a bigint column: a 4-byte integer tops out at about $21 in microcents, which one long-context request to a large model can get close to.

This table becomes your cost dashboard. A query like LlmUsage.where(created_at: 30.days.ago..).group(:tenant_id, :feature).sum(:cost_microcents) shows you which tenant and which feature the money goes to.

Enforce budgets before the call, not after

Recording usage after the fact tells you what went wrong but doesn't prevent it. To enforce a budget you have to check it before the request, while concurrent requests are running. Querying the database for each request is slow and has race conditions. A Redis counter (directly, or through the Kredis gem) is atomic and fast:

class TokenBudget
  class Exceeded < StandardError; end

  def initialize(tenant)
    @tenant = tenant
    @key = "llm_budget:#{tenant.id}:#{Date.current.strftime('%Y-%m')}"
  end

  def reserve!(tokens)
    used = redis.incrby(@key, tokens)
    redis.expire(@key, 40.days.to_i) if used == tokens
    if used > @tenant.monthly_token_limit
      redis.decrby(@key, tokens)
      raise Exceeded
    end
  end

  def reconcile!(reserved:, actual:)
    redis.incrby(@key, actual - reserved)
  end

  def over_soft_limit?
    redis.get(@key).to_i >= @tenant.monthly_token_limit * 0.8
  end
end

Reserve a pessimistic estimate up front: the input token count plus max_tokens. Once the response comes back, reconcile against the real usage. Anthropic and Google both offer token-counting endpoints, but a rough character-based heuristic is usually good enough for a reservation, because you correct it afterward. Keep it conservative: the old rule of thumb of four characters per token undercounts on Claude's current tokenizer, which Anthropic puts at about 2.5 characters per token.

Some practical refinements:

• Soft and hard limits. At 80% of the budget, send an email or notify the tenant in the app. At 100%, either fall back to a cheaper model (see Layer 3) or return a clear error. Never let the request fail silently.
• Rate limits as well as monthly caps. A monthly budget doesn't stop someone from using all of it in an hour. Add per-minute throttling with Rails 7.2+'s built-in rate_limit in controllers, or Rack::Attack, keyed by tenant.
• Cap max_tokens per feature. A title generator doesn't need 4,096 output tokens. Output tokens usually cost several times more than input tokens (five times on current Claude models), so a tight cap is the cheapest control you have.
• Tie budgets to plans. Put monthly_token_limit on the plan model, so that upgrading a plan raises the limit with no code change.

Layer 2: caching at two levels

Response caching

The cheapest request is the one you don't make. If the same input reliably produces an answer you're happy to reuse (classifying a support ticket, summarizing a document that hasn't changed, extracting fields from an uploaded invoice), cache the response:

cache_key = ["llm", @feature, PROMPT_VERSION, model, max_tokens, Digest::SHA256.hexdigest(messages.to_json)]
Rails.cache.fetch(cache_key, expires_in: 7.days) do
  # Cache a plain hash rather than the SDK's response object
  client.messages.create(model: model, max_tokens: max_tokens, messages: messages).to_h
end

Include the model and a prompt version in the key. That way, a prompt change or model upgrade invalidates the cache on its own instead of serving stale answers. Solid Cache, the default production cache store in Rails 8, makes large, long-lived caches cheap because it keeps entries in the database on disk rather than in RAM.

Don't use response caching for chat or anything personalized, where the same input should legitimately produce different output. Use it for tasks that work like pure functions.

Provider-side prompt caching

Response caching only helps when the whole input repeats. Prompt caching helps when the beginning of the input repeats, which is the common case: a long system prompt, a tool schema, few-shot examples or a large document that a user asks several questions about.

Anthropic, OpenAI and Google all support some form of this. With Anthropic you mark a breakpoint with cache_control, and later requests that share the identical prefix are billed for cache reads at a small fraction of the normal input price (10% or less, depending on the model). The cache lives for five minutes by default. Writing to it costs 1.25× the normal input price, or 2× if you choose the one-hour lifetime, so it pays off once the prefix is reused. Prompts shorter than a model-specific minimum (512 tokens on current Claude models) aren't cached at all. OpenAI applies prefix caching automatically above a minimum prompt length.

To get the most from it, structure your prompts so that stable content comes first and variable content comes last. Anthropic builds the cached prefix in the order tools, then system, then messages:

1. Tool definitions and system instructions (identical across all tenants)
2. Tenant-specific context such as a knowledge base or style guide (identical across one tenant's requests)
3. The user's actual message (different every time)

A common mistake is putting Time.current or a request ID in the system prompt. That changes the prefix on every call and means you never get a cache hit. Watch the cache-read token columns in your usage table. If they stay at zero, something in your prefix is changing between requests, or your prefix is below the minimum cacheable length.

Layer 3: model routing

Most LLM features don't need your most capable model. Classification, extraction, short rewrites and routing decisions run well on small, fast models such as Claude Haiku 5.5, which costs a fraction of what frontier-tier models do. Use the large models for tasks that need them: multi-step reasoning, long-form generation and complex tool use.

Put that decision in one place:

class ModelRouter
  TIERS = {
    fast:     "claude-haiku-5-5",
    standard: "claude-sonnet-5-5",
    frontier: "claude-opus-5-5"
  }.freeze
  # Judgement call: claude-fable-5-1 sits above Opus 5.5 for demanding reasoning;
  # Opus 5.5 is kept here because Anthropic recommends it as the default for most workloads.

  def self.pick(tier:, tenant:)
    tier = :fast if TokenBudget.new(tenant).over_soft_limit?
    tier = :standard if tier == :frontier && !tenant.plan.frontier_models?
    TIERS.fetch(tier)
  end
end

This gives you three levers:

• Static routing by feature. Each call site declares the tier it needs. This is easy to reason about and covers most of the savings.
• Plan-based routing. Premium tenants get the frontier model, everyone else gets the standard one. That makes your margins match your pricing.
• Degradation instead of denial. When a tenant passes their soft budget, quietly downgrade them to the fast tier. That's usually a better experience than a hard error.

A more advanced option is cascading: try the cheap model first, check its output (did the JSON parse, did it pass the schema, did it report high confidence?), and only escalate to a larger model on failure. This works well for structured extraction, where most inputs are easy. Your existing prompt test suite becomes the evaluation harness here. Run the suite against each tier and route each feature to the cheapest tier that passes.

Keep model IDs in configuration, not scattered through the code. Providers release new models and retire old ones regularly. With a single router, a model migration is a one-line change that you can test against your prompt suite before deploying.

Putting it together

When a request goes through the full stack, it:

1. Enters LlmGateway with a tenant and feature label
2. Checks the response cache and returns immediately on a hit
3. Gets a model from ModelRouter based on feature, plan and budget state
4. Reserves estimated tokens against the tenant's Redis budget
5. Sends a prompt with stable content first, so the provider's prompt cache applies
6. Records actual usage, including cache reads, to llm_usages and reconciles the budget

None of these steps is complicated, and together they make LLM spend something you can forecast. With this in place, you can see what a feature costs before you launch it, offer usage-based plans with confidence, and contain a runaway integration as a capped, rate-limited incident instead of a surprise on the invoice.

Build the gateway and the usage table first, even before you add budgets. Once you can see where the tokens go, the right limits are usually obvious.