Added includes() Everywhere to Fix N+1. Made Everything Slower. Eager Loading Isn’t Always the Answer.

Subtitle: When the N+1 cure is worse than the disease—and knowing when to eager load vs accept simple queries

Viral hook: “N+1 bad, includes good.” Wrong. Sometimes 100 tiny queries beat one massive join. Profile first.

The pain: we “fixed” N+1 and our P95 tripled

We inherited a Rails 7.1 app that listed Projects with recent Tasks, Owners, and aggregated Comments. Bullet was screaming about N+1 on the index page, so we did the “responsible” thing and added includes everywhere:

# app/controllers/projects_controller.rb (before)
# We thought eager loading would be universally better.
@projects = Project.order(created_at: :desc)
                 .includes(:owner, tasks: [:assignee, :comments])
                 .limit(50)

One deploy later:

• P95 latency: 260ms → 1.8s on /projects (50K DAU sample, 24h)
• DB time per request: 110ms → 1.3s (Postgres 14, EXPLAIN ANALYZE verified)
• Peak memory (web dyno): 420MB → 1.1GB (gc pressure up, more Ruby objects)

We cut queries from ~400 tiny selects to ~18 “smart” ones. It still got slower. Why? Because we fetched far more rows than we used, fanned out across left joins, and materialized huge result sets Ruby had to dedupe into associations.

Lesson learned: N+1 is not a moral failure. It’s a trade‑off between round‑trips and result set size. If you don’t access most of the associations, eager loading is wasted work.

What includes, preload, and eager_load actually do (and when)

Active Record has three paths that look similar but shape very different queries:

# All Rails 7.1+; Ruby 3.2+

# 1) includes: Rails decides preload vs eager_load based on usage
Project.includes(:owner)
# WHY: Let Rails choose; good default when you *might* access owner

# 2) preload: *always* separate SELECTs for associations
Project.preload(:owner)
# WHY: Avoids huge JOINs when base scope is complex; keeps result sets small

# 3) eager_load: forces LEFT OUTER JOIN
Project.eager_load(:owner)
# WHY: Needed if you filter/sort by columns of the association; one big query

Rails turns includes into eager_load (JOINs) when you reference associated tables in the same relation (e.g., order("owners.name") or where(owners: { active: true })). That JOIN multiplies rows by the number of children and can explode payload size.

Rule of thumb:

• Use preload when you will touch the association for each parent but don’t filter by it.
• Use eager_load only when you must SELECT/ORDER/WHERE across associations.
• Use includes when you’re not sure; then verify in logs which path Rails took.

When eager loading helps vs hurts (with numbers)

Helps when the access pattern is dense and predictable.

# “Show page”: we *always* render owner and two recent tasks
@project = Project.includes(:owner, tasks: :assignee).find(params[:id])
# WHY: Show pages have dense usage; extra queries are certain, so preloading avoids round trips

• Before: 1 base query + 5 small N+1 bursts → ~180ms DB time
• After (preload) : 3 queries total → ~70ms DB time, P95 210ms → 140ms

Hurts when the access pattern is sparse or fan‑out is high.

# “Index page”: we render 50 projects but show tasks only for the first 5 via partials
@projects = Project.order(created_at: :desc)
                   .preload(:owner, tasks: [:assignee, :comments])
                   .limit(50)
# WHY: Preloading every task/comments for all 50 drags huge datasets we won't render

• Before: ~400 queries, each <3ms, result sets tiny → P95 260ms
• After (preload) : ~20 queries but one tasks query returns ~120k rows → P95 1.8s, memory +680MB

If you only expand tasks for 10% of rows (feature flags, collapsed sections, or mobile), eager loading is pure overhead.

A safer pattern: selective preloading, lean columns, and caps

Start with the base relation without eager loading. Add what the view actually needs, and keep it narrow.

# controllers/projects_controller.rb
scope = Project.order(created_at: :desc).limit(50)

# Only preload what the template always uses
scope = scope.preload(:owner) # WHY: owner name/avatar always shown

# Defer heavy collections; fetch on demand in partials via small queries
@projects = scope

In the partial, prefer bounded follow‑ups over massive preloads:

# app/views/projects/_project.html.erb
<%# WHY: cap fan-out; small, predictable queries beat one explosion %>
<% recent_tasks = project.tasks.order(created_at: :desc).limit(3).includes(:assignee) %>
<%= render recent_tasks %>

Also trim what you select, especially on joins:

# Only the columns you render; avoid SELECT * across joins
@projects = Project.select(:id, :name, :owner_id, :created_at)
                   .order(created_at: :desc)
                   .preload(:owner)

If you must join, keep it surgical:

# We sort by owners.name so JOIN is required; keep payload lean
@projects = Project.joins(:owner)
                   .merge(User.select(:id, :name))
                   .select('projects.id, projects.name, projects.created_at, users.name AS owner_name')
                   .order('users.name ASC')
# WHY: Explicit select avoids row bloat and Ruby object churn during eager_load

Guideline: cap child collections (limit, windowed queries, pagination) before reaching for eager loading.

Profiling workflow that actually works (and keeps you honest)

Tools we used in production:

• Bullet to catch accidental N+1—then we decided deliberately when to ignore.
• Rack Mini Profiler to see SQL timings inline; we turned off the Rails query cache for tests to avoid rose‑colored numbers.
• EXPLAIN ANALYZE on suspect queries to confirm whether JOIN fan‑out or bad indexes were the real culprits.
• PgHero to spot missing indexes and high bloat.
• Scout APM/New Relic for end‑to‑end P95 and memory trends per action.

A concrete workflow we now follow on every index page:

1. Measure baseline with no eager loading. Record P50/P95, SQL count, allocated objects.
2. Add preload only for associations rendered for all rows. Re‑measure.
3. If sorting/filtering across associations, switch to eager_load for that specific association and add explicit select.
4. Cap collections (e.g., 3 tasks per project). Move the rest behind Show more with Ajax or Turbo.
5. Delete unnecessary eager loads. If Bullet complains for a code path that runs 10% of the time, consider a safe ignore.

In one real screen, this process yielded:

• DB rows returned per request: ~135k → ~8.1k
• Allocated Ruby objects: ~1.2M → ~210k
• P95: 1.9s → 280ms (steady for 7 days, 60k page views)

Common mistake we shipped (and how we fixed it)

We once replaced a simple joins(:owner) with includes(:owner) to silence Bullet, then added order('users.name') later. Rails implicitly flipped to eager_load (LEFT JOIN), returning duplicate project rows for each matching owner row during pagination.

Impact: users saw inconsistent pagination (missing/duplicated projects), cache keys churned, and P95 jumped 300ms → 1.4s.

Fix:

# 1) Make intent explicit
@projects = Project.eager_load(:owner)
                   .select('DISTINCT projects.id, projects.name, projects.created_at')
                   .order('users.name ASC')
# WHY: DISTINCT prevents duplicate parents introduced by JOIN fan-out

# 2) Or: keep JOIN but avoid eager loading altogether
@projects = Project.joins(:owner)
                   .select('projects.id, projects.name, projects.created_at, users.name AS owner_name')
                   .order('users.name ASC')
# WHY: We only render owner name; extra owner objects provide no value

Takeaway: don’t let includes change query shape behind your back. When ordering or filtering on associated columns, say eager_load or joins explicitly and control the payload.

When not to eager load (explicit trade‑offs)

Don’t eager load when:

• High fan‑out, low usage: You render 50 parents and only expand children for 3–5 of them.
• You need only scalars: A joins + select for owner_name is cheaper than building full Owner objects.
• You paginate heavily: Preloading large collections across pages wastes work; fetch per page instead.
• Cache is strong: If a fragment cache hides most child rendering, eager loading prefetches data you won’t use.

Consider eager loading when:

• Show pages or dense components: You always access the same associations.
• Hot loops: Rendering 100 rows and touching the same association on each; preload saves RTTs.
• You must sort/filter by associations: Use eager_load or joins deliberately.

Pragmatic patterns that aged well for us

# 1) Use counter caches instead of preloading big collections
# WHY: reading a small integer is cheaper than dragging children; keeps index fast
class AddCounters < ActiveRecord::Migration[7.1]
  def change
    add_column :projects, :tasks_count, :integer, default: 0, null: false
    add_column :projects, :comments_count, :integer, default: 0, null: false
  end
end

# 2) Batch when you *must* touch many children (background work)
# WHY: keep memory flat and avoid timeouts; web should stay snappy
Project.in_batches(of: 100).each_record do |project|
  project.tasks.select(:id).find_each(batch_size: 100) do |task|
    # do small updates
  end
end

# 3) Paginate aggressively on index screens
# WHY: capping result size is the simplest performance lever
@projects = Project.order(created_at: :desc).page(params[:page]).per(20) # Pagy/Kaminari

# 4) Use .load_async (Rails 7.1) sparingly for independent queries
# WHY: hide latency of separate selects without JOIN fan-out
@projects = Project.order(created_at: :desc).limit(50)
owners    = User.where(id: @projects.pluck(:owner_id)).load_async

Final Thoughts

includes isn’t bad. Blind includes is. Performance is context:

• If a page touches most associations for most rows, preload is great.
• If you filter/order by association columns, use eager_load or joins and keep the SELECT list tight.
• If you only render a few children sometimes, accept small N+1 bursts and cap them with limit/pagination.

Profile first, then choose. Our worst regression came from treating N+1 warnings as absolutes. Once we matched eager loading to actual view usage, we cut rows returned per request by >90%, stabilized memory, and landed P95 ~280ms without silencing Bullet. That’s the boring, durable kind of fast.