Fixtures never

Recently I did something I haven’t done for a long, long, long time — start up a Rails project from scratch.

Well, I didn’t even really go “from scratch”, but from a jumping-off point, using Jumpstart. But let’s say that’s close enough to scratch.

Since I’ve spent so much of my career jumping into existing projects, I ran into some surprises. Some of those things are new — I’m still trying to get used to Rails 8 and the Solid* suite of tools and their extra databases. Some of those things are very, very old. Like test fixtures.

What are fixtures? They’re sample data for use in tests. They’re “fixed” data, things you can depend on.

That sounds pretty good, right? You need data to test against, and you want it to be known and dependable. Well, you’ll get no argument from me on the first part there (because what does your code do if it’s not operating on data?), but I have some significant problems with “known” and “dependable”. I’d say at best that’s a double-edged sword.

Fixtures can be so nice and easy when you’re starting out, but they can grow and grow and become more convoluted until you’re dealing with a tangled web you’re fighting your way through (and hopefully, out of). I hesitate to say this is a definite outcome and not just a probable or possible one, but I’ve been around for a while and I’ve seen it far too many times. At this point, I feel it’s just going to happen.

For some of the problems, we can go directly to the documentation and see what it says.

The testing environment will automatically load all the fixtures into the database before each test. To ensure consistent data, the environment deletes the fixtures before running the load.

This is very helpful, but the trouble with something being dependable is that you come to depend on it. In my recent conversations about troubles with fixtures, I gave what I felt was a contrived example of the problem by saying you might have a test that queries users and expects a list of 5, and how can you possibly know why that result is expected? Well, this contrived example is actually the first example in the documentation:

require "test_helper"

class WebSiteTest < ActiveSupport::TestCase
  test "web_site_count" do
    assert_equal 2, WebSite.count
  end
end

And the best part is even this can stand as an example of the problem. If you go directly to this part of the documentation and don’t scroll up to see the fixtures, there’s no indication at all why there should be two WebSites. You may argue that this is still a contrived example, but look more at the documentation. This is the explanation of how to use fixtures. This is the recommended way. You’re supposed to define your test data in one place and use it in another, where you can’t see it.

This sort of thing is why I often say “fixtures never”. The problem is that the tests and the fixtures grow together and become coupled. Essentially, what you start to do is test your fixtures and not the code. With this set of example fixed data, you come to depend on it and test against what you know is there, and when the data changes, tests will start to fail. Tests will fail not because what they’re nominally testing is no longer working, but because they’re testing against expected values. Maybe you’ll change a fixture to test something new and a different test will fail because of that change. Maybe you’ll add a fixture because you don’t want to cause that problem and now a test will fail because it’s getting more results than it expected. Maybe you’ll do better than all this and you’ll have a morass of interconnected fixture data that doesn’t exercise all the cases you need. Or maybe you’ll have an even bigger morass because you are exercising all those cases.

Maybe you’ll do even better and won’t run into these problems. If so, you’re better than I am, and pretty much every developer I know. And I argue you’re probably using up a lot of your energy and attention on managing your test data and not your tests or even your code.

So what can you do instead? You use factories. In this case, FactoryBot (with the Rails integration). What’s a factory? It’s a builder, where you define how to build or create an example of an object, rather than set out a bunch of examples yourself.

Let’s say you need a bunch of users for testing. With fixtures, it would look something like this:

one:
  email: one@example.com
  first_name: User
  last_name: One
  
two:
  email: two@example.com
  first_name: User
  last_name: Two
  
admin:
  email: admin@example.com
  first_name: Admin
  last_name: User
  admin: true

And you would get them in your tests with calls like users(:one) or users(:admin).

With FactoryBot, you would define a factory like so

FactoryBot.define do
  sequence(:email) { |n| “user#{n}@example.com” }
  sequence(:last_name, ‘Aa’)


  factory :user do
    email
    first_name { “User” }
    last_name


    trait :admin do
      admin { true }
    end
  end
end

And then you get a user in your tests with just create(:user), as many times as you want. Need an admin? That’s just create(:user, :admin).

Me, I prefer to generate lesser-known information using Faker, so my factory would look more like

FactoryBot.define do
  factory :user do
    email { Faker::Internet.email }
    first_name { Faker::Name.first_name }
    last_name { Faker::Name.last_name }


    trait :admin do
      admin { true }
    end
  end
end

There can be some drawbacks to this, but I think well-written and-used factories circumvent those nicely.

So what’s the point? The point is this builder knows how to build your object, you can concentrate on just what you need for the tests, and you can feel confident in your tests being isolated and self-contained.

And you only need one definition. This create method can take arguments to override values, so If you need something slightly different in a test (like if you’re trying to test a specific value, exercise a uniqueness validation, or whatever) all you need to do is create(:user, email: "specific@thing.net"). No creating even more fixtures.

And if your model changes, say there’s a new attribute or a validation changes, you only need to make a change in one place, not for every fixture. How can you lose?

Well, I’m not going to pretend that factories are the be-all, end-all. I started this with “fixtures never”, but that’s the short, glib, simple thing. Fixtures have their places.

• Creating an involved, interconnected network of objects. I’d say it’s best if you can avoid this, but maybe your data model just looks like that — because of plain messiness or the true complexity of what you’re doing. Factories can handle this, but it can be messy.
• Repeatability — if you want to run a bunch of tests on the same set of data. Maybe there’s a specific configuration that surfaces a problem, or there’s an edge case that needs to be covered.
• Speed — fixtures will load everything up at the beginning, instead of creating objects as you ask for them.

With all of these, fixtures aren’t required — factories can work, possibly using helper methods or setup blocks. But that is getting away from the pure joy of how factories help you and into the trouble of hidden magic or secret setup.

It also gets into another distinction I’d like to discuss: seeds vs. fixtures vs. “test data”. That’s for next time.

If you’re looking for a team to help you discover the right thing to build and help you build it, get in touch.