It’s a question we face as test-driven ruby programmers: Should we use mock objects or real objects in our tests?
Both approaches have trade-offs, and their biggest downsides both have to do with wasting programmer time. If you test with real objects, then your tests run slowly (especially if you use an ORM that binds your domain objects tightly to the database like ActiveRecord). Your tests hit the database, and this is slow. There are other sources of slowness, but nothing has anywhere near as great an effect as hitting the DB.
If you test with mock objects, once your app has any kind of complexity, your refactoring and test writing processes become slow. This is not immediately apparent when you start using mock objects. But as you start writing more and more code, eventually you start having to come up with a crazy number of mock expectations just to test some of your methods. It is true that this is good feedback that the class you’re testing presents too complex an interface to other collaborating objects, or that it collaborates with too many objects, etc. What starts simple will eventually become too complex, and at some point you’re going to need to refactor.
More insidious than this, however, is the effect this web of mocks you’ve wrapped yourself in has on refactoring. You don’t notice how thoroughly you’ve painted yourself into a corner until you want to refactor some ugly aspect of a core class that collaborates with many objects in your system. Suddenly all of those collaborating objects’ tests break because they expect certain method calls from this core object. These tests break because you’ve changed a method signature, a method name, or even worse just an implementation, because no matter what those BDDers tell you, if you test with mocks, to too great a degree that means you’re testing implementation, not behavior (or is that behaviour).
So, now you have to go through all of those mock-based tests and “correct” them, i.e. change their expectations so that they fit the new method name/signature/implementation. This is horrible. The whole point of tests during refactoring is to verify that your refactoring hasn’t changed the behavior of the system (that being half of the definition of refactoring). The tests should pass before you refactor, and they should pass after you refactor. Not only does this break the fundamental refactoring process, it also can take a lot of time, because you have to remember the context of each of those test cases that you have to fix.
You can do something about slow running tests that hit a database (in extreme cases you can use a faster in-memory database, or even parallelize your tests). Of course they won’t run as fast as they would if they didn’t hit a database, but in my opinion it’s something you can live with. Dav Yaginuma has a good suggestion for what to do with this time: Go write that email you need to write, go take the bathroom break, go walk around the office and stretch your legs. It’s not like that’s really wasted time. It is wasted time, however, if you’re sitting there squinting at the screen fixing all of your mock expectations. You can’t do anything else with that time.
I’m kind of half-convinced now that people who advocate the heavy use of mocking either have really nice IDEs that make fixing the expectations a breeze, or they don’t refactor. And if they’re using Ruby that means they don’t refactor. Ok, tongue out of cheek. Seriously, I’d love to hear from folks who have used and continue to use mocks heavily on long running projects, to hear how they handle the refactoring issue. I have pretty much sworn off mocks except in old-school traditional cases (“mocking out an external dependency too expensive to call directly”) because of it.
I totally agree with you. Mock objects have always seemed super awkward to me. I think that when you test your application, it should mimic how it’s going to behave in the real world as much as possible. Otherwise, you’re not really testing it fully. And I think in-memory databases are the way to go. If you use an ORM (though I haven’t used one that I loved), then it should be seamless.
The most recent version of RSpec now supports ‘stub_model’s. These use real objects that created at the time of use and cannot be saved to the database. They have the advantage of allowing you to stub out attributes and still be able to call actual methods on the object. This way you end up only stubbing the data and not the behavior associated with an object.
Hehe, it’s a long running debate. Mocks vs. fixtures. Rspec brought many advantages that I like, such as being able to test views and helpers and controllers all independently of each other. Oftentimes with the right granularity you can make those tests pretty lean and mean.
Having said this, mocks are no end-all be-all as some of the fervent proponents have you believe. Mocking association proxies are a pain, for example. I’ve not yet heard a good answer to this.
Some projects out there are trying to centralize the mocks, these may be worth a look.
On a wider perspective, though, I’ve observed a very similar problem to what you’re describing–you make one implementation or API change–and all hell breaks loose with your tests, even in a fixtures-based test environment, and this typically happened when new fixtures were added to add more variety to the demo data, so as to model a new case. The upshot conclusion from this for me has always been that data and tests were too tightly coupled. So, not knowing anything about your application, I wonder if your test granularity (and thus code granularity) is too coarse?
Test granularity: definitely not too coarse : )
When we first started testing, we were bitten by the “add a fixture and a bunch of random tests break” phenomenon a few times. That was one of our motivations to move heavily into using mock-based testing. After mock-based testing bit us also, we eventually converged on the following approach to test data: use fixtures for things that are really basic and fundamental, and therefore won’t need to be added to often; use the ObjectMother pattern (which goes by many other names, basically a test data factory) for one-off objects with specific states needed by your tests; use mocks when test data setup is just too arduous, or if needed for performance or some kind of external dependency.
Thanks for your comment!