- Prefer real objects, or fakes over mocks. It will make your tests usually more robust.
- Use mocks when you must: to avoid networking, or other flaky things such as storage.
- Use mocks for “output only objects”, for example listeners, or when verifying the output for some logging. (But, prefer a good fake)
- Use mocks when you “need to get shit done”, it’s the easiest way to add tests in an area that has almost none, and the code is not designed to be easily testable. But remember this is tech debt, and try to migrate towards real objects over time.
That’s my short advice I told many times. So might as well comment with it here.
They explicitly call out clocks as a source of non-determinism that probably should be mocked, but I'll re-use them as an example anyway because everyone is familiar with them: it's extraordinarily useful for the tests to execute nearly immediately rather than actually waiting on a clock, and rare behavior like a clock running backward, two consecutive timings being identical within the clock's resolution, or whatever other weird artifacts that your code should handle are definitely better explicitly tested rather than not mocking the clock. Other domain-specific interfaces are often similarly able to exhibit a weird edge case that ought to be explicitly tested (rather than accidentally relying on a "nice" implementation) if you really want to unit test the callers and not integration test the coupled system.
Let's just assume the absurd notion that a choice of creating tech debt or changing a common interface is a real choice. If there was a serious change that could not be accounted for with easy test changes, I'm not the only one to see the tests commented out with a "TODO: fix these". Developers tend to be pragmatic.
If you want to change code you will always measure the effect it will have on the code and tests are incidental, not the primary concern. Make it work. Make it right. Make it fast. I want to be able to trigger all code paths and exceptions (make it good). Using a real dependency, I would be left in the unfortunate situation of depending on knowledge of the internals of that dependency. It may not allow me to execute specific paths via pure configuration at all.
I don't think using real dependencies is a good idea at all for unit tests. Integration tests are a different and I do fear that they are being confused.
Rather than a granular test on a single class, test the orchestration of many classes. You end up hitting big % of code. As always, depends on the project. If we're building a rocket ship, you need both granular testing and coarse testing.
As an example, consider a single API that uses ~3 services, and these 3 services have underneath from 1 to 5 other internal or external dependencies (such as time, an external API service, a DB repository, and so on). How can I test this API, the 3 services, and their underneath dependencies without exponential paths to test - I want to be able to cover all the paths of my code, and ensure that my test only tests a single thing (either the API, the service, or the dependency interface); otherwise, it is not an unit test.
I always felt like that these type of tests without mocks works super-nice in nice situations without any external, or even complex but internal, dependency; otherwise, it becomes very very hard to test ONLY what I want, and not all the dependencies underneath.
Mocks allow me to stub the behaviour of a service/dependency that I can test in a separate fashion, covering all the paths, and ensuring that each unit test covers a single unit of my code, and not the integration of all my components.
For fakes, spin up the real thing. If you’re not able to model your database transactions deterministically, then your transactions could themselves be flawed and tests are great way to catch that.
Deterministic tests are not a goal in and of themselves. Controlled non determinism is valuable. This is popularized in various frameworks under the names of property checking and fuzzing which will let you know the seed to use for the failure for example so that while the runs don’t have the exact same input/output for every invocation, you get better coverage of your test space and can revisit the problematic points at any time. If you’re doing numeric simulation, make sure you are using a PRNG that’s seedable and that you log the seed at the start if you’re using a seed (and make sure time is an input parameter). Why is this technique valuable? You transitively get increasing code coverage for free through CI/coworkers running the tests AND you have a way to investigate issues sanely.
The better advice is: continue to isolate unit tests away from real dependencies, and ALSO have integration tests that test the way the package connects to dependencies (and the other packages in the software dependent on it).
So if I write my own sort algorithm I am not allowed to unit test that class unless I make it public.
But if the sort algorithm is to solve a specific sub problem in a library there is no need to make it public - it may not make much sense.
So I had to test it “through” its consumer class(es) that is public. For illustration sake lets say a MVC controller mocked up to the eyeballs with ORM mocks, logging mocks etc.
This bugged me because having direct access to a functional core I can quickly amplify the number of test cases against it and find hidden potential bugs much quicker.
So you make models and build automation test frameworks and labs full of prototype equipment so you can run 10,000 tests an hour, 24/7. You automate setting up networks and nodes and passing live traffic to test detectors and rule sets. You use custom hardware to emulate ISP-levels of real traffic.
You can't really mock anything. You have to be sure each function will perform as you describe. So most of this testing is end-to-end testing. Using a mock wouldn't tell you if the millions of code paths are working correctly; they'd mostly just tell you if the syntax/arguments/etc of a call were correct. Unit tests are basically one step above evaluating that your code compiles correctly, but it's not testing the code works as expected. End-to-end tests are what you can rely on, because it's real traffic to real devices.
That's gonna look different for a lot of your apps, but for web apps that means getting real familiar with synthetic tests and how to test with different browsers. For SDKs it means end-to-end tests for the whole stack of functions, which is a lot more testing than you may have expected. For APIs it means getting the consumers of your APIs involved in end-to-end testing. It also means "testing in production", in the sense of spinning up new production deployments and using those as your end-to-end testing targets (rather than having a dedicated "dev/test/cert" environment which is always deviating from production). This can take a significant effort to adapt in legacy systems, but for Greenfield IaC projects is not very hard to set up.
If a unit test requires an external dependency then just use an integration test (or check it’s covered by system tests) and leave it at that.
When fakes are owned by anyone else, e.g. the folks writing the system under test, there's a high risk that the semantics of the fake and the real implementation will diverge, rendering the tests that use the fakes much less useful.
Testing "the real thing" sounds fun until the budget for new features skyrockets, because several man-weeks are needed to get decent coverage. Your client will hate it, and your client's clients will hate it even more.