Author: Jochen Wuttke What
Test Engineers do at Google . In this post, I talk about one aspect of the
work TEs may do: building and improving test infrastructure to make engineers
more productive.
Refurbishing legacy systems makes new tools necessary
flaky
tests , but had little time to implement new features. The goal for the
rewrite was to learn from the legacy system and to build something that was
easier to maintain and extend. As the team's TE, my job was to understand what
caused the high maintenance cost and how to improve on it. I found two main
causes:
Tight coupling and insufficient abstraction made unit testing very hard, and
as a consequence, a lot of end-to-end tests served as functional tests of that
code.
The infrastructure used for the end-to-end tests had no good way to create
and inject fakes or mocks for these services. As a result, the tests had to run
the large number of servers for all these external dependencies. This led to
very large and brittle tests that our existing
test execution infrastructure was not able to handle reliably.
Exploring solutions
RPC
callsgraph (not
tree !) of
dependencies, which all had to be up and running for the end-to-end test. The
new model changes how we test client and service integration. Instead of running
the client on inputs that will somehow trigger RPC calls, we write unit tests
for the code making method calls to the RPC stub. The stub
itself is mocked with a common mocking framework like Mockito in Java. For each
such test, a second test verifies that the data used to drive that mock "makes
sense" to the actual service. This is also done with a unit test, where a replay
client uses the same data the RPC mock uses to call the RPC handler method of
the service.
Adoption
The new tests are as effective in finding bugs as the end-to-end tests are.
The new tests run in about 3 minutes instead of 30 minutes for the
end-to-end tests.
The client side tests are 0% flaky. The verification tests are usually less
flaky than the end-to-end tests, and never more.
Additionally, the new tests are unit tests, so you can run them in your IDE and
step through them to debug. These results allowed us to run the end-to-end tests
very rarely, only to detect misconfigurations of the interacting services, but
not as functional tests.
How do you mark a test failure as flaky? Do you have an automated/intelligent system that flags a test run failure as flaky or do you do it manually?
ReplyDeletehttps://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
DeleteVery interesting. The replay technique sounds like an interesting alternative/variation to contract tests. I gave a presentation last week that makes very similar recommendations to this article: https://skillsmatter.com/skillscasts/8567-testable-software-architecture
ReplyDeleteSounds like the consumer driven contracts idea implemented here: https://docs.pact.io - cool in theory, but hard to write readable tests for in practice.
ReplyDelete