Jest is running thousands of tests at Facebook at all times, either through continuous integration or invoked by engineers manually during development. This worked well for years even as the people working on Jest moved on to other projects within Facebook.
Jest is a bit different from most test runners. We designed it to work well in the context of Facebook's infrastructure:
- Sandboxing Another feature of Jest that's important to Facebook is how it virtualizes the test environment and wraps
requirein order to sandbox code execution and isolate individual tests. We're even working on making Jest more modular so we can take advantage of this functionality in other non-testing related use cases.
As a result of these unique constraints, Jest may never be able to be as fast as other test runners when running on our entire suite of tests. However, engineers rarely need to run Jest on our entire test suite. Powered by static analysis in the node-haste project – we've been able to make the default mode for running Jest at Facebook
jest --onlyChanged, or
jest -o. In this mode we build a reverse dependency graph to find only the affected tests that need to be run based on the modules that have been changed.
Most of the time our static analysis determines that more than one test needs to be run. The number of affected tests can be anywhere from a couple of tests to thousands. In order to speed this process up Jest parallelizes test runs across workers. This is great because most of Facebook's development happens on remote servers with many CPU cores.
Recently we noticed Jest often seemed stuck “Waiting for 3 tests” for up to a minute toward the end of a run. It turned out we had a few really slow tests in our codebase that were dominating the test runtime. While we were able to speed these individual tests up significantly, we also made a change in how Jest schedules test runs. Previously we used to schedule test runs based on file system traversal, which was actually quite random. Here is an example of 11 tests in gray blocks over two workers. The size of the block is the runtime of the test:
We were randomly running a mix of fast and slow tests, and one of our slowest tests ended up running as almost all the other tests were completed, during which the second worker sat idle.
We made a change to schedule tests based on their file size which is usually a good proxy for how long a test is going to take. A test with a few thousand lines of code likely takes longer than a test with 15 lines of code. While this sped up the entire test run by about 10%, we ended up finding a better heuristic: now Jest stores the runtime of each test in a cache and on subsequent runs, it schedules the slowest tests to run first. Overall this helped improve the runtime of all tests by about 20%.
Here is an example of the same test run from before with better scheduling:
Because we are running slow tests first, Jest can sometimes seem to take a long time to start up – we only print results after the first test has completed. For the future we are planning to run previously failed tests first, because getting that info to developers as quickly as possible matters the most.
If you have written tests using Jasmine before, they probably look like this:
One special thing we do in Jest is reset the entire module registry after every single test (call to
it) to make sure tests don't depend on each other. Before Jest, individual tests would depend on each other and internal module state often leaked between them. As engineers removed, reordered or refactored tests, some of them started to fail, making it hard for people to understand what was going on.
Every single test in Jest receives a fresh new copy of all modules, including new versions of all mocked dependencies which take a lot of time to generate for each test. A side effect of this is that we had to call
require manually before every test, like this:
We built a babel transform called inline-requires that removes top-level require statements and inlines them in code. For example, the line
const sum = require('sum'); will be removed from code, but every use of
sum in the file will be replaced by
require('sum'). With this transform we can write tests just like you'd expect in Jasmine and the code gets transformed into this:
A great side-effect of inline requires is that we only require the modules that we actually use within the test itself, instead of all the modules we used in the entire file.
Which leads to another optimization: lazy mocking. The idea is to only mock modules on demand, which combined with inline requires saves us from mocking a lot of modules and all their recursive dependencies.
We were able to update all tests using a codemod in no time – it was a simple 50,000 line code change. Inline requires and lazy mocking improved the test runtime by 50%.
For now, if you'd like to use this transform in Jest you'll have to add it manually to your Babel configuration. We are working on ways to make this easier to opt-in.
The open source version of Jest used to be a fork of our internal version, and we'd sync Jest out only once every couple of months. This was a painful manual process that required fixing up many tests every time. We recently upgraded Jest and brought parity to all platforms (iOS, Android and web) and then enabled our sync process. Now, every change to Jest in open source is run against all of our internal tests, and there's only a single version of Jest that's consistent everywhere.
When trying to improve performance, it's important to also dive into the systems that sit above and below your system. In the case of Jest, things like Node.js and the test files themselves, for example. One of the first things we did was to update Node.js at Facebook from the years-old 0.10 to iojs and subsequently to Node 4. The new version of V8 helped improve performance and was quite easy to upgrade to.
We noticed that the
path module in Node.js is slow when making thousands of path operations which was fixed in Node 5.7. Until we drop support for Node 4 internally at Facebook, we'll ship our own version of the fastpath module.
We next started questioning the outdated node-haste. As mentioned before, the entire project has to be parsed for
@providesModule headers to build a dependency graph. When this system was originally built,
node_modules didn't exist and our file system crawler wasn't excluding them properly.
In previous versions, Jest would actually read every file in
node_modules – which contributed to the slow startup time of Jest. When we picked up Jest again we replaced the entire project with a new implementation, based on react-native's packager. The startup time of Jest is now less than a second even on large projects. The react-native team, specifically David, Amjad and Martin did an outstanding job on this project.
A lot of the above changes improved the test runtime by 10% or sometimes even 50%. We started at a runtime of about 10 minutes for all tests, and without these improvements we'd probably be at around 20 minutes by now. After these improvements, though, it now consistently takes around 1 minute and 35 seconds to run all our tests!
More importantly, adding new tests causes total runtime to grow very slowly. Engineers can write and run more tests without feeling the costs.
With Jest's recent 0.9 release and performance improvements from the node-haste2 integration, the runtime of the Relay framework's test suite went down from 60 seconds to about 25 and the react-native test suite now finishes in less than ten seconds on a 13” MacBook Pro.
We're very happy with the wins we've seen so far, and we're going to keep working on Jest and making it better. If you are curious about contributing to Jest, feel free get in touch on GitHub, Discord or Facebook :)