Cumulative time is increasing by executing tests through time. Is every lost second important?
Integration tests that are running in a container, i.e. Spring Framework or J2EE server, are executed slower than plain unit tests. Plain unit test can be 100 times faster.
When the integration test is started, it has to instantiate the container, which is a big startup time cost. If tests are using a database, the database has to be started and populated with some data. When more integration tests are started to run, the startup cost can be reduced. The container could be instantiated only once and changes made in the database can be in a transaction.
Time difference between plain unit tests and an integration tests that are running in the container
Let’s say that a plain unit test is executed in 10 milliseconds. Let’s say that an integration test is executed in 5 seconds and every additional integration test is executed in 50 milliseconds.
When you run 1000 tests, plain unit tests are executed in 10 seconds, integration tests are executed in 55 seconds. Time difference is 45 seconds.
When you run one test, a plain unit test is executed in 10 milliseconds, a integration test is executed in 5 seconds. Time difference is 5 seconds.
Direct time costs
Execution of all tests are less frequent than execution of one test. If we frequent execute one test at the time, cumulative time cost is increasing, in case plain unit tests by 10 milliseconds, in case integration tests by 5 seconds.
Indirect time cost
Long execution of test can influence on the mind flow. If tests are executed before commit or build, we have already completed a functionality and have ended thinking about it. In this case long execution has low effect on the mind flow. If have executed the test, when we are implementing the functionality, and the test need a long time to execute, the mind flow could be interrupted. In this case we need additional time in order to establish the mind flow.
Conclusion
If you have an option, write plain unit tests, which are executed fast. Think about, if you can make integration tests without containers.