Constant time CI with CodeBuild

At recruiterbox, we try to practice test driven development — our test coverage is above 80%. We have a mix of unit tests, integration tests and functional tests. Our ratio of unit tests to integration tests is not great but reasonable.

In any case, as we added more and more features to the product, the total time to run our test suite kept growing. About 2 years ago, it became too difficult to run all tests in local. At the time, we setup a Jenkins based system with 4 slaves to parallelise tests across these machines. With time, we increased the number of slaves to about 10 and kept the time to run tests to ~20 minutes. We also upgraded this setup to have 2 parallel pipelines of 10 machines each to accommodate increasing number of developers and more code being developed. If more than 2 developers pushed their code at the same time, they would have to wait their turn to run their tests.

However, this setup was very wasteful because the load on the slaves was very spiky at specific times of the day. It was difficult to predict patterns in this and setting up auto-scaling required too much work. Since there were only 2 parallel pipelines to run our test suite, during peak loads, the queue to run tests was also becoming too big. Sometimes developers had to wait 4 hours to run their tests. This was unacceptable waste of developer time.

We came across CodeBuild when we were just considering moving to EC2 Container Service with auto scaling turned on. As you can imagine CodeBuild felt like a life saver. We can get almost infinite parallelisation (upto a safety limit we can choose). As more and more tests are added, we can increase the number of containers to keep the time to run tests constant and it can keep scaling to as many parallel pipelines as we want based on spiky loads or as we add more team members.

In summary, kudos to the AWS team for building an awesome service like CodeBuild.