DevOps Core Tenet: Automated Testing
This is the fourth and final post in a series of blog posts about our core tenets of DevOps. For more information on our approach to DevOps, visit our DevOps Page.
The final core tenet of DevOps is Automated Testing. The goal of this tenet of two-fold: to reduce the amount of time dedicated to manual testing, and to ensure that quality is built into a solution’s core and continually evaluated.
In an organization that has successfully adopted Continuous Delivery, automated testing takes several different forms. While a specific testing plan is generally developed for each experience in coordination with an organization’s overall testing standards, it generally will include several key elements:
- Automated unit and syntax testing through a continuous integration server on each commit to the version control repository. This can be either a standalone server like Jenkins, Atlassian Bamboo, or JetBrains TeamCity or a continuous integration service such as Travis CI, Circle CI, or Codeship.
- Automated functional testing through a solution like Selenium, TestComplete, HP Enterprise UFT, or eggPlant.
- Automated smoke and load testing on an environment identical to the production environment.
As continuous delivery becomes more prevalent and automated testing continues to progress, the tools have matured. Entire organizations, like SauceLabs, have emerged to provide an entire suite of tools to enable organizations to rapidly adopt automated testing in the cloud. In addition, providers like Amazon Web Services have added automated testing of mobile applications through Device Farm to their already extensive list of solutions. With these advancements in place, even the smallest of organizations can take advantage of automated testing at scale.
As with all of our four core tenets, automated testing has many benefits for both business and development:
- Automated testing can reduce the man-hours required to validate a release to production. Every single hour required for manual testing and automation is an hour that could be automated. For most organizations, this is where ROI can be realized the fastest in a shift to DevOps.
- Automated testing can eliminate most functional defects in production. Once automated testing becomes standardized in the development process, acceptance criteria on user stories should be validated through automated tests that are written during the sprint in which the story is implemented. With this in place, most functional defects are discovered by the development team and the in-sprint QA engineer.
- Automated testing can continually validate non-functional requirements in the release process. With automated testing in the release process, non-functional requirements (NFR’s) such as performance and scalability can also be automated and validated in an environment identical to production. This allows for a team to set specific NFR’s across a project and ensure compliance prior to release.
Without an automated testing strategy and approach, quality simply becomes one of many goals that must be weighed for a digital solution. Unfortunately, for most organizations, when the overall goals are weighed, functionality and time to market win out over quality. To ensure that quality and reliability are not a goal but core components, you will need to embrace this core tenet and the solutions that enable it.
If you are interested in adopting an automated testing approach for your organization, you may be interested in a DevOps Maturity Assessment.