So you have configured a new build for your ASP.NET application: the source code compiles without errors, there are no unit tests failing, the deployment package is generated and published as an artifact – great! Now it’s time to deploy it. Everything seems to be fine (no errors logged), but when you try to run your application you get an YSOD like this one:
There are many things that can go wrong with a deployment, so it is important to configure your deployment pipeline to verify that the deployment itself was successful. You can do it by using smoke tests.
Smoke-Test your deployments
From Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (emphasis is mine):
When you deploy your application, you should have an automated script that does a smoke test to make sure that it is up and running. This could be as simple as launching the application and checking to make sure that the main screen comes up with the expected content. Your smoke test should also check that any services your application depends on are up and running – such as a database, messaging bus, or external service.
The smoke test, or deployment test, is probably the most important test to write once you have a unit test suite up and running – indeed, it’s arguably even more important. It gives you confidence that your application actually runs. If it doesn’t run, your smoke test should be able to give you some basic diagnostics as to whether your application is down because something it depends on is not working.
Ok, it should be obvious by now how important smoke tests are, but what and how exactly should you test? In the scenario below I will explain you how I configured the smoke tests for some applications in my previous company.
An example – configuring smoke tests for a REST API
Consider a REST API (ASP.NET Web API) with the following dependencies:
- SQL Server database
- Redis Cache Server
- A third-party API
In order to be able to test if a deployment went fine you can do the following:
- Add Ping methods to each controller of the API
- Add a Ping controller to test the dependencies of the API
- Add the unit tests
- Configure the deployment pipeline to run the tests
1. Add a ping method to each controller of the API
This should be accessible as an HTTP GET request that returns the a JSON object containing the system date, a basic description and eventually other data such as the environment, release version, etc. This method should return an HTTP status code 200 (OK) in case of success, or HTTP status code 500 (Internal Server Error) in case of error/exception.
Even though this might seem pointless/unnecessary, this test is more important than it seems. You might be able to detect runtime errors such as the one from the screenshot above (System.IO.FileNotFoundException – Could not find file or assembly) or DI/IoC errors (e.g. dependencies not configured properly in the bootstrapper).
2. Add a Ping controller to test the dependencies of the API
As mentioned before, each method should be accessible as an HTTP GET request that returns the a JSON object and returns status code 200 (OK) in case of success, or status code 500 (Internal Server Error) in case of error.
To test the database all you need to do is a simple query such as getting the system date. This way you can check if the database is up and running, and if there are no permission issues (such as expired passwords, firewall issues, etc).
Testing the cache server is similar – I usually add a temporary item to the cache and try to get the same item. Again, you can check if the cache server is up and running and if your application is able to access it (correct credentials, firewall issues, etc).
To test the third-party API you can try to invoke a Ping method, in case there is one available. In my previous company we started to add a Ping method to each API/Controler we created (SOAP Web Services or REST APIs), which can make your life easier. In case there is no Ping method available just try to invoke the simplest/fastest service available (ideally a read-only service that won’t change data).
In case of error/exception in any of the these methods (database, cache server, third-party API) return HTTP status code 500 and eventually the exception message (not the entire stack trace), in order to understand what might have gone wrong. Be careful not to expose sensitive information in your HTTP response.
3. Add the unit tests
Configuring the unit tests is easy – you can use the HttpClient object to Call a Web API From a .NET Client and check if the HTTP status code of the response is 200 (OK). Also, in case you’re returning custom data such as the environment, you can add a test to check if the value is correct for the current environment.
4. Configure the deployment pipeline to run the tests
Add a new task to your deployment pipeline to run the tests, right after the deployment. If any of these tests fail we should fail the deployment and prevent any new deployments to the failing environment, until the problem is solved.
Note: remember to configure your unit tests to use environment variables such as the base URL of the API (and others). This way you can use different configuration settings, for each environment.
Final notes
Hopefully I was able to show you how important smoke tests are, in the context of CI/CD.
Having a green build is just the first step but it’s not enough. Deployments are part of the QA and testing process, and smoke testing should be the last step of the deployment pipeline. This way you will be able to check if the application is up and running.
Also, it’s extremely important to add a smoke test for each service your application depends on: databases, cache servers, service buses, third-party APIs (Paypal, Realex, …), email servers, microservices, and others.
To finish, the Ping methods I suggested above can be run outside the context of a deployment. You can also configure some sort of monitoring application to invoke these same methods frequently (e.g. every 10 or 20 minutes) to check if the application is up and running 😉
Rui Jarimba 