Standardized Development Cycle
local changes > testing > code review > build system > deployment
The majority of micro services failures are caused by bugs introduced during the development phase. In order to reduce these bugs, we need to establish a proper development cycle that begins when the developer commits a change:
- unit tests should be run;
- the code review must be performed by two or more developers;
- the build system should run the integration tests and ensure that everything is ok;
- if the build is green, the change can be merged and deployed to a staging environment.
Stable and Reliable Deployment Pipeline
staging, canary and production rollouts
We don’t want bad deployments. There’s no reason to still ear the famous sentence: “works on my machine”.
The most reliable deployments are those that pass several steps before reaching the production servers. That’s why we should deploy on several environments in order to ensure that our changes don’t break anything.
We should keep in mind that deploying changes to a microservice multiple times per day will compromise the stability and reliability of our microservice.
if you can’t measure, you don’t know it works
Monitoring is one of the key principles of availability.
If the key metrics of our microservice aren’t tracked, we can’t predict any kind of failure until the failure itself occurs.
We should capture the metrics emitted by our microservice, log them, graph them and alert them on special situations.
Distributed Logging and Tracing
SSH in Prod? You’re doing it wrong.
A production-ready microservice must trace every request through the entire stack. If we ensure that, afterwards we can retrieve important information about the system.
The log itself must be as rich as possible in order to determine what went wrong and where things fell apart. But we need to remember that logging is expensive. Make sure you don’t deploy debug logs into production!
Logging also needs to be available, scalable, easily accessible and searchable. Please, no more SSH in Prod.
if it moves, draw it
Every microservice must have at least one dashboard where all the key metrics are collected and displayed.
Anyone should be able to look at it and know immediately whether or not the micro service is working correctly.
Although, we should never be watching the dashboard in order to detect any kind of incident. That will be a sign that our alerting system is wrongly designed.
Invert the Inverted Pyramid
more unit and contract tests, less end-to-end tests
Unit tests and contract tests should be at the bottom of the pyramid. They are smaller, more focused, and they run very quickly.
E2E tests take more time to setup and run. That’s why they should be on top of the pyramid!
An heavy test suite will be more difficult to maintain, slow to run, and will not pinpoint the reported errors. We will end up debugging our own tests.
faster tests lead to faster feedback
In order for CI and CD to be practical, the full test suite should run every time a developer commits a change and the developer should have the feedback in a matter of minutes.
If we have a slow test suite, the team will probably skip this procedure and run the test suite from time to time instead of running when a developer commits a change.
If running tests on every commit is too slow, the solution is not to run the tests less often but instead to fix the situation so the test suite runs more quickly. The most obvious solutions is to ensure that our microservice is properly covered by unit tests, that are way faster than the others, and reduce the amount of slower tests.
No More Silos
between development and operations
Silos come from a lack of communication and collaboration between teams, not simply from a separation of duties.
Currently, the developers work ends to soon and the operations work starts too soon. We should give more end-to-end responsibilities to developers.
Instead of “works on dev, now it’s ops problem” we should move to “you build it, you run it!”
Self-Sufficient Software Engineers
fewer rockstars, more orchestras
The developers need to be responsible for not just writing code but also for managing the entire life-cycle of applications, ensuring its health, maintainability and observability.
This includes being responsible for deployments, rollbacks, monitoring and debugging, in addition to bug fixes and new feature development.