Usually when called in to a fire fighting situation where the performance and scale of the system is going to shit, it is critical to understand the right treatment to apply to the problem. Understanding the Precision vs Accuracy tradeoff is critical in prescribing the right medicine for your problems. So when should you go for accuracy and when should you pursue precision ?
When you need to be in the right ballpark of the solution within an order of magnitude go for accuracy. When you need to be precise to the individual digit level strive for precision. In most situations you can guess that accuracy comes first, precision comes later. Another critical tradeoff is performance vs scale. Your application may scale great but perform like crap and vice-versa. Scale needs to be achieved at the right level of end user performance.
In general the higher you are in the abstraction stack the more performance gains you will get so Architecture changes will yield a 10x more benefit than JVM or GC tuning which will yield a 10x more benefit than tuning assembly code and so on … If it is the database tier that you think is the problem - then you can put in multiple shock absorbers 1. caches 2. queues 3. partitioning first and focused on instead tuning the startup memory and app start times.
Always understand the top constraint and problem you are solving for. You should always prioritize to solve the top constraint and the best way to determine the top constraint is to take a system level holistic view - draw out a system map and take a look at all the queues and waits in the system.
Visualize the system as a bucket of wait queues - Something like the picture below. An Event Storming exercise can help suss out this system map and visualize the API/Date stream.
Here are the top five performance mitigation suggestions based on past experience
When you need to be in the right ballpark of the solution within an order of magnitude go for accuracy. When you need to be precise to the individual digit level strive for precision. In most situations you can guess that accuracy comes first, precision comes later. Another critical tradeoff is performance vs scale. Your application may scale great but perform like crap and vice-versa. Scale needs to be achieved at the right level of end user performance.
In general the higher you are in the abstraction stack the more performance gains you will get so Architecture changes will yield a 10x more benefit than JVM or GC tuning which will yield a 10x more benefit than tuning assembly code and so on … If it is the database tier that you think is the problem - then you can put in multiple shock absorbers 1. caches 2. queues 3. partitioning first and focused on instead tuning the startup memory and app start times.
Always understand the top constraint and problem you are solving for. You should always prioritize to solve the top constraint and the best way to determine the top constraint is to take a system level holistic view - draw out a system map and take a look at all the queues and waits in the system.
Visualize the system as a bucket of wait queues - Something like the picture below. An Event Storming exercise can help suss out this system map and visualize the API/Date stream.
Here are the top five performance mitigation suggestions based on past experience
- Classpath Bloat: Check to see if the Classpath is bloated. Spring boot autoconfigures stuff that sometimes you don’t need and bloated class path leads to misconfiguration of threadpools and libraries. As a remedy put the dependencies (pom.xml or build.gradle) through a grinder. We have 25 step checklist if you want details. This will also reduce the overall memory footprint due to library misconfiguration. Developers load up a lot of dependencies and libraries in the app and the full implications of memory bloat and runtime misconfiguration are not understood till later
- Startup time: If you want your app to startup quickly see this list https://cloud.rohitkelapure.com/2020/04/start-your-spring-apps-in-milliseconds.html
- Memory: If the app is memory constrained ensure that GC is tuned correctly. Employ verbose GC tracing. see https://blog.heaphero.io/2019/11/18/memory-wasted-by-spring-boot-application/#4A7023 and https://heaphero.io/
- External integration tuning including outboud DB, HTTP Calls & Messaging. Connection Pool Tuning. See https://github.com/pbelathur/spring-boot-performance-analysis. In general examine any outbound connection/integration to any database, messaging queue. Circuit Breakers& metrics on every outbound call to see the health of the outbound connection
- Latency/ Response Time analysis to see where & wether time is spent on the platform /app /network/disk Use the latency-troubleshooter app to hunt down latency hogs. https://docs.cloudfoundry.org/adminguide/troubleshooting_slow_requests.html and https://community.pivotal.io/s/article/How-to-troubleshoot-app-access-issues
For a detailed checklist see
HAPPY Performance Hunting!
HAPPY Performance Hunting!
