Home > general > blog > Performance Matters - OneOps Release 17.04.12

17 April 2017

Performance Matters - OneOps Release 17.04.12

Manfred Moser

One of the often overlooked fact of web application design and usability, and even generally any application is that performance matters. It matters a lot. If actions take too long users get distracted with other tasks, while waiting and loose focus. And then they make mistakes and its all the applications fault. With this release we are bringing some incredible performance improvements to our users.

Changes

The changes from this release don’t look like much, but a lot of effort went into measuring current behavior, figuring out problems and coming up with solutions. One of the often requested performance issues are deployment plan generation and deployment execution. For large complex assemblies we used to observe plan generation times of 20 minutes. Complex deployments with more than 50 steps for really large, critical application was taking between 30 minutes and up to 2 hours.

Collecting key metrics impactng performance such as woAssemblyTime, scriptExecution, semaphoreWaitTime, InterStepRequestDelay and others allowed us to create charts and dashboards to determine slow components and the impact artifact size and other criteria.

The changes we implemented include steps such as

  • Increased local threads to 20
  • Removed synchronized block
  • Configured cache for controller
  • Fixed relation caching issue
  • Optimized deployment plan generation algorithm
  • Optimized queries

Results

These changes are now deployed and with our users and the improvements we are seeing are drastic. The following chart shows the decline in deployment plan generation times across multiple assemblies:

Deployment Plan Performance Overall

One specifically large assemblies has seen especially dramatic improvments and our users are loving it.

Deployment Plan Performance Example

As you can see we achieve 80% reduction in deployment times on average. One of the most complicate designs in production used to take over 20 minutes for the plan generation. Now that time is down to down to less than 2 minutes - more than a factor 10 improvement. Even in the most extreme cases we see at least a 3 times speed improvement.

Next Steps

Of course, we are not going to stop here. We still a whole host of improvements we are exploring and working on such as

  • Explore storing pre-baked images
  • Recipe optimization
  • Database performance tuning
  • Separating display and cmsdb components
  • Introduce non-blocking deployments

Bug Fixes and Improvements

  • Performance improvement for controller by enabling md caching - PR

Source Release Tags

Enjoy,

OneOps Team