Reducing MySQL Replica Lag by 90% and Lag Catchup time by 97%

Chapter 1: Setting the context

At HomeLane, we have a single MySQL DBMS which acts as both, a data lake and a data warehouse. As one might expect, we schedule ELTs to pull data from our production DB’s into the data lake and then run aggregation jobs to prepare the data for analytics. This aggregated data is then shared with teams via a replica. This architecture keeps things simple and suffices for our use cases. Recently though, we ran into a problem with this setup where we started seeing increased replica lag times. This didn’t abruptly happen in one day but it built over time and reached a point where user experience started to greatly suffer. The result of this increased wait times, sometimes of up to 6 hours, to see updated data which frustrated a lot of users and so this needed to be addressed.

»
Author's profile picture 7-m on mysql, replication, and lag

Collecting Spring Repository metrics using Spring AOP

I have been working for quite sometime on optimizing API’s here at HomeLane. As we grow, it becomes crucial to make sure that our API’s have the lowest possible latency and are written in a performant way. The former depends on many factors, some being out of our control but the latter is in our very hands. In this post I discuss how I collected metrics for our Spring repositories to aid in optimization.

»
Author's profile picture 7-m on spring and db-metrics

Programmatic access to on-demand profiling in Java

When it comes to performance optimization, profiling an application is the only way to gain detailed insights into CPU, memory and IO utilization. Things are pretty straightforward when working locally. You run your app in your favorite ide, hook your favorite profiler to it and life’s good™. But, things soon get complicated when you want to profile apps deployed on another machine or in a container . Issues like firewalls, port forwarding, security, JMX etc. start cropping up and life isn’t as it used to be. This is what I’ll try to alleviate in this blog.

»
Author's profile picture 7-m on profiling

On Performance testing

Brief

  • What is performance testing
  • Deciding on the type of performance test
  • Deciding on tools and metrics
  • Creating a synthetic load
  • Collect metrics by running the synthetic load
  • Analyzing results and fixing bottlenecks
»
Author's profile picture 7-m on testing