Last week I uncovered a tricky performance problem on Scout I wanted to share.
In seemingly random fashion, when accessing my account home page, the render time was 5-6 seconds – way too slow. Other times, the render time would be far less than a second.
Here’s what I did to debug:
- Tailed my production log
- Opened up the mysql console
- Generated a slow request
- Ran the mysql SHOW PROCESSLIST command
Plain as day, I saw a query taking 30+ seconds. A simple mysql EXPLAIN on the query revealed a missing index. A new index fixed the issue. This query was outside the Rails application – a background job – so I never saw the query when optimizing the application.
Checkout the impact on the DB server load:
It’s a great lesson that you really need the complete picture of your infrastructure when something goes wrong – just looking at the Rails application often doesn’t cut it.
P.S. You’ll love what’s coming up with Scout – finding these problems gets a lot easier.