No better title to get everyone's attention - well, any Drupal developer out there who once pledged never to touch any code of core when running sites in production. I used to be one of them, but I stepped over to the Dark side recently.
For a fairly large community site (think OG, node access control, private messages etc), we were getting in trouble giving authenticated users a decent browsing experience. The worst page was taking 16 (yes, sixteen) seconds to load and others were getting in trouble too now that the number of users and nodes was growing rapidly. Not acceptable of course, so we went searching for solutions:
- Advcache: the advcache module comes with a few patches but we only applied the one that caches URL aliases per page. While drupal_lookup_path is generally fast, it's sometimes scary to see it appear 300 times in the query log for one page. With cache_get('cache_path') you suddenly feel much better.
- Memcache: it's possible to swap the default caching implementation, so this seemed like a good solution. It took me a few clicks to realize that it actually makes loading pages extremely faster and seeing another 100 queries less, no further discussion. Since this is the first time we're using it, we opted to use memcache.db.inc which still writes the cache to the database so in case memcache dies we still have a fallback.
- Block cache alter: I'm biased since I actually maintain the module and I don't feel any pain hacking block module :)
- Rewriting one query generated by views that took around 11 seconds to give back its results. We analyzed what data we wanted and wrote our own block implementation. I'm not critising views here, in fact, we still have a lot of views pages and blocks in the site which run perfectly fine. It's up to a developer to take a good look at the mysql logs and see which ones are getting in trouble and fix them if needed.
After every step, done in the same order listed above, we recorded the numbers that Yslow gave us which you can see in this picture. I still regret not logging the number of queries that dropped after every step; some pages which sometimes needed 600 queries now only need 60-70 db hits to generate the page for an authenticated user. The conclusion is that our server now has less load, pages are generated faster so we have a happy user now - and hacker too in my case.
I hope I don't have to apply patches anymore though in D7, so I'll start following http://drupal.org/node/367257 (and swappable path.inc and handlers) even more closely than I did right now - we all should!