Hack Drupal core!

No better title to get everyone's attention - well, any Drupal developer out there who once pledged never to touch any code of core when running sites in production. I used to be one of them, but I stepped over to the Dark side recently.

For a fairly large community site (think OG, node access control, private messages etc), we were getting in trouble giving authenticated users a decent browsing experience. The worst page was taking 16 (yes, sixteen) seconds to load and others were getting in trouble too now that the number of users and nodes was growing rapidly. Not acceptable of course, so we went searching for solutions:

  • Advcache: the advcache module comes with a few patches but we only applied the one that caches URL aliases per page. While drupal_lookup_path is generally fast, it's sometimes scary to see it appear 300 times in the query log for one page. With cache_get('cache_path') you suddenly feel much better.
  • Memcache: it's possible to swap the default caching implementation, so this seemed like a good solution. It took me a few clicks to realize that it actually makes loading pages extremely faster and seeing another 100 queries less, no further discussion. Since this is the first time we're using it, we opted to use memcache.db.inc which still writes the cache to the database so in case memcache dies we still have a fallback.
  • Block cache alter: I'm biased since I actually maintain the module and I don't feel any pain hacking block module :)
  • Rewriting one query generated by views that took around 11 seconds to give back its results. We analyzed what data we wanted and wrote our own block implementation. I'm not critising views here, in fact, we still have a lot of views pages and blocks in the site which run perfectly fine. It's up to a developer to take a good look at the mysql logs and see which ones are getting in trouble and fix them if needed.

After every step, done in the same order listed above, we recorded the numbers that Yslow gave us which you can see in this picture. I still regret not logging the number of queries that dropped after every step; some pages which sometimes needed 600 queries now only need 60-70 db hits to generate the page for an authenticated user. The conclusion is that our server now has less load, pages are generated faster so we have a happy user now - and hacker too in my case.

I hope I don't have to apply patches anymore though in D7, so I'll start following http://drupal.org/node/367257 (and swappable path.inc and handlers) even more closely than I did right now - we all should!

AttachmentSize
Yslow results79.46 KB

Comments

Submitted by Brian Vuyk on April 17, 2009 - 22:13

My only concern would be remembering to reapply all the patches you've added yourself when you upgrade. Perhaps it is worth grabbing core via a cvs checkout and having it auto-merge the patches you've added with core.

Submitted by grendzy on April 17, 2009 - 22:55

The CVS method is a good one. You also need a solid process for documenting and maintaining your patchsets.

I also have few qualms about applying bug fixes that haven't yet been committed or released to core (provided I've tested them carefully). Here are two that make it into most of my sites:

For core enhancements, you're right that we need to be involved in the D7 queue to fix whatever required hacking in the first place.

Submitted by dalin on April 19, 2009 - 12:17

For all of our hacks (core or contrib) we document the rationale in /sites/all/hacks/hacklist.txt . We also generate patches for each hack that live in /sites/all/hacks . We use CVS to install everything (and store the whole codebase in SVN). This makes things much more maintainable.

Submitted by Khalid on April 19, 2009 - 18:18

Scaling of Drupal based intranets is becoming more of a challenge as Drupal is becoming more popular and it is widely used in such applications where all the users are logged in. In such cases, we find that the scaling is possible by eliminating bottlenecks. We have an in house setup where we can subject a copy of the client's web site to stress with simulated load of logged in users, and find where the slowness is.

For custom patches, upgrading within 6.x (or 5.x) is possible if you use our procedure to upgrade Drupal via automatic patches generated from official releases.

Skip to the bottom of the article and use the shell script at the bottom.

This will preserve any patches you have, yet keep you using the update module as well.

You are here