Plumbing 101 - Fixing my leaky Intertubes

So this will be a short one, but I promised Yuval I would write it up.

My recent big refactoring project at work has been largely about switching a mad-cap collection of YAML config files, database tables, poorly inferred relationships and other sorts of random insanity that accumulates over 5 years of maintaining and extending an application, and inverting this unmaintainable mess into a nice clean happy KiokuDB based set of objects.  

So all is going well until the other day when I upped the number of FCGI backends and the concurrency exposed  a leak in my objects which was not visible when using only the 1 FCGI backend for development. So the first thing I did was to turn to my unit tests and see if I could detect the leak outside of my web-app using Test::Memory::Cycle. Of course I didn't find one there, that would have been too easy, it was buried deeper inside the web-app. So I then enlisted Yuval for help since his knowledge of the Perl guts far exceeds what mine will ever be. 

Unfortunately for me, this application pre-dates Catalyst so I could not use Catalyst::Model::KiokuDB, but I was able to cargo-cult the core of that module and stuff it into the homegrown web framework that this did run on. So while this didn't solve the problem it did allow me to watch the problem happen and gather nice statistics.  I can't stress how important it is to get the your monitoring tools set up first so your not digging through false positives and/or useless information. Yuval and I spent a fair amount of time tweaking this until we finally found the right settings that gave us the perfect balance of information. 

After this it was a lot of playing around with Devel::FindRef and Scalar::Util::weaken until I found the right settings and all my leaks were gone. One particularly evil leak was closures passed into Template::Toolkit params that referenced themselves inside the closure. This resulted in the entire template object leaking, including all associated parameters (in one case this meant hundred of objects). I realized as I was fixing this, that this leak (and a few others) had existed for probably the entire 5 years this application has been running. Only now that it was leaking KiokuDB objects and causing visible issues did I actually notice. 

So while I don't feel that I am now some kind of Master Plumber or anything, I do feel confident enough to fix a leaky intertube here or there. And as always I am amazed by the flexibility of Perl and the wonders that CPAN provides.  It is an odd combination of pride and shame to have finally cleaned up 5 year old leaks. 

Anyway, back to work ...