On Frameworks ...

So I was reading zby's latest blog post on frameworks and libraries. On some level I agree with him that often times frameworks are basically just codified "convention", but I really don't see this as a bad thing, and here is why.

As I see it, the purpose of a library is to provide generic re-usable code that can be used in many different contexts. The purpose of a framework is to provide a set of guidelines/conventions/best-practices which a developer can then build upon, therefore avoiding the need to set those guidelines/dictate those conventions/determine those best practices themselves.

Some frameworks are more opinionated and complex (RoR, Catalyst, etc) and push/force you into their way of thinking, the benefit being that they also provide you with lots of building blocks that often times are zero-conf. While others (Dancer, Web::Simple, Mojolicious::Lite, etc) are more focused and purposefully simple, they leave many problems unsolved therefore allowing more freedom for the developer. Each of these approaches has merit and neither is truly superior in all contexts, which brings me to Plack and why I think this explosion of "frameworks" is a really good thing.

The ubiquity of Plack means that all these frameworks can be (fairly) easily used within the same application (or set of applications). It should (eventually) be very easy to have your Catalyst application running next to your Dancer application, running next to your WebNano application, all sharing the same session data, user information, etc. through Plack::Middleware components and mounted under a single Plack::App::URLMap and controlled easily using plackup.

Perl Oasis Slides

So I finally found some time this weekend to annotate my slides from OPW and get them uploaded to the interweb. The theme of the talk was two fold. First was that we all stand on top of the shoulders of those who came before us and that Open Source is a great context in which to do that, both when building on top of computer science concepts and when borrowing lessons learned from other language communities. The second half the talk was about my recent experiments using Plack (which is the Perl Web Server, in case you hadn't heard), Bread::Board, Path::Router and Moose, which I combined together as an experiment I call OX.

Plack - The Perl Web Server

Wow, in less then a week it seems that Plack has climbed to number seven in Google search results for Perl Web Server. It is only a matter of time before the hivemind that is Google realizes what we already know, that Plack is *the* Perl Web Server for the new millennium!

Perl Oasis 2010

So this weekend I went down to the Perl Oasis (a.k.a - Orlando Perl Workshop, or OPW for short), which is a small one day conference run by Chris Prather and his wife Jamie. I had missed it last year, but I heard so many good reviews of it I decided that this year I couldn't miss it, and I am really glad I didn't because I had a really great time.

There were lots of great people, great talks and great conversation. Some highlights include miyagawa's Plack talk (Plack really should be the top search result for "Perl Web Server"), Lucas Oman gave a really interesting talk about trying to implement Roles in PHP, the always entertaining Matt Trout gave a talk in the form of an epic poem and many more.

I find that I really enjoy the small and short conferences, they are usually on a weekend (less work juggling to do) and everything is a group activity enjoyed by all (much easier to coordinate activities when you don't have 300+ attendees). This conference, along with the always fun Pittsburgh Perl Workshop are now both on my "can't miss" list.

Re: Re: Re: Moose Or No Moose

This has been a crazy week here in Moose-land as you might have read. I would like to attempt to sum this all up so that we can all move on and get back to programming (which is sooooooo much more fun then blogging).

Adam Kennedy responded to all this in his recent use.perl post, and then later came into the #moose IRC channel to discuss. In that post I think that he did an excellent job of answering the "Should Padre use Moose" question. His conclusion seemed to be "Not Yet" which I can respect given his well thought out reasons. I was also happy to see that Adam felt Moose had the "air of inevitability" about it, and he felt it was only a matter of time till the Moose community solved our startup time and memory efficiency issues.

For the record, I was never actually suggesting that Padre should immediately be re-written in Moose, but more that Padre and Moose might benefit from one another, and would those benefits perhaps outweigh the costs. For Adam (and I assume the rest of the Padre team) the answer was "Not Yet", that the costs were too high and that benefits not compelling enough for them.

Then this morning I arrived at my desk to see Daisuke Maki had posted his 2 cents about this debate. Two lines in Daisuke's post jumped out at me. The first was this one:

I'd really like them [the Moose develoers] to make a more thought-out response, and act like you actually care about the parties involved that have a slightly different view than yours

And the second was:

That will make my life easier. I can point people to the Moose roadmap or whatever and say, "look, they care. they /will/ address your concerns. let's just install Moose in your system for now, yeah?" and go on hacking with Moose.

So while I don't have any official roapmap yet, I will say that "we are working on it" and "we do care". If nothing more, this recent debate has served to light a fire under the Moose development team and forced us to take a closer look at ourselves. It is a challenge and one that we are ready to take on.

The first step is to admit that you have a problem.

So last night (after a challenge from Adam on IRC) Jesse (doy) Luehrs added memory usage stats to Shawn (Sartak) Moore's original script which produced this data and then Cory (gphat) Watson used his Chart::Clicker module, which is written in Moose ;) to produce this graph:

The purple line is startup time and the yellow line is memory usage. (As I pointed out before, that nice downward drop at the very end there is a direct result of the Japan Perl Associations sponsorship of Goro Fuji).

The second step is to do something about it.

A big part of the memory and CPU hit is from the meta-layer. These are all the objects that Moose has to create for your class, and then all the code Moose compiles and evals to make sure things are fast at runtime. This is largely a one-time (compile time) cost and if you never call the "meta" method on your Moose class then you never use it again. We currently have two projects going on to help reduce or even eliminate some of these costs.

A recent post by fREW Schmidt announced that he is currently working to refactor the Moose test suite in such a way that the meta-level tests are optional. This is to help make way for Matt Trout's proposed "Antlers" project, which aims to "compile" Moose classes such that the meta-layer (the part that consumes most of the memory and takes so much time at startup) is not loaded or used unless you actually specifically request it by calling the "meta" method itself.

This idea has been kicking around the Moose team's heads for a while, in fact it is an evolution/mutation of Yuval Kogman's MooseX::Compile module. When Yuval wrote his version over a year and a half ago, Moose did not have the proper features internally and so the project was shelved. Now we feel that Moose is ready, and with fREW's test suite refactoring, it should only be a matter of tuits before this can become a reality.

The second is a combination of Goro Fuji's work and some XS-ification that Yuval Kogman, Florian (rafl) Ragwitz and a few other Moose hackers have been doing. The aim of this is to eliminate much of the need to do the code evaling for accessor and constructor generation by moving them to XS. Since we don't want you to pay for any more features then your using, Moose currently will compile only as much accessor and constructor code as you need and eval it into existence (which gets expensive). It is the hope that by moving that code to XS we can eliminate the speed penalty of un-used features and therefore not need to custom compile anymore.

The third step is to recruit new cult members.

Actually I am not sure if that is the 3rd step or not, but it sounds good to me. So if you love Moose and have some spare tuits lying around, please come and help pick some low hanging fruits and make this a reality.

Moose Startup Time over Time

I actually owe marcus an apology.

You were correct in that Moose startup time has not gotten significantly faster since 0.01. After our exchange Sartak decided to actually take a look and see. So he wrote this script, which produced this data, which jhannah then promptly turned into this graph (reproduced below).

It seems that since the 0.15 release we have mostly stayed within the 0.2 to 0.3 second range. It is interesting to look at the Changelogs for both Moose and Class::MOP you can actually see the feature additions or refactorings that correspond with the peaks and valleys.

Some of the most recent speedup is the direct result of Goro (gfx) Fuji's work on Class::MOP and Moose as sponsored by the Japan Perl Association. Much thanks to all involved in that.

Re: Re: Moose Or No Moose

Chris Prather recently asked "who is generally complaining about the slings and arrows of outrageous dependencies", which got me thinking (and I will get back to that thought in a moment).

Dave Cross also left a comment on my last blog post clarifying the relationship between his Array::Compare module and Test::Warn. Seems that the author of Test::Warn actually removed the usage of Array::Compare in a dev release several months ago, but simply forgot to remove it from the META.yml dependency listing. While I still think it is important to look downstream before you port to Moose, this clarification got me thinking as well.

Next comes Jonathan Rockway and his emacs fetish. Jonathan left a comment on Dave's blog which if you didn't know Jonathan, might be mistaken for an attempt to ignite the eternal Emacs vs. VI flamewar. But if you look beyond the emacs fanboi-ness of the comment you will see that Jonathan is (as usual) really making an excellent point about Padre, its extensibility as an editor, its dependencies and the choices its authors are making in relation to those things. This got me thinking about what Moose might be able to bring to the Parde party, good or bad.

So, now for my thoughts ...

To answer Chris's question, there are at least the two people who commented in the RT bug. The first is Adam Kennedy, who loves all things fast and tiny and has long complained about Moose's appetite for CPU and memory. And the second is Mark Stosberg who is a big advocate of vanilla CGI on which he and I have long disagreed. It is not surprising that neither of them like Moose. Are they representative of some kind of majority? Or just religious extremists worshipping at the altar of slim computing? It is hard to say, they both have valid concerns, but history and trends are pretty clear in how bloated software and ever improving hardware are constantly pushing one another forward (Moore's Law FTW).

So, as I said before, I agree-to-disagree with Adam and on his points, but the more I think about it, I think perhaps he is just knee-jerking here and possibly even using RT in anger. In the RT bug he says

This adds a huge amount of additional dependencies

Now sure this is true in relation to Array::Compare, but we all know that Adam is really talking about Padre. This is a little ridiculous considering Moose (on 5.10.0) has only 21 depenencies, 6 of which are core modules and has a 92% chance of installing (it would be 100%, but List::MoreUtils is dragging us down). While Padre (also on 5.10.0) has 104 dependencies, 32 of which are core modules and only has a 24% chance of installing (and requires a threaded Perl as well). So really how many dependencies would Moose add to Padre? Of the non-core dependencies they share 5 dependencies, so 21 - 6 core - 5 shared = 10, which when added to Padre's 62 non-core deps brings the grand total to 72.

Now, you might say "Wait a minute, thats a non-trivial amount of additional dependencies", and in fact you would be correct. However, the likelyhood that Moose is already installed is fairly high and growing by the day as it's popularity increases and already popular modules like Catalyst start using it. You see, Moose is not like other CPAN modules which simply provide a specific feature for a specific need, but instead is a tool to extend the language itself and which you use to create other CPAN modules. Because of this, the chance of having Moose already installed, either directly or because you install one of the approximately 600+ modules that use it, is pretty high.

Adam's next point about startup time is valid, this has long been an issue with Moose. We have made great strides on this issue over the years, but there is still a ways to go. How much this truly affects Padre though is debatable, as Jonathan Rockway pointed out, how many times a day/week/month are you really going to be starting up your text editor? Is startup time for a featureful IDE with possibly many plugins really that big of a deal? It certainly hasn't seemed to slow down the adoption for things like Eclipse.

And finally, to Adam's last comment ...

If this can't be fixed, we're going to need to remove Array::Compare (and everything that uses it) from Padre

this was actually the part that concerned me most because of what Adam is saying to all the authors of Padre's 62 non-core dependencies. While I do believe CPAN developers should look downstream before adding dependencies in the spirit of being a good neighbor, I think it is a little unreasonable for those upstream to dictate what is and is not an acceptable dependency.

Sure Dave's Array::Compare is a straightforward module and the benefits Moose brought to it were fairly minimal. But there are some pretty complex modules in Padre's dependency list, many of which could probably benefit greatly from Moose. And what about Padre plugins? Are they allowed to use Moose? What if they become popular plugins and the Padre folks want to merge them into the core? Do they have to shed the Moose before being allowed in? Where does this insanity end!?!?!

See, Moose is growing in popularity a lot lately. The number of CPAN modules that use Moose have been growing at a very steady pace. The #moose@irc.perl.org channel regularly has over 200 people in it. If you read the Perl Ironman blogs, you have no doubt seen a lot of Moose there lately. The number of Moose related questions on Perlmonks has been increasing lately (I should know I answer many of them). There were 16 talks at YAPC::NA this year tagged with Moose (including a 6 hour Moose course) which made Moose one of the largest tags in the cloud. This was followed by 7 talks at YAPC::EU this year.There will be approx 5 or so talks at the upcoming YAPC::Asia (including the Moose course again given by Sartak). Dave Rolsky will be giving a Moose course at the Italian Perl Workshop. Moose is also central to the EPO's extended core effort. We even get a fair amount of twitter traffic (for whatever that is worth).

Because of this, I really think Moose has proved itself not to be just a fad, especially considering this momentum has been steadily growing over 3 and a half years now. While it might be reasonable of Adam to request upstream deps not use Moose right now, how much longer before this becomes a real problem for Padre? Will the culture of NIH set in? Should Padre embrace the Moose future now? Has anyone even benchmarked/profiled how much of an impact Moose would have on Padre?

So Internets, what do you think?