Re: Re: Re: Moose Or No Moose

This has been a crazy week here in Moose-land as you might have read. I would like to attempt to sum this all up so that we can all move on and get back to programming (which is sooooooo much more fun then blogging).

Adam Kennedy responded to all this in his recent use.perl post, and then later came into the #moose IRC channel to discuss. In that post I think that he did an excellent job of answering the "Should Padre use Moose" question. His conclusion seemed to be "Not Yet" which I can respect given his well thought out reasons. I was also happy to see that Adam felt Moose had the "air of inevitability" about it, and he felt it was only a matter of time till the Moose community solved our startup time and memory efficiency issues.

For the record, I was never actually suggesting that Padre should immediately be re-written in Moose, but more that Padre and Moose might benefit from one another, and would those benefits perhaps outweigh the costs. For Adam (and I assume the rest of the Padre team) the answer was "Not Yet", that the costs were too high and that benefits not compelling enough for them.

Then this morning I arrived at my desk to see Daisuke Maki had posted his 2 cents about this debate. Two lines in Daisuke's post jumped out at me. The first was this one:

I'd really like them [the Moose develoers] to make a more thought-out response, and act like you actually care about the parties involved that have a slightly different view than yours

And the second was:

That will make my life easier. I can point people to the Moose roadmap or whatever and say, "look, they care. they /will/ address your concerns. let's just install Moose in your system for now, yeah?" and go on hacking with Moose.

So while I don't have any official roapmap yet, I will say that "we are working on it" and "we do care". If nothing more, this recent debate has served to light a fire under the Moose development team and forced us to take a closer look at ourselves. It is a challenge and one that we are ready to take on.

The first step is to admit that you have a problem.

So last night (after a challenge from Adam on IRC) Jesse (doy) Luehrs added memory usage stats to Shawn (Sartak) Moore's original script which produced this data and then Cory (gphat) Watson used his Chart::Clicker module, which is written in Moose ;) to produce this graph:

The purple line is startup time and the yellow line is memory usage. (As I pointed out before, that nice downward drop at the very end there is a direct result of the Japan Perl Associations sponsorship of Goro Fuji).

The second step is to do something about it.

A big part of the memory and CPU hit is from the meta-layer. These are all the objects that Moose has to create for your class, and then all the code Moose compiles and evals to make sure things are fast at runtime. This is largely a one-time (compile time) cost and if you never call the "meta" method on your Moose class then you never use it again. We currently have two projects going on to help reduce or even eliminate some of these costs.

A recent post by fREW Schmidt announced that he is currently working to refactor the Moose test suite in such a way that the meta-level tests are optional. This is to help make way for Matt Trout's proposed "Antlers" project, which aims to "compile" Moose classes such that the meta-layer (the part that consumes most of the memory and takes so much time at startup) is not loaded or used unless you actually specifically request it by calling the "meta" method itself.

This idea has been kicking around the Moose team's heads for a while, in fact it is an evolution/mutation of Yuval Kogman's MooseX::Compile module. When Yuval wrote his version over a year and a half ago, Moose did not have the proper features internally and so the project was shelved. Now we feel that Moose is ready, and with fREW's test suite refactoring, it should only be a matter of tuits before this can become a reality.

The second is a combination of Goro Fuji's work and some XS-ification that Yuval Kogman, Florian (rafl) Ragwitz and a few other Moose hackers have been doing. The aim of this is to eliminate much of the need to do the code evaling for accessor and constructor generation by moving them to XS. Since we don't want you to pay for any more features then your using, Moose currently will compile only as much accessor and constructor code as you need and eval it into existence (which gets expensive). It is the hope that by moving that code to XS we can eliminate the speed penalty of un-used features and therefore not need to custom compile anymore.

The third step is to recruit new cult members.

Actually I am not sure if that is the 3rd step or not, but it sounds good to me. So if you love Moose and have some spare tuits lying around, please come and help pick some low hanging fruits and make this a reality.

Moose Startup Time over Time

I actually owe marcus an apology.

You were correct in that Moose startup time has not gotten significantly faster since 0.01. After our exchange Sartak decided to actually take a look and see. So he wrote this script, which produced this data, which jhannah then promptly turned into this graph (reproduced below).

It seems that since the 0.15 release we have mostly stayed within the 0.2 to 0.3 second range. It is interesting to look at the Changelogs for both Moose and Class::MOP you can actually see the feature additions or refactorings that correspond with the peaks and valleys.

Some of the most recent speedup is the direct result of Goro (gfx) Fuji's work on Class::MOP and Moose as sponsored by the Japan Perl Association. Much thanks to all involved in that.

Re: Re: Moose Or No Moose

Chris Prather recently asked "who is generally complaining about the slings and arrows of outrageous dependencies", which got me thinking (and I will get back to that thought in a moment).

Dave Cross also left a comment on my last blog post clarifying the relationship between his Array::Compare module and Test::Warn. Seems that the author of Test::Warn actually removed the usage of Array::Compare in a dev release several months ago, but simply forgot to remove it from the META.yml dependency listing. While I still think it is important to look downstream before you port to Moose, this clarification got me thinking as well.

Next comes Jonathan Rockway and his emacs fetish. Jonathan left a comment on Dave's blog which if you didn't know Jonathan, might be mistaken for an attempt to ignite the eternal Emacs vs. VI flamewar. But if you look beyond the emacs fanboi-ness of the comment you will see that Jonathan is (as usual) really making an excellent point about Padre, its extensibility as an editor, its dependencies and the choices its authors are making in relation to those things. This got me thinking about what Moose might be able to bring to the Parde party, good or bad.

So, now for my thoughts ...

To answer Chris's question, there are at least the two people who commented in the RT bug. The first is Adam Kennedy, who loves all things fast and tiny and has long complained about Moose's appetite for CPU and memory. And the second is Mark Stosberg who is a big advocate of vanilla CGI on which he and I have long disagreed. It is not surprising that neither of them like Moose. Are they representative of some kind of majority? Or just religious extremists worshipping at the altar of slim computing? It is hard to say, they both have valid concerns, but history and trends are pretty clear in how bloated software and ever improving hardware are constantly pushing one another forward (Moore's Law FTW).

So, as I said before, I agree-to-disagree with Adam and on his points, but the more I think about it, I think perhaps he is just knee-jerking here and possibly even using RT in anger. In the RT bug he says

This adds a huge amount of additional dependencies

Now sure this is true in relation to Array::Compare, but we all know that Adam is really talking about Padre. This is a little ridiculous considering Moose (on 5.10.0) has only 21 depenencies, 6 of which are core modules and has a 92% chance of installing (it would be 100%, but List::MoreUtils is dragging us down). While Padre (also on 5.10.0) has 104 dependencies, 32 of which are core modules and only has a 24% chance of installing (and requires a threaded Perl as well). So really how many dependencies would Moose add to Padre? Of the non-core dependencies they share 5 dependencies, so 21 - 6 core - 5 shared = 10, which when added to Padre's 62 non-core deps brings the grand total to 72.

Now, you might say "Wait a minute, thats a non-trivial amount of additional dependencies", and in fact you would be correct. However, the likelyhood that Moose is already installed is fairly high and growing by the day as it's popularity increases and already popular modules like Catalyst start using it. You see, Moose is not like other CPAN modules which simply provide a specific feature for a specific need, but instead is a tool to extend the language itself and which you use to create other CPAN modules. Because of this, the chance of having Moose already installed, either directly or because you install one of the approximately 600+ modules that use it, is pretty high.

Adam's next point about startup time is valid, this has long been an issue with Moose. We have made great strides on this issue over the years, but there is still a ways to go. How much this truly affects Padre though is debatable, as Jonathan Rockway pointed out, how many times a day/week/month are you really going to be starting up your text editor? Is startup time for a featureful IDE with possibly many plugins really that big of a deal? It certainly hasn't seemed to slow down the adoption for things like Eclipse.

And finally, to Adam's last comment ...

If this can't be fixed, we're going to need to remove Array::Compare (and everything that uses it) from Padre

this was actually the part that concerned me most because of what Adam is saying to all the authors of Padre's 62 non-core dependencies. While I do believe CPAN developers should look downstream before adding dependencies in the spirit of being a good neighbor, I think it is a little unreasonable for those upstream to dictate what is and is not an acceptable dependency.

Sure Dave's Array::Compare is a straightforward module and the benefits Moose brought to it were fairly minimal. But there are some pretty complex modules in Padre's dependency list, many of which could probably benefit greatly from Moose. And what about Padre plugins? Are they allowed to use Moose? What if they become popular plugins and the Padre folks want to merge them into the core? Do they have to shed the Moose before being allowed in? Where does this insanity end!?!?!

See, Moose is growing in popularity a lot lately. The number of CPAN modules that use Moose have been growing at a very steady pace. The #moose@irc.perl.org channel regularly has over 200 people in it. If you read the Perl Ironman blogs, you have no doubt seen a lot of Moose there lately. The number of Moose related questions on Perlmonks has been increasing lately (I should know I answer many of them). There were 16 talks at YAPC::NA this year tagged with Moose (including a 6 hour Moose course) which made Moose one of the largest tags in the cloud. This was followed by 7 talks at YAPC::EU this year.There will be approx 5 or so talks at the upcoming YAPC::Asia (including the Moose course again given by Sartak). Dave Rolsky will be giving a Moose course at the Italian Perl Workshop. Moose is also central to the EPO's extended core effort. We even get a fair amount of twitter traffic (for whatever that is worth).

Because of this, I really think Moose has proved itself not to be just a fad, especially considering this momentum has been steadily growing over 3 and a half years now. While it might be reasonable of Adam to request upstream deps not use Moose right now, how much longer before this becomes a real problem for Padre? Will the culture of NIH set in? Should Padre embrace the Moose future now? Has anyone even benchmarked/profiled how much of an impact Moose would have on Padre?

So Internets, what do you think?

Re: Moose Or No Moose

Moose love controversy, in fact you could say that they are natures attention whores (second only to those nasty hairless apes that seem to be everywhere!). If they aren't endorsing controversial politicians, having kinky sex or crashing expensive cars then you can likely find them partying all night at the hottest clubs. And of course all this attention leads not only to a lot of imitation but even some haters.

The same can be said of the Moose perl module (well, the controversy part anyway, modules can't have sex or drive cars, DUH!). The latest controversy is surrounding a blog post by David Cross in which he tells the story of a RT "bug" report he got after porting his Array::Compare module to Moose.

The bug report was submitted by Adam Kennedy who has a well known fetish for all things tiny. While I do not share Adam's love for CPU/Memory conservation, I do respect his efforts in trying to keep some of the critical tools of our beloved Perl infrastructure slim and easily installable. And in fact, in this case I do have to agree with Adam, before you port an existing module to Moose it helps to look downstream a little.

The benefits Moose provides for a module as straightforward as Array::Compare are actually pretty minimal, it is only using a few simple attributes and not much more. But Moose does come with a well known startup cost (don't let the haters tell you we are slow, once we get up and running things are plenty fast enough and getting faster), which Array::Compare was passing onto all its dependencies. One of those (now former) dependencies was Test::Warn which itself has 101 modules that in turn depend on it in their test suites. The result is that this simple change with minimal overall benefits has imposed a cost on a sizable chunk of the CPAN.

Now you could argue that Padre - The Perl IDE (the project that initially sparked all this) is an IDE and like other IDEs (Eclipse, Visual Studio, etc) it should take forever to start up so as to allow developers a leisurely morning coffee break. Or you could argue that it is Dave's module and he can do whatever he wants with it, downstream dependencies be damned. You could even say that Moose will eventually be fast enough so people should stop whining and just be patient. But in the end, we (CPAN) are a large and heavily interdependent community and I believe we should be respectful to our neighbors as much as possible.

Anyway, Nuff Said,... Peace out to all mah homeys attending YAPC::Asia, wish I could have come this year.