John Long has just written a great piece about a possible directory structure overhaul to make Rails more modular. Discussion is raging over on rails-core, and the response has been polite but lukewarm. I think two results are inevitable. First, rails core will reject this idea because it diverges too much from the original vision behind Rails. Second, the goal of Rails modularization will continue to be pursued outside of core by projects such as Engines, and of course less revolutionary patches that further this goal will continue to be accepted.

What’s the problem with modularization?

High-level components have been debated ad nauseam. David’s stance is that it quickly becomes more trouble than it’s worth. As a programmer, designer, and perfectionist I totally agree. John Long’s proposal addresses primarily the distribution issue of component integration, but the elephant in the room is the logical integration; how will the code work together?

Rails has never addressed this problem; mostly because there’s no obvious solution. Rails encapsulates a well known design pattern, and distills down 10 years of web development best practices into an extremely usable framework. The beauty of Rails comes from the deep understanding of a problem domain.

If you’re developing a web application, there’s a good bet you’ll need what Rails is cooking. Your needs may be simpler or more complex, but chances are you’re still using the same type of functionality Rails provides.

The difference with high-level components is that they are so much more specific. It’s not enough to say, “I need components for my website.” You have to ask all kinds of difficult questions about how the components interact and fit together within your infrastructure. A successful component system needs to agree in advance on a whole slew of complex issues.

But if you’re developing a custom greenfield application with a focus on usability, then you need the freedom to answer these questions any way you want. 37Signals can’t afford compromises in their application architecture. Fortunately they don’t have to compromise because they’re building applications for thousands of customers rather than scraping together a razor-thin budget from a single client. Is it any wonder they transitioned from a consulting firm to hosted solutions company?

So what about successful modular systems?

I think David goes a little far with his assertion that “By the time [high-level components] become interesting, their fitting will require more work than creating something from scratch.” There are plenty of successful general purpose systems built using a modular architecture. My favorite by far is Drupal.

Drupal is a great example because it has a well thought out architecture designed to be customized by programmers, while still being useful to non-programmers right out of the box. Say you want to build a site that has a blog, photo gallery, and forum. These are all fairly standard needs. Millions of people want to build websites with these and other common features. With Drupal they are all modules, and you can install them and be up and running in minutes. Sure, it may not be custom designed and optimized to an exact business goal, but it all works, and it’s all integrated. In Rails it’s gonna take a lot longer; even if you find individual apps that do everything you need, good luck integrating them.

What makes Drupal interesting is how far it goes out of its way to maintain flexibility. Aside from the configurable options supplied by each module, the entire core system uses a series of hooks that allow interaction between modules. The result is that you can write a module that modifies the forms and data types of other modules, re-arranges menus and navigation, and a whole lot of other black magic. It really mitigates the functionality lock-in that comes with using a pre-packaged application.

In order to make this all work in a suitably generic fashion, Drupal imposes conventions, lots and lots of conventions. The most fundamental decision was that every data type is considered a node. Supplemental data types might have their own database tables, but they all are represented in the nodes table so they can be uniformly processed and hooked into by other modules. Other examples are the predefined page template areas, and navigation hook. You’re not just adding links to a template here, basically every aspect of development goes through some predefined structure.

This approach is extremely powerful, but of course comes with significant downsides:

• There’s always overhead for features and functionality you don’t need.
• Because everything is happening through an impersonal set of hooks, database chattiness can become an intractable problem.
• For every customization that is made easier, something is made harder, often to the point of utter impracticality.
• Estimating the cost of customization is often very difficult.
• Rarely will any module be exactly perfect, compromise is the rule not the exception.

Why modular development is ultimately necessary.

If you want to create the best web application with the best interface, high-level components won’t get you there. As programmers and interface designers, our focus is on perfection; someone else’s vision of a different (more general) problem is never going to be a perfect fit.

However most needs in the world aren’t unique, and a cheap “good enough” solution is actually more useful to the majority. If there are 10,000 people out there with the budget and vision to create a really kickass web app, there are 10,000,000 people who just want a website that does a few common things. For them, the difference between a brilliantly optimized and usable application, and a vanilla PhpBB forum is negligible. The real value to the user has very little to do with the icing on the cake of usability, but rather the underlying communication or functionality. A brilliant or horrible interface will impact usage, but users will muddle through and quickly acclimatize to a poor interface if they perceive value—MySpace being the canonical example. That thought is a definite buzzkill for us passionate web people. If we’re lucky enough to choose our projects then we may able to avoid the unpleasant truth, but we’d do well not to ignore it.

So how does this fit in with Rails?

Over the past couple years, Rails has proven itself as a better way to build websites. It sets the standard. Even though I still use Drupal or other PHP apps, I’m always secretly wishing some of its internals were as smooth as Rails. I’ve managed to work myself into a position of primarily working with Rails, but that doesn’t change the economics of the industry. The majority of web projects out there are still budgeted under $10,000, and that doesn’t buy you a lot of custom work, Rails or not.

Currently the right tool for these jobs isn’t on Rails or even Ruby—even if you want to use one of the excellent open-source Rails apps out there, you’re looking at an expensive memory footprint for what might only be one aspect of a website. You’ll probably end up on PHP, ASP or possibly Java. But it’s still all web apps. A modular system would sit very nicely on top of Rails. Given the passion in the Rails community, it seems inevitable that such projects will spring up and eventually see widespread adoption.

Rails core is not the place for these projects to live, but they will inspire core patches. Opinionated or not, the core team are reasonable people. They will make concessions to help meet these demonstrated needs, even if they have no direct interest in that style of development.

So where does John Long’s latest idea fit into all this? I’m not sure. Philosophically it just seems too far removed to fit into rails core. But it also points to a poorly addressed need of the Rails community. Now that Rails has been around a while, I think some of the heated rhetoric can be left behind. Rails is infrastructure, so it will work fine as the foundation for a modular system. I have no idea what that might look like, but with more developers learning Rails every day, it’s only a matter of time.