Towards a straight up Drupal development workflow for the working stiff – can't we all just follow industry standards?

I need to write an installation profile for Project Flow and Tracker in order to publish it right on http://drupal.org so that it can be contributed to the community. My goal however is to avoid coming up with one of those monolithic install profiles whose drupal core or modules and themes you can never update on your own without breaking everything, whose functionality you have to take as is on an all or nothing basis, and which is generally speaking unmaintainable and has to be replaced en bloc (sometimes en bric) whenever there is a new release (i.e. erase everything, copy in the new release, run update.php...).

The promise of Contexts and Features driven development was that you got to develop incrementally with everything in code (modular, versionable, automatically buildable), then with the help of Drush Make and the Install Profile API  you could make a modern installation profile which simply installs the features and their dependencies, and reverts them to make sure they are properly enabled. And that this cycle can be repeated transparently as the app evolves, as part of its development workflow.

But the reality is that features are the coolest thing in town, but once you knock on that mistress' door, you are imprisoned in her magic forever; and you don't wind up with reusable self-contained components for all your pains. In this article we are going to find out why that is and come to some conclusions as to what can be done about it (under the circumstances). I will present the problem and share conclusions as to how best practices can best be approximated in Drupal app development, and will illustrate conclusions I have come to as to the most sustainable method of creating an installation profile in the context of an ongoing application project.

I touched on this recently in Features driven development -- the nitty gritty and thoughts on the future of sustainable "in code" development, but now I want to systematize my ideas and figure out what to do about it.

The problem

The most important thing to consider is that Features as development workflow breaks a fundamental law in Software Developoment:

Software development law #1: there is no one-to-one relationship between use cases (features, functional requirements, user stories, etc.) and project configuration artifacts (modules, themes, menus, blocks, panels, settings, etc.). A single project artifact will serve as part of the implementation of many use cases; a single use case will be implemented in many project artifacts.

Now, the definition of "feature" given at http://drupal.org/project/features is "a collection of Drupal entities which taken together satisfy a certain use-case", implying that given another use case, there is another "feature" that can satisfy (implement, support) it. But this concept is fatally flawed in the context of any software app development, including that of Drupal apps, since if I package some functionality in a feature, any of the project artifacts contained therein are "imprisoned": if a content type forms part of the implementation of business objects contained in Use Case 1, and they are in Feature 1, even though the same content type is germain to Use Case 2, it cannot be included in Feature 2, because it's already in Feature 1. And the same goes for settings, permissions, etc. and things get even more complicated as the UI comes into play (blocks, contexts or panels and their contexts, panes, etc., obviously are "re-used" and exist as dependencies for multiple use cases).

This is why despite the fact that contexts, features and kits have been proposed and wildly acclaimed (by myself also) to solve the features dependencies problem (note: this highly informative article from the folks at  FunnyMonkey shows that the only (incomplete) solution is painstaking and artesanal features refactoring), the sad truth is that contexts and features becomes just another "wall of magic" (see comments and updates at foot of page of this very insightful article by Jan Ploski; although written earlier this year, it is still right on in its honest attempt to share actual dev workflow usage, which is very refreshing). Behind that wall of magic, you know something is happening, but you don't know what it is or even if you have it, while a team of rocket scientists need to be spending some quality features refactoring time, to come up with a semblance of self-contained re-usable and software configuration (version control and build) maintainable artifact sets packaged in modules and capable of being invoked by an installation profile.

This is true with or without the kit guidelines (the "library"?) and the monolithic "Build kit". Both support a very specific set of use cases, and set up rigid rules for what should and should not be considered part of the library kit, something which will actually vary enormously according to the app architecture itself. And the Build kit starts out by hacking Drupal 7 core before it's even been released and therefore really locks you in.

None of this can emerge as the fruit of an agile iterative and incremental team workflow, nor can it be popped back into a straightforward workflow as the app (and the install profile) evolves over time.

A very real problem, really

So, now that I wish to publish Project Flow & Tracker as a distribution profile right on http://drupal.org I find myself in a quandary which just should not be happening. Although there exist installation guidelines and several worthy examples to learn from (Innovation news, and the features based VoiceBox, Spaces 3 Demo, etc.) and even though I can install and enable contexts and features packaging modules from the install profile, the features I am invoking do not compartmentalize functionality in any reusable or maintainable manner; and Contexts and Features, which are great for many purposes and a godsend and a joy, do not provide any solution for the famous Drupal developoment workflow problem after all.

Conclusions and ToDo's

So what conclusions can I draw in order to publish an installation profile for Project Flow & Tracker as something honestly capable of emerging as the result of an agile process, as part of real everyday work which follows industrial app building standards so that it is built as much as possible out of reusable elements, and may be customized in a modular fashion and subject to ongoing maintenance?

My conclusions are that not only do I need to separate UI features from "core library" or business object features, I actually need to strip features right out of the app entirely and use existing exportable tools (Views Bulk export, Ctools Bulk export, Strongarm, etc.), the only exception being where currently features are the only way to export app elements, or else make it much more straightforward. And even then they would be project artifacts capable of supporting multiple use cases, and not a single "feature".

What are best practices in the software development industry anyway?

Best practices involve, along with the others mentioned throughout this article, bidirectional traceablity from business and use case model (captured requirements) all the way down to the chain of committed changesets, in order to be in a position to quantify socially useful work expressed as project artifact deliverables. In other words, given a user story, I should be able to visualize a log of committed changesets (definitely on the Project Flow & Tracker roadmap!). And I should be able to visualize dependencies using a build tool (I miss maven!).

Eureka: just export everything you can into project artifacts and use your version control system and build tool (and Project Flow & Tracker!) in order to achieve bidirectional traceability!

How can these best practices be applied to Drupal, specifically Project Flow & Tracker?

  1. Strip out all features from the project (easier said than done when in code and not in database).
  2. Rebuild functionality on the basis of modules built with exportables.
  3. Use Views bulk export and Ctools bulk export and CCK export and whatever you can find to get stuff into code artifact exportables, contained in one of the following:
    1. Installation profile.
    2. Module install script.
    3. Module update script.
    4. A minimalist feature if there's no other way to do it, or if it is after all is said and done, the most straightforward way to do it.
  4. These modules must contain (in their .info files) explicit and non-mutually exclusive dependencies.
  • That's why they cannot be features.
  1. Make sure these exportables based modules are as compatible as possible with the Install Profile API, and certainly invocable by an installation profile.
  2. Base ourselves basically on this workflow:

    http://nuvole.org/blog/2010/aug/24/features-based-development-workflow

    1. Recommended by Development Seed: "Nuvole's Andrea Pescetti explains how to integrate Features into your workflow"
    2. But from slide 21 on, actually explains a much better alternative (also used by the dev folks at economist.com) which works whether you are using features or not:
      1. Start out with project.make and project.profile
      2. Use hook_update_N() for incremental development in order to put configuration in code
      3. This can be done with teams.
      4. As suggested elsewhere, refactor your build script (project.makefile) often!
  3. Enhance this approach with automated tools which can export configuration artifacts into code autmatically and have it captured by the version control system.
    1. Generate the drush make file for the project directly from the Drupal instance.
    2. Generate install profile

Will report back on how I publish the Project Flow & Tracker installation profile!

I will certainly let you all know what I did, and also, what I plan on doing in the future for a working stiff's straight up web app dev workflow compatible with industry best practices.

Bibliography

Still reading through Victor,

Still reading through Victor, but just wanted to get this thought out -- my reaction to this:

"So what conclusions can I draw in order to publish an installation profile for Project Flow & Tracker as something honestly capable of emerging as the result of an agile process..."

I admit that we're having a tough time working with features, but we're capturing features as we work, and I can only imagine how difficult it would be to build the whole site, and THEN try to cram it into features. We've found that many of our approached to completing stories (ie. most obviously being which modules we use) are partially determined by the considering featurization during the process as part of our definition of done.

But hey, we're still working through projects with successive approximations of what we think might be the ideal, which would appear to be total featurization. Not there yet, that's for sure. And perhaps the last mile just isn't worth walking... it's hard to say.

Environment module to extend the site-as-install-profile further

Amazing summary Victor. Hadn't read your stuff in awhile, but came across you again in the #ows mailing list and came to see what was up. Seems we're heading down lots of the same trails.

Still need to catch up on the whole article, but I think one piece of the puzzle that you might find helpful is Grayside's Environment module (http://www.dgo.to/environment). Basically allows you to set a flag in settings.php, and define rules programmatically for enabling and disabling modules. Hoping to use it to control features while deploying the same "binary" (either tarball or deb package) to different environments in a build pipeline --say, with a feature set called feature_analytics_int, feature_analytics_stg and feature_analytics_prod. So an install profile isn't just a config definition of the whole site, but the whole site across all environments. So the commit to the install profile repo triggers a "build" (drush make and site-install), which is then packaged up in the first stage and stored in a repository (tarball in filesystem or deb in apt repo). A successful build in this commit stage triggers various testing stages separately (provision server, deploy site, run unit/acceptance/capacity tests, destroy), and if successful, deploys to another server, settings.php is symlinked in with --say-- a staging env flag, and appropriate features are disabled while others enabled by the environment module. This last bit happens on each successive environment right up to production, using all the same deploy scripts that have been tested extensively.

Anyhow, we're doing some exploration of these things for the Federated General Assembly development process, if you're interested :)

http://projects.occupytechnology.org/issues/86

-- patcon

This article cited in James Sansbury's Lullabot features article

Towards a straight up Drupal development workflow for the working stiff – can't we all just follow industry standards? has been cited in "The Features Module" ( http://www.lullabot.com/articles/features-module#comment-8876 ). He advocates an "educated" approach to features, which we here at awebfactory have actually been using also together with installation profiles.

Thanks for your article, James!

I agree

I agree with you Victor, we are currently building some sites at my company using make, profile and update hooks and I think it's the right way!

Great to know TN is using Drupal

Felicitaciones!

Excellent write-up

Thanks for taking the time to put together this article. It is extremely well-written and makes me think, as I am sure it will with other people, which is an important thing at this stage in Drupal's development.

unification

Very good article. Also, when working with a system where everything is already in code (Ruby on Rails in my instance), the development workflow is very simple:

  1. work on something
  2. commit

With the current state of Drupal, there's a large middle step that involves a lot of thinking and figuring out how to export things before committing them.

  1. work on a something
  2. make a mental list of all affected parts
  3. is this already a part of a "feature" or an exportable? then re-export it
  4. if it isn't, should it be a part of an existing feature or a new one? then add it and export it
  5. is it something that's not supported by exportables? then write an update hook or record a macro
  6. commit

So, there's a lot of extra effort required, which makes me want to do it less. Also, I'm never quite sure if I accidentally left out a variable or form submission. Ideally, as a developer I shouldn't be concerned with the middle steps. Maybe that's the way we should be heading for Drupal 8.x...

/drifter/

We need a Drupal workflow people can use

Thanks for your thoughtful comment.

On the question of there only being two steps upon using RoR or similar (Django, Cake, etc.), if we add bidirectional traceability in as an objective (given a use case, what is its impact on project artifacts (code, etc.)), we do need to be working from a project management system. Apart from which, even so, we need to be taking into account some kind of build system (Capistrano, Maven, etc.) if we are not only coding, but also delivering our work into some kind of deployment and/or change configuration system.

In other words, the version control system cannot perform all of your deployment/build tasks for you. For example, even back to a C Language development environment, the makefile has to be edited, dependencies checked, and a build system needs to make sure supporting libraries are installed.

So while we are at it, the extra steps for the Drupal workflow might be snuck in relatively cheap, if we get automated tools.

So, on the elimination of your steps 2-5, I am quite confident that with time the Drupal community can easily come up with the necessary automation tools (Chaos Tools Suite exportables being the core of what I hope will be a burgeoning area).

The future of Drupal as a development framework depends on it.

On the question of never quite knowing whether you left something out while carrying out all of the manual exporting, in her recent article, Sacha Chua proposes a test driven approach:

"Because we’ll be using Features to share our changes instead of working off SQL backups, I need to make sure that I’ve included all the relevant components in the features I create. One way to test that is to use Backup and Migrate to save my configured database (just in case!), load a previous backup, enable the feature, and confirm that everything works as expected."

See http://sachachua.com/blog/2010/10/drupal-features-and-drush-updating-our...

Well, I've pretty well established why features are inadequate for sharing changes in an agile, incremental and iterative approach (i.e. without considerable hand-made refactoring of the features files to manage dependencies), but I do agree with Sacha that an automated approach using Simple Test, Selenium and/or Hudson would work well, once it were documented and popularized so real dev workers will use it.

Again, thanks for your comments,

Victor Kane

"There is no problem that

"There is no problem that another layer of abstraction cannot solve" - unknown

Perhaps all that we need is to have a new thing called "configuration" (a view, variable, panel, content type, etc.). Configurations are what gets exported. A feature is a package of one or more configurations. A configuration can belong to one or more features.

P.S. Your CAPTCHA makes me think waaay too hard.

Drupal needs an apt-get type packaging system!

"There is no problem that avoiding the re-invention of the wheel cannot solve" - unknown.

Of course: We are going round in circles here: Drupal needs an apt-get type packaging system, where re-usable dependencies of any kind are cleanly separated from the functional package itself!

Victor

Excellent idea of configurations as building blocks of features

But... the key is for one configuration to be able to be included in more than one feature as a dependency. The code would be in the configuration, as you say, and its inclusion in a "feature" would be simply a pointer-like dependency. Cool.

What I am exploring right now is that changesets in a version control system are themselves configuration elements! So the dependency is recorded as a part of using the version control system and of using a project manager such as Project Flow & Tracker. Raw material for reports there at least. Then Project Flow & Tracker would connect these items with the user stories for traceability and for re-use.

In other words, I don't know that "features" should actually be like modules you enable in the system; but rather perhaps a re-usable "view" of configuration elements, a view which can exist in a building tool outside the system.

So I don't think there should be a "feature" element at all as something enabled in a system. Because even though it might sound nifty for little trucks to deliver functionality from vertical monopolies to people's sites, they are inherently unmaintainable in terms of their configurability with other parts of the system, versions, etc., they are often lock-in. It's a way of locking people in to monolithic development rather than offering tools for developers.

But if one wanted to, your idea would be a good one, since removing a feature wouldn't break other parts of the system.

Another thought: the "proper" way could be to capture this via UML diagrams (modeling) which could have some kind of "round trip engineering" to actually generate code, and integrate with code afterwards. Wow, that would be cool.

Anyway, fascinating discussion. Thanks for your contribution, whomever you are!

BTW, thanks to my CAPTCHA I don't get hardly any spam at all, so sorry for the inconvenience, but it has to be here.

Victor