Drupal.org

Drupal 9.0.0-beta2 was released this week and there are only 63 days to go for Drupal 9.0.0's release. Many product developers, distribution authors and so on are now looking at what is left to be Drupal 9 compatible and which of their dependencies are doing what where. So effectively tracking a list of drupal.org issues somewhere is on many people's minds. While you can follow issues on drupal.org that only works for your personal needs. Actually Surabhi Gokte asked me yesterday about tools for group issue tracking, and I decided to write up a quick blog post because the answers are likely useful for many people. Here are two public and two possibly private ways to track drupal.org issues. I've used each in some form throughout my years with the project and you may find some of them useful.

Public: use a drupal.org META issue

Drupal.org project developers, this is a good way to track your tasks belonging to a larger goal like Drupal 9 compatibility. Create a regular drupal.org issue for your project and use the issue link filter. [#123456] is the format to use to link to issues and get their title and even status colors represented on your issue. The only thing you need to keep an eye on is that issue summaries are cached, so as the status/title of listed issues are updated, that is not reflected on the summary. As you are likely to adjust the list as things get done and new things are discovered, the status and titles of the listed issues should get updated in the meantime. An example of this technique is [META] Requirements for tagging Drupal 9.0.0-beta1. This technique allows anyone with a drupal.org account to help maintain your list of issues.

Once you have a META issue like this for Drupal 9 compatibility, please link it from your Drupal 9 plan on your project page, see:

65 days to go until the Drupal 9 release on June 3, 2020.https://t.co/JmRQ2Cdaun project maintainers, now is a great time to indicate your Drupal 9 plans! Edit your project page and fill in the Drupal 9 plan field to help contributors help you the most effectively pic.twitter.com/QrBpZCDgiW

— The Drop is Always Moving (@DropIsMoving) March 30, 2020

Public: use Contrib Kanban

Contrib Kanban is a great service developed and ran by Matt Glaman of Centarro. You can register and create custom boards based on a list of issues, such as this kanban board for Umami's Spanish translation. Only you will be allowed to maintain your issue list but the results are shown very visually and dynamically in a kanban board style.

Public/Private: use any tool with the Drupal issue Chrome browser extension

Matthew Grasmick built the Drupal Issue Chrome browser extension which turns links to Drupal.org issues on any webpage to their colored and titled forms very much like in the META issue method. This will pull the data for issues live, but only people with the extension installed will see the titles and statuses. You can use this for private tracking or public tracking. We use it at Acquia to track some drupal.org issues through Jira tickets.

Public/Private: use a scripted spreadsheet

This method could work in any scriptable spreadsheet system. We used it extensively at Acquia with Google sheets and I believe it originates from Andrea Soper (ZenDoodles) and Jess (xjm) from several years ago. Set it up like this:

1. Assuming an empty spreadsheet, designate a column for issue numbers. You will use this column to enter the ID of issues you want to track. Let's assume this is column E for the rest of the example. Also let's assume you use the first row for header text for the columns.
2. Add this to F2: =IF(E2="","", regexreplace(importxml(hyperlink( concat("https://www.drupal.org/node/", E2)), "//*[@id='block-project-issue-issue-metadata']/div/div/div/div/div/text()"), "\n","")). This column will be your issue status.
3. Add this to G2: =if(E2="","", hyperlink( concat("http://www.drupal.org/node/",E2),concat("#",E2))). This will be your issue link.
4. Add this to H2: =if(E2="","", substitute(regexreplace(importxml(hyperlink( concat("https://www.drupal.org/node/", E2)), "//title/text()"),"\n",""), " | Drupal.org","")). This will be your issue title.

At least in Google sheets, you can hold-drag these values to fill in the rest of columns F-H with appropriate computed values. You can also apply conditional styling to status values, etc. A clear benefit of this approach is that you can assign tabular metadata to issues, like your private severity or priority information or assignments within your company as to who is planning to look into it without getting into a priority argument on the issue.

By the way it would definitely be more reliable to pull this data from the Drupal.org REST API. We used this method extensively for years at Acquia to track high priority issues leading up to Drupal core releases. However we are not currently using this method and I did not go to convert the logic from our old sheets, so improvements welcome.

These are methods I used that are still available and work fine. Do you have better ways to do issue list tracking? Let me know in the comments.

Drupal.org provides an amazingly flexible issue queue and is the backbone of most community activity around code, community, policies, drupal.org itself and so on. Each issue has a priority value which can be one of Critical, Major, Normal and Minor. Even more interesting is the tagging system we use with some commonly used tags like 'beta blocker' or 'beta target' or 'revisit before release' which add extra priority on top of the single value field. The drupal.org issues however don't lend themselves to supporting working on your priorities. Here are some options and tools I used so far that help solve this issue.

A couple weeks ago, Dries posted a call for sponsors for efforts to upgrade the Drupal.org site with its existing functionality and design to Drupal 6 and then start work on the redesign, new features and theme implementation. We are on the first phase at the moment, and sitting at the boardroom of One Laptop Per Child in Cambridge, Massachusetts. We are getting to second day's close, and already achieved an impressive amount of progress. Here are some of the most exciting items.

We need to admit, these days, people are more reliant on contributed modules then Drupal core. What is in Drupal core is taken as a given and some requested new feature would ever come in a contributed module in a reasonable timeframe for a Drupal site builder. People are not waiting for Drupal 7 to get a feature, people are active in the issue queues to get things done sooner. Drupal core is supposed to lay the foundation and be stable for long, while contributed modules can experiment, change, and add new features on a much more active pace. So as a result, Drupal core can keep using the "will be released sometime" approach, and it does well with Drupal 7 allowing for more improvements by not keeping the "every 12 months" release cycle intact. With contributed modules on the other hand, much more frequent releases are anticipated with new features introduced through a shorter timeframe.

What's common in both, is that for longer lasting development, overseeing progress on major areas like the file API, the database layer and so on is important, while on shorter release cycles, knowing what is required for the release to be fixed, and whether the project is on track or not could be key to get contributors. So having (A) overviews of project progress and (B) checklists of things to be done is increasingly needed on drupal.org. There are a few things people did about these needs recently, and all have their advantages and disadvantages. There are some reasonably new approaches, so I decided to highlight the ways I see people approach project planning on drupal.org.

Drupal.org runs the infamous project* module family to help micro (and not so micro) projects run around the Drupal core. These include modules, themes and translations as well as installation profiles. Currently, drupal.org hosts more then four thousand (yes, more then 4000) projects, with support for a CVS space for hosting the code with branching and tagging, as well as a release system tied to those CVS concepts. Projects can also have an issue queue, which people can submit issues with, covering bug fix requests and suggestions as well as new feature requests. So a project's issue queue can be quite busy, but it is the only "core" way to track what needs to be done, who is assigned to these tasks, who is working on them, what is the current status and so on.

So eventually people started to experiment building new tools on top of project issues to help manage project planning, overviews and checklists. Here is a hopefully comprehensive list of what tools people built on top:

On Drupal.org, user HongPong asked a valid question about our Drupal.org upgrade/redesign fundraiser:

I am curious if these codesprint efforts will be useful for other projects. I think it's fantastic so many people are organizing, but I wonder if whatever form the bulk of this takes will be released - or is it mostly a one-off effort that is too customized to be generally useful elsewhere?

I thought it would be useful to have a blog post about this instead of just sending the reply in the comments, since I think this is an important topic to talk about.

• As Niel pointed out, the modules used on Drupal.org will get attention. I've just started to use the "drupal.org upgrade" tag over the weekend to mark issues requiring an attention for the Drupal.org upgrade: http://drupal.org/project/issues-term/346 General purpose existing modules used on the site including simplenews, comment_upload, comment_alter_taxonomy, image or imagefield, and so on will be tested and fixed where required. New modules will be used on Drupal.org and will get similar attention: CCK and Views are the first obvious contenders (right, Drupal.org is not using these modules as of now).
• Drupal.org runs some custom core patches to improve performance by handling master and replicated databases. We will take a deep look on these changes and will implement a similar or better solution for the upgraded site based on Drupal 6. Our caching setup will have similar attention. Having this insight we will be able to document what best practice we found to scale drupal.org. This will join documentation on ways to make Drupal more scalable. Such efforts on Drupal.org also helped improve general performance in Drupal itself (both current stable and current development versions) in the past.
• The upgrade is planned to omit some of the current features on Drupal.org such as the current search implementation and the custom Drupal based distributed login system. OpenID and a new search implementation are about to be implemented in place instead. It will be easier to find stuff on drupal.org for your day-to-day development and you will be able to reuse your Drupal.org account on any OpenID supporting site.
• These items above all happen as products of our upgrade, even if there is no redesign. While all of the above items are great by themselves, the redesign will be the biggest hit. It will help you use a more fine grained navigation, move around the Drupal.org subsites seamlessly, find and understand modules needed for your projects, and so on. But what's even better is that it will be a site you can point possible customers to. It will help you market your services, consulting, books, courses and so on more effectively, since the Drupal site itself will do a better job to help the community and the project market itself and flourish in the times ahead of us. If you earn (part of) your living from Drupal related work, this will give you a boost. If you do not yet earn (part of) your living from Drupal, you'll be more tempted then ever to do so.

We are making exciting things happen, but your help is still vital to make this a reality. You can help by donating some money to get the expert teams together (use the ChipIn widget in this post) or by contributing to the sprints yourself either via the issues listed at http://drupal.org/project/issues-term/346 or by coming to the sprints. Thanks for being part of getting this huge improvement into production!

I've helped recently to unfork the Bluebeach theme (the theme used on Drupal.org and subsites) which was used with different code on drupal.org and groups.drupal.org. So now both sites can use the same source code for their theming. I've also ported the changes to the Drupal 6 port of the theme which was done by Earl Miles earlier, based on the Drupal 5 version. These were all in anticipation of doing upgrades of drupal.org and subsites with the existing theme before going to the next step with Mark Boulton's redesign.

Dries Buytaert recently posted his Fields in core code sprint debrief, in which he mentions toying with the idea of organizing a Drupal.org upgrade sprint at Acquia. This is what Dries has to say:

All things considered, this [Fields in core] sprint was a big success, and I'm now toying with organizing a "drupal.org upgrade" sprint at Acquia during the last week of January. The goal would be to upgrade drupal.org from Drupal 5 to Drupal 6, and make progress on the drupal.org redesign work. Is anyone interested in participating in or helping fund this sprint? If so, more soon.

From my previous involvement and blog posts, it is probably not big news that I am very interested in helping out with that. Are you?

In first of a series of posts, I'd like to go ahead and talk about project handling functionality, one of the most important tools behind Drupal.org. At this moment, Drupal.org is running Drupal 5, and a big chunk of modules which don't have a Drupal 6 version to migrate to on Drupal.org is the project module family: project (also includes project_release and project_usage), project_issue, pift_server, cvslog and even comment_upload.

Except the comment_upload module (which allows file uploads on comments in a general way), maintenance for these modules are headed by Derek Wright and Chad Phillips. An outstanding thing about these modules is that they keep improving and being adjusted to user needs. Automated testing integration tools were developed and keep improving, so patches submitted against Drupal 7 get automated testing. This is just plain great. However, all this huge amount of motion is going on in the Drupal 5 version of the module. And given that Drupal.org needs a stable environment, it takes considerable effort to maintain a stable Drupal 5 branch with all these feature improvements and changes coming in.

While these modules do not even have a Drupal 6 branch yet, Adam Light went ahead and worked on a Drupal 6 port for project module. He hosts this in his own private Subversion repository (see http://drupal.org/node/157694#comment-891587 and the rest of the comments there). Since he started off long ago from a then current version of the module and implemented Views integration (instead of the one-off SQL based pages in the Drupal 5 version), the Drupal 6 port has a largely refactored codebase and does not carry the improvements made to the Drupal 5 version since then.

The lead maintainers however are at this point more interested in working on a new stable release for Drupal 5, given that some bigger changes they are planning to make would be easier to manage on their own instead of as part of a bigger porting and migration work to a new Drupal version and to a Views based backend. This gets us to a message of "please help with a new stable Drupal 5 release of project module before Drupal 6 work can be considered". While these patches are relatively big, they are far from how big of a monster patch is the Drupal 6 upgrade. All-in-all the possibly awkward conclusion is that maintainers look for help with the Drupal 5 version before Drupal 6 work can be started.

For concrete action items, Adam Light summarizes it well:

Let me explain the situation a little here:

There are still some things that we'd like to get fixed in the Drupal 5.x branch of the project and project issue modules before we branch for Drupal 6. See http://groups.drupal.org/node/10865 and http://groups.drupal.org/node/16069. One big issue that affects all of project* land is #98278: project* namespace bugs in $node. The project* maintainers have agreed that it would be best to fix this problem *before* branching, because if we try to fix it during the port that will make testing the port even more difficult than it will be.

Creating a D6 branch itself will not really unblock things. Hunmonk or I could also do that ourselves. However, the ported code that is currently in my SVN repository contains a *lot* of changes, and all of us agree that those changes should not just blindly be checked into the project cvs repository without at least some review.

Yes, we all realize that this is a non-ideal situation, and that the port is moving much slower than anyone of us would like. The best way anyone could help move the port forward would be by helping to write or review patches for any issue not yet finished in the list of things to do before the next 5.x release. Mostly, that means reviewing the $node namespace patches in the issue I linked to above. The unfortunate part of this issue is that it is huge, but really boring. And none of the sites run by the project* maintainers use a combination of modules that actually causes this bug to reveal itself. But at the same time, we realize that lots of project* users *do* use such a combination of modules (eg. they use pathauto with project), and so we need to fix this bug soon.

So there are numerous big issues affecting Drupal.org which will be solved as part of the Drupal 6 port, but the main issue holding back the port from even starting is an issue which does not even affect drupal.org (and therefore is not going forward on any reasonable speed).

In summary, there are difficulties in how improvements on drupal.org are expected by some people right now instead of after an upgrade, and the maintainers are taking on work on these items; and issues not affecting drupal.org holding back our most important upgrade ever. So you can help at least three ways with the project module upgrade:

• Put away some of your cool feature ideas for project modules on drupal.org for now. Let's focus on porting and bugfixing or we are not going to get over new feature requests anytime soon.
• Help test patches against the Drupal 5 version of project modules to fix long standing bugs. See http://groups.drupal.org/node/10865 and http://groups.drupal.org/node/16069
• Help test and fix existing issues in the Drupal 6 port of project module. It at least has Views integration issues coming from the RC2 API changes in Views. See my pending patch at http://drupal.org/node/157694#comment-1069892 which still needs work.

We definitely need your help in many ways. Let's do good for the drupal.org upgrade / redesign!

As Tiffany published in the drupal.org post titled Drupal.org redesign officially underway in September, Mark Boulton design's activity with the drupal.org redesign is tied to a timeline, and will end in one week as originally planned. Whether this deadline is actually met or not, the fifth iteration of the prototype was posted on the groups.drupal.org group related to the redesign a few days ago.

Drupal.org (and its whole site family) is being redesigned, thanks to heroic efforts by the Drupal Association. The site family grew quite big (and is still growing) so there is a lot of stuff to do. You probably have your own gripes on how it should work, so now is the time to get involved! Just as you'd expect from good architects when redesigning your home, Leisa and Mark are running a series of blog posts (just watch Drupal Planet to see them) to discuss details of the redesign, and understand our needs. Anyone can put in their opinions, and it is their job to filter it down and produce useful results in a relatively short time.

I started contributing with a one-on-one interview with Leisa at Drupalcon Szeged 2008. This event was a good opportunity to interview lots of different type of people on how they use the site, what are their problems and happy moments with it. If you have not been to Szeged, or have not been able to get an interview, there is no reason to step back. You can also contribute with comments on various blog posts (just as I do), taking part in the online card sort and submitting your own wireframe suggestions.

I've just decided today to sit down and make sure my opinion gets into the pipe, so I submitted this simplistic wireframe (click on it and get to Flickr to see the notes):

My goal with the suggestions is to get rid of the blog-look for the home page and get prime-time for more important stuff by highlighting them by topic area. We don't need such a long explanation on what Drupal is, if people can instantly see, that Drupal events are happening near them, they can buy books to hold in their hands, people build cool sites with Drupal and they can earn money with Drupal as well, within a thriving community, which takes software and security seriously. If you break this sentence down, several boxes come up for the homepage to highlight security bulletins, showcases, the Drupal Planet, events, and so on. These all describe Drupal's several aspects themselves, leaving the intro itself to a short explanation. I believe this kind of hub homepage would finally get us to a state where we can tell people to just go to the drupal.org homepage and get a decent overview of Drupal.

Do you think you can do better then me or have other ideas to get highlighted in the redesign process? Why not participate? Come wireframe with Leisa.