Drupal 9

I am an initiative coordinator on Drupal 9, here is my writing on tools and processes getting there.

Four ways to track a list of Drupal issues a group of you cares about

Drupal 9.0.0-beta2 was released this week and there are only 63 days to go for Drupal 9.0.0's release. Many product developers, distribution authors and so on are now looking at what is left to be Drupal 9 compatible and which of their dependencies are doing what where. So effectively tracking a list of drupal.org issues somewhere is on many people's minds. While you can follow issues on drupal.org that only works for your personal needs. Actually Surabhi Gokte asked me yesterday about tools for group issue tracking, and I decided to write up a quick blog post because the answers are likely useful for many people. Here are two public and two possibly private ways to track drupal.org issues. I've used each in some form throughout my years with the project and you may find some of them useful.

Public: use a drupal.org META issue

Drupal.org project developers, this is a good way to track your tasks belonging to a larger goal like Drupal 9 compatibility. Create a regular drupal.org issue for your project and use the issue link filter. [#123456] is the format to use to link to issues and get their title and even status colors represented on your issue. The only thing you need to keep an eye on is that issue summaries are cached, so as the status/title of listed issues are updated, that is not reflected on the summary. As you are likely to adjust the list as things get done and new things are discovered, the status and titles of the listed issues should get updated in the meantime. An example of this technique is [META] Requirements for tagging Drupal 9.0.0-beta1. This technique allows anyone with a drupal.org account to help maintain your list of issues.

Once you have a META issue like this for Drupal 9 compatibility, please link it from your Drupal 9 plan on your project page, see:

Public: use Contrib Kanban

Contrib Kanban is a great service developed and ran by Matt Glaman of Centarro. You can register and create custom boards based on a list of issues, such as this kanban board for Umami's Spanish translation. Only you will be allowed to maintain your issue list but the results are shown very visually and dynamically in a kanban board style.

Public/Private: use any tool with the Drupal issue Chrome browser extension

Matthew Grasmick built the Drupal Issue Chrome browser extension which turns links to Drupal.org issues on any webpage to their colored and titled forms very much like in the META issue method. This will pull the data for issues live, but only people with the extension installed will see the titles and statuses. You can use this for private tracking or public tracking. We use it at Acquia to track some drupal.org issues through Jira tickets.

Public/Private: use a scripted spreadsheet

This method could work in any scriptable spreadsheet system. We used it extensively at Acquia with Google sheets and I believe it originates from Andrea Soper (ZenDoodles) and Jess (xjm) from several years ago. Set it up like this:

  1. Assuming an empty spreadsheet, designate a column for issue numbers. You will use this column to enter the ID of issues you want to track. Let's assume this is column E for the rest of the example. Also let's assume you use the first row for header text for the columns.
  2. Add this to F2: =IF(E2="","", regexreplace(importxml(hyperlink( concat("https://www.drupal.org/node/", E2)), "//*[@id='block-project-issue-issue-metadata']/div/div/div/div/div/text()"), "\n","")). This column will be your issue status.
  3. Add this to G2: =if(E2="","", hyperlink( concat("http://www.drupal.org/node/",E2),concat("#",E2))). This will be your issue link.
  4. Add this to H2: =if(E2="","", substitute(regexreplace(importxml(hyperlink( concat("https://www.drupal.org/node/", E2)), "//title/text()"),"\n",""), " | Drupal.org","")). This will be your issue title.

At least in Google sheets, you can hold-drag these values to fill in the rest of columns F-H with appropriate computed values. You can also apply conditional styling to status values, etc. A clear benefit of this approach is that you can assign tabular metadata to issues, like your private severity or priority information or assignments within your company as to who is planning to look into it without getting into a priority argument on the issue.

By the way it would definitely be more reliable to pull this data from the Drupal.org REST API. We used this method extensively for years at Acquia to track high priority issues leading up to Drupal core releases. However we are not currently using this method and I did not go to convert the logic from our old sheets, so improvements welcome.

These are methods I used that are still available and work fine. Do you have better ways to do issue list tracking? Let me know in the comments.

Join us at the free virtual MidCamp this week to learn about and improve Drupal 9

I heard a lot of good things about Midwest Drupal Camp (MidCamp) throughout the years. I also had the chance to follow Avi Schwab sharing best practices from it at various events. It is one of the real model events. I never had a chance to attend unfortunately.

In 2020, due to the ongoing pandemic, they did not have the opportunity to run the camp as usual. While that is sad, they decided to turn it around entirely and made MidCamp all digital and free to attend. This means you and I can attend finally and not just admire the event from afar! Let's do that! There is no registration or tickets required anymore, so sign up for details via email, or keep up on the site.

MidCamp - No ticket or registration required

Great Drupal 9 content

While there are lots of great sessions to participate in, I am really excited about the Drupal 9 content. Presentations will be streamed over Zoom with links being provided later in the week.

Let's get contributing

Following sessions on Thursday and Friday, there will be a virtual contribution day on Saturday from 3pm to 9pm UTC. I am really excited about these three contribution topics and would love to see you join in them! I will swap a workday out this week with Saturday to participate in this event and still have personal time for my family while they all stay at home due to the pandemic.

Olivero theme screenshot

  • Working on automation rules to fix deprecation issues in Drupal 8 projects. This is the practical application of Dan Montgomery's session. Dan and Ofer Shaal will be available and lead this topic. Rector saves a lot of manual work by automating code upgrades from Drupal 8 to Drupal 9. The immediate goal is to create Rector rules for the 15 most popular deprecations. Those 15 rules will cover 50% of all Drupal deprecations so a lot less manual work would be needed!
  • Work on contributed project Drupal 9 compatibility. Applying the above tooling combined with manual work to make contributed projects compatible with Drupal 9. Mike Lutz will be leading this topic. I would also love to help anyone interested. I maintain the overview of contributed project readiness on dev.acquia.com (thanks for the source data to the Drupal Association), which can be used to find what to work on effectively.
  • Improve the proposed new Drupal 9 default frontend theme Olivero. Various members of the team will be available to work on outstanding issues for the proposed new Olivero frontend theme. Putra Bonaccorsi, Aubrey Sambor and Brian Perry indicated they would be there.

Contribution will be coordinated on Slack I believe, more information to come later this week. Sign up for details via email, or keep up on the site. See you (virtually) there!

A whole new version of the open source "State of Drupal 9" slideshow, present it yourself!

Ten months ago I created the first version of "State of Drupal 9" and published it as an open source slideshow for anyone to review and present. It has been presented by myself and various other people since then. I kept it up to date and I got lots of feedback to improve it. Since we are closing in on the Drupal 9.0.0-beta1 release where milestone dates are better defined and the release itself is quite close, I decided to rework the whole slideshow with a more practical approach in mind focusing first on what happens to Drupal 7 and outlining the concrete practical upgrade paths for what people should do coming from Drupal 8.

I was slated to present this new version of it at DrupalCamp London, DevDays Ghent and DrupalCon Minneapolis (with Amber Himes Matz). Due to the ongoing pandemic I did not attend DrupalCamp London anymore and further events are being gradually cancelled or postponed. So I don't know if I or someone else will have a chance to present this in person anytime soon. (Which I think is for the better of our health). However, you can still review the content yourself and present it virtually on a remote Drupal meetup or conference or at your virtual company meeting. I did present the slides for DrupalCamp London remotely myself and the camp made the recording kindly available:

I added plenty of speaker notes. You can present/review it from https://slides.com/gaborhojtsy/state-of-drupal9 or fork it (even with a free slides.com registration) and replace the "About me" slide as well as the opening and closing thanks slide with your attribution. Please keep the rest of the attribution intact. Instructions are in the slides.

I'll keep this version updated from now on and archived the old version with a note pointing to this.

Drupal 9 release scenarios, the first beta deadline in 6 weeks and how you can help!

As Dries Buytaert explained in his Plan for Drupal 9 post at the end of 2018 (emphasis mine):

Drupal 8's biggest dependency is Symfony 3, which has an end-of-life date in November 2021. This means that after November 2021, security bugs in Symfony 3 will not get fixed. Therefore, we have to end-of-life Drupal 8 no later than November 2021. Or put differently, by November 2021, everyone should be on Drupal 9.

Working backwards from November 2021, we'd like to give site owners at least one year to upgrade from Drupal 8 to Drupal 9. While we could release Drupal 9 in December 2020, we decided it was better to try to release Drupal 9 on June 3, 2020. This gives site owners 18 months to upgrade. Plus, it also gives the Drupal core contributors an extra buffer in case we can't finish Drupal 9 in time for a summer release.

Here we are 14 months later and while most people took the June 3 release date and took it for granted, it is still not guaranteed! However you can help in various ways to make it much more likely!

Late last fall, we focused on defining what it means if we cannot make the June 3, 2020 release despite our best efforts and what is an early indicator that tells us we are going to miss it. First of all, that meant defining requirements for the first alpha release and requirements for the first beta (API complete) release. Also we needed to set some expectations as to when do we want to see the API-complete beta release to give enough time for the ecosystem to test it and find important problems in time. Based on how soon the beta requirements are met, there are three release scenarios and the best case ending in the June 3, 2020 release date for Drupal 9 has a beta deadline in 6 weeks! Yes, 42 days!

Alpha requirements simplified

The key requirements for the first alpha release are simple. We wanted to update the key dependencies: Symfony to version 4.4 and Twig to version 2, as well as remove frontend polyfills that were not needed and remove most of jQuery UI (which were already deprecated in Drupal 8). This gets our most important dependencies up to shape to what will be in Drupal 9. We also made it possible for contributed projects to depend on Drupal 8 and 9 at the same time, so they will not need to branch for Drupal 9 support.

There are two outstanding things for the Drupal 9 alpha:

  1. Drupal.org does not yet have an automated packaging pipeline that conforms to all the recent composer related improvements and therefore making core releases is error prone. I don't believe you can help with this at this time, the Drupal Association is hard at work on this.
  2. Drupal core should use a major version agnostic update feed for projects which is already being provided by Drupal.org but the core code to consume it is still in the works. While this is actively being worked on, reviews are always helpful. This will make sure Drupal 9's Update Status gets only contributed projects that are actually Drupal 9 compatible, while contributed modules will not need to establish a Drupal 9 branch.

Beta requirements simplified

The beta requirements are a bit more complicated and longer of course because we are looking at being API complete here. Once again, for the June 3, 2020 release date, we need these done in 6 weeks! The issue lists must haves and should haves, however the should have issues should be considered must-have for the June 3, 2020 release date and would only be reconsidered later if that date cannot be met. Here is a simplified rundown of the beta requirements:

  1. We want to keep dependencies up to date. There is no concrete pressing issue here at the moment that I know, but this really depends on how our dependencies evolve.
  2. We'd like to remove all the deprecated APIs themselves. Last year I built a graph to track this, and it shows nicely that we are down to half of them remaining (yay!), but still quite enough to deal with. There are various outstanding issues you can help with here.
  3. We want to make sure people can update to Drupal 9 from Drupal 8 by resolving critical upgrade path bugs. If you cannot update to a later version of Drupal 8 due to some critical bug, then you will be stuck on your version of Drupal 8. Not good. These include views, layout_discovery, taxonomy, menu_content, etc. related issues. All of them need help. If you are on an older version and can reproduce the problems, that is useful. If you have experience in these areas, your input would be useful.
  4. It will only be possible to update to Drupal 9 from Drupal 8.8 or 8.9, so all older update paths and their tests should be removed. Older versions of Drupal 8 will themselves be unsupported already at the time of Drupal 9's release. This issue is getting close but needs reviews.
  5. No new security or data integrity issues should be in Drupal 9. If there are any, they should be resolved. I don't know of any issues at the moment here.
  6. The API should be complete. There are no critical API additions or changes that I know of at the moment in this general category.
  7. We want to make sure people can migrate from Drupal 6/7 to Drupal 9. This needs the remaining multilingual migration paths to go stable. This is an area where we posted several call to actions, but still need your help. There are proposed migration paths for node translations and entity translations that respect revisions but they need at least code reviews to make sure they are good. Otherwise if you had content translations with revisions, the migration will not be correct. Without that, multilingual migrations will not go stable.
  8. PHP requirements should be finalised. It is likely at this time that Drupal 9 will require PHP 7.3 that is being worked on currently and could use a review.
  9. Database requirements should be finalised in terms of MySQL/MariaDB/Percona and PostgreSQL. Both issues need data as to which distributions and hosts support certain versions.
  10. The right security update information should be provided for users taking the one year support cycle and long term support of the last Drupal 8 release. This could also use reviews.
  11. We should put Drupal's base theme on a track so that it can evolve in Drupal 9 finally. This involves creating a new stable9 theme and decoupling the core themes from Classy. Various issues to help with here.
  12. Drupal.org should support multi-core compatibility eg. on project pages, localize.drupal.org, etc. This work is currently deprioritised by the Drupal Association due to the focus on the packaging pipeline blocker that I listed first. Once this gets attention, it will likely uncover core issues to resolve as well.

The two areas that receive the least attention at the moment are upgrade path blockers and the stability of the multilingual migration path, so those two are the most pressing where we need your help!

While the above is a complete rundown of the current beta requirements, it may change later on, so refer to the beta requirements issue later on for up to date information.

What happens if all the above are not done by end of February (in six weeks)

If all goes well, with your help, we'll be done with all the above in six weeks. If that does not work out, we have a plan B and a plan C. Here is how those options unfold. If beta requirements are only done two months later by end of April, then Drupal 9's first beta will be released on the first week of May and Drupal 9 is to be released on August 5, 2020. If the beta requirements are only done by end of August (four more months later), than the first beta will be released than with a Drupal 9 release date of December 2, 2020. In this case a Drupal 8.10 may also be released if needed. These dates are spelled out well in the Drupal core release cycle overview. I created this visual to help understand the alternate timelines:

Drupal 9 release scenarios visualised

While I think it is reassuring that we have plans for what happens if our initial date targets don't work out, unfortunately the end of life date at the end of next year for Drupal 8 is not movable because it is based off of Symfony 3's end of life. So the sooner we can make Drupal 9 happen (while meeting the upgrade and stability requirements) the better. What are you going to work on to help?

Helping with contributed projects

Based on the PHP deprecations our tools can identify, 43% of contributed projects would only need info.yml file updates to be Drupal 9 compatible. An additional 41% of the remaining projects have only issues that are resolvable now (even while keeping support for Drupal 8.7). Solving those anytime between now and Drupal 9's release (whenever it is) would put us to almost six thousand contributed projects compatible with Drupal 9 on day one! While that is a very idealistic number, helping with contributed projects is nonetheless a great avenue to contribute to Drupal 9 readiness. Help at the Drupal Global Contribution Weekend at end of next week or anytime before and after! I prepared a quickstart guide for this occasion.

Drupal 9 compatibility contribution quickstart guide for Drupal Global Contribution Weekend 2020

This year Drupal Global Contribution Weekend is on January 24-26, 2020 with such varied locations as Delhi, Novosibirsk, Ghent, Frankfurt, Milan, Zurich, Lutsk, London (on two continents!), Boston, Minneapolis, etc. Wow! It is truly a global gathering! With Drupal 9 planned to be released later this year, what better to focus on, than making drupal.org projects Drupal 9 ready?

To help you do that I went in and updated my open source State of Drupal 9 talk. People can use this to present at any location to get people up to speed about Drupal 9. If you need a video recording of it, there is one from DrupalCamp Belarus in May 2019. While the content got slightly updated since then, the recording should help get it.

After or instead of presenting that session, I thought a quickstart guide would be really useful to help people get started with contributing. While this looks like a colorful guide you would print, it is actually full of useful links (some to my earlier blog posts for details), so I suggest you use it in digital form.

What are you planning to do for Drupal Global Contribution Weekend this year?