Drupal 8

Let's automate deprecated API use removal from Drupal projects!

In his State of Drupal keynote at DrupalCon Amsterdam, Dries Buytaert showed once again some tools to use to prepare for Drupal 9 including the Upgrade Status module. To me the process is even more interesting than the tools, because it is entirely different from the last upgrade. As I wrote last week, you now make a lot of incremental improvements on your existing Drupal 8 site that makes the gap to Drupal 9 significantly smaller.

It is a new mindset to look at your Drupal 8 site to improve in preparation for Drupal 9 and we have tools to identify those improvements to make. However Dries also mentioned that we are not quite there to automate those improvements. Back in May Dezső Biczó of Pronovix built out a proof of concept integration with Rector that implements a sample set of refactors, including changes for global constants and best effort solutions to some EntityManager uses, a UrlGeneratorTrait and a drupal_set_message() rector. While the extent of impact of the global constant rectors are not yet known due to limitations in our tools not finding them yet, the rest of the implemented rectors are definitely tapping into the top list of deprecated APIs.

Unfortunately slightly after he posted his blog post about the proof of concept, Dezső did not have time for this project anymore. I think this tool could be of huge help in removing deprecated API use in your projects and thus making them more modern Drupal 8 projects (while being more if not already entirely compatible with Drupal 9). However, we need contributors to cover more of the deprecated APIs, especially around file_*() and db_*() functions. If we can figure out rectors for most of the top 15 errors (and Dezső already did some of them), we could cover over half of all deprecated API use in contributed projects:

Donut chart of top 15 usages of deprecated APIs on drupal.org

To top that off, I also think a simplytest.me style online service would be very useful to run drupal8-rector on your drupal.org project and suggest a patch to apply to remove deprecated APIs. Even better if it allows custom code to be uploaded so people can use it for that too. The more we can automate these tasks the easier the transition of Drupal 8 to 9 will be.

Submit issues and pull requests in the drupal8-rector project to improve API coverage. Look for me on Drupal slack if you get stuck and I'll try to help. I'd also love to talk to you if you want to set up an automated service. Let's do this!

Drupal 8.7.7+ will support extensions compatible with both Drupal 8 and 9!

Drupal 8.7.7 expected later this week is the first Drupal release to support extensions compatible with multiple major versions of Drupal. This is huge!

Drupal 9 is planned for June 3, 2020 and we are aiming to make the transition as smooth as possible. In fact we build Drupal 9 in Drupal 8 with API deprecations and optional support for new dependencies (Symfony 4.4 is getting very close, while Drupal 8.7.0 was already Twig 2 compatible). Extensions following these deprecations and making sure they are compatible with the updated dependencies could have codebases that are both compatible with Drupal 8 and 9 at the same time, meaning less support burden and earlier Drupal 9 compatibility. In March 48% of contributed projects on Drupal.org were already "Drupal 9 compatible" (as we could detect at the time), and that increased 54% by July (highfive!). These projects will now have one more thing to do, a one line info.yml file addition, to allow their existing codebase to also run on Drupal 9.

The core: 8.x key did not cut it anymore

Drupal 8 looks at extension .info.yml files and checks for the core: 8.x key to check if the extension can be used with Drupal 8. Unfortunately this was not adequate already due to API additions in Drupal 8 minor releases. You could not specify an extension was only compatible with Drupal 8.4+. People devised clever tricks with dependency on certain versions of the system module, but that was still limiting due to inability to depend on specific patch releases. In case your extension depended on a bugfix being present, you cannot set a dependency on Drupal 8.4.1+ in any way.

Introducing the core_version_requirement key

While solving this issue it became apparent that solving it with the existing core: 8.x key is going to cause backwards compatibility problems with existing Drupal 8 sites. The format of the value could not be changed without causing issues, so a new core_version_requirement key was introduced. This now allows to depend on major and minor versions of core as well as specify multiple version requirements even between multiple minor Drupal versions. For example if you would need a bugfix that is to be included in both Drupal 8.7.23 and 8.8.3 (imaginary future versions) but you wouldn't care for the minor version, then you would use the following, making your module incompatible with versions before the particular bugfix you need on both minor branches, as well as incompatible with all older Drupal 8 branches:

name: My Module
type: module
core_version_requirement: ~8.7.23 || ~8.8.3

Such precision was never possible before!

The new feature is made available in 8.7.7 to help prepare for Drupal 9, as this makes it possible to also declare Drupal 9 compatibility with the existing codebase you already have. Drupal 8.7.x will get security support up until the planned release of Drupal 9 on June 3, 2020. For extensions that only support Drupal 8.7.7 and later you can now exclusively use the core_version_requirement key. If the extension is also Drupal 9 compatible:

name: My Module
type: module
core_version_requirement: ^8.7.7 || ^9

For extensions that also support older versions, you should keep the core: key as well. The core_version_requirement key will be ignored by older Drupal versions, and new versions will validate that you specified non-conflicting requirements in the two keys (with regards to Drupal 8 support) so for extensions supporting older core versions, core_version_requirement also needs to be permissive, but could also allow compatibility with Drupal 9 too:

name: My Module
type: module
core: 8.x
core_version_requirement: ^8 || ^9

Read more about this change in the change notice. There is also work ongoing to support dependency metadata from composer.json to drive Drupal's dependency resolution as well. That will coexist with this simpler approach and will provide a more composer-native solution for those who already adopted composer for their extensions. In the meantime at least core_version_requirement will limit what Drupal would install while composer.json requirements will limit what composer will download. Because both Composer and Drupal use the term install, but those don't mean the same thing, here is a comparison chart:

  core: 8.x system module dependency core_version_requirement composer.json dependencies
Can be used from Drupal 8.0.0+ Drupal 8.0.0+ Drupal 8.7.7+ Drupal 8.0.0+
Allows minor version precision No Yes Yes Yes
Allows patch version precision No No Yes Yes
Allows multiple constraints No No Yes Yes
Limits composer install (download) No No No Yes
Limits Drupal 8 install Yes Yes Yes No, unless explicit support is added.
Allows Drupal 8 and 9 install with same codebase No No Yes No, unless explicit support is added.

What happens to Drupal.org project version numbers then?

So a 8.x-1.3 version of an extension on drupal.org may only be compatible with a subset of core, but that was already the case earlier. Now an 8.x-1.3 version of an extension on drupal.org may also be compatible with Drupal 9. How will this work? Drupal's composer integration already ignores the 8.x prefix from version numbers and only cares about the project specific version. Drupal core already only installs extensions where the core compatibility requirements are met regardless of which version number was the package you used to download (8.x modules may be only compatible with 8.4+ already). So the ultimate plan is to do away with the 8.x prefix in extension numbers entirely on drupal.org and support semantic versions for contributed extensions instead. In the short term, the Drupal.org project browser also needs to be updated to take the dependency metadata and use that to power compatibility filters, instead of merely using the version number prefix.

Improving Drupal 8's usability and the impact

We started regular Drupal usability meetings twice a week almost a year ago in March 2016. That is a long time and we succeeded in supporting many key initiatives in this time, including reviews on new media handling and library functionality, feedback on workflow user experience, outside-in editing and place block functionality. We helped set scope for the changes required to inline form errors on its way to stability. Those are all supporting existing teams working on their respective features where user interfaces are involved.

However, we also started to look at some Drupal components and whether we can gradually improve them. One of the biggest tasks we took on was redesigning the status page, where Drupal's system information is presented and errors and warnings are printed for site owners to resolve. While that looks like a huge monster issue, Roy Scholten in fact posted a breakdown of how the process itself went. If we were to start a fresh issue (which we should have), the process would be much easier to follow and would be more visible. The result is quite remarkable:

New status page in Drupal 8.3

Checking on Drupal 8's rapid innovation promises

Starting with Drupal 8, we decided to make more rapid innovation possible by releasing minor versions every 6 months that may come with new features and backwards compatible changes. Now that we released Drupal 8.1.0 and almost 8.2.0 as well, how did we do? Also what else is possible and what is blocking us to make those moves? What do all the changes mean for how might Drupal 9 unfold?

Dries Buytaert posted last Wednesday The transformation of Drupal 8 for continuous innovation and on the same day I presented Checking on Drupal 8's rapid innovation promises at DrupalCon Dublin. Here is a video recording of my session, which should be good for those looking to get to know Drupal's release process and schedule, as well as how we made it possible to experiment within Drupal core directly with Drupal 8. While I did hope for more discussion on the possibilities within Drupal 8 with the participants, somehow the discussion pretty much ended up focusing on Drupal 9, when it should be released and how much change should it come with.

Want to get issues resolved in Drupal core? Find community with an initiative!

In my previous post I explained why there will be a Drupal 9 even though we have previously unseen possibilities to add new things within Drupal 8.x.y. Now I'd like to dispel another myth, that initiatives are only there to add those new things.

Drupal 8 introduced initiatives to the core development process with the intention that even core development became too big to follow, understand or really get involved with in general. However because there are key areas that people want to work in, it makes sense to set up focused groups to organize work in those areas and support each other in those smaller groups. So initiatives like Configuration Management, Views in Core, Web Services, Multilingual, etc. were set up and mostly worked well, not in small part because it is easier to devote yourself to improving web services capabilities or multilingual support as opposed to "make Drupal better". Too abstract goals are harder to sign up for, a team with a thousand people is harder to feel a member of.

Given the success of this approach, even after the release of Drupal 8.0.0, we continued using this model and there are now several groups of people working on making things happen in Drupal 8.x. Ongoing initiatives include API-first, Media, Migrate, Content Workflows and so on. Several of these are primarily working on fixing bugs and plugging holes. A significant part of Migrate and API-first work to date was about fixing bugs and implementing originally intended functionality for example.

The wonder of these initiatives is they are all groups of dedicated people who are really passionate about that topic. They not only have plan or meta issues linked in the roadmap but also have issue tags and have regular meeting times. The Drupal 8 core calendar is full of meetings happening almost every single workday (that said, somehow people prefer Wednesdays and avoid Fridays).

If you have an issue involving usability, a bug with a Drupal web service API, a missing migration feature and so on, your best choice is to bring it to the teams already focused on the topics. The number and diverse areas of teams already in place gives you a very good chance that whatever you are intending to work on is somehow related to one or more of them. And since no issue will get done by one person (you need a reviewer and a committer at minimum), your only way to get something resolved is to seek interested parties as soon as possible. Does it sound like you are demanding time from these folks unfairly? I don't think so. As long as you are genuinely interested to solve the problem at hand, you are in fact contributing to the team which is for the benefit of everyone. And who knows, maybe you quickly become an integral team member as well.

Thanks for contributing and happy team-match finding!

Ps. If your issue is no match for an existing team, the friendly folks at #drupal-contribute in IRC are also there to help.