This year Drupal Global Contribution Weekend is on January 24-26, 2020 with such varied locations as Delhi, Novosibirsk, Ghent, Frankfurt, Milan, Zurich, Lutsk, London (on two continents!), Boston, Minneapolis, etc. Wow! It is truly a global gathering! With Drupal 9 planned to be released later this year, what better to focus on, than making drupal.org projects Drupal 9 ready?
After or instead of presenting that session, I thought a quickstart guide would be really useful to help people get started with contributing. While this looks like a colorful guide you would print, it is actually full of useful links (some to my earlier blog posts for details), so I suggest you use it in digital form.
What are you planning to do for Drupal Global Contribution Weekend this year?
It is a new mindset to look at your Drupal 8 site to improve in preparation for Drupal 9 and we have tools to identify those improvements to make. However Dries also mentioned that we are not quite there to automate those improvements. Back in May Dezső Biczó of Pronovix built out a proof of concept integration with Rector that implements a sample set of refactors, including changes for global constants and best effort solutions to some EntityManager uses, a UrlGeneratorTrait and a drupal_set_message() rector. While the extent of impact of the global constant rectors are not yet known due to limitations in our tools not finding them yet, the rest of the implemented rectors are definitely tapping into the top list of deprecated APIs.
Unfortunately slightly after he posted his blog post about the proof of concept, Dezső did not have time for this project anymore. I think this tool could be of huge help in removing deprecated API use in your projects and thus making them more modern Drupal 8 projects (while being more if not already entirely compatible with Drupal 9). However, we need contributors to cover more of the deprecated APIs, especially around file_*() and db_*() functions. If we can figure out rectors for most of the top 15 errors (and Dezső already did some of them), we could cover over half of all deprecated API use in contributed projects:
Image
To top that off, I also think a simplytest.me style online service would be very useful to run drupal8-rector on your drupal.org project and suggest a patch to apply to remove deprecated APIs. Even better if it allows custom code to be uploaded so people can use it for that too. The more we can automate these tasks the easier the transition of Drupal 8 to 9 will be.
I've had various deep discussions with contributed module maintainers recently about their process to update code to Drupal 9 and one point struck me. We are so attached to "Make it ready for Drupal 9" that a key point of the message may be lost. Check out this section of the State of Drupal keynote from DrupalCon Amsterdam 2019 where Dries Buytaert showcases Johanna's relatively simple site that she prepares for the Drupal 9 upgrade entirely in Drupal 8. Notice that she does all the steps in Drupal 8 other than the final Drupal 9 upgrade itself:
This is the base principle of the process towards Drupal 9, making your Drupal 8 site better and more prepared, so the move to Drupal 9 itself at the end is a relatively small step and you got a better Drupal 8 site in the meantime. You are not jumping over the fence all at once, but in gradual steps. I thought a comparison with Drupal 6 to 7, 7 to 8 and 8 to 9 would help, since people may have assumptions or prior experiences with those, so its worth looking at how our new process compares to the two previous transitions.
Removing deprecated API use from Drupal 8 contributed projects to get them ready for Drupal 9.
Updating core dependencies and removing deprecated APIs from Drupal 9 itself.
Improve the tools and documentation that help people prepare for Drupal 9.
On October 31st, join on Slack even if you are there in person but definitely if you are remote. We'll meet in the #d9readiness channel on Drupal Slack. If you’re at Drupalcon Amsterdam, come to Europe 2 Forum!
To remove deprecated API use from contributed projects:
Help people who want to learn how to remove deprecated API use from projects, or remove some deprecated API use yourself! In minutes, you too could help make Drupal 9! Here are tips and tricks for the most common deprecated APIs: https://github.com/mglaman/drupal-check/wiki/Deprecation-Error-Solutions.
If you're a module developer who would like to get a better chance to having your module reviewed / patched by a new contributor there, please create (or tag an existing) issue with the Drupal 9 compatibility + Amsterdam2019 tags!
To remove the deprecated APIs themselves, update dependencies in Drupal 9 or improve the deprecated API checking tools and documentation, please ask for specifics in the Slack channel or in person. The Drupal 9 tables will be easy to find!
You can also help spread the word by retweeting this:
Drupal 8.7.7 expected later this week is the first Drupal release to support extensions compatible with multiple major versions of Drupal. This is huge!
Drupal 9 is planned for June 3, 2020 and we are aiming to make the transition as smooth as possible. In fact we build Drupal 9 in Drupal 8 with API deprecations and optional support for new dependencies (Symfony 4.4 is getting very close, while Drupal 8.7.0 was already Twig 2 compatible). Extensions following these deprecations and making sure they are compatible with the updated dependencies could have codebases that are both compatible with Drupal 8 and 9 at the same time, meaning less support burden and earlier Drupal 9 compatibility. In March 48% of contributed projects on Drupal.org were already "Drupal 9 compatible" (as we could detect at the time), and that increased 54% by July (highfive!). These projects will now have one more thing to do, a one line info.yml file addition, to allow their existing codebase to also run on Drupal 9.
✋Drupal contrib maintainers, thank you! You are already contributing a lot to a successful Drupal 9. Projects that are "compatible with Drupal 9" (as much as we know now) are up to 54% from 48% since March. Deprecated API use is decreasing overall.
Drupal 8 looks at extension .info.yml files and checks for the core: 8.x key to check if the extension can be used with Drupal 8. Unfortunately this was not adequate already due to API additions in Drupal 8 minor releases. You could not specify an extension was only compatible with Drupal 8.4+. People devised clever tricks with dependency on certain versions of the system module, but that was still limiting due to inability to depend on specific patch releases. In case your extension depended on a bugfix being present, you cannot set a dependency on Drupal 8.4.1+ in any way.
Introducing the core_version_requirement key
While solving this issue it became apparent that solving it with the existing core: 8.x key is going to cause backwards compatibility problems with existing Drupal 8 sites. The format of the value could not be changed without causing issues, so a new core_version_requirement key was introduced. This now allows to depend on major and minor versions of core as well as specify multiple version requirements even between multiple minor Drupal versions. For example if you would need a bugfix that is to be included in both Drupal 8.7.23 and 8.8.3 (imaginary future versions) but you wouldn't care for the minor version, then you would use the following, making your module incompatible with versions before the particular bugfix you need on both minor branches, as well as incompatible with all older Drupal 8 branches:
name: My Module type: module core_version_requirement: ~8.7.23 || ~8.8.3
Such precision was never possible before!
The new feature is made available in 8.7.7 to help prepare for Drupal 9, as this makes it possible to also declare Drupal 9 compatibility with the existing codebase you already have. Drupal 8.7.x will get security support up until the planned release of Drupal 9 on June 3, 2020. For extensions that only support Drupal 8.7.7 and later you can now exclusively use the core_version_requirement key. If the extension is also Drupal 9 compatible:
name: My Module type: module core_version_requirement: ^8.7.7 || ^9
For extensions that also support older versions, you should keep the core: key as well. The core_version_requirement key will be ignored by older Drupal versions, and new versions will validate that you specified non-conflicting requirements in the two keys (with regards to Drupal 8 support) so for extensions supporting older core versions, core_version_requirement also needs to be permissive, but could also allow compatibility with Drupal 9 too:
Read more about this change in the change notice. There is also work ongoing to support dependency metadata from composer.json to drive Drupal's dependency resolution as well. That will coexist with this simpler approach and will provide a more composer-native solution for those who already adopted composer for their extensions. In the meantime at least core_version_requirement will limit what Drupal would install while composer.json requirements will limit what composer will download. Because both Composer and Drupal use the term install, but those don't mean the same thing, here is a comparison chart:
What happens to Drupal.org project version numbers then?
So a 8.x-1.3 version of an extension on drupal.org may only be compatible with a subset of core, but that was already the case earlier. Now an 8.x-1.3 version of an extension on drupal.org may also be compatible with Drupal 9. How will this work? Drupal's composer integration already ignores the 8.x prefix from version numbers and only cares about the project specific version. Drupal core already only installs extensions where the core compatibility requirements are met regardless of which version number was the package you used to download (8.x modules may be only compatible with 8.4+ already). So the ultimate plan is to do away with the 8.x prefix in extension numbers entirely on drupal.org and support semantic versions for contributed extensions instead. In the short term, the Drupal.org project browser also needs to be updated to take the dependency metadata and use that to power compatibility filters, instead of merely using the version number prefix.
Of the total of 23750 deprecated API use instances found in 7561 projects, 29% of them were instances of drupal_set_message(). So statistically speaking if you stop using this single function, you are already 29% on your way to Drupal 9 compatibility (likely more for most projects).
On top of the 29% of drupal_set_message(), there is another 47% of various other API uses that can be fixed now. This means that those deprecated APIs were marked before Drupal 8.6.0 was released and are therefore in currently unsupported Drupal core versions. So stopping the use of them would still keep your code compatible with all currently supported Drupal core versions. In other words, as of today, drupal.org project maintainers can resolve three quarters of the outstanding code changes for Drupal 9 compatibility. Considering we are still ten months from before Drupal 9's planned release date, this is quite good news!
You are not a drupal.org project maintainer but want to help? Yay! Based on the above it may be tempting to run to the issue queue and submit issues for fixing all the things. Good plan! One thing to keep in mind though is to work with the maintainers of projects so your help benefits the project most effectively. Drupal.org project owners may specify Drupal 9 plan information that should help you engage with them the best way (use the right meta issue, know of their timeline plans, and so on). Check the project page of projects you are interested to be involved with to make sure you contribute the best way.
https://t.co/hf2ENvlZSo projects can now specify Drupal 9 porting information, so *you* can direct *your* contributors to provide the most valuable help on the way to Drupal 9, fund the process or just step back (for now). Edit your project to help your contributors help you! pic.twitter.com/l1OWwOllBK
— The Drop is Always Moving (@DropIsMoving) May 21, 2019
More to come
I think the data is super-interesting and I plan to do more with it, for example cross-referencing with project usage. Stay tuned for more information in the future. In the meantime all the source data is available for your mining as well.
Correction: An earlier version of this post said there were 43% additional fixable deprecations for a total of 72%. Thanks to Ryan Aslett for corrections, the correct numbers are 47% and 76% respectively. Images and text fixed.
Disclaimer: The data is based on the state of contributed projects on July 29, 2019 based on Drupal core's 8.8.x development branch on July 29, 2019. As contributed module maintainers commit fixes, the data will get better. As core introduces more deprecations (until Drupal 8.8.0-alpha1), the data could get worse. There may also be phpstan Drupal integration bugs or missing features, such as not finding uses of deprecated global constants yet. This is a snapshot as the tools and state of code on all sides evolve, the resulting data will be different.
On our way to Drupal 9 every site will need to take care of making updates to their custom code as well as updating their contributed projects. However this time, instead of needing to rewrite code, only smaller changes are needed. Most contributed modules will only need to deal with a couple changes. Collaborating with project maintainers is the best way to get to Drupal 9. The first beta of the Upgrade Status module alongside recent drupal.org changes focus on making this much easier.
Upgrade Status beta provides better insight into Drupal 9 readiness
Take the first beta of the Upgrade Status module and run it on your site. It will provide executive summaries of results about all scanned projects and lets you inspect each individually.
Custom and contributed projects are grouped and summarised separately. You should be able to do all needed changes to your custom code, while for contributed projects you should keep them up to date in your environment and work with the maintainers to get to Drupal 9. The later is facilitated by displaying available update information inline and by pulling the Drupal 9 plan information from drupal.org projects and displaying it directly on the page.
This is how the summary looks like after scanning a few projects:
Image
Digging deeper from the executive summary, you can review each error separately. The beta release now categorizes issues found to actionable (Fix now) and non-actionable (Fix later) categories with a Check manually category for items where it cannot decide based on available information. For custom projects, any deprecation is fixable that has replacements in your environment while for contributed projects supporting all core versions with security support the window is shifted by a year. Only deprecations from two or more releases earlier can be fixed (compared to the latest Drupal release) while keeping Drupal core support. So somewhat ironically, Upgrade Status itself has deprecated API uses that it cannot yet fix (alongside ones it could fix, but we have them for testing purposes specifically):
Own a Drupal.org project? Direct contributors to help you the way you prefer!
If you own a Drupal.org project that has Drupal 8 code, you should specify your Drupal 9 plans. It is worth spending time to fill in this field to direct contributors to the best way you prefer them help you, so contributions can be a win-win for you and your users alike. Whether it is a META issue you plan to collect work or a specific time in the future you will start looking at Drupal 9 deprecations or a funding need to be able to move forward, letting the world know is important. This allows others to engage with you the way you prefer them to. Additionally to it being displayed in Upgrade Status's summary it is also displayed directly on your project page!
Go edit your project and find the Drupal 9 porting info field to fill in. Some suggestions are provided for typical scenarios:
Image
This will then be displayed on your project page alongside usage and security coverage information. For example, check it out on the Upgrade Status project page.
Thanks
Special thanks for dedicated contributors and testers of the Upgrade Status module who helped us get to beta, especially Karl Fritsche (bio.logis), Nancy Rackleff (ThinkShout), Tsegaselassie Tadesse (Axelerant), Bram Goffings (Nascom), Travis Clark (Worthington Libraries), Mats Blakstad (Globalbility), Tony Wilson (UNC Pembroke), Alex Pott (AcroMedia, Thunder), Charlie ChX Negyesi (Smartsheet), Meike Jung (hexabinær Kommunikation). Thanks to Neil Drumm (Drupal Association) and Angela Byron (Acquia) for collaboration on the Drupal 9 plan field.
Before I even presented the slideshow first, I already got improvement suggestions from people presenting it elsewhere. So the first time I got to present this session in Minsk, Belarus at the kind invitation of DrupalCamp Belarus organizers, it was already improved from when I first published it. I will keep updating the slides and present where I can in the coming months. Please, do it yourself too! Translate, shorten, lengthen, etc. to match your audience and timing needs, as long as the key messages get through. Be an important contributor to Drupal 9 this way!
Dwayne created an updated process and a new repository this week with fresh data. I was still curious so I delved right into the data and started mining it. A key question I was interested in is how much of the most widespread deprecations are actionable right now. An actionable deprecation is something core deprecated in a previous version that is not supported anymore, so you can update your code to remove the use of that API. Currently Drupal 8.6 and 8.7 are supported, so deprecations there should only be acted on for your custom code. However deprecations in and before 8.5 are entirely fine to act on.
First I counted the top list of deprecated APIs used from Dwayne's data across all of contributed projects. Then I wrote a script to collate api.drupal.org documentation to the deprecation notices. Ideally phpstan itself would report these messages directly and Matt Glaman is working on that. However since that is still blocked on the phpstan side, one needs a different data source to find the deprecation documentation for each occurrence, so I took to api.drupal.org to get that for now. Once that is found, we can categorize the deprecations into actionable, not actionable and actionable for custom code only. For the later case you know which core version you are using, and that should be an up to date minor version. So you don't need to deal with what core branches the community supports otherwise.
The results look really promising so far in terms of how much contributed modules can make progress on even today. If all already actionable deprecations get resolved, there will be very little left at least of the deprecations we already know.
As with all interesting data sets, this summary is just the tip of the iceberg. There is huge potential to mine this data set for other uses, such as finding modules that potential contributors at an event could contribute fixes to. I don't know yet how much I can continue to work with this data myself, and of course others doing analysis of their own would be more than welcome.
Are you a drupal.org project maintainer? Now would be a good time to fill in your Drupal 9 porting information in your project, so you can let contributors know how to best engage with your in the process towards Drupal 9.
https://t.co/hf2ENvlZSo projects can now specify Drupal 9 porting information, so *you* can direct *your* contributors to provide the most valuable help on the way to Drupal 9, fund the process or just step back (for now). Edit your project to help your contributors help you! pic.twitter.com/l1OWwOllBK
— The Drop is Always Moving (@DropIsMoving) May 21, 2019
Disclaimer: The data is based on the state of contributed projects on May 20, 2019 based on Drupal core's 8.8.x development branch on May 20, 2019. The lookup on api.drupal.org was not perfect and I found some bugs there that are being resolved as well so the data gets more accurate. Also, as contributed modules will get updated, there will be less uses of deprecated APIs. As core will introduce more deprecations, the data could get worse. There may also be phpstan Drupal integration bugs or missing features, such as not finding uses of deprecated global constants yet. This is a snapshot as the tools and state of code on all sides evolve, the resulting data will be different.
