How to automate testing whether your Drupal 8 module is incompatible with Drupal 9?

Drupal 9 is planned to be only 18 months away now, wow! It is already being built in Drupal 8 by marking APIs to be removed in Drupal 9 as deprecated and eventually upgrading some dependency version requirements where needed. Once the Drupal 9 git branch will be open, you will be able to test directly against Drupal 9. That should not stop you from assessing the compatibility of your module with Drupal 9 now. To prepare for compatibility with Drupal 9, you need to keep up with deprecated functionality and watch out for upgraded dependencies (when we know which are those exactly). Of these two, automation can go a long way to help you keep up with deprecated APIs.

What does deprecated code look like?

Deprecations include a trigger_error() call that allows catching when you use deprecated functionality. Here is an abbreviated example from the documentation:

<?php
/**
 * [...]
 */
function drupal_clear_css_cache() {
  @
trigger_error('drupal_clear_css_cache() is deprecated in Drupal 8.0.0 and will be removed before Drupal 9.0.0. Use \Drupal\Core\Asset\AssetCollectionOptimizerInterface::deleteAll(). See https://www.drupal.org/node/2317841.', E_USER_DEPRECATED);
  \
Drupal::service('asset.css.collection_optimizer')->deleteAll();
}
?>

The trigger_error() call here only fires when the error reporting level includes E_USER_DEPRECATED errors, and the message provides full explanation as to what exactly is deprecated, which Drupal version will stop providing it and what should be used instead.

Unfortunately not all deprecated code is yet updated to this standard, but help is more than welcome. We were already deprecating APIs before we instituted the clever trigger_error() format, and the old deprecations need some work to be updated. However usages of those that are already marked are possible to catch with automated testing.

Make tests catch usage of deprecated APIs

Drupal.org has a great automated testing system. Whenever someone uploads a patch for consideration, the test suite of the corresponding project runs and results are posted back on the issue. Changes that fail with the test suite of the project are never committed. You can use this system to try out how far is your contributed module from working on what we currently know to be Drupal 9. Let's see how this works.

First of all, your module should have automated testing enabled. Make sure you have at least one test in your project and configure automated testing for your module on the Automated testing tab of your project page.

Drupal.org's automated testing is driven from an (optional) drupalci.yml file in projects. Core's drupalci.yml file for example always exposes usages of deprecated APIs in test fails. On the other hand, contributed modules are not tested for usages of deprecated functionality by default. (Note no suppress-deprecations: false clauses in the file).

However you can choose to add a drupalci.yml file to your project repository to expose these failures or keep an issue on your project queue with a patch including the file to learn about the failures. Adding the file to your project outright ensures that the project keeps being Drupal 9 compatible as much as possible. On the other hand once a new minor version of Drupal 8 comes out, new deprecations could fail existing releases of your module in this case. If the newly deprecated APIs were replaced with newly added APIs, then fixing those failures would mean you increase the Drupal minor version requirement of your module, thus not being compatible with older minor versions. That may be problematic. For now you can at least add a testing issue to see what you are up to to begin with.

To make your contributed module tests fail when your module uses deprecated functionality, you need to take the default build.yml file for contributed projects and follow the instructions on drupal.org to take the assessment part of the file and add a suppress-deprecations: false directive to each testing section. The resulting file looks like as follows (everything being a straight copy-paste from the assessment section of the default build.yml file other than the deprecations directives). Save this in a drupalci.yml file in the root of your project:

build:
  assessment:
    validate_codebase:
      phplint:
      container_composer:
      csslint:
      eslint:
      phpcs:
    testing:
      run_tests.standard:
        types: 'Simpletest,PHPUnit-Unit,PHPUnit-Kernel,PHPUnit-Functional'
        # Test for Drupal 9 compatibility
        suppress-deprecations: false
      run_tests.js:
        concurrency: 1
        types: 'PHPUnit-FunctionalJavascript'
        # Test for Drupal 9 compatibility
        suppress-deprecations: false
      nightwatchjs:

If you already had a drupalci.yml file, then adapt its content adding the suppress-deprecations: false directives only. Create a patch file out of it for your project, or take mine from https://www.drupal.org/files/issues/2018-12-17/3020957-2.patch, which should be universally applicable to projects that did not have a drupalci.yml before. Once the patch is uploaded to an issue and set to needs review, you will get deprecations results.

My results and what do they mean

I tried this out on Configuration inspector, a module I help maintain, that provides insights about configuration structures and schemas for developers. As a developer focused module, it would be key to have it up to date with Drupal 9 so it will be trivial to use when the time comes. And the results were green, everything passed.

Ok, does that mean my module is Drupal 9 compatible? Well, not really. This method is only capable to tell me if there are incompatibilities, a green result does not guarantee that the module is Drupal 9 compatible. Why?

  • The method is dependent on all of core (and any other contrib dependencies) marking their deprecated code properly. As stated above this is not yet complete even for core. Help with the core issue and encouragement for contributed module authors to adopt the same deprecation format is welcome.
  • There will be more deprecations until Drupal 9 is released, and we don't know yet the full extent of those, so the result can only tell us so much about the present state.
  • Even in the present state, the results only represent code that my contributed module's test suite actually ran from my module. If my test suite is not very thorough (which in this case unfortunately is the case), then I cannot trust this green result either that it proves Drupal 9 compatibility. I know you are better in test coverage for your contributed modules, so your results would be better.
  • Drupal 9 will bring updated dependencies like Symfony 4 or 5, possibly Twig 2, etc. Usage of those dependency APIs directly would only fail tests if they would use the same trigger_error() deprecation mechanism. I believe that is not the case, for example, Symfony has its own Deprecation Detector that uses code sniffing instead of a test suite.

On the other hand if there are failures, those should be genuine compatibility issues that you should fix for Drupal 9 compatibility. Therefore the title of this post: How to automate testing whether your Drupal 8 module is incompatible with Drupal 9?

Keep in mind that some failures may be results of contributed module dependencies of your module not being up to date. You may not be able to do anything with those failures directly, however letting the maintainer know and submitting fixes to those projects is very welcome.

How will this improve in the future?

As more code in core and contributed modules is marked properly deprecated with the current standard, more of the incompatibilities will be surfaced by this testing method. Also a checkbox for Drupal 9 deprecation testing is planned to be added to the automated testing configuration screen of projects, to make it easier to configure environments to test for deprecated code use. That will make the drupalci.yml creation of this tutorial obsolete, but all caveats and conclusions will still apply.

How does this test work out for your module?

Add new comment