Traditionally Drupal was developed in two parallel branches. Take the example of Drupal 7 and 8. There was the supported stable branch of Drupal 7 that received bugfixes only and there was a development branch of Drupal 8 where developers went free to change and improve things the way they have seen best. As the development of Drupal 8 was nearing release, contributed module and theme authors were asked to take a look and release new versions as well in preparation. While the new branch was built with great care, it was not proven yet in various production environments. It also often took quite some time for contributed modules and themes to update.
Drupal 9 is planned to be only 18 months away now, wow! It is already being built in Drupal 8 by marking APIs to be removed in Drupal 9 as deprecated and eventually upgrading some dependency version requirements where needed. Once the Drupal 9 git branch will be open, you will be able to test directly against Drupal 9. That should not stop you from assessing the compatibility of your module with Drupal 9 now. To prepare for compatibility with Drupal 9, you need to keep up with deprecated functionality and watch out for upgraded dependencies (when we know which are those exactly). Of these two, automation can go a long way to help you keep up with deprecated APIs.
Earlier this week Steve Burge posted the intriguingly titled There Will Never be a Drupal 9. While that sure makes you click on the article, it is not quite true.
Drupal 8.0.0 made several big changes but among the biggest is the adoption of semantic versioning with scheduled releases.
Scheduled releases were decided to happen around twice a year. And indeed, Drupal 8.1.0 was released on time, Drupal 8.2.0 is in beta and Drupal 8.3.x is already open for development and got some changes committed that Drupal 8.2.x will never have. So this works pretty well so far.
As for semantic versioning, that is not a Drupalism either, see http://semver.org/. It basically means that we have three levels of version numbers now with clearly defined roles. We increment the last number when we make backwards compatible bug fixes. We increment the middle number when we add new functionality in a backwards compatible way. We did that with 8.1.0 and are about to do it with 8.2.0 later this year. And we would increment the first number (go from 8.x.x to 9.0.0) when we make backwards incompatible changes.
So long as you are on some version of Drupal 8, things need to be backwards compatible, so we can just add new things. This still allows us to modernize APIs by extending an old one in a backwards compatible way or introducing a new modern API alongside an old one and deprecate (but not remove!) the old one. This means that after a while there may be multiple parallel APIs to send emails, create routes, migrate content, expose web services and so on, and it will be an increasingly bigger mess.
There must be a balance between increasing that mess in the interest of backwards compatibility and cleaning it up to make developer's lives easier, software faster, tests easier to write and faster to run and so on. Given that the new APIs deprecate the old ones, developers are informed about upcoming changes ahead of time, and should have plenty of time to adapt their modules, themes, distributions. There may even be changes that are not possible in Drupal 8 with parallel APIs, but we don't yet have an example of that.
After that Drupal 9 could just be about removing the bad old ways and keeping the good new ways of doing things and the first Drupal 9 release could be the same as the last Drupal 8 release with the cruft removed. What would make you move to Drupal 9 then? Well, new Drupal 8 improvements would stop happening and Drupal 9.1 will have new features again.
While this is not a policy set in stone, Dries Buytaert had this to say about the topic right after his DrupalCon Barcelona keynote in the Q&A almost a year ago:
