The content translation problem

One of the great goals of the Drupal 8 Multilingual Initiative (D8MI for short) is to have one unified system for content translation. The basic problem is that with Drupal 7, you have two ways to translate content: copy nodes for different language versions (with the built-in Content translation module) or save different languages under one entity (with the built-in multilingual fields capability). Although the later does not have a user interface in core, the API is there, so well respecting contributed modules need to support both. The reality is that many modules support neither, because node copies are combersome and field language support is painful.

This is both a user and a developer problem. Users need to decide their translation methods up front, and both methods have their advantages and limitations. Node copies allow for best workflow because they have authors, publication status, permissions, core search support, etc. all a given. Field language on the other hand works better with relations (when signing up for nodes, putting nodes into a common menu, etc.) as well as sharing values between translations (product images, non-translated attributes, etc.). The grand plan for Drupal 8 is to figure out a way for a system that marries the advantages of all as possible and have one better configurable system instead of two independent systems. This should make it easier for users and developers alike to work with multilingual entities.

This is an extremely simple idea, yet the implementation is lagging behind enourmously.

Plans to solve this problem

The following figure illustrates major steps we need to do to get to the desired results:

To be able to unify the two systems, we need every bit of a node/entity be able to vary by language (except the identifier of the entity of course), so you can have titles for different languages, authors, publication status, creation time, etc. for different languages. These have direct implications on the Drupal permission system, menu item visibility, repeatability of widgets for properties, etc. This is the biggest part of the work, since it requires all properties that are currently stored on the entity base tables to be varied by language, essentially bringing their storage much closer to how fields are stored. While we have been discussing the ins and outs of this for years, we have not made progress in implementing it, however, it is a very key area and would need subject experts to chime in to make the whole plan possible.

• An epic discussion was taking place at http://drupal.org/node/1498634 to decide on the implementation path
• An approach was proposed to convert the test entity first: http://drupal.org/node/1658712 (needs more help)
• This is just the beginnig of the beginning. A huge set of issues will spawn from this to convert other entity types, make EFQ support this property variance, make places querying properties directly use the new approaches, etc.

For all the side effects in the permissions, search, etc. systems, I started with implementations but more help is needed for the complex cases. I don't want to make this complex for you, but if it is forced on Drupal 8 without proper consideration and wide feedback it will be a pain to use, so why not work with the proposals now?

• Runtime support for language is already added in node_access: http://drupal.org/node/1658814
• Runtime language is also already added in _node_revision_access: http://drupal.org/node/1667614
• There is a large need for eyes on the making node access records language aware: http://drupal.org/node/1658846
• Search module should have had field language support already in Drupal 7, but its not a simple question, starter patch at http://drupal.org/node/1669876 but needs lots of work.

Of course if all properties are made to be stored much more like fields, the developer nightmares associated with field data access and modification come to haunt us even for properties, right? Well, we have solution proposals for that too, but they are mostly in the works and need eyes as well.

• We got general entity getter/setters in core so you can request properties in different languages already: http://drupal.org/node/1277776 (this is used by the runtime node_access() language support patch above in anticipation of property language support landing later)
• There is a lot more to do to make property/field access simple. Fago is leading an unofficial initiative to improve the developer experience tremendously by introducing a common property API that would give us lots of "syntactic sugar" to make refering to and working with fields/properties much easier. This needs more hands and reviewers too and is not close to complete yet: http://drupal.org/node/1346214
• We obviously need to start using the API as properties become translatable, like making node title queried as translatable (http://drupal.org/node/1616962) and making term names accessed in a multilingual context (http://drupal.org/node/1616972); these are again just the tip of the iceberg, just two properties of just two basic entities, yet lots of work went into trying to make them work

Having all properties translatable and the API unified and simplified and related core features aware of languages under entities is still not enough. We need to migrate the existing node copy translation data to the new system. We are not there yet to work on this one, but if we are able to introduce full property translation in Drupal 8, this is no question part of the requirements. It would also let us finally drop the node copy based translation UI and API from core.

Finally, we need a user interface to translate entities to different languages. Currently entity forms focus on editing the original version of the entity and contributed modules are required to generate translation forms. Making the entity form language-aware is a much saner way to approach the problem since that integrates better with contributed modules as well, be it pathauto or some other entity add-on that works with the entity form.

• An essential step for making entity translation possible is to introduce entity form controllers: http://drupal.org/node/1499596 - this is getting close to be done hopefully but needs love nonetheless

What will fail from this plan?

Let's face it, we are 4.5 months (that is just 20 weeks if I'm counting right) away from feature freeze, and most of the above are heavy-duty features. Its not in the realm of cleanup if we need to remove a module or introduce major storage and API changes for properties. So it is hard to say at this point that this plan will fully succeed. Especially that you dear readers are not in the issue queues helping us make it happen.

Another way this could fail obviously is we do make the changes happen but we cripple the storage and/or APIs in ways that will make it harder, not easier to work with Drupal. Again, you can ensure that this does not happen by directly being involved with the issues. We want to do what is best but we also have some requirements to fulfill which evidently will make Drupal more complex at places. It will do more, but will not stay as easy as it was.

Just 4.5 months is an extremely limited timeframe to make all this happen, and we really need everybody interested in making multilingual content support simpler and more unified to step up your game and get involved. Although Drupal 8 is far from being released, the deadlines set up don't really allow us to plan for putting in any features beyond the end of November, that is the cutoff. If you planned to help out next year, there will be code cleanup tasks, but no new features involved as per the announced Drupal 8 timeline, so it will just be late to work on most exciting new stuff.

If you want to get involved now, the above issues should provide plenty of opportunities. If you need any personal guidance, I can connect you with the right people to help you get on board either through my personal contact form or on any Drupal 8 Multilingual Initiative meeting (next coming up on the 18th of July). We are also organizing a mega-sprint (most likely the last for Drupal 8 Multilingual efforts this year) around Drupalcon Munich, 3 days before and 3 days after the main session days. See http://groups.drupal.org/node/238003 for more information. But waiting until then to get involved is wasting 5 weeks (out of 20 left), just keep that in mind.

Do you want to be involved shaping how Drupal 8 is going to support multilingual sites or just be a user of what was served for you, for all its good and bad? There is no better time to choose.