The content translation problem

One of the great goals of the Drupal 8 Multilingual Initiative (D8MI for short) is to have one unified system for content translation. The basic problem is that with Drupal 7, you have two ways to translate content: copy nodes for different language versions (with the built-in Content translation module) or save different languages under one entity (with the built-in multilingual fields capability). Although the later does not have a user interface in core, the API is there, so well respecting contributed modules need to support both. The reality is that many modules support neither, because node copies are combersome and field language support is painful.

This is both a user and a developer problem. Users need to decide their translation methods up front, and both methods have their advantages and limitations. Node copies allow for best workflow because they have authors, publication status, permissions, core search support, etc. all a given. Field language on the other hand works better with relations (when signing up for nodes, putting nodes into a common menu, etc.) as well as sharing values between translations (product images, non-translated attributes, etc.). The grand plan for Drupal 8 is to figure out a way for a system that marries the advantages of all as possible and have one better configurable system instead of two independent systems. This should make it easier for users and developers alike to work with multilingual entities.

This is an extremely simple idea, yet the implementation is lagging behind enourmously.

I've highlighted some great developments around multilingual Drupal in my Denver core conversation talk (see the slides and video for details on where Drupal 8 multilingual improvements are going or check out the interview with me on Modules Unraveled from last week). First of all, there are some great new developments with each, second these got little publicity if anything, so I wanted to broadcast the good news here too.

Drupal 7 Multilingual Sites book is out

Image

The Drupal 7 Multilingual Sites book is out from Packt Publishing! Jose Reyero (Internationalization module maintainer) was a technical reviewer, and I've been involved with the book as a technical reviewer in parallel (without monetary compensation), so even if you might not recognize the name of the author (Kristen Pol), I can assure you that it is a strong book. It starts off from the basics and reaches as far as it comprehensively can in the small size. It goes through some complex topics like tips and tricks for Views, Panels and SEO, and even includes comprehensive tables for which module to use for translating what. Some topics were cut out due to the size boundaries but you can already read one of those as an article on Kristen's blog on Drupal Commerce localization. The book should be a life-saver for anybody starting out building multilingual Drupal 7 sites and even those who already got started but stumbled into some seemingly unsolvable issues. Admittedly as the book goes into some of the more advanced topics, you'll face note boxes more and more as it references issues where fixes for problems are in the works. Help is always welcome.

Ps. Kristen is happy to donate ruffle copies of the book for your events, see https://twitter.com/#!/kristen_pol/status/193752939515490304

New Translation Management Tool module

While there are relatively good tools to translate content, configuration and the user interface in Drupal, larger scale setups will require more complex translation workflows with queues for people working on translations and integration with outside services and subcontractors for translation. Remember the Translation Management module? Well, that was a bit monolithic solution to this problem, did not integrate with common Drupal components such as Views and Rules and was never fully ported to Drupal 7. There was no community behind it and as the developers stopped working on it, there was no solution for people to look to.

Enter Translation Management Tool (tmgmt) a solution along the Drupal way (TM) to the same problems. It is component based, already integrates with more services and is backed by several companies in Europe. It was bootstrapped earlier this year in Europe and is actively developed ever since. For further icing on the cake, a great project to improve and build on the module was accepted for this year's Google Summer of Code, so we are going to see even more awesomeness coming from there!

Drupal as base implementation in European Union multilingual project

Carsten and Carina from Cocomore held a discussion at Drupalcon Denver incorporating their announcement of the EU funded "MultilingualWeb - Language Technologies" (MultilingualWeb-LT) project for the Drupal community. The project is managed by a W3C Working Group and its goal is "to combine several existing technologies and standards to close gaps in the technical localization chain by better integrating localization service providers and translation technologies". It is not just a standards effort, the initial implementation is going to happen based on Drupal! Yes! We hope to work with this project on Drupal 8 improvements if the timeline allows and they certainly plan to make heavy use of the tmgmt module on Drupal 7 which should lead to more improvements and fixes there.

xjm has a great guide for doing interdiffs using two branches for the original patch and your new changes. I must admit I'm lazier than that, and have a much simpler process for doing interdiffs on patch updates. Of course this only works if you work on one patch at a time. Branches are better when you work on different things and want to keep those things around. With that, here is my simpler interdiff workflow.

I've had the great opportunity to share my experience navigating the waters of Drupal core development at at DrupalCon Denver last month. My talk "Thrown Into a Shark Pond? A Guide for Surviving Core Development and Even Enjoying It" was possibly a little sensationally titled, although every Drupal core developers have their ups and downs and sometimes people do feel like they are in a shark attack. I planned to provide good ways forward from different ways that ideas can be blocked from inception through implementation to getting it into core.

Image

When preparing for the session, I realized I'm going to explain a somewhat complicated tree with different decision points and states. I wanted my session to be a useful and clear explanation and let people focus on tips and tricks instead of piecing together this tree in their head, so I decided to design a handout for the attendees (PDF, 250k). This turned out to be pretty great I think, and I got lots of content feedback from xjm, webchick, Moshe Weitzman, Kieran Lal and even Dries at various stages of drafting it. (Getting it printed on-site was a herculean undertaking, but that is really due to the printing shop services available.) At the end, each attendee got a nice color copy of this that they could bring home (and the leftovers I had were distributed at the new contributors sprint at the end of the conference). After all I decided to not theme the talk or the handouts with sharks, in hopes that the handouts would be much more easily reusable later just as well.

You can also watch the recording of the session below, and download the slides (PDF, 7.8M).

After reviewing language support and translation for many of Drupal's pieces, we arrived at a pretty complex question, building multilingual navigation. The question is especially of importance because we often need to put translated content in menus, and the cross of translation of content and translation of menus can easily get us into the woods. Let's build some simple solutions for different use cases to see how to think of multilingual menus.

I know many of you faced the goal explained above. There are tools of different levels of involvement and there is of course no ready-baked answer to this question, but here is my best take so far for the current two active versions and Drupal 8 in development.

The three areas of Drupal language support

(A) First off, you can run a single language foreign language website without a need for content or configuration translation. Because the Drupal user interface comes in a flavor of English, you'll need to translate that. But all your content and configuration can be entered in your language, so you are fine there.

(B) Second, if you need to mark your content with language information, such as if you are running a multilingual blog, where you post in different languages, but will not translate your posts, you need language assignment with multiple language options.

(C) Finally, if you need to have the same content translated, the same navigation replicated or similar navigation produced for different languages, you need translation for your content and configuration.

The three types of data for Drupal translation

When it comes to translation, Drupal data can be separated to three buckets: (1) User interface (2) Content (3) Configuration. Drupal has very extensive support for user interface translation, I'd say too much support for content translation and usually not so bright support for configuration translation. Let's enumerate what Drupal has on offer for each piece.

As some of you might be aware, a group of talented and very determined people sprinted in Berlin about a month ago just to improve the i18n module for almost a week. A lot of great improvements made it in including tests, translatable contact forms and even some great usability improvements. Jose Reyero and Friederike Schmiedebach have great posts about the sprint.

Before continuing my article series, I wanted to touch quickly on the usability improvements, because my previous piece presented a pretty grim picture of textgroup based translation solutions such as blocks. Well, i18n module is still using that backend, but the new user interface improvements will make the translation process much more transparent and a lot easier to work with when managing Drupal configuration.

My last post where I've explained how Internationalization module re-implements some of Field API and where it does not do that it misses crucial functionality did not get much discussion. Therefore I decided to turn the key point at the end to the center of discussion: that either Drupal core will do fields for all user input (content and configuration alike, all through form your site name to your views empty text), or i18n module needs to do it in contrib. There is a clear need for input widgets, validators, permission handling, storage and output formatters and rendering used consistently. If it is not done by core, it will keep being a bolted-on half-failing approach despite best efforts in contrib. Please discuss at http://groups.drupal.org/node/154434

The other important post that we need your input on is about removing all UI strings from code. There are various issues with having them in code, while there are also various disadvantages to removing them from there. There are performance, translatability and even user experience concerns involved. This post is already getting some discussion, but we need much more. This could be a huge, fundamental change, so all your input is welcome. Don't say we did not ask you. Please discuss at http://groups.drupal.org/node/154394

Your input helps shape Drupal 8 and how Drupal supports building multilingual sites for years to come. Have your voice heard now!

There is great discussion forming on my previous posts on exportables and user provided text as well as the dangers of using t() for user editable data, and I can only hope we can keep that up! In that tradition, I'm cross-posting this piece to groups.drupal.org as well for discussion.

Regular readers could find this boring, but let's reiterate the three working modes that all objects should ideally be able to handle in Drupal to support multilingual site building.

1. Being able to mark an object as in one language.
2. Being able to mark an object as in one language and relate it to others as being a translation set. This is useful when you want to use the different language objects in different relations, track their history separately, have different permissions and workflows for them, etc.
3. Being able to translate pieces of the object that need translation and leave the rest alone. Load the right language variant of the object dynamically as needed. This is very useful for keeping external relations intact and sharing common fields between translations effortlessly.

There are certain things, where not all of these make sense. For the site's name for example, people would probably only use either (1) or (3). For a block for example, people should be able to use either based on their needs. (2) is useful to place blocks differently on translated pages, (3) is good to keep the placement consistent without effort. This can be different on a per-block basis. Same applies to nodes, menus, taxonomy, views, rules, and so on.

In my previous post titled Drupal's multilingual problem - why t() is the wrong answer posted on my blog and on groups.drupal.org for feedback, I've detailed issues with using t() as a translation tool for "user provided data". This post goes into some further details, a discussion of current solutions which could form basis for discussion of future solutions.

How can we even tell the difference between code and user provided translatables?

It is fair to assume that many multilingual sites will not have English as their default language (many not even as any of their supported languages), so we cannot assume that blocks, menus, and so on are entered in English. However, source code based strings are considered part of the user interface, and as such assumed to be written in English. What does this has to do with default configurations set up by modules and How do we reconcile this with the growing popularity of exportables and features (as in Feature module generated versioned export packages)? Let's look at these two questions.