i18n

As some of you might be aware, a group of talented and very determined people sprinted in Berlin about a month ago just to improve the i18n module for almost a week. A lot of great improvements made it in including tests, translatable contact forms and even some great usability improvements. Jose Reyero and Friederike Schmiedebach have great posts about the sprint.

Before continuing my article series, I wanted to touch quickly on the usability improvements, because my previous piece presented a pretty grim picture of textgroup based translation solutions such as blocks. Well, i18n module is still using that backend, but the new user interface improvements will make the translation process much more transparent and a lot easier to work with when managing Drupal configuration.

My last post where I've explained how Internationalization module re-implements some of Field API and where it does not do that it misses crucial functionality did not get much discussion. Therefore I decided to turn the key point at the end to the center of discussion: that either Drupal core will do fields for all user input (content and configuration alike, all through form your site name to your views empty text), or i18n module needs to do it in contrib. There is a clear need for input widgets, validators, permission handling, storage and output formatters and rendering used consistently. If it is not done by core, it will keep being a bolted-on half-failing approach despite best efforts in contrib. Please discuss at http://groups.drupal.org/node/154434

The other important post that we need your input on is about removing all UI strings from code. There are various issues with having them in code, while there are also various disadvantages to removing them from there. There are performance, translatability and even user experience concerns involved. This post is already getting some discussion, but we need much more. This could be a huge, fundamental change, so all your input is welcome. Don't say we did not ask you. Please discuss at http://groups.drupal.org/node/154394

Your input helps shape Drupal 8 and how Drupal supports building multilingual sites for years to come. Have your voice heard now!

There is great discussion forming on my previous posts on exportables and user provided text as well as the dangers of using t() for user editable data, and I can only hope we can keep that up! In that tradition, I'm cross-posting this piece to groups.drupal.org as well for discussion.

Regular readers could find this boring, but let's reiterate the three working modes that all objects should ideally be able to handle in Drupal to support multilingual site building.

1. Being able to mark an object as in one language.
2. Being able to mark an object as in one language and relate it to others as being a translation set. This is useful when you want to use the different language objects in different relations, track their history separately, have different permissions and workflows for them, etc.
3. Being able to translate pieces of the object that need translation and leave the rest alone. Load the right language variant of the object dynamically as needed. This is very useful for keeping external relations intact and sharing common fields between translations effortlessly.

There are certain things, where not all of these make sense. For the site's name for example, people would probably only use either (1) or (3). For a block for example, people should be able to use either based on their needs. (2) is useful to place blocks differently on translated pages, (3) is good to keep the placement consistent without effort. This can be different on a per-block basis. Same applies to nodes, menus, taxonomy, views, rules, and so on.

In my previous post titled Drupal's multilingual problem - why t() is the wrong answer posted on my blog and on groups.drupal.org for feedback, I've detailed issues with using t() as a translation tool for "user provided data". This post goes into some further details, a discussion of current solutions which could form basis for discussion of future solutions.

How can we even tell the difference between code and user provided translatables?

It is fair to assume that many multilingual sites will not have English as their default language (many not even as any of their supported languages), so we cannot assume that blocks, menus, and so on are entered in English. However, source code based strings are considered part of the user interface, and as such assumed to be written in English. What does this has to do with default configurations set up by modules and How do we reconcile this with the growing popularity of exportables and features (as in Feature module generated versioned export packages)? Let's look at these two questions.

Drupal is a great system to run foreign language websites on. The core itself is written in English and modules and themes are expected to follow suit. For developers, very simple wrapper functions are available to mark your translatable strings and let Drupal translate them to whatever language needed. These are the famous t(), the less famous format_plural() and a whole family of other functions. See my cheat sheet (PDF) and the drupal.org documentation for more on this.

Then there is "the other side", whatever does not come from code. Drupal works pretty well and very consistent if you want all of those to be in a foreign language (i.e. not English), but not in multiple languages (any of which can be English at that point). Drupal only has direct multilingual support in nodes (+ fields of entitites) and for path aliases. But life with Drupal means you work with all kinds of other objects like blocks, views, rules, content types, etc which are not "language-aware".

Unfortunately for building multilingual Drupal sites, this is the biggest problem that needs to be worked around. The contributed Internationalization module attempts to fill in the gaps, provide language associations and different workflows for translating these language-unaware objects. This works to some degree, but is really not easy without much help from the modules implementing these objects.

In 2003 I selected Drupal to replace an aging PostNuke system on a Hungarian web developer community site (weblabor.hu) that I helped ignite a few years before. Of course I needed it fully translated to Hungarian. If you were using Drupal back then, you may remember that adding a new language to your site meant running ALTER TABLE queries on your locale tables to add new columns, editing your settings file and handling database dumps of your translations (which was the only way to share them too).

Drupal has come a long way since then.

I just got back from Drupalcamp Stockholm 2011, which was an action-packed two days for me to say the least. Due to a busy schedule, I was only able to arrive last minute the night before and leave just right after my sessions on the second day. Once again I decided to do a multilingual session all over again starting from the drawing board. There are lots of new things happening plus I think I'm developing better models to explain the components involved, so I think it was a good idea to build a new presentation. I've also presented on Drupal Security on behalf of the security team, which I hope turned out to be a very informative introduction to some of the most important things to look for when securing sites.

With the basics of node and site settings translation behind us, we are getting to the more complex parts, at least in terms of the user interfaces involved. While with node translation you get a tab on each node to translate it (regardless of setting up translation sets or using translatable fields), and with settings translation, you get quick jump links, the subsystems that work with textgroups will require a better understanding of how the Drupal systems relate.

Three ways to think about language support

When people want to have language support for their site, they typically think of one of three things:

1. Being able to mark an object as in one language. With node translation this was achieved by language enabling nodes.
2. Being able to mark an object as in one language and relate it to others as being a translation set. For nodes, this is supported by Drupal core's content translation module.
3. Finally, being able to translate pieces of the object that need translation and leave the rest alone. Load the right language variant of the object dynamically as needed. In the case of nodes, this is achieved with the contributed entity_translation module (formerly translation.module).

The first case is great when you don't need to translate the object, the second is great when you need to use translations in different contexts (for nodes, you can maintain a separate comment set, put in different menus, etc). The last is great when you want to maintain the object the same way regardless of language. This might be great for an e-commerce site. Read part 4 of my blog post series for exact details for nodes.

Applying this to blocks, the i18n module provides functionality (1) and (3), but not (2) at this point. Translation set support is being implemented for various objects (menus, paths, etc. are already covered by i18n), but not blocks yet. (1) is very simple to use, but (3) will be a real pain if you don't read this blog post...

While Drupal improves its multilingual features with every version and there were numerous improvements with Drupal 6 and 7 especially, there are still lots of things for which contributed modules are needed, and multilingual support is not consistent (neither in some cases usable) with these modules. There is sizable customization and glue-code building required.

Karsten Frohwein thought to take these problems and organize a group of contributors to take a deep look at them, conceptualize on better solutions and do actual implementation in a concentrated environment. Thankfully, as with great ideas, he got lots of support from companies and individuals interested, so the Internationalization SprintCamp is a go between the 11th and 15th of May 2011 in Berlin. Key contributors are being confirmed one by one, so this event is promising to be a great and high energy one for improving multilingual features and fixing bugs. There is place for up to 20 people, so we are looking for developers who can join and help. Contact Karsten through the Impressum page or leave a comment here with your contact information (such as drupal.org username or user URL).

The sprint is sponsored by the German Drupal-Initiative e. V., Comm-press, undpaul Drupal Development, Acquia and others. We are planning to do couch-surfing to reduce costs and increase fun. If you'd like to help sponsor, contact Karsten.

Ps. we will hang around in the #drupal-i18n IRC channel throughout the sprint, and distribute information and guidance for anybody who'd like to join and help virtually. See http://drupal.org/irc for more information on Drupal's IRC channels.

I just noticed the new #translatable module popping up via a note in a Drupal issue. Historically there was i18n module, which provided the de-facto way to build multilanguage sites. Then through some smaller modules, competition and somewhat different thinking came in with localizer module.