localization

I've intended to announce this development at Drupalcon San Francisco but unfortunately the session on this was merged with a more general i18n session which was coupled by the ash cloud above Europe, so I could not go. Evidently, collaborative software translation is not a mainstream topic. On the other hand, I keep receiving requests of the general applicability of the tools Drupal.org uses about every two weeks, and this interest always amazes me. While the localization server tool used on localize.drupal.org grew out of needs of the Drupal community, the solutions were architected to be useful in a general purpose software translation environment. While the architecture was there, it was lacking useful UI controls to just run it as a generic software translation tool.

Existing non-Drupal users like the Gallery 2/3 project and the Musescore desktop app utilize custom data connector modules, which the localization server nicely supports, allowing for custom code to gather data for translation. Gallery even uses a custom Localization client port for clients to submit translations to the server, even though their software is not Drupal based. However, translating arbitrary software without writing your custom connector code was not possible earlier.

Finally, the promise of a centralized localization interface for Drupal modules and themes looks to be coming true. I've started work on this project around two years ago under Google Summer of Code sponsorship and was continuing maintenance and improvements ever since. While I was spreading the word on it, not many people signed up to help clean up some possible performance problems, so it did not make into Drupal.org yet.

However, earlier this year I've got reviews from some key people in the infrastructure team, especially Gerhard Killesreiter, who persuaded me that setting this up is more important then it not being perfect yet. Software is evolving matter anyway, and we should improve as we see the problems. So I've started to set up localize.drupal.org. While we work out some of the kinks like single sign-on with drupal.org (one of the promises of the drupal.org redesign which will be delivered here), I thought it would be a good idea to discuss the implications.

I've had the luck again to join Jose A. Reyero and present about the multilanguage features of Drupal and its contributions at Drupalcon DC last week. I've had a presentation on the topic at Drupalcon Szeged last August, and we had a session on the topic at Drupalcon Boston last March with Jose. So looking back it almost felt like we are going to repeat ourselves.

What made this time special however is that we have a huge amount of experience gathered from users of the modules and Drupal core itself, and we see our strengths, real use cases and problems better. Previous sessions covered the concepts, but this time we had the fantastic Roger Lopez join us as third panelist, who talked about the Drupal 6 based multilanguage platform used to host Britney Spears', Pink's and other Sony stars' sites already. There was a nice Drupal.org front page post on their solutions and contributions while we were in DC. It is well worth a read!

So the panel ended up with me talking about Drupal core and some broader issues, Jose talking about Drupal 7 plans and i18n features and Roger talking about solving some of the tasks they have faced on Drupal 6 with core, the well known contributed modules and some custom development. He also called for involvement in some of the unresolved issues in his presentation. Finally we took some great questions and wrapped up.

I believe this was our best Drupal multilanguage talk so far, but unfortunately it was not videotaped. So all we can share with you are our slides in presentation order. Enjoy!

• Gábor Hojtsy: Drupal core multilanguage (3.7MB PDF)
• Jose A. Reyero: Multilingual Drupal (1.4 MB PDF)
• Roger Lopez: Multilingual Drupal in the wild (1.5 MB PDF)

This story started almost a year ago, when I published my cheat sheet for the Drupal 6 localization API. Although Drupal 6 was not ready at that time, the localization API was as stable that the cheat sheet is useful without modification even today.

My intention with the cheat sheet was to start a localization API guide on Drupal.org and get the intricate details of this API documented for the general good. Over the past few weeks, i've managed to have time to actually sit down and document best practices and tips for these functions, and published the Localization API guide as part of the Drupal.org developer handbook (it is worth to check out the printer friendly version for a quick glance). While some parts of the guide are still under discussion and finalization (and I still plan two pages: one on emails and the other one on pointers for people looking to translate user provided data), the guide is pretty much complete as far as localizing the interface goes.

Another side of that old blog post of mine was new Translation template extractor support for the coder module. Well, that was basically tapping the existing errors into coder and make you figure out the rest. The existing error messages in extractor were however quite cryptic, like Invalid marker: t($joe). This is not really helpful in finding out what is the issue at hand, when you are not familiar with the finer details. This was unhelpful for both module authors and code reviewers, who were eager to fix these problems. I got several support requests in the extractor issue queue to clarify guidelines. So updating the error messages was clearly in order.

The result of these two efforts is that the latest development version of the extractor on the 6.x-2.x branch (update from CVS or wait for today's tarball to materialize) now supports nicely understandable error messages for coder module (and way better error messages for its standalone mode just as well) with links to the actual documentation explaining the underlying causes and details. This will hopefully end up in a new release very soon.

So do you have any excuses left to not write nicely translatable Drupal module interfaces?

It looks like the list of sessions for DrupalCon Boston is finalized, so I am happy to announce, that we are going to have a Multilanguage Drupal: a status report and a discussion session, which is going to cover the current state of Drupal 6 and a short overview of contributed modules, and should end up in a vibrant discussion on where Drupal 7 is headed as far as language support goes. There is a huge interest in multilingual support with around 20 modules hosted on drupal.org already. Come and discuss where Drupal is heading, Drupal 7 is in need of hands to advance in this area.

While most of what Drupal core lacks is user entered content translation and localization, and the above session will focus on this, I also added a BoF suggestion which deals with (built-in) interface localization exclusively. Localization tools for Drupal teams and users is expected to focus on tools like l10n_client and l10n_server and related technologies.

In my working hours, I am busy with better support for WYSIWYG editors in Drupal 7 these days, so I am co-hosting a working group BoF with Doug Green titled WYSIWYG Working Group for 7.x core which should be a discussion of proposals on fixing current WYSIWYG integration problems and weaknesses.

At last but not least, Kristof Van Tomme is proposing Szeged, Hungary for DrupalCon Europe 2008, and he intends to hold a discussion BoF on this. The Drupal Association also intends to have a discussion meeting (not open for the public) on the next DrupalCon, so whether this BoF happens is still to be seen. In any case, I am one of the firm supporters of a DrupalCon in Szeged, and I am confident Kristof would be able to lead effectively to get it done in good quality. The easily digestable version of the proposal is up at Proposing Szeged, Hungary for Drupalcon Europe 2008 (look for the attached PDF).

And, well, honestly this is all just peanuts to what all DrupalCon Boston has to offer. So if you are still wondering, whether to go or not to go, make sure you reserve your place! It's a must.

Through the development of the Localization Server project, I decided that it is important that we use icons instead of boring text links especially that we need to communicate lots of different things and provide action buttons for multiple options in a small space.

We do not (yet?) have a graphics artist to help out here, so it turned out that whatever icon set we choose, there will be some problem with the icons size, the exact set of icons available, their color, and so on. So it occured to me that we have a huge set of symbols already in the Unicode character set which Drupal is using, so why not use those as icons?

GMail's labels, Mint's Peppermill site and others already use a trick to wrap a few tags with specific margins to get a rounded cornered button feel, and putting a Unicode symbol in as text makes for a useful button. It is definitely not as perfect as specially tailored icons, but it allows for a few neat things. Let's see...

In my short free hours the last few days, I was brainstorming on new features for the translation template extractor (this little module which extracts translatable strings from Drupal modules) to make both the translators and Drupal coders life easier. Today I am proud to announce, that I released the old stable code as Potx-5.x-1.0 and Potx-6.x-1.0 (which signifies that the development code was quite stable for some time now) and wandered to implement new features for the 2.0 versions of the modules. From today, the 6.x-2.0-dev branch contains the two new features I developed the last few days:

• The module now extracts translation templates for themes too, not only modules. This was an obvious feature request, but the original implementation was quite shortsighted, so the relevant part needed a full code rethink to support themes. This is good for translators.
• The bigger news for module and theme developers is that potx now comes with (experimental) coder module integration. For those who have not heard about coder module, this little piece of software helps you to upgrade modules and ensure they conform to coding guidelines. It even helps you avoid some common security problems. But until now, it did not help you review your translatability errors. In fact, I got bug reports on the translation template extractor that if a module passed coder's review, it should not have any localization errors. Well, when used together with potx-6.x-2-dev, coder module now offers a new code review option. You can check translatability errors of your modules right there!

How can we make this even better? Well, there are still some TODO items for potx module, which will be implemented later (and I am sure people would like to see a 5.x-2.0-dev backport of the new features), but obviously people will not be better if told they make mistakes, if we don't tell them what to do instead. So I sat down and carefully crafted the Drupal 6 translation cheat sheet for your consumption. This fine piece contains the PHP and JavaScript interface translation API functions as well as the functions used in the installer (such as .install files and install profiles). I also collected the three most common errors and provided two tools to help you ensure you do as best as you can. This cheat sheet also includes explanation of the different placeholder syntaxes used in t()-ed strings, which even I have not been able to get used to still.

I hope you will find the new features and the cheat sheet useful, and take some extra time to ensure your modules are properly coded for interface translation, when you upgrade them for Drupal 6. Remember, we are going to have a "multilingual release" with all the new language features, so it becomes increasingly important that contributed modules use the interface translation API properly.

Update: Replaced the file with the 1.1 version, as I noticed that the !html placeholder needs a security warning to ensure people are aware that usage of this placeholder is not advised.

Happy hacking!

As a maintainer of Drupal's locale module I try to find creative ways to help people localize their sites. Our focus in Drupal 6 was on more features for content translation and interface translation imports, while the built-in locale interface was nearly untouched. We even complicated it a bit with the textgroups feature which might or might not get used by contributed modules at the end.

In a previous post, I announced the new localization client module which strives to solve some of the problems with the built-in locale module translation interface by bringing an AJAX powered widget close to the site translator. While this module is a very good looking way to solve the translation problem, it has two weaknesses:

• You can only translate what you see on the site pages you browse by. Some text is only shown in emergency, when form values are not filled properly, when some backend data is not accessible, etc. Some text is even restricted to different user groups. So you can only translate the most visible parts of your site.
• Closely connected, but slightly different issue is that you cannot translate strings with plural versions at once. If your page shows 3 years ago, you can translate @count years ago but not 1 year ago (the singular form) or @count[2] years ago and friends, which are used when the language in use has more then two plural forms. The Drupal database gives no clue in relating these for translation, so we cannot help users intending to translate all these at once.

Although locale module provides a more complete solution, allowing you to have a translation percentage overview as well as filter untranslated strings and work on them, you are still restricted to the same old, hard to use interface. If you'd like to improve on the interface issue, you can switch to use potx module to extract Gettext translation templates from your modules, then use some desktop Gettext editor which suits your taste and then import the translation back to your site. For most people though, the "favorite Gettext PO editor" question is like asking about the best time to go to the dentist. If we can do better, then why not?