Return all supported language pairs

[tomayac]

Developers often need to populate UI to, for example, let the user pick from any of the supported language pairs. Currently, this requires probing one by one if a desired language pair is supported. It'd be more ergonomic if there was one function to get all pairs. (Note that this function should not include data regarding each pair's availability to avoid fingerprinting, purely whether the API is capable at all of translating a given pair.)

[michaelwasserman]

What might be the most useful format here?

While verbose, A->B directional pairs or lists [{C: [D, E]},...] might most faithfully represent a static listing of ['available', 'downloading', 'downloadable'].includes(await Translator.availability({sourceLanguage: 'A', targetLanguage: 'B'}))

Other options might include bidirectional F<->G pairs, or a simple singular 'supported' list [H, I, J,...], if clients can assume that all pairings thereof would be supported. While Chrome's implementation supports all pairings amongst its supported set, that may not be strictly true of any possible impl.

[tomayac]

I'd err on the conservative side of not assuming all combinations to be supported. Instead, I'd list explicitly the combinations that work. [{C: [D, E]},...] seems to be the most representative of the common use case where you have one known source language and you want to know to what target languages you can translate it.

[michaelwasserman]

Can clients reasonably also obtain the [list of sourceLanguages]->knownTargetLanguage from that format?

[tomayac]

const translationCapabilities = {
  "en": ["fr", "es", "de"],
  "fr": ["en", "es"],
  "es": ["en", "fr"],
  "de": ["en"]
};

const getSourcesForTarget = (data, targetLang) => {
  return Object.keys(data).filter(sourceLang => 
    data[sourceLang].includes(targetLang)
  );
};

const sources = getSourcesForTarget(translationCapabilities, "en");

[michaelwasserman]

One more q: Should the list include BCP 47 subtags (e.g. "en-GB" or "ja-Kana")? I'd suggest those would only be included when the subtag mapping differs from the parent tag.

For example:

• "en": ["es-SP"] signals support for any/unspecified "en" to "es-SP", but explicitly not "es-MX", etc.
• "en": ["es"], "en-US": ["jp"] signals support for any "en" to any "es", but only "en-US" can be translated to "jp"

Or perhaps subtags are omitted unless the caller requests them, and support is presumed equivalent for all subtags?
Thoughts appreciated!

[tomayac]

I think we need to always include them, as the popular ones, like (Portuguese Brazilian vs. Portuguese) pt-BR/pt-PT or (Chinese Tradition vs. Simplified) zh-TW/zh-CN, will be important differentiators, and it's hopefully cheap to also include less popular ones (which we might not have support for today).

[aphillips]

Some language differences are importantly differentiated by script or region subtags, so you should definitely define the tags used as BCP47 language tags... or more properly they should be defined as being language ranges. A language range is a language tag used in matching (see RFC4647 or UTR35).

The reason for the distinction is that users and documents will generally want to have a fully qualified locale (en-GB, en-US, en-PH, etc.) for formatting purposes, but this API will not profit by listing every available locale and its corresponding set of target locales. So an item like "en": ["fr", "es", "de"], is perfectly sensible for selecting the language arc.

In practice, the flavor of French, Spanish, or German produced by the example will probably be a specific one. In some cases, where this is the dominant variety, additional subtags won't help. For example, fr is very likely to be fr-FR. If it produced Quebecois, the tag fr-CA might be needed. The variety of Spanish is probably interesting, as Iberian Spanish (es-ES) is different from a Latin American Spanish (es-419) is different from some specific local variation es-AR?)

Some languages are customarily written in more than one script (by which I mean, any document is in a single script, but different populations have a different script preference). zh-Hans vs zh-Hant (Simplified vs. Traditional Chinese) is one example. Another would be languages written in different scripts, such as sr-Cyrl vs. sr-Latn (Serbian in Cyrillic or Latin script), in which the difference is so pronounced as to make the difference important.

Finally, it may be important to provide tailoring of the translation capability. It's fine to select en->fr, but I may wish to tailor the output to use fr-BE formatting or query the translation engine for which variety is specifically being produced (zh-Hant-HK)

[michaelwasserman]

Supporting expressiveness sgtm, so long as it represents meaningful distinctions.

Implementers may unfortunately ignore subtags (as Chrome currently does), using equivalent algorithms/models for pt-BR and pt-PT. So, I'd expect Chrome to omit those subtags from this proposed static list, while an implementer with differentiated support could include them. Sg?

Here's a rough first draft of IDL, keeping in mind that (1) clients likely want the list without instantiating a session (2) implementers might need to throw an exception before some state is initialized like with LanguageModel.params():

interface Translator {
  // Yields a map of tags from supported source languages to respective supported target languages.
  static Promise<maplike<DOMString, sequence<DOMString>>> getSupportedLanguages();

[aphillips]

Implementers may unfortunately ignore subtags (as Chrome currently does), using equivalent algorithms/models for pt-BR and pt-PT

It's often the case that the same language input model is used for multiple varieties (such as spelling differences in US vs. UK English). The subtags aren't being ignored in such a case. This API should omit them because they do not affect performance.

As noted, the output language generally can only be "one flavo(u)r" and this might (only) be indicated by the API in cases where it makes a difference (zh-Hans for example). Casual readers of the spec should not be mislead into thinking that only the primary language subtag is permitted. Casual implementers should not trim the language tags without reason. This has been a problem in the past, hence my calling it out carefully here. In practice, the additional specificity should only appear where it is needed to distinguish the output, not because the API can produce every imaginable variety.

BAD:

"en-US": ["fr-FR", "fr-CA", "fr-CH", "fr-BE" ... ],
"en-GB": ["fr-FR", "fr-CA", "fr-CH", "fr-BE" ... ],
...
"en-XX": ["fr-FR", "fr-CA", "fr-CH", "fr-BE" ... ],

GOOD:

"en": [ "fr", "zh-Hans", "zh-Hant", "de" ... ],

The suggested IDL looks good, but I would suggest a slight change in the comment to make clear that the "source languages" is actually only a single language range. I use the word "tag" instead of the less familiar "range" here:

interface Translator {
  // Yields a map from source language tag to respective sequence of target language tags
  static Promise<maplike<DOMString, sequence<DOMString>>> getSupportedLanguages();

[byJim]

I've been testing language pair availability in Chrome's Translation API

Repository: https://github.com/byJim/chrome-translator-availability/

The out/results.json file contains verified availability results for multiple language pairs. In the format:
"en": [ "fr", "zh-Hans", "zh-Hant", "de" ... ],