What is language detection?

Language identification is the first step in any text analysis or natural language processing pipeline. If the language of a document is misidentified, all subsequent language specific models will produce inaccurate results. Errors at this stage of analysis can snowball and result in invalid results, as when an English language analyzer is applied to French text. It is vital to identify the language of each document, and establish whether any sections are in another language. Depending on the country and culture, it is very common for documents to contain multiple language regions.

Whether you are building a search engine or analyzing product reviews, Rosette® Language Identifier answers not just, Is it Korean or French, but is it North Korean or South Korean?

What do I need from language detection?

Detecting the language of a monolingual document is widely solved by using statistical profiles of languages, but there are other instances:

• Multilingual documents — What about a critical two-line email written in French with a 12-line legal footer in English? That email might fool many language identifiers into tagging the email as English, but Rosette detects language regions within each document, correctly tagging the language of each section to unlock the information inside.
• Short texts — From a few words to a sentence, short texts appear everywhere, from social media, image captions, news headlines, email subject lines, and tweets to metadata, keywords, queries, files, and logs. Using statistical profiles only, short text language detection is challenging. Rosette uses a special short string language detection algorithm to overcome this issue with script and word awareness.

Language identification categorizes content and improves search results especially for multilingual documents, or anything with text: social media, image captions, news headlines, email subject lines, tweets, metadata, keywords, queries, files, logs, and more.

Basis Technology leads the pack

Our language identifier software outperforms the competition in detecting the language of:

• Short texts — Such as the language of tweets and queries (from one to three words to a full sentence)
• Multilingual documents — Rosette recognizes the dominant language in a body of text, as well as smaller sections in different languages, and tags the various language regions
• Transliterated texts — At times, non-Latin languages (such as Arabic) may appear in Arabic script or Latin characters, and Rosette recognizes both
• Dialect detection — Rosette distinguishes between the dialects of North Korea and South Korea.

Product highlights

• Detects 56 languages
• Supports 18 language scripts (e.g., Latin and Cyrillic)
• Identifies 364 language/encoding pairs
• Reports the dominant language of a document
• Detects different language regions within multilingual documents
• Delivers high accuracy based on as few as one to three words
• Cloud, on-premise, and search plugin deployments

Input

The input data may be in any of 364 language-encoding-script combinations, and involve 56 languages, 48 encodings, and 18 writing scripts. The language identifier uses an n-gram algorithm to detect language. Each of the 155 built-in profiles contains the quad-grams (i.e., four consecutive bytes) that are most frequently encountered in documents in a given language, encoding, and script. The default number of n-grams is 10,000 for double-byte encodings, and 5,000 for single-byte encodings.

Technology

When input text is submitted for detection, a similar n-gram profile is built based on that data. The input profile is then compared with all the built-in profiles (a vector distance measure between the input profile and the built-in profile is calculated). The built-in language profiles are returned in ascending order from the most likely language (i.e., the built-in profile with the shortest distance from the input text’s profile).

Confidence

Our language identifier returns a confidence score ranging from 0 to 1 with each language result. It is a measurement that you can use as a threshold for flagging results that are “too close to be sure.”

Language Boundary Locator

Digital text is often composed of multiple languages within the same document, presenting a challenge to computers and humans. RLI enriches the text with start and end markers for each language placed within multilingual documents — even if all the languages are written in the same script, such as English, French, German, or Italian. Boundaries of each writing system are also detected, such as Latin, Cyrillic, Japanese Kana, or Chinese Hanzi.

Short String Language Detection

For a number of languages, the language identifier uses additional proprietary algorithms for detecting the language of short strings (140 characters or less).

Sample output:

{
  "languageDetections": [
    {
      "language": "spa",
      "confidence": 0.38719602327387076
    },
    {
      "language": "eng",
      "confidence": 0.32699986625091865
    },
    {
      "language": "por",
      "confidence": 0.05569054210624943
    },
    {
      "language": "deu",
      "confidence": 0.030069489878380328
    },
    {
      "language": "swe",
      "confidence": 0.027734757034048835
    }
  ]
}