The Sixth Multilingual AI Round Table at LocWorld Monterey 2024 

AI is transforming and redefining the localization industry. While the goal of identifying actionable  items has become attainable, in practice, aligning the potential of AI with actual business requirements as well as their implementation environments can be a daunting task. The focus of this roundtable is on applying both human and machine intelligence at the enterprise level, within a broader context that integrates symbolic AI into deep learning.

1. Data representations in deep learning

First things first, what is intelligence? How is deep learning related to adjacent concepts such as machine intelligence, human intelligence, and non-intelligent automation? According to Yann LeCun, Yoshua Bengio & Geoffrey Hinton (2015), the definitive feature of an ‘intelligent’ deep learning system is its generalization capabilities, to produce sensible answers in response to new inputs that it never encountered during training. This definition also applies to human intelligence.

Deep learning weaves one of the simplest, yet most efficient and intricate networks to transmit information across layers, including from human to machine, within machine, and finally from machine to human. The feed-forward movement from an encoder to a decoder is a transformation process of data representations. The input and output layers consist of natural language symbols such as words, whereas the intermediate layers in between contain representations of the training data with multiple levels of abstraction. Data in these hidden, inner layers is vectorized into mathematical vectors in a k-dimensional space. Semantic features of data fed into the input layer are ‘learned’, i.e., automatically discovered, by neural networks, rather than pre-determined by experts. Vectors or other forms of mathematical representations are subsymbolic. They are the true native languages of machines.

2. Symbolic approach to AI

 The theory of data transmission in deep learning applies to human communication. However, the vehicle humans employ is primarily symbolic. We use words to think, communicate with others, and organize knowledge bases. Knowledge has both internal and external states. On the one hand, knowledge is internalized and personalized as structured information for an individual. On the other hand, knowledge can be externalized and represented as a shareable model of data representations. For humans, knowledge models can evolve over a lifetime. For machines, when knowledge is represented in such a way that can be read and computed, they can act on their own to leverage external knowledge.

The above ability of machines is essential for symbolic artificial intelligence, which is based on high-level, human-readable symbolic representations of problems, logic and searches. Focusing on the processing and manipulation of symbols or concepts, rather than numerical data, symbolic AI is deductive, top-down, and rule-based. Symbolic AI techniques can conduct a series of logic-like reasoning steps over language-like representations. Examples of symbolic AI include expert systems based on production rules, automated reasoning and planning, semantic webs and knowledge graphs.

Deep learning and symbolic AI vary in terms of their reliance on data and computation. Deep learning requires very little engineering by hand and can easily take advantage of increases in the amount of available computation and data (LeCun et al., 2015). Conversely, symbolic AI tends to be abstract, and often labor-intensive, lending itself to re-use in multiple tasks, which promotes data efficiency.

3. Cross-lingual Large Language Models (XLMs) and machine translation

 In a deep learning approach, the scale of language models plays a significant role in boosting the model capacity and conducting multitask learning. The assumption is when a neural language model is trained on a ‘sufficiently large’ and diverse dataset, it can perform well across many domains and tasks. The release of the Transformer in 2017 is a breakthrough in scaling up language models trained on multilingual data, to a degree that these models can learn cross-lingual relations and transfer learning across languages.

A XLM model can generate language-independent vectors. The zero-shot cross-lingual transfer capabilities allow a language model to be adapted in a zero-shot manner to make predictions in a language that the model has not been trained on. These XLMs can be used as a foundation model, which, if further fine tuned, can conduct specific natural language tasks.

Machine translation is a typical task to fine tune a foundation model, which can be done by adding a thin layer to an open source large language model  (LLM) or XLM. Unlike the conventional neural MT solutions, a system driven by LLMs relies on foundation models to create pre-trained language representations. Then this system will be trained on the downstream tasks by simply fine-tuning all pretrained parameters. Fine tuning ensures the foundation model is task-aware during this process, to achieve effective transfer learning results. Furthermore, some MT companies have started to build their own proprietary LLMs to conduct machine translation.

4. Integrating symbolic AI within deep learning: Knowledge graphs and RAG

In practice, even the largest pre-trained Transformers did not achieve top performance on specific tasks without external knowledge. For example, in a cross-lingual question answering (CLQA) task, knowledge graphs applied to retrieval augmented generation (RAG), can effectively integrate data from external sources into LLMs.

A knowledge graph is simply a knowledge base organized as a graph. A graph model consists of nodes and edges, where the nodes represent concepts or classes, and the edges represent relationships between them. Knowledge graphs enable sharing and reuse of knowledge, not only among people, but also with computer systems. Compared with conventional relational databases, graphs provide more dense relationships, and capture more contextually derived meaning.

RAG with knowledge graphs helps enhance the performance of LLMs, by retrieving the most relevant information from external databases and incorporating it into the generated output. A naive RAG is based on terms, which are natural language symbols. Such an approach aims to match the terms in a query to those in document(s). This type of information retrieval can capture the relationship between query and data on the surface, symbolic level. To unleash more machine power, terms and concepts as well as associated relationships, must be converted to dense vectors. In doing so, machines can integrate refined external knowledge based on queries into their own learning mechanism. In this sense, symbolic AI is machine-augmented human intelligence.

5. Enterprise-level AI implementation – quality parameters

Here quality is a broad concept, including not only localization/linguistic quality assurance of an AI-driven or a hybrid localization process, but also special concerns about AI implementation, such as security, risks and ethical issues.

In real-life enterprise-level implementation, it is all about control. A simple function of adding a record such as an extracted term may be associated with multiple restriction rules. For example, what the term looks like, who can enter the term, as well as how to verify the term. Again, the goal is not for humans to manually work on top of each record, but enable symbolic AI or other algorithms to automate this process. Humans are monitoring and communicating with this high-speed process through data collection and analysis.

Humans are stepping into an exciting new era, with their intelligence being magnified, as well as developing a greater capacity to think about who we are and what we can do. This subsymbolic space may be an analogy of the subconscious mind. It is time for us to dive deeper into our own world, on condition that we keep the ‘above’ world, the integrated AI world, well under control.

References

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015). https://doi.org/10.1038/nature14539.

Acknowledgement

I would like to thank Megan Reid for helping me not only improve the writing of this article, but also understand how a reader without technical background feels about the content.

Join us in Monterey, California for The Localization Institute’s Global Toolbox, where I will be hosting the Multilingual AI Round Table: Integrated AI – Deep Learning and Beyond on October 28 2024. Tickets to LocWorld52 are not required to attend this Round Table.