Calibrating a model means aligning the confidence predictions with output probabilities. Essentially, when the model’s softmax output for a certain class (or a certain token in the generative scenario) is high, we want the model to be correct most of the time. Likewise, when the confidence is low, the model should have a hard time choosing the correct class. Often, this is not the case. Simply training the model using the cross-entropy loss for the generative next token prediction does not anyhow guarantee that the model will be calibrated. Thus, calibrated models have the confidence and probability distributions aligned, i.e. high confidence examples are mostly classified correctly and vice versa.

There are numerous simple techniques for calibrating the classification problems, such as text classification or named entity recognition. Histogram binning, isotonic regression, and temperature scaling are just some of the techniques. However, such approaches cannot be used for generative models for three reasons:

1. the number of classes is the size of the vocabulary, which is usually ~50k tokens (classes);

2. ground truth text can be tokenized differently compared to the output tokens of our model, even though they, when combined, make up the same text. Example: ground truth "SCHMITZ" can be obtained via different combinations of generated tokens, for instance, S, CH, MIT, Z and S, CH, M, IT, Z.

3. output tokens need to be aligned with the tokenized ground truth. Example i): ground truth: S, CH, MIT, Z; obtained: SCH, N, IT, Z (only 1 letter differs but 3/4 tokens are incorrect). Example ii): ground truth: S, CH, MIT, Z; obtained: S, CH, N, IT, Z (lengths differ and it is unclear how to align the two).

Generative calibration

A different approach needs to be taken for generative models. A paper examining this field is Calibrating sequence likelihood improves conditional language generation. The idea is to utilize the beam search from the model’s generate function, which is normally used during inference, to get multiple candidate outputs for a single image input. The procedure is as follows:

• After fine-tuning your generative model, introduce a calibration stage, which is also done on the training data.

• Choose a similarity function that will later be used to either rank the candidates for loss calculation (rank loss) or to determine the scale of the loss (margin loss). Character Error Rate (CER) was used for this metric.

• Using the input pixel values, y ground truth and possible y target candidates outputted by the model's generate function, calculate the calibration loss. This loss aims to align models' decoded candidate's sequence likelihood according to their similarity with the ground truth target sequence, using the CER metric. Rank loss optimizes the ranking order of positive and negative candidate pairs y+, y− uniformly sampled where s(y+, y; x) > s(y−, y; x). Margin loss maximizes the sequence probability gap of positive and negative candidate pairs. List-rank and reward losses gave the worst results in the paper, so they weren't considered.

  

• (Optional) Apply the regularization loss to prevent models from deviating significantly from their fine-tuned MLE objective. Cross entropy is the standard fine-tuning MLE objective. KL divergence directly minimizes the probability distribution distance between the calibrated model and the fine-tuned model at each token on the observed target sequence. The main difference is cross entropy loss regularizes the model toward the gold reference while KL divergence regularizes the model toward the fine-tuned-only model. Aligning the probabilities of an initial (fine-tuned) and calibrated model is difficult due to the often phenomenon of different output lengths. However, to calculate the KL divergence loss, these probability tensors need to be of the same length. Thus, KL divergence loss wasn't considered.

The main manipulation was in different loss calculations. Initial experimentation showed that CE regularization loss is not beneficial for the TrOCR task. In total, 6 model variations were evaluated: initial (fine-tuned) model, rank loss, rank loss with if clause variants 1, 2, 3, and margin loss.

Rank loss and margin loss are calculated as explained in the above picture equations. Calibration loss using rank loss with if clause variants 1, 2, and 3 are calculated as follows:

• Variant 1

  

• Variant 2

  

• Variant 3

  

These loss variations are not mentioned in the paper but were examined from our side. There were several reasons for this modification of the rank loss from the paper. First, when both y candidates are correct (have CER 0), we don’t necessarily want to increase the generation probability of one of them, while decreasing the other. This is what happens with the default rank loss function. Instead, we motivate an increase in generation probabilities of either the first beam (variants 2 and 3) or both beams (variant 1). Second, when both y candidates are incorrect, we don’t necessarily want to increase the probability of any of them. Instead, we reduce both beams (variants 1 and 3) or just the first one (variant 2). Third, when the candidates are incorrect and an improvement between them, i.e. absolute difference of CERs, is minimal, we again might not want to increase one while decreasing the other (variants 1 and 2). For these reasons, different rank loss variations were added to mitigate these effects.

Results

Below is the overview of the performance of each of the models. All models, except the first (uncalibrated) one, are calibrated on the train set. All models are tested on the test set. Originally published trocr-base-handwritten model and IAM dataset were used. An example is skipped during evaluation if any of the predicted tokens' probability is less than 50%. These examples are still included in the graphs, but are below denoted as skipped. Examples that aren’t skipped and have any CER > 0 are classified as incorrect. Although TrOCR is a generative model with a vision encoder and text decoder, the behavior differs from the tasks used in the paper. Namely, they used abstractive summarization, question answering, and structured data-to-text datasets. These are more open-ended than the TrOCR task which has a strict ground truth. Thus, a different calibration loss proved to work better in our scenario.

Initial (fine-tuned) model

Average CER: 0.0385
Average sequence probability: 0.5496
Correct : 902
Incorrect: 408
Skipped : 551

Calibrated - rank loss

Average CER: 0.0334
Average sequence probability: 0.6627
Correct : 1001
Incorrect: 451
Skipped : 409

Calibrated - rank loss if clause variant 1

Average CER: 0.0313
Average sequence probability: 0.4965
Correct : 972
Incorrect: 375
Skipped : 514

Calibrated - rank loss if clause variant 2

Average CER: 0.0338
Average sequence probability: 0.6724
Correct : 1001
Incorrect: 480
Skipped : 380

Calibrated - rank loss if clause variant 3

Average CER: 0.0385
Average sequence probability: 0.6386
Correct : 972
Incorrect: 461
Skipped : 428

Calibrated - margin loss

Average CER: 0.0299
Average sequence probability: 0.6321
Correct : 1059
Incorrect: 381
Skipped : 421

Conclusion

Generative models also proved to be suitable for calibration. For the TrOCR problem specifically, margin loss yielded the best results. Compared to the initial, fine-tuned-only model, test set metrics improved by a large margin. Average CER decreased from 0.0385 to 0.0299 (22.3% decrease), average sequence probability increased from 0.5496 to 0.6321, number of correctly classified examples increased by 17.4%, number of incorrect ones decreased by 6.6%, and number of skipped examples dropped by 23.6%. Its graph also shows a more desirable 2-D CER-probability example distribution compared to the initially fine-tuned-only model. Thus, the method proved useful, not only for calibrating a generative model but also for increasing its performance in general.

The project is open-sourced and available on GitHub.

Prompt engineering

Prompt engineering has been applied to various tasks, from arithmetic reasoning to domain-specific question answering. At its most fundamental level, a prompt is the input to the language model that shapes the response of the model. For instance, the input “In the afternoon it will …” yields a response such as “In the afternoon, it will typically become warmer and brighter as the day progresses, assuming it's a typical sunny day. However, the specific weather conditions can vary depending on your location, time of year, and local climate patterns.” In the same vein, prompts can serve various purposes, including facilitating question answering (Figure 1) and fostering creativity in tasks like crafting poems (Figure 2). 

Figure 1: Question answering task using ChatGPT

Figure 2: Creative writing task using ChatGPT

As task intricacy escalates, there's a commensurate increase in prompt complexity. This frequently necessitates task decomposition, nuanced contextualization, and the meticulous selection of phrasing within the prompt.

Several techniques for prompt engineering leverage the well-known capabilities of deep neural networks, such as zero-shot and few-shot learning. Zero-shot learning is the ability of deep networks to produce correct solutions for unseen tasks. In the realm of prompt engineering, this entails providing a task description and a solitary input example to solicit the desired output. For more intricate tasks, though, significant enhancements in model performance can be achieved by furnishing a small set of input-output examples, thereby creating a few-shot learning scenario. Figures 2 and 3 illustrate concrete instances of zero-shot and few-shot learning in action.

Figure 3: Zero-shot learning task with ChatGPT

Figure 4: Few-shot learning task with ChatGPT

Overall, when developing quality prompts, there is no one-size-fits-all solution. Crafting a high-quality prompt is a dynamic and iterative process tailored to specific use cases. Nevertheless, there have been efforts to formulate and systemise effective prompt creation methods, resulting in the formulation of prompt patterns,  a prompt engineering equivalent of design patterns. 

Subscribe now

Application of prompt engineering to a personalised language learning task

This article delves into prompt engineering techniques for the development of a language learning chatbot powered by GPT models. The objective is to create a highly adaptable chatbot tailored for language learning and practice via prompt engineering. Users should have the flexibility to select their desired language, language proficiency level, and the context in which they want to engage with the chatbot. These three key components collectively shape the conversational context. The ultimate goal is for the chatbot to seamlessly generate coherent and contextually appropriate conversations that align with the chosen language proficiency level. Illustrative conversations conducted with the LinguaAI chatbot can be found in Figures 5 and 6.

Figure 5: Conversation with LinguaAI “In a restaurant”. The conversation is carried out in English on B2 level.

Figure 6: Conversation with LinguaAI “In a restaurant”. The conversation is carried out in French on A2 level.

Before going into detail about implementation methods, it is imperative to underscore the core objectives and essential attributes that we seek to achieve with our solution. The fundamental requirement is that the conversation follows the prescribed dialogue format, yielding a single, grammatically correct response per chatbot prompt. We'll refer to these as "format requirements" henceforth. Furthermore, we wish the chatbot to possess common language skills such as fluency and coherence and we wish the language proficiency level to be adhered to. Lastly, the chatbot should be able to embody its designated role effectively, employing common phrases, vocabulary, and a speech style befitting the assigned persona or context.

Taking into account the qualities of an effective language-learning chatbot as outlined above, the proposed solution comprises the following key subtasks:

• Asserting user inputs

• Inferring conversation roles from the setting description

• Main chatbot dialogue generation

• Chatbot response refinements

The LinguaAI chatbot architecture, outlined in Figure 7, consists of several components, each responsible for carrying out one of the aforementioned subtasks.

Figure 7. LinguaAI chatbot architecture

Asserting user inputs

Given the open-ended nature of the user's description of the conversational context, it is essential to verify its suitability and clarity before moving forward. To address this, the solution involves the implementation of a straightforward binary classifier integrated with a Large Language Model (LLM). In this setup, an output of "1" signifies that the input qualifies as a situation, place, or activity description, typically involving two interlocutors, while an output of "0" indicates that it does not meet these criteria.

Role inference

Once the user input has been validated, the context is then passed to the role inference model. The purpose of this step is to enhance the natural flow of the dialogue and facilitate understanding of the task for both the model and the user. This is achieved by inferring two distinct, natural roles, representing the participants in the discussion tailored to the given context. The role designated to guide the conversation is assigned to the chatbot, while the other role is allocated to the user. This allocation ensures a more coherent and intuitive interaction.

Main chatbot dialogue generation

After role inference, we have all relevant information at our disposal and we are able to commence the dialogue. To initiate the dialogue, a zero-shot task definition is used as a prompt. The information included in the prompt is the full conversation context, the Persona Pattern instructions, the CEFR language level definition and the format guidelines. Every subsequent prompt is expected to be a user’s message. In response, the chatbot generates a single conversational line as its reply.

LLM response refinement

Following the initial response generated by the primary chatbot, a series of targeted refinements is applied to the response. This iterative refinement process is motivated by the fact that it results in improved performance of Large Language Models (LLMs) compared to the baseline zero-shot approach.

The goal of individual refinements is to ensure the outlined properties of a good language-learning chatbot are met. The first refinement in the pipeline is designed to confirm that the assigned role is adhered to, while the subsequent refinement focuses on ensuring that the appropriate language proficiency level is maintained.

We have observed that common language skills such as fluency and coherence and the format are satisfied for the vast majority of text produced by the model, so no refinement was performed for this purpose.

The fundamental role fitness and language level correlation

Our refinements are primarily directed at two crucial properties: language proficiency level and role fitness. It's essential to emphasise the inverse relationship between these two properties. We prioritise language proficiency level because it forms the foundation of the language learning process, and in most cases, generic roles do not significantly suffer as a result. However, for more specific roles, such as mimicking a particular individual, there may be a trade-off where some nuances in style and tone of the text could be sacrificed in favour of maintaining the desired language proficiency level.

Refinement prompts alternative evaluation

Recognizing that the input space for a refinement pipeline is virtually limitless and that there are multiple equally valid prompt solutions, we found that simply experimenting with prompts to assess refinement quality was no longer sufficient. Our objective was to demonstrate that refining responses genuinely enhances their quality in terms of role fitness and language proficiency level matching.

To achieve this, we created a dataset that consists of 18 dialogues, with three dialogues for each CEFR language level, for each refinement prompt alternative. These datasets were automatically generated by supplying a chatbot without a refinement pipeline to act as a user. The evaluation process involved the use of two evaluation bots developed through prompt engineering. Each message generated from the assessed prompt received a score, and the scores were then averaged for each conversation. This approach allowed for a more comprehensive and objective assessment of refinement quality. The results somewhat differed depending on the GPT model used.

When dealing with GPT-4, it was observed that all refinements and initial responses exhibited a near-perfect language proficiency level. However, refinements introduced only moderate improvements in terms of role fitness.

On the other hand, with GPT-3.5-turbo, refinements led to significant enhancements in both language proficiency and role fitness criteria. Nevertheless, GPT-3.5-turbo encountered difficulties in maintaining text simplicity for the A1 and A2 language levels. To address this issue, a text simplification refinement bot was added to the end of the pipeline. This bot is designed to activate exclusively for A1 and A2 language levels, ensuring that the text remains basic and easy to understand for users at these proficiency levels.

Challenges Encountered During the Development Process

Fine-tuning the temperature setting to suit the specific task at hand is a common practice when working with Large Language Models (LLMs). However, when adjusting the temperature, we encountered a dual effect. Increasing the temperature elevates the likelihood of format requirements not being met, but it also introduces more variability in the chatbot's responses. This variability allows two dialogues with identical context to evolve in various directions, which is valuable for the language learning process.

In the end, we opted for a moderate temperature setting for the model. It's important to note that this issue was more pronounced with GPT-3.5 compared to GPT-4.

Furthermore, it was challenging to develop an evaluation process for prompt performance. To automate the score assignment process, the quality of evaluation bots had to be manually checked beforehand. This was performed using small manually crafted datasets targeting the edge cases. Given that a certain lack of trust should be present for such manual construction, the results of the evaluation process have been taken with a grain of salt. Several best-performing prompt alternatives have been further examined in a real-use setting before making small adjustments to prompts and selecting the most effective one.

Future developments

There are several ways to extend the functionality of the chatbot and improve its user utility for language learning. 

• Feedback constitutes a foundational stage in the learning process. User messages can be assessed and constructive feedback can be generated to enhance the user's experience.

• Speaking and listening are crucial aspects of learning a foreign language because they help learners develop their oral communication skills and fluency, enabling them to engage in real-life conversations and effectively convey their thoughts and ideas. These skills also enhance comprehension, pronunciation, and cultural understanding, making the language learning experience more immersive and practical. Therefore, to further enhance LinguaAI, it's essential to incorporate the ability to record and listen to messages, coupled with robust text-to-speech and speech-to-text technologies. 

• The current refinement process involves initiating an API call for each refinement, leading to slower response times for users. It is imperative to explore strategies that can enhance response speed, either by minimising the frequency of API calls or by considering alternative approaches.

• Adjusting the temperature parameter is essential to influence the diverse characteristics of the generated responses.

Key takeaways

Prompt engineering with LLMs such as GPT has allowed us to quickly develop complex AI based solutions. Building a customizable chatbot for language learning relies on several different prompt engineering approaches, from zero-shot learning to iterative refinements. The development process involves user input assessment, role inference, main chatbot dialogue generation, and response refinements. Notably, the interplay between language level and role fitness is crucial and adjustments in temperature settings influence the chatbot's behaviour. To enhance the language learning experience for users, it's beneficial to expand LinguaAI with feedback, progress tracking and audio capabilities. These features can provide users with valuable insights into their language learning journey and help them make meaningful improvements over time.

Background and motivation

Since the introduction of the pre-train and fine-tune transfer learning paradigm introduced by Devlin et al. (2019), the approach has been preserved until now. There are now different generative methods, but they scale well with an increase in model size (Lester et al., 2021), not making them accessible to everyone. The problem with the mentioned transfer learning paradigm is that it requires fine-tuning a new large language model (LLM) for each new task, which is unsustainable in terms of time, storage, and energy.

We would like to find an efficient, yet effective multi-task learning (MTL) method which would be able to handle learning multiple tasks using less time and storage space, eventually leading to less energy consumption. All this should be achieved while preserving the performance as in single-task learning (STL). The method should be easily accessible to all small to medium-sized companies, so it should be able to run on a single or few mid-range graphics processing units (GPUs). For that reason, the method should be discriminative, while generative ones require too large LLMs, e.g. 175B parameters for GPT-3 (Brown et al., 2020) and 540B for Flan-PaLM (Chung et al., 2022). Additionally, as such companies often face distribution shifts over time and continuously-incoming requirements from their clients, the method should be liable to continual learning (CL).

In this blog, we write about finding such a method, which will be able to handle training multiple tasks jointly while also being liable to training new ones continuously. This CL does not necessarily have to be as good as lifelong learning, but only needs to cover a period of 6-12 months of new requirements. After such a period, the MTL can again be re-trained using all the previous MTL data and the newly available CL data. This method should take less time, use less storage space, be able to train on a single or few mid-range GPUs, and keep the performance at least on par with STL.

Data

Two types of tasks were examined in this work — sequence classification (CLS) and token classification, i.e. NER. Three datasets were available for the CLS task, while four were available for the NER task, resulting in a total of seven datasets. These datasets were then split into MTL and CL (data-, task-, and class-incremental), resulting in the total of 10 datasets. A multilingual version of BERT was used to to the presence of multiple languages.

Methodology

Adapters

Adapters take a completely different approach to MTL. Instead of sharing as many parameters as possible, as done in the shared encoder approach, adapters share no parameters at all, besides the pre-trained model. Alternatively, adapters train as few parameters as possible and inject them cleverly into the Transformer architecture. This makes adapters modular, less resource-intensive, and faster to train. Most importantly, performance reported for the GLUE benchmark (Wang et al., 2018) does not suffer and stays on par with STL (Houlsby et al., 2019). Two of the function-composition methods will be examined in this work — bottleneck adapter (Pfeiffer et al., 2020c) and Compacter++ (Karimi Mahabadi et al., 2021).

Pfeiffer configuration

The architecture of the bottleneck adapter will be the same as the one initially introduced in Houlsby et al. (2019), with one exception. Instead of inserting the adapter after both MHA and FF sub-layers, it will only be inserted after the latter. The reason is a two times reduction in trainable parameters, while the performance on GLUE and SuperGLUE even increases slightly (Karimi Mahabadi et al., 2021). A visual representation of an adapter is shown in Figure 1, where only the adapter after the Transformer’s FF sub-layer is used.

Compacter++ configuration

Compacter++ is inserted in the same place as the aforementioned adapter, but the parameter formulation is different. It uses parameterized hyper-complex multiplication (PHM) layers. These layers have a similar form as FC layers, with the key difference being that the weights are learned as a sum of Kronecker products. They decompose adapters into A and B matrices, where A is shared across all adapter layers, while B matrices have adapter-specific parameters. Additionally, they parameterize B as a low-rank matrix. By combining the two, low-rank parameterized hypercomplex multiplication (LPHM) layer is shown on Figure 2 and formulated here:

Adapter Fusion

Once all the adapters for the MTL part of the training have been trained, one could ask themselves if these adapters can somehow be utilized for learning an incoming CL task. The AdapterFusion (Pfeiffer et al., 2020a) approach deals with training a fusion of previously trained adapters in order to solve a single target task. It is a different approach to transfer learning. Figure 3 shows how the Adapter Fusion is inserted into each Transformer layer. Newly introduced fusion parameters learn to combine previously trained adapters as a dynamic function of the target task data. Similarly to attention (Vaswani et al., 2017), a contextual activation of each adapter is learned. Given the adapters and target task data, AdapterFusion learns a parameterized mixer of the available adapters.

Hypernetworks

Hypernetwork is a bridge between the shared encoder and adapters — it shares parameters through a hypernetwork, but also keeps and generates task-specific parameters. A simple visual overview of the hypernetwork approach is shown in Figure 4. Task embeddings are kept task-specific, while the hypernetwork, depending on the given embedding and its own parameters, generates target model parameters. In our case, a target model is the bottleneck adapter.

A hypernetwork approach that we used is from Üstün et al. (2022). The architecture is shown in Figure 4. Besides the task embedding, their work also introduces language and layer embeddings. Separating the task embeddings from language embeddings enables transfer to arbitrary task-language combinations at any time. In our experiments, language embeddings were kept the same for all the tasks, but were also be trained end-to-end. Layer embeddings were introduced in order to avoid having a different hypernetwork at each Transformer layer. This way, layer embeddings significantly reduce the number of parameters and allow for information sharing across layers. During training, only the corresponding task and language embedding were updated for each batch, depending on the task and language that the batch was sampled from. Additionally, layer embedding corresponding to the current Transformer layer is also updated. Task, language, and layer embeddings are concatenated and fed into a source projector network Ps, consisting of two FF layers and a ReLU activation. The idea of the Ps component is to reduce the size of the input embedding and potentially learn the interactions between the task, language, and layer embeddings.

Besides the FF down- and up-projection matrices, a layer norm is also generated and trained via a hypernetwork. A slightly larger learning rate is used due to randomly initialized weights in a hypernetwork. A pre-trained encoder-based model is frozen. In order to achieve heterogeneous batches, gradients are accumulated over 4 steps before making an optimizer step. This way, a shared hypernetwork achieves gradient directions from multiple tasks before eventually following these directions. This should mitigate catastrophic interference, which is already largely mitigated by not sharing the parameters directly. Instead, a proxy, i.e. a hypernetwork, is shared. Instead of size-proportional sampling, temperature-based sampling with T=2 was used. Additionally, all CLS losses were divided by 5.0 and each point-loss by log(n), where n is the number of possible labels for the corresponding task. This was done to ensure a more stable training, regardless of the task type or number of possible labels.

When continuously training new tasks on top of the jointly trained hypernetwork model, CL regularization loss is added. This loss term is calculated using the adapter and layer normalization parameters generated by the hypernetwork, not the hypernetwork and embedding parameters directly. The reason is that the change in embedding and hypernetwork parameters is not important, as long as the parameters generated by these components do not change by much. The generated parameters are the ones being used in the model’s forward pass when the textual input is given.

There were two similar CL regularization methods examined, named type 1 and type 2 CL methods. Type 1 CL method penalized the sum of squared differences between the generated parameters before and after each optimizer step. This approach is expected to suffer in performance due to generated parameters changing slightly in each step during training. This leads to regularization loss being calculated using the generated parameters before the latest optimizer step, which could largely differ from the actually generated parameters before the beginning of training. Thus, the type 2 CL method compares the parameters generated before the start of training with the currently generated ones and calculates the loss between the two.

Results

Overview

The overview of the results is shown in Table 1. It can be seen that STL, adapters, Adapter Fusion, and hypernetwork all perform similarly within one macro average F1-score point. The hypernetwork CL performance downgrade is also shown in the table. The results shown are after two rounds of CL with three and two tasks respectively. The performance downgrade is reasonable for high-resource NER tasks, but becomes overly amplified for low-resource CLS datasets (130-475 training examples). A shared encoder approach has significantly lower results and suffers from an inability to efficiently continuously learn new tasks.

STL does not show any superiority over adapters and hypernetworks. Additionally, training it takes longer and requires more trainable parameters, as shown in Table 2. Although a compacter++ configuration is slightly more efficient than a Pfeiffer one, the results are notably different (see Table 3). Adapter Fusion, other than the fact that it does not reach the performance of ST-adapters, is also longer to train and much more parameter heavy due to the fusion of all the available adapters. Here, it should be noted that a fusion for CL CLS tasks consisted of two previously trained CLS adapters, while a fusion for CL NER tasks consisted of three NER adapters. These are small numbers, and a fusion can be made of even more adapters, which then accordingly requires longer training and inference times. A shared encoder approach does use more parameters than other non-STL methods, due to the whole encoder. Its training time is additionally the shortest, but due to the usage of 16-bit mixed precision. Hypernetwork ensures 91.38% number of trainable parameter reduction. Training time is only reduced by 15%, making adapters faster to train. This training time increases during CL, especially when there are many MTL tasks whose forgetting should be mitigated.

A longer hypernetwork CL training time due to CL regularization loss also needs to be taken into account as a downside. Generating all the possible previous adapters of all the previous tasks after each optimizer step is an expensive operation.

Adapters

A performance comparison of the Pfeiffer and compacter++ adapter configuration is shown in Table 3. Pfeiffer configuration, the one that does not further reduce the number of parameters by using LPHM layers, consistently outperforms the compacter++ configuration.

Adapter Fusion

Adapter Fusion test set performance comparison is shown in Table 4. As expected, the fusion of Pfeiffer configuration adapters also performs better or equal for all the datasets than the one of compacter++ adapters.

Conclusion

Adapters and hypernetworks proved as more efficient and equally effective approaches as STL. The adapter approach treats each MTL and CL task with a separate adapter, while hypernetwork separates MTL and CL phases. Adapters with Pfeiffer configuration reached a performance of STL. The hypernetwork forgetting rate for previously trained tasks was up to 2 macro average F1-score points, while low-resource CLS tasks with only 130-475 training examples faced losses of up to 35 percentage points. When continuously training a hypernetwork on new tasks, they mostly reached satisfactory performances, with extremely low-resource CLS tasks again struggling.

The proposed adapters approach is limiting in its similarity to STL, with a reduction of 28.4% in training time and 99.16% in trainable parameters. A separate adapter is deployed for each task. On the other hand, hypernetwork approach reduces the training time by 15% and trainable parameters by 91.38% and separates the joint MTL and CL phases. Additionally, a hypernetwork allows for some valuable knowledge sharing. However, during the CL phase, it struggles with low-resource CLS tasks trained during both MTL and CL phases.

These findings can be practically incorporated in companies serving many clients and their use cases and requirements. It allows them to train the models faster using only one or few mid-range GPUs, save the models using less storage, integrate them easier with ML systems, and continually train them, all whilst preserving an on-par performance with STL.

Document-level entity relation extraction presents unique challenges due to complex dependencies and context beyond individual sentences. Entities may exhibit relationships across different sections of a document, making it essential to capture meaningful connections at a broader scope.

By investigating state-of-the-art generative models, we aim to enhance our understanding of entity relations within documents and unlock the potential for more comprehensive information extraction.

The dataset consists of 4883 annotated invoice-type documents written in German, where each document has been manually annotated with entity groups. We split the dataset into train, validation, and test datasets, with 3418 documents, 722 documents, and 733 documents, respectively. The dataset consists of the following features:

1. Document: This feature is a dictionary containing the document's ID and filename, providing essential metadata for easy identification and organization.

2. Tokens: Represented as a list of strings, this feature holds the tokens present in the document, allowing for a detailed analysis of the textual content.

3. Token IDs: Each token in the document is assigned a unique ID, enabling efficient indexing and retrieval of information.

4. Page IDs and Line IDs: These features are lists of integers that represent the page numbers and line numbers where the tokens appear within the document, providing spatial context to the data.

5. Annotated Entities: This feature is a list of dictionaries that represent the named entities identified within the document. Each dictionary contains information such as the entity's name, instances of the entity (with unique IDs and corresponding text), and the token IDs associated with each instance.

6. Annotated Entity Groups: This feature is a list of dictionaries that represent the entity groups found within the document. Each dictionary includes the group's name, instances of the group (with unique IDs and associated entity IDs), and the entity IDs corresponding to each group instance.

Minimal preprocessing was performed, primarily involving structuring the annotated entity groups by converting entity IDs to their corresponding words from the text.

One of the major challenges encountered during preprocessing was the constraint of fitting the entire document within the models' input, given their maximum input length of 512 tokens. Despite attempts to chunk the documents and their associated groups, it was found that approximately 94% of the documents had groups that resided within the first 512 tokens. The inclusion of empty chunks not only prolonged the training time but also introduced considerable noise to the dataset. Consequently, the decision was made to extract only the initial 512 tokens from each document, ensuring the feasibility of the task while maintaining data integrity.

Task Modeling

This study approaches relation extraction as the extraction of entity groups from a document, where each group consists of an item or concept along with associated numerical values. Unlike tasks relying on semantic associations, here, the relationships depend primarily on the text layout. This structural aspect adds complexity, demanding an effective understanding of text hierarchy to identify and extract accurate entity groups. By framing relation extraction in this context, the study aims to explore targeted approaches to address these unique challenges and enhance the accuracy of relation extraction algorithms and models.

Experiments

In this study, we conducted three experiments to explore different approaches for relation extraction (RE) in the context of document-level entity relation extraction. Each experiment aimed to investigate the effectiveness of specific methods in identifying and grouping entities based on their relationships within the document text.

1. NER + RE Experiment: In the first experiment, we combined Named Entity Recognition (NER) with RE. This approach simulates a realistic scenario where entities are recognized and labeled within the document text. By leveraging NER, we aimed to add complexity to the task, requiring the model to recognize entities and their relationships based on the provided textual context.

2. Tags + RE Experiment: The second experiment utilized specific tags to emphasize entities within the document text. This approach eliminated the need for a separate NER task, as entities were explicitly marked with tags. By doing so, we aimed to simplify the process of entity recognition and focus solely on the RE aspect of the task.

3. Entities + RE Experiment: In the third experiment, we explored the use of annotated entities within the document while disregarding the entire document text. The focus here was on the arrangement and grouping of identified entities to uncover underlying patterns and dependencies. By isolating the entities and their relationships, we aimed to understand the influence of entity information alone on the RE task.

Prompting

In addition to the three experiments, we utilized prompting, a technique involving specific instructions or queries to guide the language model's generation process. Prompting serves as a context for coherent and relevant responses. The model can operate in both zero-shot and one-shot settings.

In zero-shot, the model generates responses without any specific examples, relying solely on the provided instructions. In one-shot, a single example is given to guide the model's comprehension and generation.

The choice of prompt significantly influences the model's output, making it a crucial factor in our study. By incorporating prompting, we gained insights into how it affects the model's performance in document-level entity relation extraction, enhancing our understanding of its adaptability and responsiveness to different prompts.

We employed three types of prompts corresponding to each experiment:

1. Experiment 1 (NER + RE) Prompt: "Find all entities from the invoice-type document written in German. After finding all the words that are entities, group all entities that are related according to the document text. Don't include dates and personal information such as email addresses, IBAN, phone numbers, and addresses. Group ONLY entities that are related, not all of them. Related entities are entities that represent a row of a table and relate to the same item. Groups are usually made of one entity that resembles the item or a concept, and the rest of the group are numbers that relate to it. The output must be a list of lists, each list within the main one is a group of entities that belong together."

2. Experiment 2 (RE) Prompt: "Extracted entities are: "{annotated_entities}" from the invoice-type document written in German. After finding all the words that are entities, group all entities that are related according to the document text. Don't include dates and personal information such as email addresses, IBAN, phone numbers, and addresses. Group ONLY entities that are related, not all of them. Related entities are entities that represent a row of a table and relate to the same item. Groups are usually made of one entity that resembles the item or a concept, and the rest of the group are numbers that relate to it. The output must be a list of lists, each list within the main one is a group of entities that belong together."

3. Experiment 3 (Entities + RE) Prompt: "Extracted entities are: "{annotated_entities}" from the invoice-type document written in German. Group all entities that are related. Don't include dates and personal information such as email addresses, IBAN, phone numbers, and addresses. Group ONLY entities that are related, not all of them. Related entities are entities that represent a row of a table and relate to the same item. Groups are usually made of one entity that resembles the item or a concept, and the rest of the group are numbers that relate to it. The output must be a list of lists, each list within the main one is a group of entities that belong together."

Models

The models selected were based on their relevance and performance in natural language processing tasks. Each model is introduced, along with its architecture, strengths, and limitations, to assess its effectiveness in capturing and extracting entity relations within documents.

1. T5 (Text-to-Text Transfer Transformers): T5 is a transformer-based neural network architecture that has gained significant attention for its impressive performance on various language tasks. It uses a sequence-to-sequence framework for text conversion and can be fine-tuned on specific tasks. Its flexibility and efficiency make it a promising candidate for document-level entity relation extraction.

2. Multilingual T5: An extension of T5, multilingual T5 models can process multiple languages and generate text in any of those languages. They offer the advantage of handling multiple languages without the need for separate models, reducing computational costs for multilingual NLP systems.

3. FLAN-T5: FLAN-T5 builds on the original T5 architecture and underwent extensive fine-tuning on over 1,800 language tasks, resulting in enhanced reasoning capabilities and promptability. Its customizability and performance across NLP tasks make it a prominent player in the field.

4. GPT (Generative Pre-trained Transformers): The GPT architecture, based on the Transformer model, revolutionized language processing with its autoregressive generation capabilities. It effectively captures long-range dependencies and contextual information, making it a powerful tool for language generation tasks.

5. GPT-3.5 Turbo: An advancement over GPT-3, GPT-3.5 Turbo demonstrates enhanced proficiency and reduced hallucinatory responses. Its versatility and integration with the user-friendly ChatGPT platform offer practical applications in various domains.

6. GPT-4: The latest iteration, GPT-4, surpasses its predecessors in accuracy and performance. It excels in handling longer contexts and has an extended token span, yielding more accurate and coherent responses. OpenAI's extensive investment in human feedback enhances GPT-4's intelligence and self-control, addressing challenges related to toxicity controls.

Results

Experimental Configuration

In model performance evaluation, mT5 surpasses both T5 and FLAN-T5 across all ROUGE metrics. FLAN-T5, despite being trained on multiple languages, exhibits similar performance to T5. Interestingly, using only annotated entities without document text consistently yields the highest performance across all models. On the other hand, the inclusion of tags does not significantly improve the model's performance.

ROUGE metrics provide valuable insights into model evaluation. Specifically, ROUGE1 scores higher than ROUGE2 scores, indicating a better match in terms of individual word or entity identification and matching. Furthermore, ROUGEL scores higher than ROUGE2, emphasizing the significance of considering the overall structure and length of relations in the evaluation process. Overall, these metrics help assess the model's ability to capture and replicate text patterns and relationships effectively.

Dataset Size

Increasing the dataset size does not lead to significant changes in performance. The highest scores are often obtained on the training dataset with only 2000 examples. This suggests that the size of the training dataset may not be the primary factor influencing performance in relation extraction tasks. Instead, other factors such as data quality, model architecture, and feature engineering may play a more significant role in determining the model's effectiveness in capturing entity relationships within documents. Therefore, focusing on optimizing these factors might yield better results in relation extraction tasks rather than solely relying on dataset size.

Prompting

Fine-tuning the mT5 model leads to consistent performance, indicating its effectiveness in capturing entity relations in document-level tasks.

However, when it comes to GPT-3.5 Turbo and GPT-4, their performance varies across different experimental setups. This highlights the importance of careful model selection for document-level entity relation extraction tasks, as different models may excel under different conditions.

In the zero-shot experiments, GPT-4 consistently outperforms GPT-3.5 Turbo across all experiment types and even surpasses the fine-tuned T5 model. This indicates that GPT-4 has a natural ability to capture entity relationships without specific fine-tuning on the task.

In the one-shot experiments, GPT-4 demonstrates superior performance compared to GPT-3.5 Turbo. However, GPT-3.5 Turbo performs better when using the one-shot method, emphasizing the significance of providing explicit examples for this model to perform well.

Overall, the results underscore the importance of carefully selecting the appropriate model for document-level entity relation extraction tasks, considering their performance across different experimental setups and the presence or absence of fine-tuning or explicit examples.

Conclusion

This study focused on document-level entity relation extraction and presented results from experiments using different models: T5, mT5, FLAN-T5, GPT-3.5 Turbo, and GPT-4. GPT-4 outperformed other models in ROUGE scores, indicating its effectiveness. However, performance varied based on experiment setups, including NER or entity tags.

The study has some limitations. ROUGE scores measure summaries, not entity relation extraction accuracy. Generalizability to other domains and datasets needs exploration. Longer documents (beyond 512 tokens) pose challenges. Future research can address these limitations and explore new strategies, enhancing model accuracy for real-world entity relation extraction tasks.

In this post, we explore how the model performs when trained using as little data as possible. The idea is that the model has already seen similar examples during the pre-training and that it needs only a little bump in the right direction.

The task that we are solving is NER using Question Answering (QA) task. NER is a task in which the model has to extract the relevant (named) entities from the text. By using the QA task, we prompt the model to extract relevant entities using a natural question to which the model has to provide an answer.

Model

The model of choice for this experiment is Fine-tuned Text-to-Text Transfer Transformer (Flan-T5). Flan-T5 is a generative transformer-based model with a full encoder-decoder architecture. This means that the encoder consumes the input text and produces a latent representation which the decoder uses to generate a new sequence. Also, the model was further fine-tuned using 1.8k various tasks. We used the base variant of the model due to its timid size of 250M parameters (for reference, GPT-4 supposedly boasts 1T parameters). However, the T5 model has its drawbacks. Most notably, the model is limited to 512 input tokens.

Data

For our experiments, we used four different proprietary datasets containing invoice-type documents in German. To follow the Q paradigm of our experiments, we formatted the dataset to contain three fields: context, question, and answer. To create the context field, we split the document into pages and collected all of the words into a single string. This helped alleviate some of the issues related to the model's input token limit. Furthermore, we formed the question by asking the model about a specific entity type present in the context. Lastly, the answer field contained all of the possible answers to a given question within the context. Here is a simple example:


'context': 'Luke Skywalker destroyed the Death Star. Young Skywalker achieved this by piloting the X-wing starfighter.'
'question': 'Who is the main character in this text?'
'answer': ['Luke Skywalker', 'Young Skywalker']

Experiments

To gauge how the reduced quantity of training data affected the performance, we performed zero- and few-shot experiments. While training the model in the zero-shot setting, we introduced examples only from the non-target dataset. For instance, if DS1 was our target dataset, we trained the model using examples from DS2, DS3, or DS4.

The zero-shot experiments that we performed can be roughly divided into two groups. In the first group are the examples in which we used all three non-target datasets for training and validation. In this group, we increased the number of training examples from each non-target dataset from 100 to 400. In the second group, we included a different combination of non-target datasets in the training set. The number of examples from each of the included non-target datasets amounted to 400.

For the few-shot setting, we introduced a limited number of training examples from the target dataset. Again, we had two families of experiments. In the first part, we included 400 examples from each of the non-target datasets and increased the number of examples from the target dataset from 100 to 400. Also, in the second group of the few-shot experiments, we kept the number of examples from the target dataset fixed at 400, while trying different combinations of the included non-target datasets. In this case, we also kept the number of examples from the non-target dataset at 400. Since subsampling can produce noise, we ran all of the aforementioned experiments 10 times which were averaged to produce the final result.

Additionally, we conducted a pure zero-shot where we did not update the model's parameters to achieve the lower-bound baseline. Also, we did the full-scale fine-tuning on the whole training set of the target dataset to obtain upper-bound baseline results.

Results

Zero-shot

Increasing the number of examples from non-target datasets group

As we can observe in Figures 2-5, the initial introduction of training examples from the non-target dataset did improve performance. However, any increase in the number of training examples resulted in only a marginal improvement. Also, for the experiments with dataset DS3 as the target dataset, we note a drop in the performance. This is probably because DS3 comprised a unique set of entity types. The model probably learned to solve the task during the early stages of training but became overfitted to the seen entity types as the number of training samples increased.

Different combinations of non-target datasets group

In experiments in which we aimed to explore how the number of included datasets impacted the performance, we did not find a link between those two. However, the results showed that the presence of certain datasets in the training set impacted the performance when tested on specific datasets. For instance, performance for target dataset DS1 proved to be higher when the model was trained using only DS4 than when DS2 or DS3 were included in the training dataset.  

Few-shot

Increasing the number of examples from the target dataset group

Figures 6-9 showcase the results of the first set of few-shot experiments. These experiments show that adding more training examples from the target dataset improves performance. Even though this is an expected result and the results are not on par with the full-scale fine-tuning, this confirms the idea that including only a limited number of examples can lead the model in the right direction.

Different combinations of non-target datasets group

In the experiments where we included different numbers of datasets into the training set, we did not observe such differences in performance as we did in the zero-shot setting. However, we noted a high standard deviation in the results among the experiment runs. We attribute this to two main causes. The first one is the random sampling. It is possible that we hit a local minimum or maximum in the performance, or that the produced sample contained some difficult examples. Secondly, it is possible that there existed some ambiguity among the documents from different datasets. When the model was trained using DS2 or DS3 and tested using DS1 it yielded a high standard deviation, which is also supported by the results from the zero-shot case.

Key takeaways

The results that we obtained in this research pose a promising prospect. Even though the performance is not at the level of fine-tuning the model using the high-resource dataset, the results showcase that the few-shot setting can be a viable option for prototyping and proof of concept solutions, or when using a low-resource training dataset. Another benefit of using a reduced dataset is a reduction in training time. For example, a single run of training in the few-shot setting took about 45 minutes, while the full-scale fine-tuning took around a day. Finally, for high-performance demands, full-scale fine-tuning remains the king, but with the arrival of bigger and more advanced models, this might change.

This blog post is focused on empowering developers to add an effective shield to their applications. It aims to provide you with the right tools and knowledge to bolster the security of your applications. Through a hands-on, step-by-step approach, we will guide you to integrate authentication into a basic web application. Our philosophy hinges on the belief that learning is best facilitated through practical examples.

1. Keycloak

Before proceeding, it's essential to be familiar with Keycloak and have an instance in use. For those who are new to Keycloak, it's recommended to spend some time understanding its purpose and capabilities. In short, Keycloak is an open-source identity and access management platform. If you're confident in setting it up, you can quickly launch Keycloak as a standalone Docker container using the following command:

docker run -p 8080:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:21.1.1 start-dev

After executing this command, you'll need to create a realm, add at least one user, and ensure Keycloak has connectivity to the Kubernetes (k8s) cluster where the application will be deployed.

In our setup, Keycloak runs in High Availability (HA) mode on a Google Kubernetes Engine (GKE) cluster, with Microsoft Azure Active Directory (AD) configured as the Identity Provider. To enable group-level mapping between Keycloak and Azure AD, we have configured it as oidc . This configuration allows users to authenticate apps/services using their Azure AD credentials via Keycloak. A detailed explanation of this setup will be left for a future blog post.

2. Gatekeeper

Gatekeeper is a straightforward authentication and authorization proxy, initially created by the Keycloak organization. As development has ceased, several projects have forked the source code and continued Gatekeeper's development. We use gogatekeeper in our implementation.

3. Implementation

Let's dive into the practical implementation - imagine you have an application that you wish to secure with authorization.

3.1. Create and Configure a Keycloak Client

First, navigate to your Keycloak instance, choose the desired realm, and then head to the Clients section. Here, create a new client (as shown in Figures 3.1 and 3.2).

Upon completing these steps, your client should now be created.

The next step involves creating Protocol Mappers for groups and audiences. Head to the Client Scopes section of your client, and click on the dedicated scope and mappers, as shown in Figure 3.3.

Next, navigate to the Mappers tab and click on Add mapper followed by By configuration. This will present you with options to select the ones you need. First, choose Group Membership as depicted in Figure 3.4.

Repeat these steps and select Audience as displayed in Figure 3.5.

Lastly, ensure that you input the Redirect URI, which informs the Keycloak client which application is authorized to utilize this client. For the time being, you can configure it with http://* and/or https://*.

3.2. Groups and Users

This section caters to individual preferences and requirements. While not prescribing a specific approach, it is worth noting that from a usability standpoint, having a dedicated group(s) linked to your Keycloak client enables more granular access levels.

For testing, it is advisable to create a group and two users: one who is part of the group and another who is not. To create a group in Keycloak, simply go to Groups and click Create group. Next, head to Users and click Create user. Once you've filled in all the necessary details, click on Join groups to add the user to the group created earlier, as illustrated in Figure 3.6.

For the second test user, do not perform this final step of adding them to the group.

After the users are created, you can set up their passwords or request them to do so during their initial login. There are numerous other configuration options available, so feel free to explore them.

3.2.1. Mapping Keycloak Group to Identity Provider Group (Optional)

This optional step is included because many individuals use Keycloak in conjunction with an Identity Provider. This means that if we want user groups configured in the Identity Provider to access our application, the appropriate mapping must be established. In our case, we use Microsoft Azure AD and map the AD group ID value to the desired group in Keycloak. To do this, go to Identity providers, select the desired provider, navigate to the Mappers tab, and create a new mapping as demonstrated in Figure 3.7.

4. Simple Web App

I have created a basic k8s application consisting of Deployment and Service resources. It's important to note that I am not using ingress, so SSL/TLS is not enabled. Since I am deploying on GKE, I am using the annotation networking.gke.io/load-balancer-type: "Internal" to obtain an internal IP. However, you may not need this for your environment.

To give you an idea of how Gatekeeper functions, we will first deploy the application without Gatekeeper and Keycloak. The figure below illustrates the app's deployment scheme without Gatekeeper in the scenario.

You can use the following YAML manifest to create Deployment and the Service:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api-test
  name: api-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-test
  template:
    metadata:
      labels:
        app: api-test
    spec:
      containers:
        - name: api-test
          image: yeasy/simple-web:latest
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              memory: "384Mi"
              cpu: "375m"
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api-test
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: api-test
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: api-test

If you prefer, you can remove the Service and instead utilize port forwarding. To apply the manifests, execute:

kubectl apply -f api-test.yaml
kubectl get pods --all-namespaces

After applying the template above, you should see the Service, Deployment,
and Pod deployed. To access the simple web app via a browser, use the Service IP.

You should see a view similar to the one displayed in Figure 4.2.

Congratulations! The simple application is now up and running.

5. Protect the Application with the gatekeeper

As we've previously learned, Gatekeeper is used as an authentication proxy. In comparison to the setup in the previous section, the new configuration should resemble the one depicted in Figure 5.1.

In the image above, you can see how the Service resource communicates with the Gatekeeper container, rather than directly with the application. The gatekeeper initiates authentication and authorization against Keycloak, either granting or denying access to the application. This means that the Gatekeeper configuration contains redirection details and authentication details.

The Gatekeeper configuration file is created within a ConfigMap, allowing it to be mounted as a volume in the pod container:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gatekeeper-config
data:
  keycloak-gatekeeper.conf: |+
    discovery-url: https://<yourkeycloakFQDN>/realms/<your-realm>
    enable-default-deny: false
    secure-cookie: false
    client-id: app-test
    client-secret: fi9cU5n5ytu2L0LF2J9u8FJNL5cNEHet
    listen: :3000
    encryption-key: AgXa7xRcoClDEU0ZDSH4X0XhL5Qy2Z2j
    redirection-url: http://10.164.0.42
    upstream-url: http://127.0.0.1:80
    resources:
    - uri: /*
      groups:
      - app-test-group
    - uri: /public/*
      white-listed: true
    - uri: /favicon
      white-listed: true
    - uri: /css/*
      white-listed: true
    - uri: /img/*
      white-listed: true

Important parts of the config are:

• discovery-url - Keycloak server realm address and path.

• client-id - The client ID we used when creating the client.

• client-secret - Obtained from the Credentials tab in the Keycloak app-test client we created. Shown in Figure 5.2.

• redirection-url - URL used by the application. In this case, the IP of the `Service`. This is an optional parameter since it defaults to the URL scheme and host, but it is important to mention when HTTPS is used, as you often want to specify where authentication should be directed.

• upstream-url - URL where Gatekeeper will forward traffic. It is `127.0.0.1` since containers inside the same pod communicate via the local network, and port 80 since our test app uses port 80.

• secure-cookie - In this case, set to false since we're using HTTP, not HTTPS.

• enable-default-deny - Indicates if we should deny all incoming requests by default and explicitly state what is allowed.

• resources - Collection of resources we want to protect/expose and the groups that can access them.

For more details about the config itself, you can check the GoGatekeeper Keycloak config.

To accommodate the changes, we need to update the deployment templates. The new YAML file should look like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api-test
  name: api-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-test
  template:
    metadata:
      labels:
        app: api-test
    spec:
      imagePullSecrets:
        - name: gitlab-registry
      containers:
        - name: api-test
          image: yeasy/simple-web:latest
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              memory: "384Mi"
              cpu: "375m"
        - name: gatekeeper
          image: quay.io/gogatekeeper/gatekeeper:2.3.1
          resources:
            requests:
              memory: "128Mi"
              cpu: "125m"
          args:
            - --config=/etc/keycloak-gatekeeper.conf
          ports:
            - containerPort: 3000
          volumeMounts:
            - name: gatekeeper-config
              mountPath: /etc/keycloak-gatekeeper.conf
              subPath: keycloak-gatekeeper.conf
      volumes:
        - name: gatekeeper-config
          configMap:
            name: gatekeeper-config
---
apiVersion: v1
kind: Service
metadata:
  name: api-test
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
  labels:
    app: api-test
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
    protocol: TCP
  selector:
    app: api-test

In this manifest, notice that the application itself has no ports exposed. The only port exposed is the Gatekeeper port. When you deploy the updated application with the Gatekeeper configuration, you should see two containers in the pod, as shown in Figure 5.3 and Figure 5.4.

If you prefer to use a stricter redirect URI, you can replace http://* with http://10.164.0.42/oauth/callback. Note that in this case, the IP used is the test application's Service IP. Typically, you should use HTTPS and a fully qualified domain name instead of the IP.

Now, when attempting to access the app, you will be redirected to the Keycloak authentication window before accessing the app, as shown in the figure below.

After logging in, you will notice that the request originated from localhost (Figure 5.6). In this case, it means that the Gatekeeper container inside the pod we deployed has contacted the application within the same pod via the local network.

Upon checking the Gatekeeper logs, you can see that it first verifies if there is an existing session token. If not, it initiates a new session if the user is authenticated properly:

2023-06-05T14:00:15.858Z    error    no session found in request, redirecting for authorization    {"error": "authentication session not found"}
2023-06-05T14:00:22.261Z    info    issuing access token for user    {"email": "test@doxray.com", "sub": "17c13f75-6c86-4aee-8d66-ce1775548967", "expires": "2023-06-05T14:15:22Z", "duration": "14m5

If we attempt to log in with another user who does not belong to the group we created and assigned in Keycloak and Gatekeeper configurations, access will be denied, as illustrated in Figure 5.7.

Upon examining the Gatekeeper logs, it becomes evident that the user is not a member of the necessary group:

2023-06-05T14:02:46.267Z    error    no session found in request, redirecting for authorization    {"error": "authentication session not found"}
2023-06-05T14:02:52.051Z    info    issuing access token for user    {"email": "test2@doxray.com", "sub": "2317d31e-adfb-4bed-8ab0-e53a2ececa0d", "expires": "2023-06-05T14:17:52Z", "duration": "14m
2023-06-05T14:02:52.093Z    warn    access denied, invalid groups    {"access": "denied", "email": "test2@doxray.com", "resource": "/*", "groups": "app-test-group"}

It is good to note that Gatekeeper allows the addition of custom forbidden pages, which may be necessary for specific use cases.

Furthermore, you can employ an Identity Provider for authentication with your application (refer to section 2.1). Observe the availability of an additional login option for Microsoft Azure AD, as seen in Figure 5.5. This feature may have already caught your attention.

6. Conclusion

Gatekeeper serves as an effective authentication proxy solution that, in appropriate use cases, can significantly reduce development time. With its customizable and configurable nature, Gatekeeper becomes easier to use as you familiarize yourself with it. As mentioned earlier, integrating Gatekeeper requires minimal changes to application authentication. You only need an additional `configMap` containing the Keycloak configuration and a `Deployment` to set up the sidecar container responsible for handling authentication.

I hope you find this information valuable and applicable to your projects.

Manually extracting this data from paper or digital documents is often time-consuming, prone to errors (typos or transposition errors), and costly as it might require a team of people to perform the task. It may take a person 5 minutes on average to read and understand the invoice, and manually type the information or copy and paste it into a destination software (e.g. ERP).

By automating the process of data extraction from invoices, businesses can reduce the time and resources required for manual data entry. Having accurate and up-to-date information about expenses and payments can help businesses make informed decisions about their finances and operations. Next to this, extracting data from invoices can help with:

• Improving cash flow and payment efficiency by reducing errors and speeding up the processing of invoices and payments

• Detecting fraud and duplicate entries by matching invoice line items to purchase orders and contracts

• Analyzing spending patterns can help businesses identify areas where they can reduce cost and optimise spending, or identify areas where they can reduce waste and improve efficiency

• Calculating CO2 emissions or carbon footprint by analyzing expenses and payments related to energy consumption and transportation.

Traditionally, automated data extraction is done using rule-based methods, such as regular expressions or template matching. These methods are inflexible and can be labor-intensive, and error-prone. Additionally, generalizability is often challenging to get right, if the rules are too specific, they may miss some cases or exceptions, or on the other hand, if the rules are too general, they may produce false positives or negatives.

Subscribe now

Template matching and rule-based approaches require a manually defined template or rule for every single invoice format and layout, and rely on data being in specific positions in a document, consistent, clear, and legible.

However, invoice documents come from many different sources, industries, or businesses, the document formats and appearance vary, and documents are often scanned (see Figure 2.). Scans can be at a low resolution and have handwritten notes, annotations, or typos and errors in the data. The ever-changing nature and variations of real-world data present further challenges for rule-based methods. For example, changes in laws and regulations might affect how and what data is displayed in a document, or a company might change the layout of the data in their invoices, add more account numbers, change visual branding, etc.

In recent years, transformer-based large language models (LLMs), have revolutionized the field of natural language processing (from entity extraction to text generation). LLMs are trained on large amounts of data from various sources and formats, enabling them to recognize patterns in the data and make predictions based on these patterns, regardless of how complex the document format might be. LLMs can leverage their general knowledge and reasoning abilities to extract data (e.g. tax rates, discounts, payment terms, etc.), handle poor-quality documents, and easily deal with documents in various languages.

Unlike rule-based methods, LLMs are not limited by strict rules, making them more flexible and less prone to errors.

However, general models like OpenAI's ChatGPT can be too broad to achieve the highest accuracy for a specific use case, like invoice data extraction, as it requires the model to understand the structure and meaning of the data in the invoices.

Therefore, for the best results, it is necessary to fine-tune these models on specific use cases such as invoice data extraction, like we do at doXray. Fine-tuning involves training the model on a smaller, specific dataset relevant to the use case. This dataset may contain examples of invoices and their corresponding data fields, or other related texts that can help the model learn about invoices. By fine-tuning the model on this dataset, we adjust the model's parameters and align it to the invoice data extraction use case. This allows the model to make more accurate predictions and extract the required data with higher precision.

At doXray, our research has shown that periodic revisions of the model and continual learning are particularly beneficial for real-world use cases (Figure 3.). This is because invoice documents can change over time and by utilizing continual learning, our model can adapt to new invoice formats and accurately extract data, even when encountering previously unseen invoice formats and data layouts. This ensures that our model remains up-to-date and effective in extracting data from invoices.

In summary, the combination of large language models, fine-tuning, and continual learning provides a powerful set of tools for accurately extracting data from invoice-type documents, ensuring the best possible value and experience for doXray’s clients.

Share

For businesses, this process is essential because it allows the AI model to learn from real-world examples and develop a better understanding of the context and nuances of human language. Annotated data helps the AI make accurate predictions when processing new, unannotated documents, ultimately improving the efficiency and effectiveness of various tasks, such as document categorization, email classification, and information extraction.

My position at doXray is that of an Annotation Team Lead which consists of carefully studying customer demands for a given use case together with the AI and business team. This position is also responsible for transferring customer demands to the rest of the annotation team and monitoring the annotation process.  

The annotation Team Lead assigns annotators to documents provided by the customer and makes a set of written rules to guide the annotation process. In case of ambiguities, a meeting is scheduled where all the issues are discussed, and annotation rules are updated accordingly. After the annotation process is completed, the annotation team lead double-checks all documents and makes sure the annotations are in accordance with customer demands. 

The overall process of managing and completing customer demands regarding annotations can be lengthy, as such having a reliable and competitive team and efficient teamwork strategy is of crucial importance.  

Our Core Values 

• Team spirit: When recruiting a new team member, we prioritize finding the best fit by outlining essential qualifications and conducting thorough interviews. Once we identify a suitable candidate, we hold a follow-up meeting to clarify their role and responsibilities. We ensure a warm welcome, introducing them to the team to foster a positive atmosphere that enhances morale and productivity for everyone involved.

• Communication: Our dedicated team prioritizes understanding customer needs and maintaining effective communication. This approach allows us to address challenges promptly and ensure all team members contribute, including newer members. We encourage open dialogue and value diverse opinions, which helps optimize deliverables for our clients. Our business prioritizes comprehending client requirements in depth when handling their documents. For complex demands, we conduct weekly meetings with our machine learning engineers to collaboratively address challenges and determine the most effective approach for various document types, ensuring high-quality results and a streamlined workflow for our team.

• Quality assurance: this is a priority for our team. We collaborate to ensure timely completion of tasks and address any delays in a dedicated meeting. After the annotation process, the team lead reviews the work for adherence to client requirements. If improvements are needed, we convene to discuss and implement solutions.

As an Annotation Team Lead managing a small or medium-sized team, striking a balance between the rewarding and challenging aspects of the role is essential. Our team remains committed to meeting customer expectations and fostering a collaborative work environment to ensure success.

void swap(int* xp, int* yp){int temp = *xp;*xp = *yp;*yp = temp;}void bubbleSort(int arr[], int n){int i, j; for (i = 0; i < n - 1; i++) for (j = 0; j < n -i -1;j++) if (arr[j] > arr[j + 1]) swap(&arr[j], &arr[j + 1]);}

A disaster! In contrast, here is the same code, properly formatted;

void swap(int* xp, int* yp){
    int temp = *xp;
    *xp = *yp;
    *yp = temp;
}

void bubbleSort(int arr[], int n){
    int i, j;
    for (i = 0; i < n - 1; i++)
        for (j = 0; j < n - i - 1; j++)
            if (arr[j] > arr[j + 1])
                swap(&arr[j], &arr[j + 1]);
}

Obviously, understanding formatted code is much easier. With a similar line of thought in mind, researchers at Microsoft developed layout language models (LayoutLMs). LayoutLMs are transformer-based models which, alongside the text, additionally have access to the layout of the document, meaning access to the document's image and the bounding boxes of every word. At the time of writing, three major LayoutLM models had been released: LayoutLM, LayoutLMv2 and LayoutLMv3, and all of them achieved state-of-the-art performance on a variety of visually-rich document understanding tasks.

The Model: LayoutLMv2/LayoutXLM

In 2020, one year after the release of the original LayoutLM paper, a paper describing LayoutLMv2 was published. The two models have similar architectures, both using three types of embeddings: text, layout and visual embeddings. Depending on the model version, the embeddings come in different flavours, but generally, these are their descriptions:

• text embedding: fixed-length vectorized representation of tokenized text,

• layout embedding: axis-aligned bounding boxes of each token and

• visual embedding: page region image encoded into a fixed-length sequence by a CNN-based visual encoder. The regions correspond to token bounding boxes from the layout embeddings.

For each token, the three embeddings are concatenated to a single vector and forwarded to the following layer.

The major difference between the two models is that LayoutLMv2 had pre-training objectives which used all embedding types, while LayoutLM didn't use visual embeddings until fine-tuning. This helped LayoutLMv2 to better learn cross-modality interactions between visual and textual information, leading to an increase in performance.

The original LayoutLMv2 was pre-trained on English documents and thus could not be used on the provided dataset of documents in German. Luckily, LayoutXLM, a multilingual version of the LayoutLMv2 exists. This model has the same architecture and pre-training objectives but is pre-trained on multilingual documents.

More detailed descriptions of LayoutLMs can be found in their respective papers, linked in the references section.

Dataset

A dataset of invoice documents is used to fine-tune the model. As previously mentioned, the documents are in German. The raw dataset consists of PDFs and their OCR scans in JSON format. The scans include bounding box information for each recognized word. Since LayoutLMv2/XLM can only take one page at a time, multi-page documents were split so that each sample consists of one page. OCR scan files contained a lot of data irrelevant to the LayoutXLM, so a script was written to extract the important data and arrange it in CSV format. This simplifies and reduces the duration of data loading. PDFs are converted to PNG files and resized to 224x224 (dimension stated in the paper). A dummy excerpt of a CSV and a document image follow:

word, x0, y0, x1, y1, label

Bei, 45, 64, 65, 76, ZERO
der, 67, 64, 101, 76, ZERO
Westschmiede, 104, 64, 121, 76, SUPPLIER
in, 123, 64, 182, 76, ZERO
Düsseldorf, 184, 64, 229, 76, CITY

During preprocessing, several issues were detected:

• LayoutXLM was pre-trained using a maximum sequence length of 512. Longer examples were truncated.

• Some bounding box coordinates were either missing or out of bounds. Since this was not a common occurrence, the corrupted coordinates were set to their corresponding page corner.

• Some OCR scans could not be matched with PDF files, probably because of typos in file names or different naming conventions. Since a large majority of samples could be matched, removing non-matches was deemed to be the best solution.

Preprocessing yielded 88.5k samples, with a predetermined 70.8k/17.7k train/evaluation split. The total number of tokens is 4.4 million. Excluding the "0" entity, there are 14 entity types, shown below:

Training and Error Evaluation

AdamW optimizer is used alongside a linear learning rate scheduler with a learning rate of 5x10^-5 and a warmup of 0.1. With 8 GB of GPU RAM, a batch size of 4 could be achieved. To compensate for the small batch size, 8 gradient accumulation steps were done between each parameter update, making the effective batch size 32.

The model is evaluated after each epoch. The best model was determined by measuring the overall macro F1-score. The model was trained for a total of 12 epochs using a GeForce GTX 1080, amounting to 53 hours of training time.

The model gave the best results on the evaluation split after 6 epochs, reaching a 0.88 f1-score.

LayoutXLM

The model achieves slightly better results compared to a BERT baseline:

BERT

Table 2: BERT results

A few things should be noted.

The similar performances of BERT and LayoutXLM may indicate that the visual information in this particular document dataset was insufficient to make a difference.

Since LayoutXLM works only with single-page samples, splitting multi-page documents might lead to a loss of valuable context. In contrast, BERT used multi-pagers as a single sample. Perhaps LayoutXLM could compare more favourably to BERT if the dataset consisted only of single-page documents.

BERTs classification scores are the result of grid search hyperparameter optimization, whereas LayoutXLM was trained with a cherry-picked hyperparameter set. A more extensive hyperparameter search could hopefully lead to even better results.

8.7k tokens were falsely classified while having a confidence score of 0.98 or higher. In almost all of these instances, the model classified an entity token as a non-entity and vice-versa, as opposed to misinterpreting one entity for some other. It was suggested that these high-confidence false classifications are the result of poor annotation. More work should be done to analyze annotation quality.

Conclusion

LayoutXLM, a multilingual version of the LayoutLMv2 model was trained on an in-house dataset of German invoices. The model outperforms a BERT baseline by a slight margin, despite BERT being optimized with a robust hyperparameter search. Several roads could be pursued in future work:

• Analysis of the invoice dataset annotation quality and its impact on model performance.

• Hyperparameter search optimization for LayoutXLM.

• Comparison with BERT on different, perhaps more visually-rich document datasets.

References

• LayoutLM: Pre-training of Text and Layout for Document Image Understanding

• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

1. Initial training, which is done on a larger amount of labelled data before the model is deployed to production

2. Continual learning, which consists of retraining the model, once it is already in production, on a monthly basis with new labelled data alongside the older data.

However, this approach poses some challenges. As new data arrives each month, the total amount of training data is constantly rising, making model training more and more expensive. Furthermore, as with any system in production, there is a possibility of data drift and over time most of the data in the dataset will be older data.

Subsampling in machine learning is a method of reducing the amount of data by selecting a subset of the original data. It is commonly done to resolve class imbalances, but in our case, subsampling was used to combat the aforementioned challenges. By subsampling older data and keeping all newer data, we want to reduce the amount of data and speed up training, while keeping the performance at the same level. Additionally, by prioritising newer, more relevant data, we want to account for data drift.

Dataset

The dataset I used during my internship contains invoice-type OCR scanned documents. There are around 42000 documents in the full dataset, with 10000 of them being recent data (less than 1 year old), and 32000 being older data (more than 1 year old). Each document has information about the supplier of the document and the time it was supplied.

The documents are already split into tokens and each token has a corresponding label. There are 15 labels in total, including the neutral ‘0’ label, which denotes that the token is outside of any named entities. The complete list of entities can be seen in Tables 2 and 3. Unlike the usual IOB-tagging scheme that is widely used for NER tasks, the tags for the beginning and inside of a named entity are not used and each label corresponds to exactly one token.

There are 32 unique document suppliers with 23 of them being present in the older data, and 29 of them in recent data. This suggests that some suppliers are new in recent data, so they will not be represented at all in the older data, but also that some suppliers have been discontinued in recent data.

Another important aspect of the dataset is the number of tokens in different documents. The shortest document length is 48 tokens, the longest is 19599 and the average is 744. As can be seen, document lengths vary greatly, but the average stays roughly the same for both older and recent data. However, this indicates that documents will have to be chunked in order to fit the BERT input, which is 512 BERT tokens.

Dataset Splits

The chart in Figure 1 represents the distribution of labelled documents per month. As can be seen, there is a larger amount of initial training data from November 2020, which is followed by 1000-2000 new documents each month, up to January 2022. Starting from then there is significantly less labelled data. If the model is trained on the full dataset, the recent data is in the minority, which poses a problem because it is likely more relevant in production.

To discover which combination of data works best, the training dataset was first split into recent and older data, with documents older than a year being in the older category, and everything newer than that in the recent category.

The dataset was then subsampled in different ways, with the model trained on the full dataset being the baseline model. Each of the subsampled datasets consists of all recent data, combined with a subsample of older data. Two different subsampling methods were used in combination with three different amounts of data, leading to 6 subsamples.

1. Data amount

   1. half as many older documents are sampled as there are recent documents

   2. the same number of older documents is sampled as there are recent documents

   3. twice as many older documents are sampled as there are recent document

2. Subsampling method

   1. documents are chosen randomly

   2. documents are chosen so that there is an approximately equal

   3. number of documents from each supplier

To ensure the validity of the results, all the datasets (including the baseline) were split into five equal parts in which 4/5 were used as a training set and the remaining 1/5 as the validation set (i.e., data was prepared for 5-fold cross-validation). This results in 6 subsampled datasets + baseline dataset = 7 datasets x 5 folds = 35 different datasets to train on.

Data Preprocessing

As mentioned in Chapter 2, BERT expects inputs to be 512 BERT tokens long. However, BERT doesn’t use word tokenization (which is the way data is currently tokenized), but wordpiece tokenization which splits words into subwords. BERT also expects the special tokens [CLS] and [SEP] at the beginning and end of each input. This means that the input can effectively be only 510 wordpieces long. To ensure this is the case, after subsampling the documents in the way described in Chapter 3, the documents were chunked.

The next step in data preprocessing is aligning the original token labels. The original labels align with the original token. However, these tokens have now been split into subwords. There are two common approaches: 

1. propagating the original label to each subword

2. aligning the label only to the first subword and giving the rest of the subwords the -100 label (which will be ignored when calculating CrossEntropyLoss). 

For this task, the second approach was used.

Training and Evaluation

The model used for the experiment was BertForTokenClassification from the HuggingFace Python library, with the ‘bert-base-multilingual-cased’ pretrained model checkpoint.

The training was done using batch sizes of 16, a learning rate of 3e-05 with an Adam optimizer, linear scheduler, and early stopping with a patience of 2. The maximum number of epochs was 15, but this limit was never reached.

To compare the results of models fine-tuned on different subsampled datasets against each other and the baseline, two evaluation datasets were prepared for each of the 5 folds. They consisted of the validation data described in Chapter 3 but filtered to only contain data from the last 6 months for the first set and the last 12 months for the second set. The metric used was the F1-score for each individual entity as well as the macro average of all entities excluding the neutral ‘0’ label. To calculate the final metrics for every dataset, the scores for each of the 5 folds were averaged.

Results

Table 1 shows the macro F1-score average on 6-month and 12-months-old evaluation data as well as the average best epoch and the approximate time needed for one epoch in minutes.

As can be seen from Table 1, all results are very similar, with the general trend being that more data leads to very slightly better F1-scores and that models achieve slightly better F1-scores on 6-month evaluation data than on 12-month evaluation data. For 6-month evaluation data, both biggest subsampled datasets outperform the baseline in terms of F1-score, but only by a very thin margin. For 12-month evaluation data, the baseline remains the model with the highest F1-score. The biggest difference in the datasets is in their training time, with the smallest of the models needing less than half the training time needed for the baseline.

Important to note here is that datasets that were subsampled according to the supplier are slightly smaller than the ones subsampled by random, due to some suppliers having slightly fewer documents than were necessary for a completely equal split between all suppliers. The shorter training times are therefore to be expected. 

Table 2 and Table 3 offer more detailed results, showing F1-scores for each individual entity. These results also show very similar results between all subsampled datasets, with none of the entities experiencing a significant performance drop or rise.

Conclusion

The main task of this internship was to find a way to reduce the amount of older training data in a way that, when used to train a new model, would maintain the F1-score of the baseline but lower the training time. This was done by creating 6 different subsampled datasets and using each one to train a BERT model, along with a baseline. These models were then compared against each other. The results showed a very small decrease in F1-scores (at most a 0.011 decrease) for datasets with a much smaller number of data (the smallest of which is around 35% of the size of the full dataset). Similarly, no notable difference was found between subsamples created at random and the subsamples created by sampling the same number of documents from each supplier.

In conclusion, subsampling older data in any of the ways tested during this internship offers a faster training time and minimal cost with respect to the F1-score.