T5 (Text-To-Text Transfer Transformer) is a transformer model that is trained in an end-to-end manner with text as input and modified text as output, in contrast to BERT-style models that can only output either a class label or a span of the input. This text-to-text formatting makes the T5 model fit for multiple NLP tasks like Summarization, Question-Answering, Machine Translation, and Classification problems.

How T5 is different from BERT?

Both T5 and BERT are trained with MLM (Masked Language Model) approach. 

What is MLM?

The MLM is a fill-in-the-blank task, where the model masks part of the input text and tries to predict what that masked word should be.

Example:

• “I like to eat peanut butter and <MASK> sandwiches,”

• “I like to eat peanut butter and jelly sandwiches,”

The only difference is that T5 replaces multiple consecutive tokens with the single Mask Keyword, unlike, BERT which uses Mask token for each word. This illustration is shown below.

About T5 Models

Google has released the pre-trained T5 text-to-text framework models which are trained on the unlabelled large text corpus called C4 (Colossal Clean Crawled Corpus) using deep learning. C4 is the web extract text of 800Gb cleaned data. The cleaning process involves deduplication, discarding incomplete sentences, and removing offensive or noisy content.

You can get these T5 pre-trained models from the HuggingFace website:

1.   T5-small with 60 million parameters.
2.   T5-base with 220 million parameters.
3.   T5-large with 770 million parameters.
4.   T5-3B with 3 billion parameters.
5.   T5-11B with 11 billion parameters.

T5 expects a prefix before the input text to understand the task given by the user. For example, “summarize:” for the summarization, “cola sentence:” for the classification, “translate English to Spanish:” for the machine translation, etc., You can have a look at the below image to understand the above illustration.

Every task we consider uses text as input to the model, which is trained to generate some target text. This allows us to use the same model, loss function, and hyperparameters across our diverse set of tasks including translation (green), linguistic acceptability (red), sentence similarity (yellow), and document summarization (blue).

Besides the improved transformer architecture and massive unsupervised training data, better decoding methods have also played an important role. Currently, the most prominent decoding methods are Greedy Search, Beam Search, Top-K Sampling, and Top-p Sampling.

Visit this link to know the detailed information about these methods.

Using T5 through the HuggingFace transformers:

HuggingFace, an open-source NLP library that helps load pre-trained models, which are similar to sci-kit learn for machine learning algorithms.

We define the content we are going to summarize. 

content = “China’s Huawei overtook Samsung Electronics as the world’s biggest seller of mobile phones in the second quarter of 2020, shipping 55.8 million devices compared to Samsung’s 53.7 million, according to data from research firm Canalys. While Huawei’s sales fell 5 per cent from the same quarter a year earlier, South Korea’s Samsung posted a bigger drop of 30 per cent, owing to disruption from the coronavirus in key markets such as Brazil, the United States and Europe, Canalys said. Huawei’s overseas shipments fell 27 per cent in Q2 from a year earlier, but the company increased its dominance of the China market which has been faster to recover from COVID-19 and where it now sells over 70 per cent of its phones. “Our business has demonstrated exceptional resilience in these difficult times,” a Huawei spokesman said. “Amidst a period of unprecedented global economic slowdown and challenges, we’re continued to grow and further our leadership position.” Nevertheless, Huawei’s position as number one seller may prove short-lived once other markets recover given it is mainly due to economic disruption, a senior Huawei employee with knowledge of the matter told Reuters. Apple is due to release its Q2 iPhone shipment data on Friday.”

Importing the necessary packages

from transformers import T5Tokenizer, T5ForConditionalGeneration

Loading the tokenizer and model architecture with weights

T5_PATH = ‘t5-large’ # T5 model name

# initialize the model architecture and weights

t5_model = T5ForConditionalGeneration.from_pretrained(T5_PATH)

# initialize the model tokenizer

t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)

The pre-trained model used here is t5-large. Other pre-trained models of t5 are discussed above.

Encode the text

# encode the text into tensor of integers using the tokenizer

inputs = tokenizer.encode(“summarize: ” + article, return_tensors=”pt”, max_length=512, padding=’max_length’, truncation=True)

Generate the summarized text and decode it

summary_ids = t5_model.generate(inputs,

                                    num_beams=int(2),

                                    no_repeat_ngram_size=3,

                                    length_penalty=2.0,

                                    min_length=min_length,

                                    max_length=max_length,

                                    early_stopping=True)

output = t5_tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)

The decoding method used here is Beam Search with num_beams value as 2.

With min_length 50 and max_length 50, the output is:

“Huawei overtakes Samsung as world’s biggest seller of mobile phones in second quarter of 2020. Company shipped 55.8 million devices compared to Samsung’s 53.7 million, Canalys says. Sales of Huawei’s”

and the time taken to generate the summary is 8.07 seconds with 16 cores CPU host.

With min_length 50 and max_length 100, the output is:

“Huawei overtakes Samsung as world’s biggest seller of mobile phones in second quarter of 2020. Company shipped 55.8 million devices compared to Samsung’s 53.7 million, Canalys says. Sales fell 5% from same quarter a year earlier, owing to disruption from coronavirus. But company increased its dominance of the china market which has been faster to recover from COVID-19.”

and the time taken to generate the summary is 14.32 seconds with 16 cores CPU host.

With min_length 100 and max_length 200, the output is:

“Huawei overtakes Samsung as world’s biggest seller of mobile phones in second quarter of 2020. Company shipped 55.8 million devices compared to Samsung’s 53.7 million, Canalys says. Sales fell 5% from same quarter a year earlier, owing to disruption from coronavirus. But Huawei increased its dominance of the china market which has been faster to recover from COVID-19.. Apple is due to release its Q2 iPhone shipment data on friday.”

and the time taken to generate the summary is 23.15 seconds with 16 cores CPU host.

As you increase any of these parameters num_beams, min_lenth, and max_length, the time taken to generate the summary is going to increase.

Conclusion

In this article, we have used the Beam Search decoding method. For a better summary, we can suggest increasing the beam value and trying the other decoding methods(Greedy Search, Beam Search, Top-K Sampling, and Top-p Sampling) mentioned. 

With Pegasus, we can only perform abstractive summarization but T5 can perform various NLP tasks like Classification tasks (eg: Sentiment Analysis), Question-Answering, Machine Translation, and Document Summarization. We recommend you go through the other NLP tasks of T5.

The post Abstractive Summarization Using Google’s T5 appeared first on Turbolab Technologies.

In this article, we explore different methods to implement this task and some of the learnings that we have come across on the way. We hope this will be helpful to other folks who would like to implement basic summarization in their data science pipeline for solving different business problems.

Python provides some excellent libraries and modules to perform Text Summarization. We will provide a simple example of generating Extractive Summarization using the Gensim and HuggingFace modules in this article.

Uses of Summarization?

It may be tempting to use summarization for all texts to get useful information from them and spend less time reading. However, for now, NLP summarization has been a successful use case in only a few areas.

Text summarization works great if a text has a lot of raw facts and can be used to filter important information from them. The NLP models can summarize long documents and represent them in small simpler sentences. News, factsheets, and mailers fall under these categories.

However, for texts where each sentence builds up upon the previous, text summarization does not work that well. Research journals, medical texts are good examples of texts where summarization might not be very successful.

Finally, if we take the case of summarizing fiction, summarization methods can work fine. However, it might miss the style and the tone of the text that the author tried to express.

Hence, Text summarization is helpful only in a handful of use cases.

Two Types Of Summarization

There are two main types of Text Summarization

Extractive

Extractive summarization methods work just like that. It takes the text, ranks all the sentences according to the understanding and relevance of the text, and presents you with the most important sentences.

This method does not create new words or phrases, it just takes the already existing words and phrases and presents only that. You can imagine this as taking a page of text and marking the most important sentences using a highlighter.

Abstractive

Abstractive summarization, on the other hand, tries to guess the meaning of the whole text and presents the meaning to you.

It creates words and phrases, puts them together in a meaningful way, and along with that, adds the most important facts found in the text. This way, abstractive summarization techniques are more complex than extractive summarization techniques and are also computationally more expensive.

Comparison with practical example

The best way to illustrate these types is through an example. Here we have run the Input Text below through both types of summarization and the results are shown below.

Input Text:

China’s Huawei overtook Samsung Electronics as the world’s biggest seller of mobile phones in the second quarter of 2020, shipping 55.8 million devices compared to Samsung’s 53.7 million, according to data from research firm Canalys. While Huawei’s sales fell 5 per cent from the same quarter a year earlier, South Korea’s Samsung posted a bigger drop of 30 per cent, owing to disruption from the coronavirus in key markets such as Brazil, the United States and Europe, Canalys said. Huawei’s overseas shipments fell 27 per cent in Q2 from a year earlier, but the company increased its dominance of the China market which has been faster to recover from COVID-19 and where it now sells over 70 per cent of its phones. “Our business has demonstrated exceptional resilience in these difficult times,” a Huawei spokesman said. “Amidst a period of unprecedented global economic slowdown and challenges, we’re continued to grow and further our leadership position.” Nevertheless, Huawei’s position as number one seller may prove short-lived once other markets recover given it is mainly due to economic disruption, a senior Huawei employee with knowledge of the matter told Reuters. Apple is due to release its Q2 iPhone shipment data on Friday.

Extractive Summarization Output:

While Huawei’s sales fell 5 per cent from the same quarter a year earlier, South Korea’s Samsung posted a bigger drop of 30 per cent, owing to disruption from the coronavirus in key markets such as Brazil, the United States and Europe, Canalys said. Huawei’s overseas shipments fell 27 per cent in Q2 from a year earlier, but the company increased its dominance of the China market which has been faster to recover from COVID-19 and where it now sells over 70 per cent of its phones.

Abstractive Summarization Output:

Huawei overtakes Samsung as world’s biggest seller of mobile phones in the second quarter of 2020. Sales of Huawei’s 55.8 million devices compared to 53.7 million for south Korea’s Samsung. Shipments overseas fell 27 per cent in Q2 from a year earlier, but company increased its dominance of the china market. Position as number one seller may prove short-lived once other markets recover, a senior Huawei employee says.

Extractive Text Summarization Using Gensim

Import the required libraries and functions:

from gensim.summarization.summarizer import summarize

from gensim.summarization.textcleaner import split_sentences

We store the article content in a variable called Input (mentioned above). Next, we have to pass it to the summarize function, the second parameter being the ratio we want the summarized text to be. We chose it as 0.4, or the summary will be around 40% of the original text.

summarize(Input, 0.4)

Output:

While Huawei’s sales fell 5 per cent from the same quarter a year earlier, South Korea’s Samsung posted a bigger drop of 30 per cent, owing to disruption from the coronavirus in key markets such as Brazil, the United States and Europe, Canalys said. Huawei’s overseas shipments fell 27 per cent in Q2 from a year earlier, but the company increased its dominance of the China market which has been faster to recover from COVID-19 and where it now sells over 70 per cent of its phones.

With the parameter split=True, you can see the output as a list of sentences.

Gensim summarization works with the TextRank algorithm. As the name suggests, it ranks texts and gives you the most important ones back.

Extractive Text Summarization Using Huggingface Transformers

We use the same article to summarize as before, but this time, we use a transformer model from Huggingface,

from transformers import pipeline

We have to load the pre-trained summarization model into the pipeline:

summarizer = pipeline(“summarization”)

Next, to use this model, we pass the text, the minimum length, and the maximum length parameters. We get the following output:

summarizer(Input, min_length=30, max_length=300)

Output:

China’s Huawei overtook Samsung Electronics as the world’s biggest seller of mobile phones in the second quarter of 2020, shipping 55.8 million devices compared to Samsung’s 53.7 million. Samsung posted a bigger drop of 30 per cent, owing to disruption from coronavirus in key markets such as Brazil, the United States and Europe.

 

Conclusion

We saw some quick examples of Extractive summarization, one using Gensim’s TextRank algorithm, and another using Huggingface’s pre-trained transformer model. In further posts, we will go over LSTM, BERT, and Google’s T5 transformer models in-depth and look at how they work to do tasks such as abstractive summarization.

The post Types of Text Summarization: Extractive and Abstractive Summarization Basics appeared first on Turbolab Technologies.