🤗 Speed up
Having began in a time when wrappers have been much less widespread, I turned accustomed to writing my very own coaching loops, which I discover simpler to debug – an strategy that 🤗 Speed up helps successfully. It proved helpful on this mission – I wasn’t completely sure of the required information and label codecs or shapes and my information didn’t match the well-organized examples typically proven in tutorials, however having full entry to intermediate computations throughout the coaching loop allowed me to iterate rapidly.
Context Size
Most tutorials recommend utilizing every sentence as a single coaching instance. Nonetheless, on this case, I made a decision an extended context could be extra appropriate as paperwork sometimes include references to a number of entities, lots of that are irrelevant (e.g. legal professionals, different collectors, case numbers). This broader context permits the mannequin to raised establish the related shopper. I used 512 tokens from every doc as one coaching instance. This can be a widespread most restrict for fashions however comfortably accommodates all entities in most of my paperwork.
Labelling of Subtokens
Within the 🤗 token classification tutorial [1], really useful strategy is:
Solely labeling the primary token of a given phrase. Assign
-100to different subtokens from the identical phrase.
Nonetheless, I discovered that the next methodology advised within the token classification tutorial of their NLP course [2] works a lot better:
Every token will get the identical label because the token that began the phrase it’s inside, since they’re a part of the identical entity. For tokens inside a phrase however not initially, we exchange the
B-withI-
Label “-100” is particular label that’s ignored by loss operate. Therefore, I used their features with minor modifications:
def align_labels_with_tokens(labels, word_ids):
new_labels = []
current_word = None
for word_id in word_ids:
if word_id != current_word:
# Begin of a brand new phrase!
current_word = word_id
label = -100 if word_id is None else labels[word_id]
new_labels.append(label)
elif word_id is None:
# Particular token
new_labels.append(-100)
else:
# Identical phrase as earlier token
label = labels[word_id]
# If the label is B-XXX we alter it to I-XXX
if label % 2 == 1:
label += 1
new_labels.append(label)return new_labels
def tokenize_and_align_labels(examples):
tokenizer = AutoTokenizer.from_pretrained("../mannequin/xlm-roberta-large")
tokenized_inputs = tokenizer(
examples["tokens"], truncation=True, is_split_into_words=True,
padding="max_length", max_length=512)
all_labels = examples["ner_tags"]
new_labels = []
for i, labels in enumerate(all_labels):
word_ids = tokenized_inputs.word_ids(i)
new_labels.append(align_labels_with_tokens(labels, word_ids))
tokenized_inputs["labels"] = new_labels
return tokenized_inputs
I additionally used their postprocess()operate:
To simplify its analysis half, we outline this
postprocess()operate that takes predictions and labels and converts them to lists of strings.
def postprocess(predictions, labels):
predictions = predictions.detach().cpu().clone().numpy()
labels = labels.detach().cpu().clone().numpy()true_labels = [[id2label[l] for l in label if l != -100] for label in labels]
true_predictions = [
[id2label[p] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
return true_predictions, true_labels
Class Weights
Incorporating class weights into the loss operate considerably improved mannequin efficiency. Whereas this adjustment could seem simple — with out it, the mannequin overemphasized the bulk “O” class — it’s surprisingly absent from most tutorials. I carried out a customized compute_weights() operate to handle this imbalance:
def compute_weights(trainset, num_labels):
c = Counter()
for t in trainset:
c += Counter(t['labels'].tolist())
weights = [sum(c.values())/(c[i]+1) for i in vary(num_labels)]
return weights
Coaching Loop
I outlined two extra features: PyTorch DataLoader() to handle batch processing, and a most important() operate to arrange distributed coaching objects and execute the coaching loop.
from speed up import Accelerator, notebook_launcher
from collections import Counter
from datasets import Dataset
from datetime import datetime
import torch
from torch.optim.lr_scheduler import ReduceLROnPlateau
from torch.nn import CrossEntropyLoss
from torch.utils.information import DataLoader
from transformers import AutoTokenizer
from transformers import AutoModelForTokenClassification
from transformers import XLMRobertaConfig, XLMRobertaForTokenClassification
from seqeval.metrics import classification_report, f1_scoredef create_dataloaders(trainset, evalset, batch_size, num_workers):
train_dataloader = DataLoader(trainset, shuffle=True,
batch_size=batch_size, num_workers=num_workers)
eval_dataloader = DataLoader(evalset, shuffle=False,
batch_size=batch_size, num_workers=num_workers)
return train_dataloader, eval_dataloader
def most important(batch_size, num_workers, epochs, model_path, dataset_tr, dataset_ev, training_type, model_params, dt):
accelerator = Accelerator(split_batches=True)
num_labels = model_params['num_labels']
# Put together information #
train_ds = Dataset.from_dict(
{"tokens": [d[2][:512] for d in dataset_tr],
"ner_tags": [d[1][:512] for d in dataset_tr]})
eval_ds = Dataset.from_dict(
{"tokens": [d[2][:512] for d in dataset_ev],
"ner_tags": [d[1][:512] for d in dataset_ev]})
trainset = train_ds.map(tokenize_and_align_labels, batched=True,
remove_columns=["tokens", "ner_tags"])
evalset = eval_ds.map(tokenize_and_align_labels, batched=True,
remove_columns=["tokens", "ner_tags"])
trainset.set_format("torch")
evalset.set_format("torch")
train_dataloader, eval_dataloader = create_dataloaders(trainset, evalset,
batch_size, num_workers)
# Sort of coaching #
if training_type=='from_scratch':
config = XLMRobertaConfig.from_pretrained(model_path, **model_params)
mannequin = XLMRobertaForTokenClassification(config)
elif training_type=='transfer_learning':
mannequin = AutoModelForTokenClassification.from_pretrained(model_path,
ignore_mismatched_sizes=True, **model_params)
for param in mannequin.parameters():
param.requires_grad=False
for param in mannequin.classifier.parameters():
param.requires_grad=True
elif training_type=='fine_tuning':
mannequin = AutoModelForTokenClassification.from_pretrained(model_path,
**model_params)
for param in mannequin.parameters():
param.requires_grad=True
for param in mannequin.classifier.parameters():
param.requires_grad=True
# Intantiate the optimizer #
optimizer = torch.optim.AdamW(params=mannequin.parameters(), lr=2e-5)
# Instantiate the training charge scheduler #
lr_scheduler = ReduceLROnPlateau(optimizer, endurance=5)
# Outline loss operate #
weights = compute_weights(trainset, num_labels)
loss_fct = CrossEntropyLoss(weight=torch.tensor(weights))
# Put together objects for distributed coaching #
loss_fct, train_dataloader, mannequin, optimizer, eval_dataloader, lr_scheduler = accelerator.put together(
loss_fct, train_dataloader, mannequin, optimizer, eval_dataloader, lr_scheduler)
# Coaching loop #
max_f1 = 0 # for early stopping
for t in vary(epochs):
# coaching
accelerator.print(f"nnEpoch {t+1}n-------------------------------")
mannequin.practice()
tr_loss = 0
preds = record()
labs = record()
for batch in train_dataloader:
outputs = mannequin(input_ids=batch['input_ids'],
attention_mask=batch['attention_mask'])
labels = batch["labels"]
loss = loss_fct(outputs.logits.view(-1, num_labels), labels.view(-1))
accelerator.backward(loss)
optimizer.step()
optimizer.zero_grad()
tr_loss += loss
predictions = outputs.logits.argmax(dim=-1)
predictions_gathered = accelerator.collect(predictions)
labels_gathered = accelerator.collect(labels)
true_predictions, true_labels = postprocess(predictions_gathered, labels_gathered)
preds.prolong(true_predictions)
labs.prolong(true_labels)
lr_scheduler.step(tr_loss)
accelerator.print(f"Prepare loss: {tr_loss/len(train_dataloader):>8f} n")
accelerator.print(classification_report(labs, preds))
# analysis
mannequin.eval()
ev_loss = 0
preds = record()
labs = record()
for batch in eval_dataloader:
with torch.no_grad():
outputs = mannequin(input_ids=batch['input_ids'],
attention_mask=batch['attention_mask'])
labels = batch["labels"]
loss = loss_fct(outputs.logits.view(-1, num_labels), labels.view(-1))
ev_loss += loss
predictions = outputs.logits.argmax(dim=-1)
predictions_gathered = accelerator.collect(predictions)
labels_gathered = accelerator.collect(labels)
true_predictions, true_labels = postprocess(predictions_gathered, labels_gathered)
preds.prolong(true_predictions)
labs.prolong(true_labels)
accelerator.print(f"Eval loss: {ev_loss/len(eval_dataloader):>8f} n")
accelerator.print(classification_report(labs, preds))
accelerator.print(f"Present Studying Price: {optimizer.param_groups[0]['lr']}")
# checkpoint finest mannequin
if f1_score(labs, preds) > max_f1:
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(mannequin)
unwrapped_model.save_pretrained(f"../mannequin/xlml_ner/{dt}/",
is_main_process=accelerator.is_main_process,
save_function=accelerator.save)
accelerator.print(f"Mannequin saved throughout {t+1}. epoch.")
max_f1 = f1_score(labs, preds)
best_epoch = t
# early stopping
if (t - best_epoch) > 10:
accelerator.print(f"Early stopping after {t+1}. epoch.")
break
accelerator.print("Executed!")
With every thing ready, the mannequin is prepared for coaching. I simply have to provoke the method:
label_list = [
"O",
"B-evcu", "I-evcu", # variable symbol of creditor
"B-rc", "I-rc", # birth ID
"B-prijmeni", "I-prijmeni", # surname
"B-jmeno", "I-jmeno", # given name
"B-datum", "I-datum", # birth date
]
id2label = {a: b for a,b in enumerate(label_list)}
label2id = {b: a for a,b in enumerate(label_list)}num_workers = 6 # variety of GPUs
batch_size = num_workers*2
epochs = 100
model_path = "../mannequin/xlm-roberta-large"
training_type = "fine_tuning" # from_scratch / transfer_learning / fine_tuning
model_params = {"id2label": id2label, "label2id": label2id, "num_labels": 11}
dt = datetime.now().strftime("%Ypercentmpercentd_percentHpercentMpercentS")
os.mkdir(f"../mannequin/xlml_ner/{dt}")
notebook_launcher(most important, args=(batch_size, num_workers, epochs, model_path,
dataset_tr, dataset_ev, training_type, model_params, dt),
num_processes=num_workers, mixed_precision="fp16", use_port="29502")
I discover utilizing notebook_launcher() handy, because it permits me to run coaching within the console and simply work with outcomes afterward.
XLM-RoBERTa base vs giant vs Small-E-Czech
I experimented with fine-tuning three fashions. The XLM-RoBERTa base mannequin [3] delivered passable efficiency, however the server capability additionally allowed me to attempt the XLM-RoBERTa giant mannequin [3], which has twice the parameters.
XLM-RoBERTa is a multilingual model of RoBERTa. It’s pre-trained on 2.5TB of filtered CommonCrawl information containing 100 languages.
The massive mannequin confirmed a slight enchancment in outcomes, so I in the end deployed it. I additionally examined Small-E-Czech [4], an Electra-small mannequin pre-trained on Czech net information, however its efficiency was poor.
Advantageous-tuning vs Switch studying vs Coaching from scratch
Along with fine-tuning (updating all mannequin weights), I examined switch studying, as it’s typically advised that coaching solely the ultimate (classification) layer might suffice.. Nonetheless, the efficiency distinction was vital, favoring full fine-tuning. I additionally tried coaching from scratch by importing solely structure of the mannequin, initializing the weights randomly, after which coaching, however as anticipated, this strategy was ineffective.
RoBERTa vs LLM (Claude 3.5 Sonnet)
I briefly explored zero-shot LLMs, although with minimal immediate engineering (so 🥱). The mannequin struggled even with primary requests, resembling (I used Czech within the precise immediate):
Discover variable image of creditor. This quantity has precisely 9 consecutive digits 0–9 with out letters or different particular characters. It’s normally preceded by one of many following abbreviations: ‘ev.č.’, ‘zn. opr’, ‘VS. O’, ‘evid. č. opr.’. Quite the opposite, I’m not considering a transaction quantity with the abbreviation ‘č.j.’. This quantity doesn’t seem typically in paperwork, it might occur that you simply will be unable to seek out it, then write ‘can not discover’. If you happen to’re undecided, write ‘undecided’.
The mannequin typically did not output the 9-digit format precisely. Put up-processing would filter out shorter numbers, however there have been many false positives 9-digit numbers.
Sometimes the mannequin inferred incorrect delivery IDs primarily based solely on delivery dates (even with temperature set to 0). Then again, it excelled at extracting names, surnames, and delivery dates.
General, even in my earlier experiments, I discovered that LLMs (on the time of writing) carry out higher with common duties however lack accuracy and reliability for particular or unconventional duties. The efficiency in figuring out the shopper was pretty related for each approaches. For inside causes, the RoBERTa mannequin was deployed.
