How to fine-tune a language model and train a classifier
from fastai2.text.all import *
from nbdev.showdoc import *
# all_slow

Finetune a pretrained Language Model

First we get our data and tokenize it.

path = untar_data(URLs.IMDB_SAMPLE)
df = pd.read_csv(path/'texts.csv')

Then we put it in a Datasets. For a language model, we don't have targets, so there is only one transform to numericalize the texts.

splits = ColSplitter()(df)
tfms = [attrgetter("text"), Tokenizer.from_df("text"), Numericalize()]
dsets = Datasets(df, [tfms], splits=splits, dl_type=LMDataLoader)

Then we use that Datasets to create a DataLoaders. Here the class of TfmdDL we need to use is LMDataLoader which will concatenate all the texts in a source (with a shuffle at each epoch for the training set), split it in bs chunks then read continuously through it.

dls = dsets.dataloaders(bs=64, seq_len=72)

Or more simply with a factory method:

dls = TextDataLoaders.from_df(df, text_col='text', is_lm=True, valid_col='is_valid')
dls.show_batch(max_n=2)
text text_
0 xxbos xxmaj the views of xxmaj earth that are claimed in this film to have been xxunk by xxup nasa have recently been compared with the historical weather data for the time of xxmaj xxunk 11 , and show a good match between the xxunk patterns in the video sequence and the actual xxunk xxunk on the day . \n\n xxmaj this would seem to undermine the entire argument put forward in xxmaj the views of xxmaj earth that are claimed in this film to have been xxunk by xxup nasa have recently been compared with the historical weather data for the time of xxmaj xxunk 11 , and show a good match between the xxunk patterns in the video sequence and the actual xxunk xxunk on the day . \n\n xxmaj this would seem to undermine the entire argument put forward in the
1 end he says today xxmaj malaria is killing thousands of xxmaj africans and that is why they ca n't catch up with us . xxmaj so which is it , xxmaj jared ? xxmaj did xxmaj malaria help the xxmaj africans by xxunk xxmaj xxunk or hurt them ? xxmaj and how come xxmaj europe did okay despite massive xxunk throughout our history ? \n\n xxmaj he also seems far too eager he says today xxmaj malaria is killing thousands of xxmaj africans and that is why they ca n't catch up with us . xxmaj so which is it , xxmaj jared ? xxmaj did xxmaj malaria help the xxmaj africans by xxunk xxmaj xxunk or hurt them ? xxmaj and how come xxmaj europe did okay despite massive xxunk throughout our history ? \n\n xxmaj he also seems far too eager to

Then we have a convenience method to directly grab a Learner from it, using the AWD_LSTM architecture.

learn = language_model_learner(dls, AWD_LSTM, metrics=[accuracy, Perplexity()], path=path, wd=0.1).to_fp16()
learn.freeze()
learn.fit_one_cycle(1, 1e-2)
epoch train_loss valid_loss accuracy perplexity time
0 4.508908 4.040500 0.274641 56.854774 00:07
learn.unfreeze()
learn.fit_one_cycle(4, 1e-2)
epoch train_loss valid_loss accuracy perplexity time
0 4.266948 4.144216 0.260595 63.068161 00:07
1 4.105926 4.083119 0.267325 59.330254 00:07
2 3.757105 4.070948 0.272230 58.612507 00:07
3 3.318740 4.130954 0.268484 62.237255 00:07

Once we have fine-tuned the pretrained language model to this corpus, we save the encoder since we will use it for the classifier.

learn.show_results()
input target pred
0 xxbos a beautiful xxunk in xxmaj london is swept off her feet by a millionaire tea plantation owner and soon finds herself married and living with him at his xxunk in xxmaj british xxmaj xxunk . xxmaj although based upon the book by xxmaj robert xxmaj xxunk , this initial set - up is highly reminiscent of xxmaj xxunk 's " xxunk " , with leading lady xxmaj xxunk xxmaj taylor xxunk a beautiful xxunk in xxmaj london is swept off her feet by a millionaire tea plantation owner and soon finds herself married and living with him at his xxunk in xxmaj british xxmaj xxunk . xxmaj although based upon the book by xxmaj robert xxmaj xxunk , this initial set - up is highly reminiscent of xxmaj xxunk 's " xxunk " , with leading lady xxmaj xxunk xxmaj taylor xxunk with xxmaj very , xxunk the xxunk . a off the xxunk and a xxunk who xxmaj . who xxunk finds a in to is in her . a local . xxmaj xxunk xxmaj xxunk . xxmaj the the on a story , xxmaj xxunk xxmaj xxunk , it is adaptation of up of a successful of xxmaj xxunk xxmaj xxmaj xxunk xxmaj . which a man xxmaj xxunk xxmaj xxunk as in
1 you love to hate " , the spoiled , xxunk son of xxmaj newman 's gangster father ; and an almost unrecognizable xxmaj xxunk xxmaj law as an especially slimy xxunk who goes on pursuit of xxmaj hanks and his son and figures very importantly in the film 's riveting second half . xxmaj but acting in a movie this xxunk is bound to take a back seat to the xxunk fireworks love to hate " , the spoiled , xxunk son of xxmaj newman 's gangster father ; and an almost unrecognizable xxmaj xxunk xxmaj law as an especially slimy xxunk who goes on pursuit of xxmaj hanks and his son and figures very importantly in the film 's riveting second half . xxmaj but acting in a movie this xxunk is bound to take a back seat to the xxunk fireworks on can the see ! xxunk xxmaj only girl xxunk , of the xxunk , xxunk , . xxmaj xxmaj xxunk - xxmaj xxunk xxmaj xxunk as xxmaj xxunk xxunk xxunk . xxunk on to and a martin and the xxunk . his a hard the the film . xxunk tale act . xxmaj the the is the xxunk that film is xxunk to be place xxunk - and the xxunk of that
2 watching the 1 1 / 2 i was like wow . xxmaj all my expectations ( for xxunk ) were broken . a truly lovely and original plot keeps you glued to your seat for the entire time . i have noticed that the cartoon was filled with so many comical moments that xxunk will apply here 100 % . \n\n i xxunk recommend seeing the cartoon . xxbos xxmaj horrible acting the 1 1 / 2 i was like wow . xxmaj all my expectations ( for xxunk ) were broken . a truly lovely and original plot keeps you glued to your seat for the entire time . i have noticed that the cartoon was filled with so many comical moments that xxunk will apply here 100 % . \n\n i xxunk recommend seeing the cartoon . xxbos xxmaj horrible acting , this movie / / 2 / think xxunk , . xxmaj it in friends were and xxmaj ) were xxunk , xxmaj lot sad movie touching movie about you interested to the seat . weeks rest movie . xxmaj think to that the ending has n't with some many characters themes that i the be to . % . xxmaj xxmaj would this this this movie . xxbos xxmaj this acting ,
3 dream . \n\n xxmaj the xxunk of classic film noir are all hit upon without any xxunk changes , for all xxunk and xxunk . xxmaj it is in the xxunk of its xxunk that xxmaj xxunk 's xxmaj french thriller has followed no example . xxmaj for the film 's xxunk , xxunk by the challenges of physical lighting that would normally be faced , have been able to begin with . \n\n xxmaj the xxunk of classic film noir are all hit upon without any xxunk changes , for all xxunk and xxunk . xxmaj it is in the xxunk of its xxunk that xxmaj xxunk 's xxmaj french thriller has followed no example . xxmaj for the film 's xxunk , xxunk by the challenges of physical lighting that would normally be faced , have been able to begin with a of xxmaj xxmaj the story of the xxmaj - xxunk xxunk over by me the doubt . . and example the and xxunk . xxmaj the 's a fact xxunk of the xxunk , the xxunk xxmaj xxunk xxunk xxmaj is been its xxunk of xxmaj the me most 's xxunk , it are the xxunk of xxunk physics , xxunk be be xxunk with xxunk a xxunk to xxunk with the
4 schlock , the way it was meant to be , unaware , clueless , and pointless . xxmaj god bless xxmaj todd xxmaj sheets . xxmaj for anyone seeking surprisingly worthwhile 90 's b - horror , xxmaj xxunk xxmaj xxunk 's xxmaj darkness should be at the top of your list . xxmaj as for xxmaj zombie xxmaj bloodbath , if you 're a xxunk who got bored xxunk around 1990 , the way it was meant to be , unaware , clueless , and pointless . xxmaj god bless xxmaj todd xxmaj sheets . xxmaj for anyone seeking surprisingly worthwhile 90 's b - horror , xxmaj xxunk xxmaj xxunk 's xxmaj darkness should be at the top of your list . xxmaj as for xxmaj zombie xxmaj bloodbath , if you 're a xxunk who got bored xxunk around 1990 , . but xxunk it comes made to be . is of that , and not . xxmaj the , this god xxmaj sheets , xxmaj he the who to entertaining movies 's , - movie , this i xxmaj xxunk is xxmaj xxunk is be xxunk least expense of the list . xxmaj it a the xxunk xxmaj bloodbath , it you want a fan , 's a of , the 's
5 the xxmaj nutcracker , rather than wearing a huge mask ( as is always done when the xxmaj balanchine production is performed onstage ) , xxmaj culkin is actually made up as the toy - he wears what looks like a bald cap , as well as a white xxunk , xxunk , and a xxunk . xxmaj he also has his face xxunk up somewhat , and the worst aspect of xxmaj nutcracker , rather than wearing a huge mask ( as is always done when the xxmaj balanchine production is performed onstage ) , xxmaj culkin is actually made up as the toy - he wears what looks like a bald cap , as well as a white xxunk , xxunk , and a xxunk . xxmaj he also has his face xxunk up somewhat , and the worst aspect of his xxmaj xxunk is the than the a xxunk xxunk . xxunk well usual said in a xxmaj xxunk 's is xxunk ) ) . the the 's xxunk xxunk to of a xxunk xxunk maker - the is like a xxunk xxunk . and well as a xxunk xxunk . a , xxunk a xxunk . xxmaj the also looks a hair xxunk , and xxunk but the xxunk is of the
6 xxunk their entertaining xxunk xxunk in a xxunk of xxunk . a xxunk oil xxunk convinces two xxmaj catholic nuns back in the 1920s to xxunk a xxmaj west xxmaj texas well . xxmaj xxunk they have blown their bucks on an ill - xxunk fantasy , the sisters xxunk the xxunk terrain with rose xxunk and xxunk xxmaj st . xxmaj xxunk 's patron saint of hopeless causes ' to xxunk their entertaining xxunk xxunk in a xxunk of xxunk . a xxunk oil xxunk convinces two xxmaj catholic nuns back in the 1920s to xxunk a xxmaj west xxmaj texas well . xxmaj xxunk they have blown their bucks on an ill - xxunk fantasy , the sisters xxunk the xxunk terrain with rose xxunk and xxunk xxmaj st . xxmaj xxunk 's patron saint of hopeless causes ' to xxunk . , xxunk xxunk . . a xxunk xxunk xxunk . xxmaj xxunk xxunk xxunk and him people xxunk xxunk that in xxmaj xxmaj and become their xxunk xxunk xxmaj xxunk xxunk - xxmaj the xxmaj have to up xxunk up the old - fitting and , xxmaj xxunk are the xxunk of and their xxunk and xxunk xxunk xxunk . xxmaj mary xxunk xxunk , , the love . trouble xxunk the
7 in the xxmaj united xxmaj states or xxmaj england . xxmaj for me the movie was simply fun and entertaining and the xxunk just left me quite cold to be honest . \n\n xxmaj the musical score by xxmaj james xxmaj xxunk is good and fun , even though it 's your average every day xxmaj xxunk stuff , it still all works perfectly for the movie and helps to make some the xxmaj united xxmaj states or xxmaj england . xxmaj for me the movie was simply fun and entertaining and the xxunk just left me quite cold to be honest . \n\n xxmaj the musical score by xxmaj james xxmaj xxunk is good and fun , even though it 's your average every day xxmaj xxunk stuff , it still all works perfectly for the movie and helps to make some of the first united xxmaj states , xxmaj xxunk . xxmaj the the , xxunk is a a to entertaining . i movie of did me xxunk disappointed . the able . xxmaj xxmaj the story score was xxmaj xxunk xxmaj xxunk is excellent but the . but if the is not own xxunk time . i xxmaj . but 's is sounds . . the rest . it it keep the good
8 is shot with pure silent technique , long xxunk xxunk of narrative without a single xxunk -- save for a few disconcerting sequences where xxmaj louise xxmaj brooks , playing a xxmaj french xxunk , is quite xxunk speaking in xxmaj english … xxmaj the only section that obviously cries out for sound is the final scene , where xxmaj brooks is watching the rushes for her test ' for a sound shot with pure silent technique , long xxunk xxunk of narrative without a single xxunk -- save for a few disconcerting sequences where xxmaj louise xxmaj brooks , playing a xxmaj french xxunk , is quite xxunk speaking in xxmaj english … xxmaj the only section that obviously cries out for sound is the final scene , where xxmaj brooks is watching the rushes for her test ' for a sound film a in a xxunk intensity . which enough , , xxunk , resorting camera xxunk . and the the xxunk minutes scenes . you xxunk xxunk brooks xxunk xxmaj a xxunk xxunk woman , is xxunk xxunk . . a english . \n\n the xxunk thing of i xxunk at in her xxunk xxmaj xxunk scene in which she xxunk xxunk xxunk a xxmaj of the xxunk - xxunk the minute -
learn.save_encoder('enc1')

Use it to train a classifier

For classification, we need to use two set of transforms: one to numericalize the texts and the other to encode the labels as categories. Note that we have to use the same vocabulary as the one used in fine-tuning the language model.

lm_vocab = dls.vocab
splits = ColSplitter()(df)
x_tfms = [attrgetter("text"), Tokenizer.from_df("text"), Numericalize(vocab=lm_vocab)]
dsets = Datasets(df, splits=splits, tfms=[x_tfms, [attrgetter("label"), Categorize()]], dl_type=SortedDL)

We once again use a subclass of TfmdDL for the dataloaders, since we want to sort the texts (sortish for the training set) by order of lengths. We also use pad_collate to create batches form texts of different lengths.

dls = dsets.dataloaders(before_batch=pad_input_chunk)

And there is a factory method, once again:

dls = TextDataLoaders.from_df(df, text_col="text", text_vocab=lm_vocab, label_col='label', valid_col='is_valid', bs=32)
dls.show_batch(max_n=2, trunc_at=60)
text category
0 xxbos xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \n\n xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , xxunk bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no negative
1 xxbos xxup the xxup shop xxup around xxup the xxup corner is one of the xxunk and most feel - good romantic comedies ever made . xxmaj there 's just no getting around that , and it 's hard to actually put one 's feeling for this film into words . xxmaj it 's not one of those films that positive

Then we once again have a convenience function to create a classifier from this DataLoaders with the AWD_LSTM architecture.

learn = text_classifier_learner(dls, AWD_LSTM, metrics=[accuracy], path=path,drop_mult=0.5)
learn = learn.load_encoder('enc1')

Then we can train with gradual unfreezing and differential learning rates.

learn.fit_one_cycle(4)
epoch train_loss valid_loss accuracy time
0 0.719304 0.604357 0.705000 00:04
1 0.612775 0.518476 0.790000 00:04
2 0.524039 0.440905 0.820000 00:04
3 0.464367 0.448220 0.800000 00:04
learn.unfreeze()
learn.opt = learn.create_opt()
learn.fit_one_cycle(8, slice(1e-5,1e-3))
epoch train_loss valid_loss accuracy time
0 0.412078 0.443219 0.810000 00:06
1 0.380434 0.528751 0.775000 00:06
2 0.369134 0.454926 0.790000 00:07
3 0.319608 0.472347 0.820000 00:07
4 0.283797 0.482769 0.785000 00:06
5 0.251635 0.496452 0.800000 00:07
6 0.224202 0.518999 0.805000 00:06
7 0.207961 0.518238 0.795000 00:07
learn.show_results(max_n=4, trunc_at=60)
text category category_
0 xxbos xxmaj i 'm sure things did n't exactly go the same way in the real life of xxmaj homer xxmaj hickam as they did in the film adaptation of his book , xxmaj rocket xxmaj boys , but the movie " october xxmaj sky " ( an xxunk of the book 's title ) is good enough to stand positive positive
1 xxbos xxmaj to review this movie , i without any doubt would have to quote that memorable scene in xxmaj tarantino 's " pulp xxmaj fiction " ( xxunk ) when xxmaj jules and xxmaj vincent are talking about xxmaj mia xxmaj wallace and what she does for a living . xxmaj jules tells xxmaj vincent that the " only negative negative
2 xxbos xxmaj how viewers react to this new " adaption " of xxmaj shirley xxmaj jackson 's book , which was promoted as xxup not being a remake of the original 1963 movie ( true enough ) , will be based , i suspect , on the following : those who were big fans of either the book or original negative negative
3 xxbos xxmaj the trouble with the book , " memoirs of a xxmaj geisha " is that it had xxmaj japanese xxunk but underneath the xxunk it was all an xxmaj american man 's way of thinking . xxmaj reading the book is like watching a magnificent ballet with great music , sets , and costumes yet performed by xxunk negative negative
learn.predict("This was a good movie")
('positive', tensor(1), tensor([0.0010, 0.9990]))
from fastai2.interpret import *
interp = Interpretation.from_learner(learn)
interp.plot_top_losses(6)
input target predicted probability loss
0 xxbos xxmaj i 'm gon na xxunk the xxunk here a bit and say i enjoyed this . xxmaj however , the cartoon is really only going to appeal to those who have very xxunk xxunk . xxmaj it 's definitely something that most people will not get , as is the nature of xxunk . \n\n the animation is horrible , but yes , that 's the point . xxmaj the main character is foul mouthed , violent , and stupid . no redeeming qualities whatsoever . his wife xxunk and xxunk , apparently just barely capable of the most basic xxunk skills . most of these stories completely lack any kind of point . \n\n but again , that 's the point xxunk \n\n xxmaj if non xxunk , foul language , and complete and utter xxunk are your thing , you 're going to love this . positive negative 0.9995180368423462 7.637721061706543
1 xxbos xxmaj this one is a little better than the first one . xxmaj it still relies on a lot of its humor which basically keeps saying that the old xxmaj bond movies were not realistic . xxmaj that wears thin after so many xxunk . xxmaj the girls were more interesting in this one . \n\n xxmaj there is a tremendous amount of total gross out humor . xxmaj hopefully one day real comedy will come back . negative positive 0.9977344274520874 6.089940071105957
2 xxbos xxmaj for anyone who may not know what a one - actor movie was like , this is the best example . xxmaj this plot is ridiculous , and really makes no sense . xxmaj it 's full of xxunk situations , hackneyed lines , melodrama , comedy … you name it ! \n\n xxmaj but xxmaj xxunk xxmaj xxunk can make anything convincing , and this movie is by no means an exception . xxmaj everyone turns in a decent performance - xxmaj xxunk xxmaj xxunk , xxmaj xxunk xxmaj xxunk , xxmaj xxunk , xxmaj om xxmaj xxunk , xxmaj xxunk xxmaj xxunk … xxmaj but it is the xxmaj xxunk who xxunk everyone with his xxunk presence . xxmaj without him , this movie would have been a non - xxunk … xxmaj the story is about xxunk / mistaken identities / misunderstandings / love / positive negative 0.9917295575141907 4.79507303237915
3 xxbos xxmaj while i count myself as a fan of the xxmaj xxunk 5 television series , the original movie that introduced the series was a weak start . xxmaj although many of the elements that would later mature and become much more compelling in the series are there , the pace of xxmaj the xxmaj gathering is slow , the makeup somewhat inadequate , and the plot confusing . xxmaj worse , the characterization in the premiere episode is poor . xxmaj although the ratings xxunk shows that many fans are willing to overlook these problems , i remember xxmaj the xxmaj gathering almost turned me off off what soon grew into a spectacular series . negative positive 0.9898333549499512 4.5886430740356445
4 xxbos xxmaj weaker entry in the xxmaj xxunk xxmaj drummond series , with xxmaj john xxmaj howard in the role . xxmaj usual funny xxunk and antics , but not much plot . xxmaj barrymore gets something to do as the inspector , xxunk xxunk to follow xxmaj drummond , xxmaj algy , and xxmaj xxunk on a wild xxunk chase ( mostly in circles ; perhaps the budget was tighter than usual ) to rescue poor xxmaj xxunk , who is being held captive by people who want to lure xxmaj drummond to his doom . xxmaj for those keeping score , in this one , xxmaj drummond is planning to ask xxmaj xxunk to marry him and xxmaj algy is worried about missing the baby 's xxunk . xxmaj it 's fun to see xxmaj algy and xxmaj xxunk dressed up as xxunk to blend in at xxmaj negative positive 0.9651989936828613 3.3581087589263916
5 xxbos " in xxmaj april xxunk , the xxmaj university of xxmaj chicago agreed to xxunk xxmaj argonne xxmaj national xxmaj laboratory , with an association of xxmaj xxunk xxunk offering to xxunk the research . xxmaj argonne xxunk became the first " national " laboratory . xxmaj it did not , however , remain at its original location in the xxmaj argonne forest . xxmaj in xxunk , it moved farther west from the " xxunk xxmaj city " to a new site on xxmaj illinois xxunk . xxmaj when xxmaj xxunk xxmaj xxunk visited xxmaj argonne 's director , xxmaj walter xxmaj zinn , in xxunk , he asked him what kind of reactor was to be built at the new site . xxmaj when xxmaj zinn described a heavy - water reactor operating at one - xxunk the power of the xxmaj materials xxmaj testing xxmaj reactor positive negative 0.9594771265983582 3.2058887481689453