Training a Language Model on WT2
from fastai2.basics import *
from fastai2.callback.all import *
from fastai2.text.all import *
# all_slow

Data

path = untar_data(URLs.WIKITEXT_TINY)

The dataset comes with the articles in two csv files, so we read it and concatenate them in one dataframe.

df_train = pd.read_csv(path/'train.csv', header=None)
df_valid = pd.read_csv(path/'test.csv', header=None)
df_all = pd.concat([df_train, df_valid])
df_all.head()
0
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....

We could tokenize it based on spaces to compare (as is usually done) but here we'll use the standard fastai tokenizer.

splits = [list(range_of(df_train)), list(range(len(df_train), len(df_all)))]
tfms = [attrgetter("text"), Tokenizer.from_df(0), Numericalize()]
dsets = Datasets(df_all, [tfms], splits=splits, dl_type=LMDataLoader)
bs,sl = 104,72
dls = dsets.dataloaders(bs=bs, seq_len=sl)
dls.show_batch()
text text_
0 xxbos = xxmaj laborintus xxup ii ( 2012 recording ) = \n▁\n▁ xxmaj laborintus xxup ii is a 2012 album by xxmaj belgian orchestra xxmaj ictus xxmaj ensemble , vocal group xxmaj nederlands xxmaj kamerkoor and xxmaj american vocalist xxmaj mike xxmaj patton . xxmaj it is a recording of the 1965 work of the same name by xxmaj italian composer xxmaj luciano xxmaj berio , which featured lyrics taken from fellow = xxmaj laborintus xxup ii ( 2012 recording ) = \n▁\n▁ xxmaj laborintus xxup ii is a 2012 album by xxmaj belgian orchestra xxmaj ictus xxmaj ensemble , vocal group xxmaj nederlands xxmaj kamerkoor and xxmaj american vocalist xxmaj mike xxmaj patton . xxmaj it is a recording of the 1965 work of the same name by xxmaj italian composer xxmaj luciano xxmaj berio , which featured lyrics taken from fellow xxmaj
1 xxmaj hot 100 , and 22 on the r & b / hip - hop xxmaj singles chart . xxmaj it also reached number six on the xxmaj adult xxmaj contemporary chart , while reaching a peak of 39 on the xxmaj adult xxmaj pop xxmaj songs chart . xxmaj in xxmaj canada , the song debuted at number 98 on the xxup rpm xxmaj singles chart , on the xxmaj july hot 100 , and 22 on the r & b / hip - hop xxmaj singles chart . xxmaj it also reached number six on the xxmaj adult xxmaj contemporary chart , while reaching a peak of 39 on the xxmaj adult xxmaj pop xxmaj songs chart . xxmaj in xxmaj canada , the song debuted at number 98 on the xxup rpm xxmaj singles chart , on the xxmaj july 22
2 a blacksmith shop . \n▁\n▁ = = = xxmaj establishment = = = \n▁\n▁ xxmaj the dates of the earliest church can not be verified , but a church building may have been constructed at the site of xxmaj old xxmaj pine xxmaj church as early as around 1814 , and possibly as early as 1792 . \n▁ xxmaj on xxmaj september 24 , 1838 , xxmaj william xxunk and his wife blacksmith shop . \n▁\n▁ = = = xxmaj establishment = = = \n▁\n▁ xxmaj the dates of the earliest church can not be verified , but a church building may have been constructed at the site of xxmaj old xxmaj pine xxmaj church as early as around 1814 , and possibly as early as 1792 . \n▁ xxmaj on xxmaj september 24 , 1838 , xxmaj william xxunk and his wife ,
3 featured xxunk ' xxmaj dirty # # # # # # # in the remix for " xxmaj fantasy . " xxmaj they feared the sudden change was completely left field for her music , and worried it would jeopardize the album 's success . xxmaj in an interview with xxmaj entertainment xxmaj weekly , xxmaj carey openly spoke of her issues with xxmaj columbia : " xxmaj everybody was like ' xxunk ' xxmaj dirty # # # # # # # in the remix for " xxmaj fantasy . " xxmaj they feared the sudden change was completely left field for her music , and worried it would jeopardize the album 's success . xxmaj in an interview with xxmaj entertainment xxmaj weekly , xxmaj carey openly spoke of her issues with xxmaj columbia : " xxmaj everybody was like ' xxmaj
4 . xxmaj on the right - hand wall , an altar dedicated to the xxmaj virgin of the xxmaj confidence is decorated with numerous churrigueresque figurines xxunk away in niches , columns and top pieces . \n▁\n▁ = = = = xxmaj chapel of the xxmaj holy xxmaj angels and xxunk = = = = \n▁\n▁ xxmaj the xxmaj chapel of the xxmaj holy xxmaj angels and xxunk ( xxmaj spanish : xxmaj on the right - hand wall , an altar dedicated to the xxmaj virgin of the xxmaj confidence is decorated with numerous churrigueresque figurines xxunk away in niches , columns and top pieces . \n▁\n▁ = = = = xxmaj chapel of the xxmaj holy xxmaj angels and xxunk = = = = \n▁\n▁ xxmaj the xxmaj chapel of the xxmaj holy xxmaj angels and xxunk ( xxmaj spanish : xxmaj
5 plot to assassinate him and sent into exile . xxmaj in the novel , the xxmaj general remembers that he had once appointed xxmaj santander to govern xxmaj colombia because he believed him to be an effective and brave soldier . xxmaj he formerly regarded xxmaj santander as " his other self , and perhaps his better self " , but by the time of the events in xxmaj the xxmaj general to assassinate him and sent into exile . xxmaj in the novel , the xxmaj general remembers that he had once appointed xxmaj santander to govern xxmaj colombia because he believed him to be an effective and brave soldier . xxmaj he formerly regarded xxmaj santander as " his other self , and perhaps his better self " , but by the time of the events in xxmaj the xxmaj general in
6 - xxunk . \n▁ xxwrep 3 xxunk describes the condition : \n▁ xxmaj strange is the madness of those into whom demon foxes enter . xxmaj sometimes they run naked shouting through the streets . xxmaj sometimes they lie down and xxunk at the mouth , and xxunk as a fox xxunk . xxmaj and on some part of the body of the possessed a moving xxunk appears under the skin , xxunk . \n▁ xxwrep 3 xxunk describes the condition : \n▁ xxmaj strange is the madness of those into whom demon foxes enter . xxmaj sometimes they run naked shouting through the streets . xxmaj sometimes they lie down and xxunk at the mouth , and xxunk as a fox xxunk . xxmaj and on some part of the body of the possessed a moving xxunk appears under the skin , which
7 different as well , and was now one - piece and striped . \n▁\n▁ = = xxmaj return = = \n▁\n▁ xxmaj from her introduction to until 1967 , xxmaj midge was marketed as xxmaj barbie 's original best friend , but no dolls were sold for about 20 years until 1988 , when xxmaj california xxmaj dream xxmaj midge was sold as part of a beach line , which used the as well , and was now one - piece and striped . \n▁\n▁ = = xxmaj return = = \n▁\n▁ xxmaj from her introduction to until 1967 , xxmaj midge was marketed as xxmaj barbie 's original best friend , but no dolls were sold for about 20 years until 1988 , when xxmaj california xxmaj dream xxmaj midge was sold as part of a beach line , which used the "
8 xxup ii 's world . xxmaj each of the four stages — wasteland , caves , dungeon , and inner xxunk — is a series of interconnected rooms , populated by monsters , traps , and items . xxmaj the game displays one room at a time in a flick - screen manner : as the protagonist leaves a room , the screen is updated to display the next . xxmaj the ii 's world . xxmaj each of the four stages — wasteland , caves , dungeon , and inner xxunk — is a series of interconnected rooms , populated by monsters , traps , and items . xxmaj the game displays one room at a time in a flick - screen manner : as the protagonist leaves a room , the screen is updated to display the next . xxmaj the connections

Model

config = awd_lstm_lm_config.copy()
config.update({'input_p': 0.6, 'output_p': 0.4, 'weight_p': 0.5, 'embed_p': 0.1, 'hidden_p': 0.2})
model = get_language_model(AWD_LSTM, len(dls.vocab), config=config)
opt_func = partial(Adam, wd=0.1, eps=1e-7)
cbs = [MixedPrecision(clip=0.1), ModelReseter, RNNRegularizer(alpha=2, beta=1)]
learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), opt_func=opt_func, cbs=cbs, metrics=[accuracy, Perplexity()])
learn.fit_one_cycle(1, 5e-3, moms=(0.8,0.7,0.8), div=10)
epoch train_loss valid_loss accuracy perplexity time
0 5.541026 5.053756 0.241910 156.609619 02:27

Full training

#learn.fit_one_cycle(90, 5e-3, moms=(0.8,0.7,0.8), div=10)