2015-FSE-Suggesting accurate method and class names(4)

sθ(.)=rcontextqisDone+bisDone

Figure2:VisualexplanationoftherepresentationandcomputationofcontextintheD-dimensionalspaceasde nedinEquation4;

the nalparagraphofSection2.2explainsthesumovertheItlocations.EachsourcecodetokenandfeaturemapstoalearnedD-dimensionalvectorincontinuousspace.Thetoken-vectorsaremultipliedwiththeposition-dependentcontextmatrixCiandsummed,thenaddedtothesumofallthefeature-vectors.TheresultingvectoristheD-dimensionalrepresentationofthecurrentsourcecodeidenti er.Finally,theinnerproductofthecontextandtheidenti ervectorsisaddedtoascalarbiasb,producingascoreforeachidenti er.Thisneuralnetworkisimplementedbymappingitsequationsintocode.isthatlogbilinearmodelsmakeitespeciallyeasytoexploitlong-distanceinformation;e.g.whenpredictingthenameofamethod,itisusefultotakeintoaccountalloftheidenti ersthatappearinthemethodbody.Wemodellong-distancecontextviaasetoffeaturefunctions,suchas“WhetheranyvariableinthecurrentmethodisnamedaddCount”,“Whetherthereturntypeofthecurrentmethodisint,”andsoon.Thelogbilinearcontextmodelcombinesthesefeatureswiththelocalcontext.

Asbefore,supposethatwearetryingtopredictacodetokentgivenasequenceofcontexttokensc=(c0,c1,...,cN).Weassumethatccontainsalloftheothertokensinthe lethatarerelevantforpredictingt;e.g.tokensfromthebodyofthemethodthattnames.Thetokensincthatarenearesttothetargettaretreatedspecially.Supposethattoccursinpositioniofthe le,thatis,ifthe leisthetokensequencet1,t2,...,thent=ti.ThenthelocalcontextisthesetoftokensthatoccurwithinKpositionsoft,thatis,theset{ti+k}for K≤k≤K,k=0.Thelocalcontextincludestokensthatoccurbothbeforeandaftert.

Theoverallformofthecontextmodelwillfollowthegeneric

cisformin(1)and(2),exceptthatthecontextrepresentationr

cusingtwode neddifferently.Inthecontextmodel,wede ner

differenttypesofcontext:localandglobal.First,thelocalcontextishandledinaverysimilarwaytothelogbilinearLM.Eachpossiblelexemevisassignedtoavectorrv∈RD,and,foreachtokentkthatoccurswithinKtokensoftinthe le,weadditsrepresentationrtkintothecontextrepresentation.

Theglobalcontextishandledusingasetoffeatures.Eachfeatureisabinaryfunctionbasedonthecontexttokensc,suchastheexamplesdescribedatthebeginningofthissection.Formally,eachfeaturefmapsacvaluetoeither0or1.MaddisonandTarlow[31]useasimilarideatorepresentfeaturesofasyntacticcontext,thatis,anodeinanAST.Here,weextendthisideatoincorporatearbitraryfeaturesoflong-distancecontexttokensc.The rstcolumnofTable4presentsthefulllistoffeaturesthatweuseinthiswork.Tolearnanembedding,weassigneachfeaturefunctiontoasinglevectorinthecontinuousspace,inthesamewayaswedidfortokens.Mathematically,letFbethesetofallfeaturesinthemodel,andletFc,foracontextc,bethesetofallfeaturesfwithf(c)=1.Thenforeachfeaturef∈F,welearnanembeddingrf∈RD,whichisincludedasaparametertothemodelinexactlythesamewaythatrtwasforthelanguagemodelingcase.

Now,wecanformallyde neacontextmodelofcodeasaprob-abilitydistributionP(t|c)thatfollowstheform(1)and(2),where c=r context,wherer contextisr

context=r

f∈Ftc

matrixthatisalsolearnedduringtraining1.Intuitively,thisequation

sumstheembeddingsofeachtokentkthatoccursneartinthe le,andsumstheembeddingsofeachfeaturefunctionfthatreturns

context,justtrue(i.e.,1)forthecontextc.Oncewehavethisvectorr

asbefore,wecanselectatokentsuchthattheprobabilityP(t|c)

ishigh,whichhappensexactlywhenrcontextqtishigh—inother

words,whentheembeddingqtoftheproposedtargettiscloseto

contextofthecontext.theembeddingr

Figure2givesavisualexplanationoftheprobabilisticmodel.This guredepictshowthemodelassignsprobabilitytothetokenisDoneiftheprecedingtwotokensarefinalbooleanandthesucceedingtwoare=false.Readingfromrighttoleft,the guredescribeshowthecontinuousembeddingofthecontextiscomputed.Followingthedashed(pink)arrows,thetokensinthelocalcontextareeachassignedtoD-dimensionalvectorsrfinal,rboolean,andsoon,whichareaddedtogether(aftermultiplicationbytheC kmatri-cesthatmodeltheeffectofdistance),toobtaintheeffectofthelocal

context.Thesolid(blue)arrowsrepresentcontextontheembeddingr

theglobalcontext,pointingfromthenamesofthefeaturefunc-tionsthatreturntruetothecontinuousembeddingsofthosefeatures.Addingthefeatureembeddingstothelocalcontextembeddings

context.Thesimilaritybetweenyieldsthe nalcontextembeddingr

thisvectorandembeddingofthetargetvectorqisDoneiscomputedusingadotproduct,whichyieldsthevalueofsθ(isDone,c)whichisnecessaryforcomputingtheprobabilityP(isDone|c)via(1).MultipleTargetTokensUptonow,wehavepresentedthemodelinthecasewherewearerenamingatargettokentthatoccursatonlyonelocation,suchasthenameofamethod.Othercases,suchaswhensuggestingvariablenames,requiretakingalloftheoccurrencesofanameintoaccount[2].Whenatokentappearsata

contextseparatelysetoflocationsIt,wecomputethecontextvectorsr

foreachtokenti,fori∈It,thenaveragethem.Whenwedothis,wecarefullyrenamealloccurrencesofttoaspecialtokencalledSELFtoremovetfromitsowncontext.

2.3SubtokenContextModelsofCode

∑

rf+

k:K≥|k|>0

∑

Ckrti+k,

(4)

Alimitationofallofthepreviousmodelsisthattheyareunabletopredictneologisms,thatis,unseenidenti ernamesthathavenotbeenusedinthetrainingset.Thereasonforthisisthatweallowthemapfromalexemevtoitsembeddingqvtobearbitrary(i.e.withoutlearningafunctionalformfortherelationship),sowehavenobasistoassigncontinuousvectorstoidenti ernamesthathavenotbeenobserved.Inthissection,wesidestepthisproblembyexploitingtheinternalstructureofidenti ernames,resultinginanewmodelwhichwecallasubtokencontextmodel.

1Notethatkcanbepositiveornegative,sothatingeneralC

2

where,asbefore,Ckisaposition-dependentD×Ddiagonalcontext=C2.

百度搜索“77cn”或“免费范文网”即可找到本站免费阅读全部范文。收藏本站方便下次阅读，免费范文网，提供经典小说综合文库2015-FSE-Suggesting accurate method and class names(4)在线全文阅读。