2015-FSE-Suggesting accurate method and class names(6)

Table1:Evaluationprojects(Java).Orderedbypopularity.

Name

GitSHADescription

elasticsearch

d3e10f9RESTSearchEngineAndroid-Universal-Image-Loader19ce389AndroidLibrary

spring-framework2bf6b41ApplicationFrameworklibgdx789aa0bGameDevFrameworkstormbc54e8eDistributedComputationzxing71d8395Barcodeimageprocessingnetty

3b1f15eNetworkAppFrameworkplatform_frameworks_basef19176fAndroidBaseFrameworkbigbluebutton02bc78cWebConferencingjunitc730003TestingFramework

rxjavacf5ae70ReactiveJVMextensionsretro tafd00fdRESTclient

clojuref437b85ProgrammingLanguagedropwizard741a161RESTfulwebserverokhttp0a19746HTTP+SPDYclientpresto6005478DistributedSQLenginemetrics4984fb6MetricsFramework

spring-bootb542aa3AppFrameworkWrapperbukkitf210234MincraftModAPI

nokgiri

a93cde6

HTML/XML/CSSparser

Inreferencetothe rsttworesearchquestionswedescribethefeaturesthatweuseandhowwecapturesubstructureinthenextsection.Wethende nitivelyanswertheseresearchquestionsbycomparingourapproachwithprevioustechniquesforsuggestingvariablenamesonabroadsoftwarecorpus.Thereislittletonoresearchthattacklesthesecondtworesearchquestionstocompareagainst.Nonetheless,weuseann-grammodelasapointofcompari-sonfornamingmethodsandclassestodemonstratetheperformanceofourapproachasthatmodelhasperformedthebestforvariablenaminginthepastandwehypothesizeisreasonableforthesenewnamingtasks.Therestofthissectiondescribesourexperimentalsetupandmethodology.

DataWepickedthetopactiveJavaGitHubprojectsonJanuary22nd2015.Weobtainedthemostpopularprojectsbytakingthesumofthez-scoresofthenumberofwatchersandforksofeachproject,usingtheGitHubArchive.Startingfromthetopproject,weselectedthetop20projectsexcludingprojectsthatwereinadomainthatwaspreviouslyselected.Wealsoincludedonlyprojectswithmorethan50collaboratorsandmorethan500commits.TheprojectsalongwithshortdescriptionsareshowninTable1.Weusedthisproceduretoselectamature,active,anddiversecorpuswithlargedevelopmentteams.Finally,wesplitthe lesuniformlyintoatraining(70%)andatest(30%)set.

MethodologyWetrainallmodelsonthetrainsetsformedoverthe lesofeachproject.Toevaluatethemodels,foreachofthetest lesandforeachvariable(allidenti ersthatresolvetothesamesymbol),methoddeclarationorclassdeclaration,wecomputethefeaturesandcontextofthelocationoftheidenti erandaskthemodeltopredicttheactualtargettokenthedeveloperused(whichisunknowntothemodel),asinAllamanisetal.[2].Intheusecasesweconsider(seeSection1),modelsthataredeployedwitha“con dence lter”,thatis,themodelwillonlypresentasuggestiontotheuserwhentheprobabilityofthetoprankednameisabovesomethreshold.Thisistoavoidannoyingtheuserwithlow-qualitysuggestions.Tore ectthisinourevaluation,wemeasurethedegreetowhichthequalityoftheresultschangesasafunctionofthethreshold.Ratherthanreportingthethreshold,whichisnotdirectlyinterpretable,weinsteadreportthesuggestionfrequency,whichisthepercentageofnamesinthetestsetforwhichthemodeldecidestomakeapredictionforagiventhreshold.

Tomeasurethequalityofasuggestion,wecomputetheF1scoreandtheaccuracyfortheretrievaloverthesubtokensofeachcorrecttoken.Thus,allmethodsaregivenpartialcreditifthepredictednameisnotanexactmatchbutsharessubtokenswiththecorrectname.F1istheharmonicmeanofprecisionandrecallandisastandardmeasure[32]becauseitconservative:asaharmonicmean,

itsvalueisin uencedmostbythelowestofprecisionandrecall.Wealsocomputetheaccuracyofeachprediction:apredictioniscorrectwhenthemodelpredictsexactly(exactmatch)theactualtoken.WhencomputingtheF1scoreforsuggestionrankk>1,wepicktheprecision,recall,andF1oftherankl≤kthatresultsinthehighestF1score.

Becausethisevaluationfocusesonpopularprojects,theresultsmaynotre ectperformanceonalowqualityprojectinwhichmanynamesarepoor.Forsuchprojects,werecommendtrainingonadifferentprojectthathashighqualitynames,butleaveevaluatingthisapproachtofuturework.Alternatively,onecouldarguethat,becausewemeasurewhetherthemodelcanreproduceexistingnames,theevaluationistooharsh:ifapredictednamedoesnotmatchtheexistingname,itcouldbeequallygood,orevenanimprovement.Nonetheless,matchingexistingnamesinhighqualityprojects,aswedo,stillprovidesevidenceofoverallsuggestionquality,and,comparedtoauserstudy,anautomaticevaluationhasthegreatadvantagethatitallowsef cientcomparisonofalargernumberofdifferentmethods.

Finally,duringtraining,wesubstituterareidenti ers,subtokensandfeatures(i.e.thoseseenlessthantwotimesinthetrainingdata)withaspecialUNKtoken.Duringtesting,whenanyofthemodelssuggeststheUNKtoken,wedonotmakeanysuggestions;thatis,theUNKtokenindicatesthatthemodelexpectsaneologismthatitcannotpredict.Forthesubtokenmodel,duringtesting,wemayproducesuggestionsthatcontainUNKsubtokens.IntheunlikelycasethatacontexttokentwithaspecialSi+k=ti(i.e.isthesametoken),wereplaceti+kELFtoken.Thismakessurethatthecontextofthemodelincludesnoinformationaboutthetargettoken.

TrainingParametersWeusedlearningrate0.07,D=50,mini-batchsize100,dropout20%[48],generated10distractorsforeachsampleforeachepochandtrainedforamaximumof25epochspickingtheparametersthatachievedthemaximumlikelihoodinaheldoutvalidationset(the10%ofthetrainingdata).ThecontextsizewassettoK=6andsubtokencontextsizewassettoM=3.Beforethetrainingstarted,parameterswere 4initializedaround0withuniformadditivenoise(scaledby10).ThebiasparametersbwereinitializedsuchthatP(t|c)matchestheempirical(unigram)distributionofthetokens(orsubtokensforthesubtokenmodel).AllthehyperparametersexceptforDweretunedusingBayesianoptimizationonbigbluebuttonformethoddeclarations.Thepa-rameterDisspecialinthatasweincreaseit,theperformanceofeachmodelincreasesmonotonically(assumingagoodvalidationset),withdiminishingreturns.Also,anincreaseinDincreasesthecomputationalcomplexityoftrainingandtestingeachmodel.WepickedD=50thatresultedinagoodtrade-offofthecomputationalcomplexityvs.performance.

4.IDENTIFIERREPRESENTATION

Firstweevaluateourmodelqualitatively,byvisualizingitsoutput.Allofthemodelsthatwehavedescribedassigntokens,features,andsubtokenstoembeddings,whicharelocationsinaD-dimensionalcontinuousspace.Theselocationshavebeenselectedbythetrain-ingproceduretoexplainstatisticalpropertiesoftokens,butitdoesnotnecessarilyfollowthattheembeddingscaptureanythingaboutthesemanticsofnames.Toexplorethisquestion,weexaminequalitativelywhethernamesthatappearsemanticallysimilartousareassignedtosimilarembeddings,byvisualizingthecontinuousembeddingsassignedtonamesfromafewprojects.Thisraisestheimmediatedif cultyofhowtovisualizevectorsinaD=50dimensionalspace.Fortunately,thereisarichliteratureinstatisticsandmachinelearningaboutdimensionalityreductionmethodsthatmaphighdimensionalvectorstotwo-dimensionalvectorswhilepre-servingimportantpropertiesoftheoriginalspace.Therearevariousideasbehindsuchtechniques,suchaspreservingdistancesorangles

百度搜索“77cn”或“免费范文网”即可找到本站免费阅读全部范文。收藏本站方便下次阅读，免费范文网，提供经典小说综合文库2015-FSE-Suggesting accurate method and class names(6)在线全文阅读。