CodeDJ: Reproducible queries over large-scale software repositories
Authors
Year
2021
Published
Leibniz International Proceedings in Informatics (LIPIcs). Saarbrücken: Dagstuhl Publishing,, 2021. p. 1-24. ISSN 1868-8969. ISBN 978-3-95977-190-0.
Type
Proceedings paper
Departments
Annotation
Analyzing massive code bases is a staple of modern software engineering research – a welcome side-effect of the advent of large-scale software repositories such as GitHub. Selecting which projects one should analyze is a labor-intensive process, and a process that can lead to biased results if the selection is not representative of the population of interest. One issue faced by researchers is that the interface exposed by software repositories only allows the most basic of queries. CodeDJ is an infrastructure for querying repositories composed of a persistent datastore, constantly updated with data acquired from GitHub, and an in-memory database with a Rust query interface. CodeDJ supports reproducibility, historical queries are answered deterministically using past states of the datastore; thus researchers can reproduce published results. To illustrate the benefits of CodeDJ, we identify biases in the data of a published study and, by repeating the analysis with new data, we demonstrate that the study’s conclusions were sensitive to the choice of projects.
Lambdulus: Teaching Lambda Calculus Practically
Authors
Máj, P.; Sliacký, J.
Year
2019
Published
SPLASH-E 2019: Proceedings of the 2019 ACM SIGPLAN Symposium on SPLASH-E. New York: ACM, 2019. p. 57-65. ISBN 9781450369893.
Type
Proceedings paper
Annotation
λ calculus is a great formal introduction to functional programming. However, its abstract nature poses a challenge for many students as they struggle both with the unfamiliarity of functional programming and with the high abstraction and minimalism of λ calculus. As a result, functional programming classes are often delayed to older students who are hoped to be better prepared for appreciating its qualities, as was the case at the authors’ university as well. 3 years ago, as part of a redesigned curriculum around programming languages education, we have decided to introduce students to various programming paradigms much sooner than before. In doing so, we faced the problem of explaining very theoretical foundational concepts to very young students in a very short time. We have monitored the achievements and shortcomings of the new course over the past years and as a result of our findings have developed Lambdulus, an interactive and visual evaluator of λ calculus expressions that encourages students to explore the mechanisms of λ calculus by treating it not as a theoretical concept, but as a programming language in its own right.
On the Impact of Programming Languages on Code Quality: A Reproduction Study
Type
Article
DOI
Annotation
In a 2014 article, Ray, Posnett, Devanbu, and Filkov claimed to have uncovered a statistically significant association between 11 programming languages and software defects in 729 projects hosted on GitHub. Specifically, their work answered four research questions relating to software defects and programming languages. With data and code provided by the authors, the present article first attempts to conduct an experimental repetition of the original study. The repetition is only partially successful, due to missing code and issues with the classification of languages. The second part of this work focuses on their main claim, the association between bugs and languages, and performs a complete, independent reanalysis of the data and of the statistical modeling steps undertaken by Ray et al. in 2014. This reanalysis uncovers a number of serious flaws that reduce the number of languages with an association with defects down from 11 to only 4. Moreover, the practical effect size is exceedingly small. These results thus undermine the conclusions of the original study. Correcting the record is important, as many subsequent works have cited the 2014 article and have asserted, without evidence, a causal link between the choice of programming language for a given task and the number of software defects. Causation is not supported by the data at hand; and, in our opinion, even after fixing the methodological flaws we uncovered, too many unaccounted sources of bias remain to hope for a meaningful comparison of bug rates across languages.