master/chapter/ch5.tex

\chapter{Evaluation}

In this chapter we will present the results of running each of the proposals discussed in this thesis on large-scale JavaScript projects. 

To evaluate this tool on existing JavaScript codebases, we have collected JavaScript projects from Github containing many or large JavaScript files.

Each case study was evaluated by running this tool on every \texttt{.js}-file in the repository,
and then collecting the number of matches found in total,
and how many files were successfully searched.
Evaluating if the transformation was correct is done by manually sampling output files,
and verifying that it passes through Babel Generate~\cite{BabelGenerate}
without errors. 

We describe below our results and observations on using our tool on the codebases of various large-scale projects that use JavaScript.

\textbf{Next.js}~\cite{NEXT.JS} is one of the largest projects on the web. It is used with React~\cite{React} to enable feature such as server-side rendering and static site generation. 

\begin{table}[H]
\begin{center}
   \begin{tabular}{|c|c|c|c|c|}
    \hline
    Proposal & Matches found & Files with matches & Files processed\\
    \hline \hline
    ``Pipeline'' & 242079 & 1912 & 3340 \\
    \hline
    ``Do Expression'' & 229 & 37 & 3340 \\
    \hline
    Await to Promise & 8 & 7 & 3340 \\
    \hline
\end{tabular} 
\end{center}
\caption{Evaluation with Next.js source code}
\label{fig:evalNextJS}
\end{table}

\textbf{Three.js}~\cite{ThreeJS} is a library for 3D rendering in JavaScript.
It is written purely in JavaScript and uses GPU for 3D calculations.

\begin{table}[H]
\begin{center}
\begin{tabular}{|c|c|c|c|}
    \hline
    Proposal & Matches found & Files with matches & Files searched \\
    \hline \hline
    Pipeline & 84803 & 1117 & 1384 \\
    \hline
    ``Do Expression'' & 248 & 36 & 1384 \\
    \hline
    Await to Promise & 13 & 7 & 1384 \\
    \hline
\end{tabular}
\end{center}
\caption{Evaluation with Three.js source code}
\label{fig:evalThreeJS}
\end{table}

\textbf{React}~\cite{React} is a graphical user interface library for JavaScript, which facilitates the creation of user interfaces for both web and native platforms. React is based upon splitting a user interface into components for simple development. It is currently one of the most popular libraries for creating web apps. 

\begin{table}[H]
\begin{center}
\begin{tabular}{|c|c|c|c|}
    \hline
    Proposal & Matches found & Files with matches & Files searched\\
    \hline \hline
    ``Pipeline'' & 16353 & 1266 & 2051 \\
    \hline
    ``Do Expression'' & 0 & 0 & 2051 \\
    \hline
    Await to Promise & 30 & 13 & 2051 \\
    \hline
\end{tabular}
\end{center}
\caption{Evaluation with React source code}
\label{fig:evalReact}
\end{table}


\textbf{Bootstrap}~\cite{Bootstrap} is a front-end framework used for creating responsive and mobile-first websites, and it comes with a variety of built-in components. This library is a good evaluation point for this thesis as it is written in ``vanilla'' JavaScript.

\begin{table}[H]
\begin{center}
\begin{tabular}{|c|c|c|c|}
    \hline
    Proposal & Matches found & Files with matches & Files searched\\
    \hline \hline
    "``Pipeline'' & 13794 & 109 & 115 \\
    \hline
    ``Do Expression'' & 0 & 0 & 115 \\
    \hline
    Await to Promise & 0 & 0 & 115 \\
    \hline
\end{tabular}
\end{center}
\caption{Evaluation with Bootstrap source code}
\label{fig:evalBootstrap}
\end{table}


\textbf{Atom}~\cite{Atom} is a text editor developed in JavaScript.

\begin{table}[H]
\begin{center}
\begin{tabular}{|c|c|c|c|}
    \hline
    Proposal & Matches found & Files with matches & Files searched\\
    \hline \hline
    ``Pipeline'' & 40606 & 361 & 401  \\
    \hline
    ``Do Expression'' & 3 & 3 & 401 \\
    \hline
    Await to Promise & 12 & 7 & 401 \\
    \hline
\end{tabular}
\end{center}
\caption{Evaluation with Atom source code}
\label{fig:evalAtom}
\end{table}

The ``Pipeline'' proposal is applicable to most files: the reason for this is that call expressions are widely used when writing JavaScript code. Our tool found matches in most files that Babel~\cite{BabelParser} managed to parse, and with manual evaluation transformations were performed correctly. 

The ``Do Expression'' proposal is not as ``applicable'' as the ``Pipeline'' proposal: this means that the amount of transformed code this specification in \DSL{} will be able to perform is expected and proven to be lower.
This is because the proposal introduces an entirely new way of writing
expression-oriented code in JavaScript.
If the code has not used the current way of writing
expression-oriented in JavaScript,
\DSL{} is limited in the amount of transformations it can perform.
Nevertheless, our tool is able to identify matches where it is applicable, and by manual verification transformations are correct. 

The imaginary ``Await to promise'' proposal also has an ``expected'' number of matches; however, we do not evaluate this proposal since it is not an official TC39 proposal.


Our tool demonstrates its capability to perform searches on large codebases,
to identify applicable code for proposals,
and to transform the code.
As can be seen from the tables above,
some of the proposals found zero matches
when evaluated on some of these codebases.
This is due to the fact that the developers of these projects
have not used the language construct the proposal is targeting.
Because of this,
no transformations can be performed.
This is especially apparent with the ``Do Expression`` proposal,
but also with the ``Await to Promise'' imaginary proposal.
This means that our tool's ability to perform transformations
depends on how widespread the adoption of the language construct targeted in a proposal is.
We can hypothesize that the amount of matches reflects the ``impact''
that design decisions made by the TC39 committee might have on established JavaScript projects and codebases.

We give examples of some of the transformations performed on these codebases in Appendix~\ref{appendix:b}.