polsevev/master
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

something

Browse source
This commit is contained in:
Rolf Martin Glomsrud 2024-06-01 01:52:28 +02:00
parent fdda0273f8
commit e34158c6db
11 changed files with 223 additions and 211 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
build/report.pdf
View file

Binary file not shown.

39
chapter/ch2.tex
View file

@ -13,11 +13,11 @@ We explain now what a proposal is, and how proposals are developed in TC39 for t
A \emph{proposal} is a suggested change to the ECMA-262 language standard. These additions to the standard have to solve some form of problem with the current version of ECMAScript. Such problems can come in many forms, and can apply to any part of the language. Examples include: a feature that is not present in the language, inconsistent parts of the language, simplification of common patterns, and so on. The proposal development process is defined in the TC39 Process Document.
\textbf{The TC39 Process Document~\cite{TC39Process}} describes each stage a proposal has to go through to be accepted into the ECMA-262 language standard.
The \textbf{TC39 Process Document}~\cite{TC39Process} describes each stage a proposal has to go through to be accepted into the ECMA-262 language standard.
The purpose of \emph{stage 0} of the process is to allow for exploration and ideation around which parts of the current version of ECMAScript can be improved, and then define a problem space for the committee to focus on improving.
\emph{stage 1} is the stage the committee will start development of a suggested proposal. The are several requirements to enter this stage: a champion, a delegate of the committee responsible for a proposal, has to be identified. A rough outline of the problem must be privded, and a general shape of a solution must be given. There must have been discussion around key algorithms, abstractions and semantics of the proposal. Exploration of potential implementation challenges and cross-cutting concerns must have been done. The final requirement is for all parts of the proposal to be captured in a public repository. Once all these requirements are met, a proposal is accepted into stage 1. During this stage, the committee will work on the design of a solution, and resolve any cross-cutting concerns discovered previously.
\emph{stage 1} is the stage the committee will start development of a suggested proposal. The are several requirements to enter this stage: a champion, a delegate of the committee responsible for a proposal, has to be identified. A rough outline of the problem must be provided, and a general shape of a solution must be given. There must have been discussion around key algorithms, abstractions and semantics of the proposal. Exploration of potential implementation challenges and cross-cutting concerns must have been done. The final requirement is for all parts of the proposal to be captured in a public repository. Once all these requirements are met, a proposal is accepted into stage 1. During this stage, the committee will work on the design of a solution, and resolve any cross-cutting concerns discovered previously.
At \emph{stage 2} a preferred solution has been identified. Requirements for a proposal to enter this stage are as follows: all high level API's and syntax must be described in the proposal document, illustrative examples created, and an initial specification text must be drafted. During this stage the following areas of the proposal are explored: refining the identified solution, deciding on minor details, and create experimental implementations.
@ -27,21 +27,16 @@ Once a proposal has been sufficiently tested and verified, it is moved to \emph{
At \emph{Stage 4} the proposal is completed and included in the next revision of the ECMA-262.
\section{AST and Babel}
\section{Abstract Syntax Trees}
\subsection*{Abstract Syntax Tree}
An abstract syntax tree is a tree representation of source code. Every node of the tree represents a construct from the source code. ASTs remove syntactic details that are present in the source code, and while maintaining the structure of the program with its tree. Each node is set to represent elements of the programming language, some common ones are statements, expressions, declarations and other language concepts. Every node type represents a grammatical construct in the language the AST was built from.
An \emph{abstract syntax tree} (AST) is a tree representation of source code. Every node of such a tree represents a construct from the source code. ASTs remove syntactic details while maintaining the \emph{structure} of the program. Each node is set to represent elements of the programming language, such as statements, expressions, declarations to name a few. Every node type represents a grammatical construct in the language the AST was built from.
ASTs are important for working with source code, they are used by almost any tool that has to represent source code in some way to perform operations with it~\cite{AST3}. This is because the structure is simpler to work with then raw text, especially when considering tools like compilers, interpreters, or code transformation tools.
ASTs are important for manipulating source code; they are used by almost any tool that has to represent source code in some way to perform operations with it~\cite{AST3}. Using ASTs is favored over raw text due to their structured nature, especially when considering tools like compilers, interpreters, or code transformation tools. ASTs are produced by language parsers. For JavaScript, one of the popular labraries used for parsing is \emph{Babel}~\cite{Babel}.
ASTs are built by language parsers. A language parser takes the raw source code of a language, and parses the code into an AST while maintaining its structure but discarding irrelevant information. A simple example of how JavaScript is parsed into an AST can be seen in Figure~\ref{ex:srcToAST}.
Babel is a JavaScript toolchain, its main usage is converting ECMASCript 2015 and newer into older versions of JavaScript. This conversion is done to increase the compatibility of JavaScript in older environments such as older browsers.
\subsection*{Babel}
Babel is a JavaScript toolchain, its main usage is converting ECMASCript 2015 and newer into older versions of JavaScript. The conversion to older versions is done to increase compatibility of JavaScript in older environments such as older browsers.
Babel has a suite of libraries used to work with JavaScript source code, each library relies on Babels AST definition~\cite{BabelAST}. The AST specification Babel uses tries to stay as true to the ECMAScript standard as possible~\cite{BabelSpecCompliant}, which has made it a recommended parser to use for proposal transpiler implementations~\cite{TC39RecommendBabel}. A simple example of how source code parsed into an AST with Babel looks like can be seen in Figure~\ref{ex:srcToAST}.
Babel has a suite of libraries used to work with JavaScript source code, each library relies on Babel's AST definition~\cite{BabelAST}. The AST specification Babel uses tries to stay as close to the ECMAScript standard as possible~\cite{BabelSpecCompliant}, which has made it a recommended parser to use for proposal transpiler implementations~\cite{TC39RecommendBabel}. A simple example of how source code parsed into an AST with Babel looks like can be seen in Figure~\ref{ex:srcToAST}.
\begin{figure}[H]
\noindent\begin{minipage}{.30\textwidth}
@ -73,28 +68,24 @@ let name = f(100);
\end{figure}
Babel's mission is to transpile newer version of JavaScript into older versions that are more compatible with different environments. It has a rich plugin system to allow a myriad of features to be enabled or disabled. This makes the parser very versatile to fit different ways of working with JavaScript source code. This plugin system is built to enable or disable several language constructs,
One of Babel's more prominent features is \texttt{@babel/parse}~\cite{BabelParser} with plugins. This library allows parsing of JavaScript experimental features. These features are usually proposals that are under development by TC39, and the development of these plugins are a part of the proposal deliberation process. This allows for experimentation as early as stage one of the proposal development process. Some examples of proposals that were first supported by Babels plugin system are "Do Expression"~\cite{Proposal:DoProposal} and "Pipeline"~\cite{Pipeline}. These proposals are both currently in the very early stage of development, with "Do Expression" being stage one, and "Pipeline" being stage 2.
To achieve compilation of newer versions into older versions, Babel uses a plugin system that allows a myriad of features to be enabled or disabled. This makes the parser versatile to fit different ways of working with JavaScript source code. Because of this, Babel allows parsing of JavaScript experimental features. These features are usually proposals that are under development by TC39, and the development of these plugins are a part of the proposal deliberation process. This allows for experimentation as early as stage 1 of the proposal development process. Some examples of proposals that were first supported by Babel's plugin system are "Do Expression"~\cite{Proposal:DoProposal} and "Pipeline"~\cite{Pipeline}. These proposals are both currently in stage 1 and 2, respectively.
\section{Source Code Querying}
Source code querying is the action of searching source code to extract some information or find specific sections of code. This is primarily done using several varying techniques, and is a core part of many tools developers use. The primary use cases for source code querying is code understanding, analysis, code navigation, enforcement of styles along with others. All these are important tools developers use when writing programs, and they all rely on some form of source code queries.
Source code querying is the action of searching source code to extract some information or find specific sections of code. Source code querying comes in many forms, the simplest of which is text search. Since source code is primarily text, one can apply text search techniques to perform a query, or a more complex approach using regular expressions (eq tools like \texttt{grep}). Both these methods do not allow for queries based on the structure of the code, and rely solely on its syntax. AST-based queries allow queries to be written based on both syntax and structure, and are generally more powerful than regular text based queries. Another technique for code querying is based on semantics of code.
The primary use cases for source code querying is code understanding, analysis, code navigation, enforcement of styles, along with others. All these are important tools developers use when writing programs, and they all rely on some form of source code queries. One such tool is Integrated Development Environments (IDEs), as these tools are created to write source code, and therefore rely on querying the source code for many of their features. One such example of code querying being used in an IDE is JetBrains structural search and replace~\cite{StructuralSearchAndReplaceJetbrains}, where queries are defined based on code structure to find and replace sections of our program.
Source code querying comes in many forms, the simplest of which is text search. Since source code is primarily text, one can apply text search techniques to perform a query, this can be regular string search like with CTRL+F in the browser, or a more complex approach using regular expressions with tools like grep. Both these methods cannot allow for queries based on the structure of the code, and rely solely on its syntax. AST based queries allow queries to be written based on both syntax and structure, and are generally more powerful than regular text based queries. Another technique for code querying is creating queries based on semantics of code. Recently, querying based on the semantics of code is more feasible by using large language models to perform the queries.
\section{Domain-Specific languages}
Source code querying is used in many areas of software development. Some of the more prevalent areas is in Integrated Development Environments (IDEs), as these tools are created to write source code, and therefore rely on querying of the source code written for many of their features. Some of these features include code navigation, static code analysis, or complex code searching. One such example of code querying being used in an IDE is Jetbrains structural search and replace~\cite{StructuralSearchAndReplaceJetbrains}, where we define queries based on code structure to find and replace sections of our program.
\section{Domain Specific languages}
Domain specific languages are computer languages specialized to a specific domain. If we compare a DSL to a general purpose language like Python, C++ or JavaScript, these GPL are not designed with a specific task in mind, but have a more general feature set to allow them to be used in a wide array of applications. What a domain is for a DSL is not so simple to define, as there is no general way to define exactly the point in which a DSL becomes a GPL and vice versa. This difference is defined more like a spectrum, in which DSL is on one end and GPL is on the other~\cite{DSLSANDHOW}.
DSL's has some clear advantages when being applied to a specific domain compared to GPL's. A DSL allows for very concise and expressive code to be written that is specifically designed for the application, in which a GPL might require specific implementations to suit the domain. Using a DSL might result in faster development because of this expressiveness within the domain, this specificity to a domain might also increase correctness. However, there are also clear disadvantages to DSL's, the restrictiveness of a DSL might become a hinderance if it is not well designed to the domain. DSL's also might have a learning curve, making the knowledge required to use them a hinderance. Developing the DSL might also be a hinderance, as a DSL requires both knowledge of the domain and knowledge of language design.
Domain specific languages are software languages specialized to a specific narrow domain~\cite{Kleppe}. The difference between DSLs and General Purpose Languages (GPLs) is defined like a spectrum, in which DSL is on one end and GPLs is on the other~\cite{DSLSANDHOW}. DSLs allow domain experts to get involved in the development process, as it is expected that a domain expert would have the capabilities to read and write DSL code. A DSL allows for very concise and expressive code to be written that is specifically designed for the domain. Using a DSL might result in faster development because of this expressiveness within the domain, this specificity to a domain might also increase correctness. However, there are also some disadvantages to DSL's, the restrictiveness of a DSL might become a hinderance if it is not well designed to represent the domain. DSL's also might have a learning curve, making the knowledge required to use them a hinderance. Developing the DSL might also be a hinderance, as a DSL requires both knowledge of the domain and knowledge of software language engineering.
\section{Language Workbenches}
\textbf{REWRITE IN A MORE GENERAL SENSE, I AM SO LOST!}
A language workbench is a tool created to facilitate the development of a computer language, such as a DSL. Language workbenches also create tooling for languages defined within them, and help with the language development process in general.
Language workbenches support generating tooling for languages, as most modern computer languages are backed by some form of tooling. This tooling comes in the form of language parsing, language servers for integrated development environments, along with other tooling for using the language created within the language workbench.

234
chapter/ch3.tex
View file

@ -1,7 +1,6 @@
\chapter{Collecting User Feedback for Syntactic Proposals}
The goal for this project is to utilize users familiarity with their own code to gain early and worthwhile user feedback on new
syntactic proposals for ECMAScript.
The goal for this project is to utilize user's familiarity with their own code to gain early and worthwhile feedback on a certain kind of ECMAScript proposals.
\section{The core idea}
@ -17,30 +16,26 @@ The program will allow the users to preview proposals way before they are part o
The way this project will use the pre-existing knowledge a user has of their own code is to use that code as base for showcasing a proposals features. Using the users own code as base requires the following steps to automatically implement the examples that showcase the proposal inside the context of the users own code.
The ide is to identify where the features and additions of a proposal could have been used. This means identifying parts of the users program that use pre-existing ECMAScript features that the proposal is interacting with and trying to solve. This will then identify all the different places in the users program the proposal can be applied. This step is called \textit{matching} in the following chapters
The idea is to identify where the features and additions of a proposal could have been used. This means identifying parts of the users program that use pre-existing ECMAScript features that the proposal is interacting with and trying to solve. This will then identify all the different places in the users program the proposal can be applied. This step is called \textit{matching} in the following chapters.
Once we have matched all the parts of the program the proposal could be applied to, the users code has to be transformed to use the proposal, this means changing the code to use a possible future version of JavaScript. This step also includes keeping the context and functionality of the users program the same, so variables and other context related concepts have to be transferred over to the transformed code.
The output of the previous step is then a set of code pairs, where one a part of the users original code, and the second is the transformed code. The transformed code is then ideally a perfect replacement for the original user code if the proposal is part of ECMAScript. These pairs are used as examples to present to the user, presented together so the user can see their original code together with the transformed code. This allows for a direct comparison and an easier time for the user to understand the proposal.
The steps outlined in this section require some way of defining matching and transforming of code. This has to be done very precisely and accurately to avoid examples that are wrong. Imprecise definition of the proposal might lead to transformed code not being a direct replacement for the code it was based upon. For this we suggest two different methods, a definition written in a custom DSL \DSL and a definition written in a self-hosted way only using ECMAScript as a language as definition language. Read more about this in SECTION HERE.
The steps outlined in this section require some way of defining matching and transforming of code. This has to be done very precisely and accurately to avoid examples that are wrong. Imprecise definition of the proposal might lead to transformed code not being a direct replacement for the code it was based upon. For this we suggest two different methods, a definition written in a custom DSL \DSL{} and a definition written in a self-hosted way only using ECMAScript as a language as definition language.
\section{Applicable proposals}
\label{sec:proposals}
A proposal for ECMAScript is a suggested change for the language, in the case of ECMAScript this comes in the form of an addition to the language, as ECMAScript does not allow for breaking changes. There are many different kinds of proposals, this project focuses exclusively on Syntactic Proposals.
\subsection{Syntactic Proposals}
A syntactic proposal, is a proposal that contains only changes to the syntax of a language. This means, the proposal contains either no, or very limited change to functionality, and no changes to semantics. This limits the scope of proposals this project is applicable to, but it also focuses solely on some of the most challenging proposals where the users of the language might have the strongest opinions.
A \emph{syntactic proposal} is a proposal that contains only changes to the syntax of a language. This means that the proposal contains either no or very limited change to functionality, and no changes to the semantics of the language. This limits the scope of proposals this project is applicable to, but it also focuses solely on some of the most challenging proposals where the users of the language might have the strongest opinions.
\subsection{Simple example of a syntactic proposal}
Consider an imaginary proposal \exProp. This proposal describes adding an optional keyword for declaring numerical variables if the expression of the declaration is a numerical literal.
This proposal will look something like this:
An example of this proposal can be seen below:
\begin{lstlisting}[language={JavaScript}, caption={Example of imaginary proposal \exProp}, label={ex:proposal}]
\begin{lstlisting}[language={JavaScript}]
// Original code
let x = 100;
let b = "Some String";
@ -52,7 +47,7 @@ let b = "Some String";
let c = 200;
\end{lstlisting}
See that in \ref{ex:proposal} the change is optional, and is not applied to the declaration of \textit{c}, but it is applied to the declaration of \textit{x}. Since the change is optional to use, and essentially is just \textit{syntax sugar}, this proposal does not make any changes to functionality or semantics, and can therefore be categorized as a syntactic proposal.
See above that the change is optional, and is not applied to the declaration of \texttt{c}, but it is applied to the declaration of \texttt{x}. Since the change is optional to use, and essentially is \textit{syntax sugar}, this proposal does not make any changes to functionality or semantics, and can therefore be categorized as a syntactic proposal.
\iffalse
\subsection{\cite{Proposal:DiscardBindings}{Discard Bindings}}
@ -61,7 +56,7 @@ The proposal \discardBindings is classified as a Syntactic Proposal, as it conta
Unpacking when doing an assignment refers to assigning internal fields of an object/array directly in the assignment rather than using a temporary variable. See \ref{ex:unpackingObject} for an example of unpacking an object and \ref{ex:unpackingArr}.
\begin{lstlisting}[language={JavaScript}, caption={Example of unpacking Object}, label={ex:unpackingObject}]
\begin{lstlisting}[language={JavaScript}, caption={Example of unpacking Object.}, label={ex:unpackingObject}]
// previous
let temp = { a:1, b:2, c:3, d:4 };
let a = temp.a;
@ -72,7 +67,7 @@ let {a,b ...rest} = { a:1, b:2, c:3, d:4 };
rest; // { c:3, d:4 }
\end{lstlisting}
\begin{lstlisting}[language={JavaScript}, caption={Example of unpacking Array}, label={ex:unpackingArr}]
\begin{lstlisting}[language={JavaScript}, caption={Example of unpacking Array.}, label={ex:unpackingArr}]
// previous
let tempArr = [ 0, 2, 3, 4 ];
let a = tempArr[0]; // 0
@ -84,19 +79,19 @@ let [a, b, _1, _2] = [ 0, 2, 3, 4 ]; // a = 0, b = 2, _1 = 3, _2 = 4
In ECMAScripts current form, it is required to assign every part of an unpacked object/array to some identifier. The current status quo is to use \_ as a sign it is meant to be discarded. This proposal suggests a specific keyword \textit{void} to be used as a signifier whatever is at that location should be discarded.
In ECMAScripts current form, it is required to assign every part of an unpacked object/array to some identifier. The current status quo is to use \_ as a sign it is meant to be discarded. This proposal suggests a specific keyword \texttt{void} to be used as a signifier whatever is at that location should be discarded.
This feature is present in other languages, such as Rust wildcards, Python wildcards and C\# using statement and discards. In most of these other languages, the concept of discard is a single \_. In ECMAScript the \_ token is a valid identifier, therefore, this proposal suggests the use of the keyword \textit{void}. This keyword is already is reserved as part of function definitions where a function is meant to have no return value.
This feature is present in other languages, such as Rust wildcards, Python wildcards and C\# using statement and discards. In most of these other languages, the concept of discard is a single \_. In ECMAScript the \_ token is a valid identifier, therefore, this proposal suggests the use of the keyword \texttt{void}. This keyword is already is reserved as part of function definitions where a function is meant to have no return value.
This proposal allows for the \textit{void} keyword to be used in a variety of contexts. Some simpler than others but all following the same pattern of allowing discarding of bindings to an identifier. It is allowed anywhere the \textit{BindingPattern}, \textit{LexicalBinding} or \textit{DestructuringAssignmentTarget} features are used in ECMAScript. This means it can be applied to unpacking of objects/arrays, in callback parameters and class methods.
This proposal allows for the \texttt{void} keyword to be used in a variety of contexts. Some simpler than others but all following the same pattern of allowing discarding of bindings to an identifier. It is allowed anywhere the \texttt{BindingPattern}, \texttt{LexicalBinding} or \texttt{DestructuringAssignmentTarget} features are used in ECMAScript. This means it can be applied to unpacking of objects/arrays, in callback parameters and class methods.
\begin{lstlisting}[language={JavaScript}, caption={Example discard binding with variable discard}]
\begin{lstlisting}[language={JavaScript}, caption={Example discard binding with variable discard.}]
using void = new UniqueLock(mutex);
// Not allowed on top level of var/let/const declarations
const void = bar(); // Illegal
\end{lstlisting}
\begin{lstlisting}[language={JavaScript}, caption={Example Object binding and assignment pattern}]
\begin{lstlisting}[language={JavaScript}, caption={Example Object binding and assignment pattern.}]
let {b:void, ...rest} = {a:1, b:2, c:3, d:4}
rest; // {a:1, c:3, d:4};
\end{lstlisting}
@ -184,15 +179,13 @@ await using void = x; // via: LexicalBinding : `void` Initializer
\end{lstlisting}
\fi
\subsection{"Pipeline" Proposal}
\subsection{``Pipeline'' Proposal}
The "Pipeline" proposal~\cite{Pipeline} is a syntactic proposal which focuses on solving problems related to nesting of function calls and other expressions that take an expression as an argument.
The ``Pipeline'' proposal~\cite{Pipeline} is a syntactic proposal which focuses on solving problems related to nesting of function calls and other expressions that take an expression as an argument.
This proposal aims to solve two problems with performing consecutive operations on a value. In \emph{ECMAScript} there are two main styles of achieving this functionality currently: nesting calls and chaining calls, each of them come with a differing set of challenges when used.
This proposal aims to solve two problems with performing consecutive operations on a value. In ECMAScript there are two main styles of achieving this functionality currently: nesting calls and chaining calls, each of them come with a differing set of challenges when used.
Nesting calls is mainly an issue related to function calls with one or more arguments. When doing many calls in sequence the result will be a \textit{deeply nested} call expression.
Using nested calls has some specific challenges related to readability. The order of calls is from right to left, which is the opposite of the natural reading direction a lot of the users of ECMAScript are used to day to day. This means it is difficult to switch the reading direction when working out which call happens in which order. When using functions with multiple arguments in the middle of the nested call, it is not intuitive to see what call its arguments belong to. These issues are the main challenges this proposal is trying to solve. There are currently ways to improve readability with nested calls, as they can be simplified by using temporary variables. While this does introduce its own set of issues, it provides some way of mitigating the readability problem. Another positive side of nested calls is they do not require a specific design to be used, and a library developer does not have to design their library around this specific call style.
Nesting calls is mainly an issue related to function calls with one or more arguments. When doing many calls in sequence the result will be a \textit{deeply nested} call expression. Using nested calls can have some specific challenges related to readability. The reading order of nested calls is from right to left rather than the regular reading direction of JavaScript code which is left to right. This means it is difficult to switch the reading direction when working out which call happens in which order. When using functions with multiple arguments in the middle of the nested call, it is not intuitive to see what call its arguments belong to. These issues are the main challenges this proposal is trying to solve. There are currently ways to improve readability with nested calls, as they can be simplified by using temporary variables. While this does introduce its own set of issues, it provides some way of mitigating the readability problem. Another positive side of nested calls is they do not require a specific design to be used, and a library developer does not have to design their library around this specific call style. In the listings below, examples of deeply nested calls with both single and multiple arguments can be seen.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
@ -209,38 +202,30 @@ f1(v5, f2(f3(v3, f4(v1, v2)), v4), v6);
\end{minipage}\hfil
Chaining solves some of these issues: indeed, as it allows for a more natural reading direction left to right when identifying the sequence of call, arguments are naturally grouped together with their respective function call, and it provides a way of untangling deep nesting. However, executing consecutive operations using chaining has its own set of challenges. To use chaining, the API of the code being called has to be designed to allow for chaining. This is not always the case however, making use of chaining when it has not been specifically designed for can be very difficult. There are also concepts in JavaScript not supported when using chaining, such as arithmetic operations, literals, \texttt{await} expressions, \texttt{yield} expressions and so on. This is because all of these concept would "break the chain", and one would have to use temporary variables.
Chaining calls solves some of these issues: indeed, as it allows for a more natural reading direction left to right when identifying the sequence of call, arguments are naturally grouped together with their respective function call, and it provides a way of untangling deep nesting. However, executing consecutive operations using chaining has its own set of challenges. To use chaining, the API of the code being called has to be designed to allow for chaining. This is not always the case however, making use of chaining when it has not been specifically designed for can be very difficult. There are also concepts in JavaScript not supported when using chaining, such as arithmetic operations, literals, \texttt{await} expressions, \texttt{yield} expressions and so on. This is because all of these concepts would break the chain of calls, and one would have to use temporary variables. In the listings below are examples of both chaining with no arguments other than \texttt{self}, and examples where additional arguments are passed during the chain.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// Chaining calls
function1().function2().function3();
// Chaining calls with multiple arguments
function1(value1).function2().function3(value2).function4();
f1().f2().f3();
\end{lstlisting}
\end{minipage}\hfil
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// Chaining calls with multiple arguments
f1().f2(v1, v2).f3(v2).f4();
\end{lstlisting}
\end{minipage}\hfil
The "Pipeline" proposal aims to combine the benefits of these two styles without the challenges each method faces.
~
The main benefit of the proposal is to allow for a similar style to chaining when chaining has not been specifically designed to be applicable. The essential idea is to use syntactic sugar to change the writing order of the calls without influencing the API of the functions. Doing so will allow each call to come in the direction of left to right, while still maintaining the modularity of deeply nested function calls.
The ``Pipeline" proposal aims to combine the benefits of these two styles without the challenges each method faces. The proposal wants to achieve a similar style to chaining when doing deeply nested calls. The idea is to use syntactic sugar to change the writing order of the calls without influencing the API of the functions. Doing so will allow each call to come in the direction of left to right, while still maintaining the modularity of deeply nested function calls.
The proposal introduces a \emph{pipe operator}, which takes the result of an expression on the left, and pipes it into an expression on the right. The location of where the result is piped to is where the topic token is located. All the specifics of the exact token used as a topic token and exactly what operator will be used as the pipe operator might be subject to change, and is currently under discussion~\cite{PipelineBikeshedding}.
The code snippets below showcase the machinery of the proposal.
The code snippets below showcase the machinery of the proposal. They are the examples used to showcase the proposal in the proposal repository~\cite{Pipeline}
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// Status quo
var loc = Object.keys(grunt.config( "uglify.all" ))[0];
\end{lstlisting}
\end{minipage}\hfil
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// With pipes
var loc = grunt.config('uglify.all') |> Object.keys(%)[0];
\end{lstlisting}
\end{minipage}\hfil
More intuitive ordering of function calls, to know exactly the order of execution.
The example below showcases a left to right ordering of function calls that follows the order of execution.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
@ -256,7 +241,7 @@ const json = pkgs[0] |> npa(%).escapedName |> await npmFetch.json(%, opts);
\end{lstlisting}
\end{minipage}\hfil
Seeing which argument is passed to which function call is is simpler when using pipes.
In the example below, we can see that functions calls with multiple arguments are supported by ``Pipeline''.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
@ -271,7 +256,8 @@ return cb(predicate) |> _.negate(%) |> _.filter(obj, %, context);
\end{lstlisting}
\end{minipage}\hfil
Can be used with any number of function arguments, as long as a single topic token is used.
In the example below, we can see the topic token can be used multiple times to pass the same expression several times on the right side of a ``Pipeline'' proposal expression.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
@ -289,18 +275,18 @@ return xf
\end{lstlisting}
\end{minipage}\hfil
Complex call expressions are unraveled with pipes.
The pipe operator is present in many other languages such as F\#~\cite{FPipeOperator}, Julia~\cite{JuliaPipe}, Elixir~\cite{ElixirPipe} and Unix Shell~\cite{BashPipeline}. The main difference between the these languages's pipe operator and the pipe operator suggested in this proposal is the result of the left side expression has to be piped into a function with a single argument, this proposal suggests a topic reference to be used instead, clearly marking where the left side result should be piped to. Proposals suggesting pipe expressions similar to how it is done in F\# have been rejected by TC39 multiple times, but were rejected both times due to syntactical concerns and technical challenges~\cite{PipelineHistory}
The pipe operator is present in many other languages such as F\#~\cite{FPipeOperator} and Julia~\cite{JuliaPipe}. The main difference between the Julia and F\# pipe operator compared to this proposal, is the result of the left side expression has to be piped into a function with a single argument, the proposal suggests a topic reference to be used in stead of requiring a function. ~
\subsection{``Do Expression'' Proposal}
\subsection{"Do Expression"}
The ``Do Expression''~\cite{Proposal:DoProposal} proposal is a proposal meant to bring a style of \textit{expression oriented programming}~\cite{ExpressionOriented} to ECMAScript. Expression oriented programming is a concept taken from functional programming which allows for combining expressions in a very free manner, resulting in a highly malleable programming experience.
The "Do Expression"~\cite{Proposal:DoProposal} proposal, is a proposal meant to bring a style of \textit{expression oriented programming}~\cite{ExpressionOriented} to ECMAScript. Expression oriented programming is a concept taken from functional programming which allows for combining expressions in a very free manner, resulting in a highly malleable programming experience.
The motivation of the "Do Expression" proposal is to allow for local scoping of a code block that is treated as an expression. Thus, complex code requiring multiple statements will be confined inside its own scope~\cite[8.2]{ecma262} and the resulting value is returned from the block implicitly as an expression, similarly to how a unnamed functions or arrow functions are currently used. To achieve this behavior in the current stable version of ECMAScript, one needs to use immediately invoked unnamed functions~\cite[15.2]{ecma262} and invoke them immediately, or use an arrow function~\cite[15.3]{ecma262}.
The motivation of the ``Do Expression'' proposal is to allow for local scoping of a code block that is treated as an expression. Thus, complex code requiring multiple statements will be confined inside its own scope~\cite[Sect. 8.2]{ecma262} and the resulting value is returned from the block implicitly as an expression, similarly to how unnamed functions or arrow functions are currently used. To achieve this behavior in the current version of ECMAScript, one needs to use immediately invoked unnamed functions~\cite[Sect. 15.2]{ecma262} or use an arrow function~\cite[Sect. 15.3]{ecma262}.
The codeblock of a \texttt{do} expression has one major difference from these equivalent functions, as it allows for implicit return of the final statement of the block, and is the resulting value of the entire \texttt{do} expression. The local scoping of this feature allows for a cleaner environment in the parent scope of the \texttt{do} expression. What is meant by this is for temporary variables and other assignments used once can be enclosed inside a limited scope within the \texttt{do} block. Allowing for a cleaner environment inside the parent scope where the \texttt{do} block is defined.
The current version of JavaScript enables the use of arrow functions with no arguments to achieve similar behavior to ``Do Expression'', an example of this can be seen in the Listing below. The main difference between arrow functions and ``Do Expression'' is the final statement/expression will implicitly return its Completion Record~\cite[Sect. 6.2.4]{ecma262}, and we de not need a return statement.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// Current status quo
@ -320,7 +306,7 @@ let x = do {
\end{lstlisting}
\end{minipage}\hfil
The current version of JavaScript enables the use of arrow functions with no arguments to achieve similar behavior to "Do Expression". The main difference in this case, is the final statement/expression will implicitly return it's Completion Record~\cite[6.2.4]{ecma262}
The example below is very similar to the one above, and uses an unnamed function~\cite[15.2]{ecma262} which is invoked immediately to produce similar behavior to the ``Do Expression'' proposal.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
@ -343,22 +329,20 @@ let x = do {
\end{lstlisting}
\end{minipage}\hfil
This example is very similar, as it uses an unnamed function~\cite[15.2]{ecma262} which is invoked immediately to produce similar behavior to the "Do Expression" proposal.
\subsection{``Await to Promise'' (imaginary proposal)}
\subsection{Await to Promise}
We discuss now an imaginary proposal that was used as a running example during the development of this thesis. This proposal is of just a pure JavaScript transformation example. The transformation this proposal is meant to display, is transforming a code using \texttt{await}~\cite[27.7.5.3]{ecma262}, into code which uses a promise~\cite[27.2]{ecma262}.
We discuss now an imaginary proposal that was used as a running example during the development of this thesis. This proposal is a pure JavaScript transformation example. The transformation this proposal is meant to display is transforming code using \texttt{await}~\cite[Sect. 27.7.5.3]{ecma262} into code which uses a promise~\cite[Sect. 27.2]{ecma262}.
To perform this transformation, we define an equivalent way of expressing an \texttt{await} expression as a promise. This means removing \texttt{await}, this expression now will return a promise, which has a function \texttt{then()}, this function is executed when the promise resolves. We pass an arrow function as argument to \texttt{then}, and append each following statement in the current scope~\cite[8.2]{ecma262} inside the block of that arrow function. This will result in equivalent behavior to using \texttt{await}.
To perform this transformation, we define an equivalent way of expressing an \texttt{await} expression as a promise. This means removing \texttt{await}, this expression now will return a promise, which has a function \texttt{then}, this function is executed when the promise resolves. We pass an arrow function as argument to \texttt{then}, and append each following statement in the current scope~\cite[Sect. 8.2]{ecma262} inside the block of that arrow function. This will result in equivalent behavior to using \texttt{await}. An example of a function using \texttt{await} can be seen below on the left. The example below on the right is the same function but is using a promise.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}[language={JavaScript}]
// Code containing await
async function a(){
let b = 9000;
let something = await asyncFunction();
let c = something + 100;
let s = await asyncF();
let c = s + 100;
return c + 1;
}
\end{lstlisting}
@ -368,38 +352,38 @@ async function a(){
// Re-written using promises
async function a(){
let b = 9000;
return asyncFunction()
.then(async (something) => {
let c = something + 100;
return asyncF()
.then(async (s) => {
let c = s + 100;
return c;
})
}
\end{lstlisting}
\end{minipage}\hfil
Transforming using this imaginary proposal, will result in a returning the expression present at the first \texttt{await} expression, with a deferred function \texttt{then}, that will execute once the expression is completed. This function \texttt{then} takes a callback containing a lambda function with a single argument. This argument shares a name with the initial \texttt{VariableDeclaration}. This is needed because we have to transfer all statements that occur after the original \texttt{await} expression into the body of the callback function. This callback function also has to be async, in case any of the statements placed into it contains \texttt{await}. This will result in equivalent behavior to the original code.
Transforming using this imaginary proposal will result in a returning the expression present at the first \texttt{await} expression, with a deferred function \texttt{then} which will execute once the expression is completed. This function takes a callback containing a lambda function with a single argument. This argument shares a name with the initial \texttt{VariableDeclaration}. This is needed because we have to transfer all statements that occur after the original \texttt{await} expression into the body of the callback function. This callback function also has to be async, in case any of the statements placed into it contains \texttt{await}. This will result in equivalent behavior to the original code.
\section{Searching user code for applicable snippets}
\section{Searching user code for applicable parts}
To identify snippets of code in the user's code where a proposal is applicable, we need some way to define patterns of code to use as a query. To do this, we have designed and implemented a domain-specific language that allows matching parts of code that is applicable to some proposal, and transforming those parts to use the features of that proposal.
To identify parts of code in the user's code where a proposal is applicable, we need some way to define patterns of code to use as a query. To do this, we have designed and implemented a domain-specific language that allows matching parts of code that is applicable to some proposal, and transforming those parts to use the features of that proposal.
\subsection{Structure of \DSL}
\subsection{Structure of \DSL{}}
\label{sec:DSLStructure}
In this section, we describe the structure of \DSL. We describe every section of the language, why each section is needed and what it is used for.
In this section, we describe the structure of \DSL{} (JavaScript Template Query Language).
\paragraph*{Proposal definition.}
\paragraph*{Proposal definition}
\DSL is designed to mimic the examples already provided in proposal descriptions~\cite{TC39Process}. These examples can be seen in each of the proposals described in Section \ref{sec:proposals}. The idea is to allow a similar kind of notation to the examples in order to define the transformations.
\DSL{} is designed to mimic the examples already provided in proposal descriptions. These examples can be seen in each of the proposals described in Section \ref{sec:proposals}. The idea is to allow a similar kind of notation to the examples in order to define the transformations.
The first part of \DSL is defining the proposal, this is done by creating a named block containing all definitions of templates used for matching alongside their respective transformation. This section is used to contain everything relating to a specific proposal and is meant for easy proposal identification by tooling.
The first part of \DSL{} is defining the proposal; this is done by creating a named block containing all definitions of templates used for matching alongside their respective transformation.
\begin{lstlisting}
proposal Pipeline_Proposal {}
\end{lstlisting}
\paragraph*{Case definition.}
Each proposal will have one or more definitions of a template for code to identify in the users codebase, and its corresponding transformation definition. These are grouped together to have a simple way of identifying the corresponding cases of matching and transformations. This section of the proposal is defined by the keyword \textit{case} and a block that contains its related fields. A proposal definition in \DSL should contain at least one \texttt{case} definition. This allows for matching many different code snippets and showcasing more of the proposal than a single concept the proposal has to offer.
\paragraph*{Case definition}
Each proposal will have one or more definitions of a template for code to identify in the users codebase, and its corresponding transformation definition. These are grouped together to have a simple way of identifying the corresponding cases of matching and transformations. This block of the proposal is defined by the keyword \texttt{case} and a block that contains its related fields. A proposal definition in \DSL{} should contain at least one \texttt{case} definition. This allows for matching many different code parts and showcasing more of the proposal than a single concept the proposal has to offer.
\begin{lstlisting}
case case_name {
@ -409,18 +393,18 @@ Each proposal will have one or more definitions of a template for code to identi
\paragraph*{Template used for matching}
To define the template used to match, we have another section defined by the keyword \textit{applicable to}. This section will contain the template defined using JavaScript with specific DSL keywords defined inside the template. This template is used to identify applicable parts of the user's code to a proposal.
To define the template used to match, we have another block defined by the keyword \texttt{applicable to}. This block will contain the template defined using JavaScript with specific DSL blocks defined inside the template. This template is used to identify applicable parts of the user's code to a proposal.
\begin{lstlisting}
applicable to {
"let a = 0;"
}
\end{lstlisting}
This \texttt{applicable to} template, will create matches on any \texttt{VariableDeclaration} that is initialized to the value 0, and is stored in an \texttt{Identifier} with name \texttt{a}.
This \texttt{applicable to} template will create matches on any \texttt{VariableDeclaration} that is initialized to the value 0, and has an \texttt{Identifier} with the name \texttt{a}.
\paragraph*{Defining the transformation}
To define the transformation that is applied to a specific matched code snippet, the keyword \textit{transform to} is used. This section is similar to the template section, however it uses the specific DSL identifiers defined in applicable to, to transfer the context of the matched user code, this allows us to keep parts of the users code important to the original context it was written in.
To define the transformation that is applied to a specific matched part of the code, the keyword \texttt{transform to} is used. This block is similar to the template block, however it uses the specific DSL identifiers defined in applicable to, to transfer the context of the matched user code, this allows us to keep parts of the users code important to the original context it was written in.
\begin{lstlisting}
transform to{
@ -431,11 +415,11 @@ transform to{
\end{lstlisting}
This transformation definition, will change any code matched to its corresponding matching definition into exactly what is defined. This means for any matches produced this code will be inserted in its place.
\paragraph*{Full definition of \DSL}
\paragraph*{Full definition of \DSL{}}
Taking all these parts of \DSL structure, defining a proposal in \DSL will look as follows.
Taking all these parts of \DSL{} structure, defining a proposal in \DSL{} will look as follows.
\begin{lstlisting}[caption={\DSL definition of a proposal}]
\begin{lstlisting}[caption={\DSL{} definition of a proposal.}]
proposal PROPOSAL_NAME {
case CASE_NAME_1 {
applicable to {
@ -455,7 +439,7 @@ proposal PROPOSAL_NAME {
}
}
\end{lstlisting}
This full example of \DSL has two \texttt{case} sections. Each \texttt{case} is applied one at a time to the user's code. The first case will try to find any \texttt{VariableDeclaration} statements, where the identifier is \texttt{b}, and the right side expression is a \texttt{Literal} with value 100. The second \texttt{case} will change any empty \texttt{console.log} expression, into a \texttt{console.dir} expression.
This complete example of \DSL{} has two \texttt{case} blocks. Each \texttt{case} is applied one at a time to the user's code. The first case will try to find any \texttt{VariableDeclaration} statements, where the identifier is \texttt{b}, and the right side expression is a \texttt{Literal} with value 100. The second \texttt{case} will change any empty \texttt{console.log} expression, into a \texttt{console.dir} expression.
\subsection{How a match and transformation is performed}
@ -463,54 +447,53 @@ This full example of \DSL has two \texttt{case} sections. Each \texttt{case} is
To perform matching and transformation of the user's code, we first have to have some way of identifying applicable user code. These applicable code sections then have to be transformed and inserted it back into the full user code definition.
\subsection*{Identifying applicable code}
\paragraph*{Identifying applicable code}
To identify sections of code a proposal is applicable to, we use \emph{templates}, which are snippets of JavaScript. These templates are used to identify and match applicable sections of a users code. A matching section for a template is one that produces an exactly equal AST structure, where each node of the AST sections has the same information contained within it. This means that templates are matched exactly against the users code, this does not really provide some way of querying the code and performing context based transformations, so for that we use \textit{wildcards} within the template.
To identify parts of code a proposal is applicable to, we use \emph{templates}, which are defined using JavaScript. These templates are used to identify and match applicable parts of a users code. A matching part for a template is one that produces an exactly equal AST structure, where each node of the AST has the same information contained within it. This means that templates are matched exactly against the users code; this does not really provide some way of querying the code and performing context based transformations, so for that we use \textit{wildcards} within the template.
Wildcards are interspliced into the template inside a block denoted by \texttt{<< >>}. Each wildcard starts with an identifier, which is a way of referring to that wildcard in the definition of the transformation template later. This allows for transferring the context of parts matched to a wildcard into the transformed output, like identifiers, parts of statements, or even entire statements, can be transferred from the original user code into the transformation template. A wildcard also contains a type expression. A type expression is a way of defining exactly the types of AST nodes a wildcard will produce a match against. These type expressions use Boolean logic together with the AST node-types from BabelJS~\cite{Babel} to create a very versatile of defining exactly what nodes a wildcard can match against.
Wildcards are interspliced into the template inside a block denoted by \texttt{<< >>}. Each wildcard starts with an identifier, which is a way of referring to that wildcard in the definition of the transformation template later. This allows for transferring the context of parts matched to a wildcard into the transformed output, like identifiers, parts of statements, or even entire statements, can be transferred from the original user code into the transformation template. A wildcard can also contains a \emph{type expression}. A type expression is a way of defining exactly the types of AST nodes a wildcard will produce a match against. These type expressions use Boolean logic together with the AST node-types from BabelJS~\cite{Babel} to create a very versatile of defining exactly what nodes a wildcard can match against.
\subsubsection*{Wildcard type expressions}
\paragraph*{Wildcard type expressions}
Wildcard expressions are used to match AST node types based on Boolean logic. This Boolean logic is based on comparison of Babel AST node types~\cite{BabelAST}. We do this because we need an accurate and expressive way of defining specifically what kinds of AST nodes a wildcard can be matched against. This means an type expression can be as simple as \texttt{VariableDeclaration}: this will match only against a node of type \texttt{VariableDeclaration}. We also special types for \texttt{Statement} for matching against a statement, and \texttt{Expression} for matching any expression.
Wildcard expressions are used to match AST node types based on Boolean logic. This Boolean logic is based on comparison of Babel AST node types~\cite{BabelAST}. We do this because we need an accurate and expressive way of defining specifically what kinds of AST nodes a wildcard can be matched against. This means a type expression can be as simple as \texttt{VariableDeclaration}: this will match only against a node of type \texttt{VariableDeclaration}. We also special types for \texttt{Statement} for matching against a statement, and \texttt{Expression} for matching any expression.
This example will allow any \texttt{CallExpression} to match against this wildcard named \texttt{expr}.
The example below will allow any node with type \texttt{CallExpression} to match against this wildcard named \texttt{expr}.
\begin{lstlisting}
<< expr: CallExpression >>
\end{lstlisting}
To make this more expressive, the type expressions support binary and unary operators.We support the following operators, \texttt{\&\&} is logical conjunction, \texttt{||} means logical disjunction,\texttt{!} is logical negation. This makes it possible to build complex type expressions, making it very expressive exactly what nodes are allowed to match against a specific wildcard.
To make this more expressive, the type expressions use binary and unary operators. The following operators are supported: \texttt{\&\&} for logical conjunction, \texttt{||} for logical disjunction,\texttt{!} for logical negation. This makes it possible to build complex type expressions, making it very expressive exactly what nodes are allowed to match against a specific wildcard.
In the first example on line 1, we want to limit the wildcard to not match against any nodes with type \texttt{VariableDeclaration}, while still allowing any other \texttt{Statement}. The example on line 2 want to avoid loop specific statements. We express this by allowing any \texttt{Statement}, but we negate the expression containing the types of loop specific statements.
In the example below on line 1, we want to limit the wildcard to not match against any nodes with type \texttt{VariableDeclaration}, while still allowing any other \texttt{Statement}. On line 2 below we want to avoid any loop-specific statements. We express this by allowing any \texttt{Statement}, but we negate the expression containing the types of loop specific statements.
\begin{lstlisting}
<< notVariableDeclaration: Statement && !VariableDeclaration >>
<< noLoopSpecificStatements: Statement && !(BreakStatement || ContinueStatement) >>
\end{lstlisting}
The wildcards support matching subsequent sibling nodes of the code against a single wildcard. We achieve this behavior done by using a Keene plus at the top level of the expression. A Keene plus means one or more, so we allow for one or more matches in order when using this token. This is useful for matching against a series of one or more specific nodes, the matching algorithm will continue to match until the type expression no longer evaluates to true.
The wildcards support matching subsequent sibling nodes of the code against a single wildcard. We achieve this behavior done by using a Kleene plus at the top level of the expression. A Kleene plus means one or more, so we allow for one or more matches in order when using this token. This is useful for matching against a series of one or more specific nodes, the matching algorithm will continue to match until the type expression no longer evaluates to true.
In the example below, we allow the wildcard to match multiple nodes with the Keene plus \texttt{+}. This example will continue to match against itself as long as the nodes are a \texttt{Statement} and at the same time is not a \texttt{ReturnStatement}.
In the example below, we allow the wildcard to match multiple nodes with the Kleene plus \texttt{+}. This example will continue to match against itself as long as the nodes are a \texttt{Statement} and at the same time is not a \texttt{ReturnStatement}. This example showcases how a wildcard that matches against many sibling nodes is written.
\begin{lstlisting}
<< statementsNoReturn : (Statement && !ReturnStatement)+ >>
\end{lstlisting}
In the example below a wildcard block is defined on the right hand side of an assignment statement. This wildcard will match against any AST node classified as a \texttt{CallExpression} or an \texttt{Identifier}. This example showcases how wildcards are interspliced with JavaScript templates.
\begin{lstlisting}
let variableName = << expr1: ((CallExpression || Identifier) && !ReturnStatement)+ >>;
\end{lstlisting}
A wildcard section is defined on the right hand side of an assignment statement. This wildcard will match against any AST node classified as a CallExpression or an Identifier.
\subsection{Transforming}
When matching sections of the users code has been found, we need some way of defining how to transform those sections to showcase a proposal. This is done using the \texttt{transform to} template. This template describes the general structure of the newly transformed code, with context from the users code by using wildcards.
When matching parts of the users code has been found, we need some way of defining how to transform those parts to showcase a proposal. This is done using the \texttt{transform to} template. This template describes the general structure of the newly transformed code.
A transformation template defines how the matches will be transformed after applicable code has been found. The transformation is a general template of the code once the match is replaced in the original AST. However, without transferring over the context from the match, this would be a template search and replace. Thus, to transfer the context from the match, wildcards are defined in this template as well. These wildcards use the same block notation found in the \texttt{applicable to} template, however they do not need to contain the types, as those are not needed in the transformation. The only required field of the wildcard is the identifier defined in \texttt{applicable to}. This is done to know which wildcard match we are taking the context from, and where to place it in the transformation template.
Transforming a variable declaration from using \texttt{let} to use \texttt{const}.
The example below showcases a transformation that transforms a variable declaration from using \texttt{let} to use \texttt{const}. We do this by defining the name of the variable as a wildcard, and the expression as a wildcard. These are referenced in the transformation to transfer context from the match into the transformation template.
\begin{lstlisting}[language={JavaScript}]
// Example applicable to template
applicable to {
@ -524,23 +507,21 @@ transform to {
\end{lstlisting}
\subsection{Using \DSL}
\subsection{Using \DSL{}}
\DSL is designed to be used at a proposal development stage, this means the users of \DSL will most likely be TC39~\cite{TC39} delegates, or otherwise relevant stakeholders.
\DSL{} is designed to closely mimic the style of the examples required in the TC39 process~\cite{TC39Process}. We chose to design it this way to specifically make this tool fit the use-case of the committee. Since the idea behind this project is to gather early user feedback on syntactic proposals, the users of this kind of til is the committee themselves, as they are the ones that want user feedback.
\DSL is designed to closely mimic the style of the examples required in the TC39 process~\cite{TC39Process}. We chose to design it this way to specifically make this tool fit the use-case of the committee. The idea behind this project is to gather early user feedback on syntactic proposals, this would mean the main users of this kind of tool is the committee themselves.
\DSL is just written using text, most Domain-specific languages have some form of tooling to make the process of using the DSL simpler and more intuitive. \DSL has an extension built for Visual Studio Code, see Figure \ref{fig:ExtensionExample}, this extension supports many common features of language servers, it supports auto completion, it will produce errors if fields are defined wrong or are missing parameters.
Writing a proposal definition in \DSL{} is done with text, most domain-specific languages have some form of tooling to make the process of using the DSL simpler and more intuitive. \DSL{} has an extension built for Visual Studio Code, see Figure \ref{fig:ExtensionExample}. This extension supports auto completion, error checking, and other common IDE features.
\begin{figure}[H]
\begin{center}
\includegraphics[width=\textwidth/2]{figures/ExtensionExample.png}
\caption{\label{fig:ExtensionExample} Writing \DSL in Visual Studio Code with extension}
\includegraphics[width={.75\textwidth}]{figures/ExtensionExample.png}
\caption{\label{fig:ExtensionExample} Writing \DSL{} in Visual Studio Code with extension.}
\end{center}
\end{figure}
The language server included with this extension performs validation of the wildcards. This allows verification of wildcard declarations in applicable to, see Figure \ref{fig:NoTypes}. If a wildcard is declared with no types, an error will be reported.
The language server included with this extension performs validation of the wildcards. This allows verification of wildcard declarations in applicable to, as shown in Figure~\ref{fig:NoTypes}. If a wildcard is declared with no types, an error will be reported.
\begin{figure}[H]
\begin{center}
@ -549,25 +530,26 @@ The language server included with this extension performs validation of the wild
\end{center}
\end{figure}
The extension automatically uses wildcard declarations in \texttt{applicable to} to verify all wildcards referenced in \texttt{transform to} are declared. If an undeclared wildcard is used, an error will be reported and the name of the undeclared wildcard will be displayed, see Figure \ref{fig:UndeclaredWildcard}.
The extension automatically uses wildcard declarations in \texttt{applicable to} in order to verify that all wildcards referenced in \texttt{transform to} are declared. If an undeclared wildcard is used, an error will be reported and the name of the undeclared wildcard will be displayed, as shown in Figure~\ref{fig:UndeclaredWildcard}.
\begin{figure}[H]
\begin{center}
\includegraphics[width=\textwidth/2]{figures/UndeclaredRef.png}
\includegraphics[width=.75\textwidth]{figures/UndeclaredRef.png}
\caption{\label{fig:UndeclaredWildcard} Error displayed with usage of undeclared wildcard.}
\end{center}
\end{figure}
\section{Using the \DSL with syntactic proposals}
\section{Using the \DSL{} with syntactic proposals}
This section contains the definitions of the proposals used to evaluate the tool created in this thesis. These definitions do not have to cover every single case where the proposal might be applicable, as they just have to be general enough to create some amount of examples that will give a representative number of matches when the transformations are applied to some relatively long user code. This is because this this tool will be used to gather feedback from user's on proposals during development. Because of this use case, it does not matter that we catch every single applicable code snippet, just that we find enough to perform a "showcase" of the proposal to the user. The most important thing is that the transformation is correct, as incorrect transformations will lead to bad feedback on the proposal.
In this section, we present specifications of the proposals described in Section~\ref{sec:proposals}. We will use these specifications to evaluate the tool created in this thesis. These specifications do not have to cover every single case where the proposal might be applicable, as they only have to be general enough to create some amount of examples that will give a representative number of matches when the transformations are applied to some relatively long user code.
This is because this tool is designed to be used by TC39 to gather feedback from user's on proposals during development. This use case means the specifications should be defined in a way that showcases the proposal. This also means it is important the transformation is correct, as incorrect transformations might lead to bad feedback on a proposal.
\subsection{"Pipeline" Proposal}
The "Pipeline" proposal is one of the proposals presented in Section \ref{sec:proposals}. This proposal is applicable to call expressions, which are used all across JavaScript. This proposal is trying to solve readability when performing deeply nested function calls.
This proposal is applicable to call expressions, and is aimed at improving code readability when performing deeply nested function calls.
\begin{lstlisting}[language={JavaScript}, caption={Example of "Pipeline" proposal definition in \DSL}, label={def:pipeline}]
\begin{lstlisting}[language={JavaScript}, caption={Example of "Pipeline" proposal definition in \DSL{}.}, label={def:pipeline}]
proposal Pipeline {
case SingleArgument {
@ -591,14 +573,14 @@ proposal Pipeline {
}
\end{lstlisting}
In the Listing \ref{def:pipeline}, the first pair definition \texttt{SingleArgument} will apply to any \texttt{CallExpression} with a single argument. We do not expressively write a \texttt{CallExpression} inside a wildcard, as we have defined the structure of a \texttt{CallExpression}. The first wildcard \texttt{someFunctionIdent}, has the types of \texttt{Identifier}, to match against single identifiers, and \texttt{MemberExpression}, to match against functions who are members of objects, i.e. \texttt{console.log}. In the transformation template, we define the structure of a function call using the pipe operator, but the wildcards change order, so the argument passed as argument \texttt{someFunctionParam} is placed on the left side of the pipe operator, and the \texttt{CallExpression} is on the right, with the topic token as the argument. This case will produce a match against all function calls with a single argument, and transform them to use the pipe operator. The main difference of the second case \texttt{TwoArgument}, is it matches against functions with exactly two arguments, and uses the first argument as the left side of the pipe operator, while the second argument remains in the function call.
In the Listing \ref{def:pipeline}, the first \texttt{case} definition \texttt{SingleArgument} will apply to any \texttt{CallExpression} with a single argument. We do not write a \texttt{CallExpression} inside a wildcard, as we have defined the structure of a \texttt{CallExpression}. The first wildcard \texttt{someFunctionIdent} has the types of \texttt{Identifier} to match against single identifiers and \texttt{MemberExpression} to match against functions who are members of objects, i.e. \texttt{console.log}. In the transformation template, we define the structure of a function call using the pipe operator, but the wildcards change order, so the argument passed as argument \texttt{someFunctionParam} is placed on the left side of the pipe operator, and the \texttt{CallExpression} is on the right, with the topic token as the argument. This case will produce a match against all function calls with a single argument, and transform them to use the pipe operator. The main difference of the second case \texttt{TwoArgument} is that it matches against functions with exactly two arguments, and uses the first argument as the left side of the pipe operator, while the second argument remains in the function call.
\subsection{"Do Expressions" Proposal}
\subsection{``Do Expression'' Proposal}
The "Do Expressions" proposal~\cite{Proposal:DoProposal} can be specified in our DSL. Due to the nature of the proposal, it is not as applicable as the "Pipeline" proposal, as it does not re-define a style that is used quite as frequently as call expressions. This means the amount of transformed code snippets this specification in \DSL will be able to perform is expected to be lower. This is due to the "Do Expression" proposal introducing an entirely new way to write expression-oriented code in JavaScript. If the user running this tool has not used the current way of writing in an expression-oriented style in JavaScript, \DSL is limited in the amount of transformations it can perform. Nevertheless, if the user has been using an expression-oriented style, \DSL will transform parts of the code.
The ``Do Expression'' proposal~\cite{Proposal:DoProposal} focuses on bringing expression-oriented programming to JavaScript.
\begin{lstlisting}[language={JavaScript}, caption={Definition of Do Proposal in \DSL}, label={def:doExpression}]
\begin{lstlisting}[language={JavaScript}, caption={Definition of Do Proposal in \DSL{}.}, label={def:doExpression}]
proposal DoExpression {
case arrowFunction {
applicable to {
@ -634,16 +616,14 @@ proposal DoExpression {
}
\end{lstlisting}
In Listing \ref{def:doExpression}, the specification of "Do Expression" proposal in \DSL can be seen. It has two cases: the first case \texttt{arrowFunction}, applies to a code snippet using an arrow function~\cite[15.3]{ecma262} with a return value. The wildcards of this template are \texttt{statements}, which is a wildcard that matches against one or more statements that are not of type \texttt{ReturnStatement}, the reason we limit the one or more match is we cannot match the final statement of the block to this wildcard, as that has to be matched against the return statement in the template. The second wildcard \texttt{returnVal} matches against any expressions. The reason for extracting the expression from the \texttt{return} statement, is to use it in the implicit return of the \texttt{do} block. In the transformation template, we replace the arrow function with with a \texttt{do} expression, this do expression has to be defined inside parenthesis, as a free floating do expression is not allowed due to ambiguous parsing against a \texttt{do {} while()} statement. We and insert the statements matched against \texttt{statements} wildcard into the block of the \texttt{do} expression, and the final statement of the block is the expression matched against the \texttt{returnVal} wildcard. This will produce an equivalent transformation of an arrow function into a \texttt{do} expression. The second case \texttt{immediatelyInvokedAnonymousFunction} follows the same principle as the first case, but is applied to immediately invoked anonymous functions, and produces the exact same output after the transformation as the first case. This is because immediately invoked anonymous functions are equivalent to arrow functions.
In Listing \ref{def:doExpression}, the specification of ``Do Expression'' proposal in \DSL{} can be seen. It has two cases: the first case \texttt{arrowFunction} applies to code using an arrow function~\cite[15.3]{ecma262} with a return value. The wildcard \texttt{statements} matches against one or more statements that are not of type \texttt{ReturnStatement}. The reason we limit the wildcard is we cannot match the final statement of the block to this wildcard, as that has to be matched against the return statement in the template. The second wildcard \texttt{returnVal} matches against any expressions; the reason for extracting the expression from the \texttt{return} statement, is to use it in the implicit return of the \texttt{do} block. In the transformation template, we replace the arrow function with with a \texttt{do} expression. This expression has to be defined inside parenthesis, as a free floating \texttt{do} expression is not allowed due to ambiguous parsing against a \texttt{do while} statement. We insert the statements matched against \texttt{statements} wildcard into the block of the \texttt{do} expression, and the final statement of the block is the expression matched against the \texttt{returnVal} wildcard. This will transform an arrow function into a \texttt{do} expression.
The second case \texttt{immediatelyInvokedAnonymousFunction} follows the same principle as the first case, but is applied to immediately invoked anonymous functions, and produces the exact same output after the transformation as the first case. This is because immediately invoked anonymous functions are equivalent to arrow functions.
\subsection{"Await to Promise" imaginary proposal}
The imaginary proposal "Await to Promise" is created to transform code snippets from using \texttt{await}, to use a promise with equivalent functionality.
This proposal was created to evaluate the tool, as it is quite difficult to define applicable code in this current template form. This definition is designed to create matches in code using await, and highlight how await could be written using a promise in stead. This actually highlights some of the issues with the current design of \DSL that will be described in Future Work.
\begin{lstlisting}[language={JavaScript}, caption={Definition of Await to Promise evaluation proposal in \DSL}, label={def:awaitToPromise}]
\begin{lstlisting}[language={JavaScript}, caption={Definition of Await to Promise evaluation proposal in \DSL{}.}, label={def:awaitToPromise}]
proposal awaitToPomise{
case single{
applicable to {
@ -663,15 +643,15 @@ proposal awaitToPomise{
}
\end{lstlisting}
The specification of "Await to Promise" in \DSL is created to match asynchronous code inside a function. It is limited to match asynchronous functions containing a single await statement, and that await statement has to be stored in a \texttt{VariableDeclaration}. The second wildcard \texttt{statements}, is designed to match all statements following the \texttt{await} statement up to the return statement. This is done to move the statements into the callback function of \texttt{then()} in the transformation. We include \texttt{\!ReturnStatement} because we do not want to consume the return as it would then be removed from the functions scope and into the callback function of \texttt{then()}. We also have to avoid matching where there exists loop specific statements such as \texttt{ContinueStatement} or \texttt{BreakStatement}.
The specification of "Await to Promise" in \DSL{} is created to match asynchronous code inside a function. It is limited to match asynchronous functions containing a single await statement, and that await statement has to be stored in a \texttt{VariableDeclaration}. The second wildcard \texttt{statements} is designed to match all statements following the \texttt{await} statement up to the return statement. This is done to move the statements into the callback function of \texttt{then()} in the transformation. We include \texttt{\!ReturnStatement} because we do not want to consume the return as it would then be removed from the functions scope and into the callback function of \texttt{then()}. We also have to avoid matching where there exists loop specific statements such as \texttt{ContinueStatement} or \texttt{BreakStatement}.
The transformation definition has to use an async function in \texttt{.then()}, as there might be more await expressions contained within \texttt{statements}.
The transformation definition has to use an async arrow function as argument for \texttt{then}, as there might be more await expressions contained within \texttt{statements}.
\section{\DSLSH}
\section{\DSL{}SH}
In this thesis, we also created an alternative way of defining proposals and their respective transformations, this is done using JavaScript as it's own meta language for the definitions. The reason for creating a way of defining proposals using JavaScript is, it allows us to limit the amount of dependencies of the tool, since we no longer rely on \DSL, and it allows for more exploration in the future work of this project.
In this thesis, we also created an alternative way of defining proposals and their respective transformations, this is done using JavaScript as it's own meta language for the definitions. The reason for creating a way of defining proposals using JavaScript is, it allows us to limit the amount of dependencies of the tool, since we no longer rely on \DSL{}, and it allows for more exploration in the future work of this project.
\DSLSH is less of an actual language, and more of a program API at the moment, it allows for defining proposals purely in JavaScript objects, which is meant to allow a more modular way of using this idea. In \DSLSH you define a \textit{prelude}, which is just a list of variable declarations that contain the type expression as a string for that given wildcard. This means we do not need to perform wildcard extraction when wanting to parse the templates used for matching and transformation.
\DSL{}SH is less of an actual language, and more of a program API at the moment, it allows for defining proposals purely in JavaScript objects, which is meant to allow a more modular way of using this idea. In \DSL{}SH you define a \texttt{prelude}, which is a list of variable declarations that contain the type expression as a string for that given wildcard. This means we do not need to perform wildcard extraction when wanting to parse the templates used for matching and transformation.
\noindent\begin{minipage}{.45\textwidth}
\begin{lstlisting}

44
chapter/ch4.tex
View file

@ -1,21 +1,21 @@
\chapter{Implementation}
In this chapter, the implementation of the tool utilizing the \DSL and \DSLSH will be presented. It will describe the overall architecture of the tool, the flow of data throughout, and how the different stages of transforming user code are completed.
In this chapter, the implementation of the tool utilizing the \DSL{} and \DSL{}SH will be presented. It will describe the overall architecture of the tool, the flow of data throughout, and how the different stages of transforming user code are completed.
\section{Architecture of the solution}
The architecture of the solution described in this thesis is illustrated in \figFull[fig:architecture]
In this tool, there exists two multiple ways to define a proposal, and each provide the same functionality, they only differ in syntax and writing-method. One can either write the definition in \DSL, or one can use the program API with \DSLSH, which is more friendly for programs to interact with.
In this tool, there exists two multiple ways to define a proposal, and each provide the same functionality, they only differ in syntax and writing-method. One can either write the definition in \DSL{}, or one can use the program API with \DSL{}SH, which is more friendly for programs to interact with.
In the architecture diagram of Figure~\ref{fig:architecture}, ellipse nodes show data passed into the program sections, and rectangular nodes is a specific section of the program. The architecture is split into seven levels, where each level is a step of the program. The initial step is the proposal definition, the definition can have two different forms, either it is \DSL code, or it can be a JavaScript object using the self hosted in \DSLSH. If we use \DSL, the second step is parsing it using Langium~\cite{Langium}, this parses the raw source code into an AST. If \DSLSH is used, we have to build the prelude, so we have to extract the wildcard definitions from JavaScript source code. At this point the two paths meet at the second step, which is wildcard extraction. At this step, if \DSL was used, the wildcards are extracted from the template. If \DSLSH was used extraction is not needed. In both cases we parse the wildcard type expressions into an AST. The third step is parsing the raw source code with Babel~\cite{Babel}. It is also at this point we parse the users source code into an AST. The fourth step is translating the Babel AST into our own custom tree structure for simpler traversal. Once all data is prepared, the fifth step is matching the user's AST against the \texttt{applicable to} template AST. Once all matches have been found, we transplant the wildcard matches into the \texttt{transform to} template, and insert it back into the users code in step six. We have at this point transformed the users code, the final step seven is generating it back into source code.
In the architecture diagram of Figure~\ref{fig:architecture}, ellipse nodes show data passed into the program sections, and rectangular nodes is a specific section of the program. The architecture is split into seven levels, where each level is a step of the program. The initial step is the proposal definition, the definition can have two different forms, either it is \DSL{} code, or it can be a JavaScript object using the self hosted in \DSL{}SH. If we use \DSL{}, the second step is parsing it using Langium~\cite{Langium}, this parses the raw source code into an AST. If \DSL{}SH is used, we have to build the prelude, so we have to extract the wildcard definitions from JavaScript source code. At this point the two paths meet at the second step, which is wildcard extraction. At this step, if \DSL{} was used, the wildcards are extracted from the template. If \DSL{}SH was used extraction is not needed. In both cases we parse the wildcard type expressions into an AST. The third step is parsing the raw source code with Babel~\cite{Babel}. It is also at this point we parse the users source code into an AST. The fourth step is translating the Babel AST into our own custom tree structure for simpler traversal. Once all data is prepared, the fifth step is matching the user's AST against the \texttt{applicable to} template AST. Once all matches have been found, we transplant the wildcard matches into the \texttt{transform to} template, and insert it back into the users code in step six. We have at this point transformed the users code, the final step seven is generating it back into source code.
\iffalse
\begin{description}
\item[\DSL Code] is the raw text definition of proposals
\item[Self-Hosted Object] is the self-hosted version in \DSLSH format
\item[1a. Langium Parser] takes raw \DSL source code, and parses it into a DSL
\item[2. Wildcard parsing] extracts the wildcards from the raw template definition in \DSL, and parse
\item[\DSL{} Code] is the raw text definition of proposals
\item[Self-Hosted Object] is the self-hosted version in \DSL{}SH format
\item[1a. Langium Parser] takes raw \DSL{} source code, and parses it into a DSL
\item[2. Wildcard parsing] extracts the wildcards from the raw template definition in \DSL{}, and parse
\item[1b. Prelude-builder] translates JavaScript prelude into array of wildcard strings
\item[3. Babel] parses the templates and the users source code into an AST
\item[4. Custom Tree Builder] translates the Babel AST structure into our tree structure
@ -62,15 +62,15 @@ In the architecture diagram of Figure~\ref{fig:architecture}, ellipse nodes show
\end{figure}
\section{Parsing \DSL using Langium}
\section{Parsing \DSL{} using Langium}
In this section, we describe the implementation of the parser for \DSL. We outline the tool Langium, used as a parser-generator to create the AST used by the tool later to perform the transformations.
In this section, we describe the implementation of the parser for \DSL{}. We outline the tool Langium, used as a parser-generator to create the AST used by the tool later to perform the transformations.
\subsection{Langium}
Langium~\cite{Langium} is a language workbench~\cite{LanguageWorkbench} primarily used to create parsers and Integrated Development Environments for domain specific languages. These kinds of parsers produce Abstract Syntax Trees that are later used to create interpreters or other tooling. In this project, we use Langium to generate an AST definition in the form of TypeScript objects. These objects and their structure are used as definitions for the tool to do matching and transformation of user code.
To generate this parser, Langium requires a definition of a grammar. A grammar is a specification that describes syntax a valid programs. The \DSL grammar describes the structure of \DSL, such as \texttt{proposals}, \texttt{cases}, \texttt{applicable to} blocks, and \texttt{transform to} blocks. A grammar in Langium starts by describing the \texttt{Model}. The model is the top entry of the grammar; this is where the description of all valid top level statements.
To generate this parser, Langium requires a definition of a grammar. A grammar is a specification that describes syntax a valid programs. The \DSL{} grammar describes the structure of \DSL{}, such as \texttt{proposals}, \texttt{cases}, \texttt{applicable to} blocks, and \texttt{transform to} blocks. A grammar in Langium starts by describing the \texttt{Model}. The model is the top entry of the grammar; this is where the description of all valid top level statements.
Contained within the \texttt{Model} rule, is one or more proposals. Each proposal is defined with the rule \texttt{Proposals}, and starts with the keyword \texttt{proposal}, followed by a name, and a code block. This rule is designed to contain every definition of a transformation related to a specific proposal. To hold every transformation definition, a proposal definition contains one or more cases.
@ -82,7 +82,7 @@ The rule \texttt{TransformTo}, is created to contain a single template used for
To define exactly what characters/tokens are legal in a specific definition, Langium uses terminals defined using regular expressions, these allow for a very specific character-set to be legal in specific keys of the AST generated by the parser generated by Langium. In the definition of \texttt{Proposal} and \texttt{Pair} the terminal \texttt{ID} is used; this terminal is limited to allow for only words and can only begin with a character of the alphabet or an underscore. In \texttt{Section} the terminal \texttt{STRING} is used, this terminal is meant to allow any valid JavaScript code and the custom DSL language described in \ref{sec:DSL_DEF}. Both these terminals defined allows Langium to determine exactly what characters are legal in each location.
\begin{lstlisting}[caption={Definition of \DSL in Langium.}, label={def:JSTQLLangium}]
\begin{lstlisting}[caption={Definition of \DSL{} in Langium.}, label={def:JSTQLLangium}]
grammar Jstql
entry Model:
@ -112,14 +112,14 @@ terminal ID: /[_a-zA-Z][\w_]*/;
terminal STRING: /"[^"]*"|'[^']*'/;
\end{lstlisting}
With \DSL, we are not implementing a programming language meant to be executed. We are using Langium to generate an AST that will be used as a markup language, similar to YAML, JSON or TOML~\cite{TOML}. The main reason for using Langium in such an unconventional way is Langium provides support for Visual Studio Code integration, and it solves the issue of parsing the definition of each proposal manually. However, with this grammar we cannot actually verify the wildcards placed in \texttt{apl\_to\_code} and \texttt{transform\_to\_code} are correctly written. To do this, we have implemented several validation rules.
With \DSL{}, we are not implementing a programming language meant to be executed. We are using Langium to generate an AST that will be used as a markup language, similar to YAML, JSON or TOML~\cite{TOML}. The main reason for using Langium in such an unconventional way is Langium provides support for Visual Studio Code integration, and it solves the issue of parsing the definition of each proposal manually. However, with this grammar we cannot actually verify the wildcards placed in \texttt{apl\_to\_code} and \texttt{transform\_to\_code} are correctly written. To do this, we have implemented several validation rules.
\subsection*{Langium Validator}
A Langium validator allows for further checks DSL code, a validator allows for the implementation of specific checks on specific parts of the grammar.
\DSL does not allow empty typed wildcard definitions in \texttt{applicable to} blocks, this means we cannot define a wildcard that allows any AST type to match against it. This is not defined within the grammar, as inside the grammar the code is defined as a \texttt{STRING} terminal. This means further checks have to be implemented using code. To do this we have a specific \texttt{Validator} implemented on the \texttt{Case} definition of the grammar. This means every time anything contained within a \texttt{Case} is updated, the language server created with Langium will perform the validation step and report any errors.
\DSL{} does not allow empty typed wildcard definitions in \texttt{applicable to} blocks, this means we cannot define a wildcard that allows any AST type to match against it. This is not defined within the grammar, as inside the grammar the code is defined as a \texttt{STRING} terminal. This means further checks have to be implemented using code. To do this we have a specific \texttt{Validator} implemented on the \texttt{Case} definition of the grammar. This means every time anything contained within a \texttt{Case} is updated, the language server created with Langium will perform the validation step and report any errors.
The validator uses \texttt{Case} as its entry point, as it allows for a checking of wildcards in both \texttt{applicable to} and \texttt{transform to}, allowing for a check for whether a wildcard identifier used in \texttt{transform to} exists in the definition of \texttt{applicable to}.
@ -147,9 +147,9 @@ export class JstqlValidator {
\subsection*{Using Langium as a parser}
Langium is designed to automatically generate extensive tool support for the language specified using its grammar. However, in our case we have to parse the \DSL definition using Langium, and then extract the Abstract syntax tree generated in order to use the information it contains.
Langium is designed to automatically generate extensive tool support for the language specified using its grammar. However, in our case we have to parse the \DSL{} definition using Langium, and then extract the Abstract syntax tree generated in order to use the information it contains.
To use the parser generated by Langium, we created a custom function \texttt{parseDSLtoAST}, which takes a string as an input (the raw \DSL code), and outputs the pure AST using the format described in the grammar, see Listing \ref{sec:DSL_DEF}. This function is exposed as a custom API for our tool to interface with. This also means our tool is dependent on the implementation of the Langium parser to function with \DSL. The implementation of \DSLSH is entirely independent.
To use the parser generated by Langium, we created a custom function \texttt{parseDSLtoAST}, which takes a string as an input (the raw \DSL{} code), and outputs the pure AST using the format described in the grammar, see Listing \ref{sec:DSL_DEF}. This function is exposed as a custom API for our tool to interface with. This also means our tool is dependent on the implementation of the Langium parser to function with \DSL{}. The implementation of \DSL{}SH is entirely independent.
When interfacing with the Langium parser to get the Langium generated AST, the exposed API function is imported into the tool, when this API is executed, the output is on the form of the Langium \texttt{Model}, which follows the same form as the grammar. This is then transformed into an internal object structure used by the tool, this structure is called \texttt{TransformRecipe}, and is then passed in to perform the actual transformation.
@ -159,9 +159,9 @@ To refer to internal DSL variables defined in \texttt{applicable to} and \texttt
\subsection*{Why not use Langium for wildcard parsing?}
Langium has support for creating a generator to output an artifact, which is some transformation applied to the AST built by the Langium parser. This suits the needs of \DSL quite well and could be used to extract the wildcards and parse the type expressions. This is the way the developers of Langium want this kind of functionality to be implemented, however, the implementation would still be mostly the same, as the parsing of the wildcards still has to be done "manually" with a custom parser. Therefore, we decided for this project to keep the parsing of the wildcards separate. If we were to use Langium generators to parse the wildcards, it would make \DSLSH dependent on Langium. This is not preferred as that would mean both ways of defining a proposal are reliant of Langium. The reason for using our own extractor is to allow for an independent way to define transformations using our tool.
Langium has support for creating a generator to output an artifact, which is some transformation applied to the AST built by the Langium parser. This suits the needs of \DSL{} quite well and could be used to extract the wildcards and parse the type expressions. This is the way the developers of Langium want this kind of functionality to be implemented, however, the implementation would still be mostly the same, as the parsing of the wildcards still has to be done "manually" with a custom parser. Therefore, we decided for this project to keep the parsing of the wildcards separate. If we were to use Langium generators to parse the wildcards, it would make \DSL{}SH dependent on Langium. This is not preferred as that would mean both ways of defining a proposal are reliant of Langium. The reason for using our own extractor is to allow for an independent way to define transformations using our tool.
\subsection*{Extracting wildcards from \DSL}
\subsection*{Extracting wildcards from \DSL{}}
To allow the use of Babel~\cite{Babel}, the wildcards present in the \texttt{applicable to} blocks and \texttt{transform to} blocks have to be parsed and replaced with some valid JavaScript. This is done by using a pre-parser that extracts the information from the wildcards and inserts an \texttt{Identifier} in their place.
@ -213,7 +213,7 @@ export function parseInternal(code: string): InternalParseResult {
\paragraph*{Parsing wildcard}
Once a wildcard has been extracted from definitions inside \DSL, they have to be parsed into a simple AST to be used when matching against the wildcard. This is accomplished by using a simple tokenizer and a recursive descent parser~\cite{RecursiveDescent}.
Once a wildcard has been extracted from definitions inside \DSL{}, they have to be parsed into a simple AST to be used when matching against the wildcard. This is accomplished by using a simple tokenizer and a recursive descent parser~\cite{RecursiveDescent}.
Our tokenizer takes the raw stream of input characters extracted from the wildcard block within the template, and determines which part is what token. Due to the very simple nature of the type expressions, no ambiguity is present with the tokens, so determining what token is meant to come at what time is quite trivial. We use a switch case on the current token, if the token is of length one we accept it and move on to the next character. If the next character is an unexpected one it will produce an error. The tokenizer also groups tokens with a \textit{token type}, this allows for an simpler parsing of the tokens later.
@ -251,13 +251,13 @@ Our recursive descent parser produces an AST, which is later used to determine w
\paragraph*{Extracting wildcards from \DSLSH}
\paragraph*{Extracting wildcards from \DSL{}SH}
The self-hosted version \DSLSH also requires some form of pre-parsing to prepare the internal DSL environment. This is relatively minor and only parsing directly with no insertion compared to \DSL.
The self-hosted version \DSL{}SH also requires some form of pre-parsing to prepare the internal DSL environment. This is relatively minor and only parsing directly with no insertion compared to \DSL{}.
To use JavaScript as the meta language, we define a \texttt{prelude} on the object used to define the transformation case. This prelude is required to consist of several \texttt{Variable declaration} statements, where the variable names are used as the internal DSL variables and right side expressions are strings that contain the type expression used to determine a match for that specific wildcard.
We use Babel to generate the AST of the \texttt{prelude} definition, this allows us to get a JavaScript object structure. Since the structure is very strictly defined, we can expect every \texttt{stmt} of \texttt{stmts} to be a variable declaration, otherwise throw an error for invalid prelude. Then the string value of each of the variable declarations is passed to the same parser used for \DSL wildcards.
We use Babel to generate the AST of the \texttt{prelude} definition, this allows us to get a JavaScript object structure. Since the structure is very strictly defined, we can expect every \texttt{stmt} of \texttt{stmts} to be a variable declaration, otherwise throw an error for invalid prelude. Then the string value of each of the variable declarations is passed to the same parser used for \DSL{} wildcards.
The reason this is preferred is it allows us to avoid having to extract the wildcards and inserting an Identifier.
@ -538,7 +538,7 @@ export interface Match {
To perform the transformation and replacement on each of the matches, we take the resulting list of matches, the template from the \texttt{transform to} section of the current case of the proposal, and the Babel AST~\cite{BabelAST} version of original code. All the transformations are then applied to the code and we use \texttt{@babel/generate}~\cite{BabelGenerate} to generate JavaScript code from the transformed AST.
An important discovery is to ensure we transform the leafs of the AST first, this is because if the transformation was applied from top to bottom, it might remove transformations done using a previous match. This means if we transform from top to bottom on the tree, we might end up with \texttt{a(b) |> c(\%)} in stead of \texttt{b |> a(\%) |> c(\%)} in the case of the pipeline proposal. This is quite easily solved in our case, as the matcher looks for matches from the top of the tree to the bottom of the tree, the matches it discovers are always in that order. Therefore when transforming, all that has to be done is reverse the list of matches, to get the ones closest to the leaves of the tree first.
An important discovery is to ensure we transform the leafs of the AST first, this is because if the transformation was applied from top to bottom, it might remove transformations done using a previous match. This means if we transform from top to bottom on the tree, we might end up with \texttt{a(b) |> c(\%)} in stead of \texttt{b |> a(\%) |> c(\%)} in the case of the ``Pipeline'' proposal. This is quite easily solved in our case, as the matcher looks for matches from the top of the tree to the bottom of the tree, the matches it discovers are always in that order. Therefore when transforming, all that has to be done is reverse the list of matches, to get the ones closest to the leaves of the tree first.
\subsubsection{Building the transformation}

52
chapter/ch5.tex
View file

@ -1,22 +1,24 @@
\chapter{Evaluation}
In this chapter we will discuss how we evaluated \DSL and its related tools. This chapter will include some testing of the tool on demo code snippets, as well as running each of the proposals discussed in this thesis on some large scale JavaScript projects.
In this chapter we will present the results of running each of the proposals discussed in this thesis on large scale JavaScript projects.
\section{Real Life source code}
To perform actual large scale trial of this program, we have collected some github projects containing many or large JavaScript files. Every JS file within the project is then passed through the entire tool, and we will evaluate it based upon the amount of matches discovered, as well as manual checking that the transformation resulted in correct code on the matches.
To evaluate this tool on real life JavaScript code, we have collected some JavaScript projects from Github containing many or large JavaScript files. Every JavaScript file within the project is transformed with our tool, and we will evaluate it based upon the amount of matches discovered, as well as manual checking that the transformation resulted in correct output code.
Each case study was evaluated by running this tool on every .js file in the repository, then collecting the number of matches found in total and how many files were successfully searched. Evaluating if the transformation was correct is done by manually sampling output files, and verifying that it passes through Babel Generate~\cite{BabelGenerate} without error.
"Pipeline"~\cite{Pipeline} is very applicable to most files, as the concept it touches (fucntion calls) is widely used all across JavaScript. This is by far the best result, and it found matches in almost all files that Babel~\cite{BabelParser} managed to parse.
``Pipeline''~\cite{Pipeline} is applicable to most files, as the concept it touches (function calls) is widely used when writing JavaScript code. Our tool found matches in almost all files that Babel~\cite{BabelParser} managed to parse, and the transformed files are correct.
The Do proposal~\cite{Proposal:DoProposal} is expected to not find as many matches, as code that has not been written in expression-oriented programming style will not produce many matches. However, this also highlights how impactful this proposal is to previously written code compared to "Pipeline".
The ``Do Expression'' proposal~\cite{Proposal:DoProposal} is not as applicable as the "Pipeline" proposal, as it does not re-define a concept that is used quite as frequently as call expressions. This means the amount of transformed code this specification in \DSL{} will be able to perform is expected and proven to be lower. This is due to the ``Do Expression'' proposal introducing an entirely new way to write expression-oriented code in JavaScript. If the code has not used the current way of writing in an expression-oriented style in JavaScript, \DSL{} is limited in the amount of transformations it can perform. Nevertheless, matches are discovered and transformed correctly.
Await to promise also has an expected number of matches, but this evaluation proposal is not meant to be real life representative. As it is limited to functions containing only a single await statement and that statement has to be a \texttt{VariableDeclaration}.
All these proposals have an impact on JavaScript, as they change how certain functionality is achieved. However, the size of the impact is very different. "Pipeline" is applicable to all function calls, this is why it produces so many more transformations than the others. From this we can deduce the impact each proposal might have on the current version of JavaScript. "Do Expression" is an interesting case, as it does produce a number of matches, and is quite applicable when ran on a large codebase. ... More
The imaginary ``Await to promise'' proposal also has an expected number of matches, but this evaluation proposal is not meant to be real life representative, and was only created to evaluate the expressiveness of our tool.
\paragraph*{Next.js}~\cite{NEXT.JS} is one of the largest projects on the web. It is used with React~\cite{React} to enable feature such as server-sire rendering and static site generation.
\textbf{Next.js}~\cite{NEXT.JS} is one of the largest projects on the web. It is used with React~\cite{React} to enable feature such as server-sire rendering and static site generation.
\begin{table}[H]
\begin{center}
@ -24,9 +26,9 @@ All these proposals have an impact on JavaScript, as they change how certain fun
\hline
Proposal & Matches found & Files with matches & Files processed\\
\hline \hline
"Pipeline" & 242079 & 1912 & 3340 \\
``Pipeline'' & 242079 & 1912 & 3340 \\
\hline
"Do" expression & 480 & 111 & 3340 \\
``Do Expression'' & 480 & 111 & 3340 \\
\hline
Await to Promise & 143 & 75 & 3340 \\
\hline
@ -72,7 +74,7 @@ const content = await (styledJsxPath |> join(%, file) |> fs.readFile(%, 'utf8'))
await (typesDir |> join(%, file) |> fs.writeFile(%, content));
}
\end{lstlisting}
\caption*{"Pipeline" transformation, taken from \texttt{next.js/packages/next/taskfile.js}}
\caption*{``Pipeline'' transformation, taken from \texttt{next.js/packages/next/taskfile.js}}
\end{figure}
\begin{figure}[H]
@ -88,12 +90,12 @@ const html = await browser.eval('document.documentElement.innerHTML');
html.match(/iframe/) ? 'fail' : 'success'
}, /success/);
\end{lstlisting}
\caption*{"Do expression" transformation, taken from \texttt{next.js/test/integration/typescript-hmr/index.test.js}}
\caption*{``Do Expression'' transformation, taken from \texttt{next.js/test/integration/typescript-hmr/index.test.js}}
\end{figure}
\paragraph*{Three.js}~\cite{ThreeJS} is a library for 3D rendering in JavaScript. It is written purely in JavaScript and uses GPU for 3D calculations. It being a popular JavaScript library, and being written in mostly pure JavaScript makes it a good case study for our tool. It currently sits at over 1 million downloads weekly.
\textbf{Three.js}~\cite{ThreeJS} is a library for 3D rendering in JavaScript. It is written purely in JavaScript and uses GPU for 3D calculations. It being a popular JavaScript library, and being written in mostly pure JavaScript makes it a good case study for our tool. It currently sits at over 1 million downloads weekly.
\begin{table}[H]
\begin{center}
@ -103,7 +105,7 @@ html.match(/iframe/) ? 'fail' : 'success'
\hline \hline
Pipeline & 84803 & 1117 & 1384 \\
\hline
"Do" expression & 277 & 55 & 1384 \\
``Do Expression'' & 277 & 55 & 1384 \\
\hline
Await to Promise & 13 & 7 & 1384 \\
\hline
@ -125,7 +127,7 @@ frameTime
\caption*{Transformation taken from \texttt{three.js/src/animation/AnimationClip.js}}
\end{figure}
\paragraph*{React}~\cite{React} is a graphical user interface library for JavaScript, it facilitates the creation of user interfaces for both web and native platforms. React is based upon splitting a user interface into components for simple development. It is currently one of the most popular libraries for creating web apps and has over 223000 stars on Github.
\textbf{React}~\cite{React} is a graphical user interface library for JavaScript, it facilitates the creation of user interfaces for both web and native platforms. React is based upon splitting a user interface into components for simple development. It is currently one of the most popular libraries for creating web apps and has over 223000 stars on Github.
\begin{table}[H]
\begin{center}
@ -133,9 +135,9 @@ frameTime
\hline
Proposal & Matches found & Files with matches & Files searched\\
\hline \hline
"Pipeline" & 16353 & 1266 & 2051 \\
``Pipeline'' & 16353 & 1266 & 2051 \\
\hline
"Do" expression & 79 & 60 & 2051 \\
``Do Expression'' & 79 & 60 & 2051 \\
\hline
Await to Promise & 30 & 13 & 2051 \\
\hline
@ -156,12 +158,12 @@ const logger = {
storagePath: __dirname |> join(%, '.progress-estimator')
} |> createLogger(%);
\end{lstlisting}
\caption*{"Pipeline" transformation, taken from \texttt{react/scripts/devtools/utils.js}}
\caption*{``Pipeline'' transformation, taken from \texttt{react/scripts/devtools/utils.js}}
\end{figure}
\paragraph*{Bootstrap}~\cite{Bootstrap} is a front-end framework used for creating responsive and mobile-first websites, it comes with a variety of built-in components, as well as a built in styling. This styling is also customizable using CSS. This library is a good evaluation point for this thesis as it is written in pure JavaScript and is used by millions of developers.
\textbf{Bootstrap}~\cite{Bootstrap} is a front-end framework used for creating responsive and mobile-first websites, it comes with a variety of built-in components, as well as a built in styling. This styling is also customizable using CSS. This library is a good evaluation point for this thesis as it is written in pure JavaScript and is used by millions of developers.
\begin{table}[H]
\begin{center}
@ -169,9 +171,9 @@ const logger = {
\hline
Proposal & Matches found & Files with matches & Files searched\\
\hline \hline
""Pipeline" & 13794 & 109 & 115 \\
"``Pipeline'' & 13794 & 109 & 115 \\
\hline
"Do" expression & 13 & 8 & 115 \\
``Do Expression'' & 13 & 8 & 115 \\
\hline
Await to Promise & 0 & 0 & 115 \\
\hline
@ -194,12 +196,12 @@ if (content |> isElement(%)) {
return;
}
\end{lstlisting}
\caption*{"Pipeline" transformation, taken from \texttt{bootstrap/js/src/util/template-factory.js}}
\caption*{``Pipeline'' transformation, taken from \texttt{bootstrap/js/src/util/template-factory.js}}
\end{figure}
\paragraph*{Atom}~\cite{Atom} is a text editor made in JavaScript using the Electron framework. It was created to give a very minimal and modular text editor. It was bought by Microsoft, and later discontinued in favor for Visual Studio Code.
\textbf{Atom}~\cite{Atom} is a text editor made in JavaScript using the Electron framework. It was created to give a very minimal and modular text editor. It was bought by Microsoft, and later discontinued in favor for Visual Studio Code.
\begin{table}[H]
\begin{center}
@ -207,9 +209,9 @@ if (content |> isElement(%)) {
\hline
Proposal & Matches found & Files with matches & Files searched\\
\hline \hline
"Pipeline" & 40606 & 361 & 401 \\
``Pipeline'' & 40606 & 361 & 401 \\
\hline
"Do" expression & 46 & 26 & 401 \\
``Do Expression'' & 46 & 26 & 401 \\
\hline
Await to Promise & 12 & 7 & 401 \\
\hline
@ -233,6 +235,6 @@ if (repo && repo.onDidDestroy) {
(() => pathForDirectory |> this.repositoryPromisesByPath.delete(%)) |> repo.onDidDestroy(%);
}
\end{lstlisting}
\caption*{"Pipeline" transformation, taken from \texttt{atom/src/project.js}}
\caption*{``Pipeline'' transformation, taken from \texttt{atom/src/project.js}}
\end{figure}

6
chapter/future_work.tex
View file

@ -6,13 +6,13 @@ In this thesis, we have developed a way to define transformations of JavaScript
\section{Future Work}
\textbf{Provide access and gather feedback} This project is build upon creating a tool for users of EcmaScript to see new proposals within their own codebase. The idea behind this is to use the users familiarity to showcase new syntactic proposals, and get valuable feedback to the committee developing the ECMA-262 standard. This means making the definitions of a proposal in \DSL and this tool available to end-users to execute using their own code. This can come in multiple forms, we suggest some ideas, such as a playground on the web, an extension for Visual Studio Code, or to be used in github pull requests.
\textbf{Provide access and gather feedback} This project is build upon creating a tool for users of EcmaScript to see new proposals within their own codebase. The idea behind this is to use the users familiarity to showcase new syntactic proposals, and get valuable feedback to the committee developing the ECMA-262 standard. This means making the definitions of a proposal in \DSL{} and this tool available to end-users to execute using their own code. This can come in multiple forms, we suggest some ideas, such as a playground on the web, an extension for Visual Studio Code, or to be used in github pull requests.
\textbf{Supporting other languages} The idea of showcasing changes to a programming language by transforming user code is not only limited to EcmaScript, and could be applied to many other programming languages using a similar development method to EcmaScript. The developers of a language could write definitions of new changes for their respective language, and use a similar tool to the one discussed in this thesis to showcase possible new changes.
\textbf{Parameterized specifications} The current form of \DSL supports writing each template as its own respective case, but multiple templates might be very similar and could be written using generics that are shared between case definitions. Introducing this might give a simpler way of writing more complex definitions of a proposal transformation by re-using generic type parameters for the wildcards used in the transformations.
\textbf{Parameterized specifications} The current form of \DSL{} supports writing each template as its own respective case, but multiple templates might be very similar and could be written using generics that are shared between case definitions. Introducing this might give a simpler way of writing more complex definitions of a proposal transformation by re-using generic type parameters for the wildcards used in the transformations.
\textbf{Fully self-hosting \DSLSH} The current version of \DSLSH relies on this tools parser to generate the AST for the type expressions used for matching by wildcards. This might make this tool more difficult to adopt for the committee. Therefore adding functionality for writing these type expressions purely in JavaScript and allowing for the use of JavaScript as its own meta language is an interesting avenue to explore.
\textbf{Fully self-hosting \DSL{}SH} The current version of \DSL{}SH relies on this tools parser to generate the AST for the type expressions used for matching by wildcards. This might make this tool more difficult to adopt for the committee. Therefore adding functionality for writing these type expressions purely in JavaScript and allowing for the use of JavaScript as its own meta language is an interesting avenue to explore.
\textbf{Support for custom proposal syntax} Currently this tool relies heavily on that a proposal is supported by Babel~\cite{Babel}. This makes the tool quite limited in what proposals could be defined and transformed due to relying on Babel for parsing the templates and generating the output code. Introducing some way of defining new syntax for a proposal in the proposal definition, and allowing for parsing JavaScript containing that specific new syntax would limit the reliance on Babel, and allow for defining proposals earlier in the development process. This can possibly be done by implementing a custom parser inside this tool that allows defining custom syntax for specific new proposals.

2
chapter/introduction.tex
View file

@ -6,7 +6,7 @@ The idea behind using the user's own code, is to lower the barrier to understand
This thesis discusses a way of defining transformations of code specifically for syntactic proposals. These transformations are used to showcase a syntactic proposal to users by transforming code the user's are already familiar with. The theory is this will help user's understand proposals faster and easier, as they are already familiar with the code the proposal is being presented in. The two proposals discussed in this thesis to evaluate the tool are "Do Expression"~\cite{Proposal:DoProposal} and "Pipeline"~\cite{Pipeline}.
The language defined in this thesis is titles \DSL, and allows for structured template queries to be written, these templates are used to search for code that is applicable to a proposal. A code snippet being applicable to a proposal, means it could have been written using that proposal. These queries come with a definition of a transformation, which will use the matched code snippet in the user's code, and transform it to use the features of the proposal. This newly transformed code can then be presented to the user along with the original unchanged code. This allows a user to see an example of the proposal being applied, and they can then give feedback on the proposal based on that.
The language defined in this thesis is titles \DSL{}, and allows for structured template queries to be written, these templates are used to search for code that is applicable to a proposal. A code snippet being applicable to a proposal, means it could have been written using that proposal. These queries come with a definition of a transformation, which will use the matched code snippet in the user's code, and transform it to use the features of the proposal. This newly transformed code can then be presented to the user along with the original unchanged code. This allows a user to see an example of the proposal being applied, and they can then give feedback on the proposal based on that.
Every template uses a notion of wildcards, which are sections of the template that is not defined using JavaScript, but a custom block containing an identifier, and a AST node type expression.

12
chapter/related_work.tex
View file

@ -14,7 +14,7 @@ Aspect oriented programming is similar to this project in that to define where \
\section{Other source code query languages}
To allow for simple analysis and refactoring of code, there exists many query languages designed to query source code. These languages use several methods to allow for querying code based on specific paradigms such as logical queries, declarative queries, or SQL-like queries. All provide similar functionality of being able to query code. In this section we will look some of these languages for querying source code, and how they relate to \DSL developed in this thesis.
To allow for simple analysis and refactoring of code, there exists many query languages designed to query source code. These languages use several methods to allow for querying code based on specific paradigms such as logical queries, declarative queries, or SQL-like queries. All provide similar functionality of being able to query code. In this section we will look some of these languages for querying source code, and how they relate to \DSL{} developed in this thesis.
\subsection{CodeQL}
@ -29,7 +29,7 @@ where c.declaresMethod("equals") and
select c.getPackage(), c
\end{lstlisting}
The syntax of writing queries in CodeQL is not similar to \DSL, as it is SQL-like, and not declarative patterns, which makes the writing experience of the two languages very different. Writing CodeQL queries are similar to querying a database, while queries written in \DSL are similar to defining an example of the structure you wish to search for.
The syntax of writing queries in CodeQL is not similar to \DSL{}, as it is SQL-like, and not declarative patterns, which makes the writing experience of the two languages very different. Writing CodeQL queries are similar to querying a database, while queries written in \DSL{} are similar to defining an example of the structure you wish to search for.
\subsection{PMD XPath}
@ -51,7 +51,7 @@ There are two queries with PMD XPath defined in the example below~\cite{PMDXPath
//VariableId[@Name = "bill" and ../../Type[@TypeImage = "short"]]
\end{lstlisting}
Comparing this tool to \DSL, we can see it is good at querying code based on structure, which our tool also excels at. The main difference is the manor of which each tool does this, \DSL uses JavaScript templates to perform the query, making writing queries simple for users as they are based in JavaScript. PMD XPath uses XPath to perform define structural queries that is quite verbose, and requires extended knowledge of the AST that is currently being queried.
Comparing this tool to \DSL{}, we can see it is good at querying code based on structure, which our tool also excels at. The main difference is the manor of which each tool does this, \DSL{} uses JavaScript templates to perform the query, making writing queries simple for users as they are based in JavaScript. PMD XPath uses XPath to perform define structural queries that is quite verbose, and requires extended knowledge of the AST that is currently being queried.
\subsection{XSL Transformations}
@ -59,13 +59,13 @@ XSLT~\cite{XSLT} is a language created to perform transformations of XML documen
XSLT is part of Extensible Stylesheets Language family of programs. The XSL language is expressed in the form of a stylesheet~\cite[1.1]{XSLT}, whose syntax is defined in XML. This language uses a template based approach to define matches on specific patterns in the source to find sections to transform. These transformations are defined by a transformation declaration that describes how the output of the match should look.
This language defines matching and transformation very similarly to \DSL, and uses the same technique, where the transformation declaration describes how the output should look, and not exactly how the transformation is performed.
This language defines matching and transformation very similarly to \DSL{}, and uses the same technique, where the transformation declaration describes how the output should look, and not exactly how the transformation is performed.
\subsection{Jackpot}
Jackpot~\cite{Jackpot} is a query language created for the Apache Netbeans platform~\cite{ApacheNetBeans}, it has since been mostly renamed to Java Declarative Hints Language, we will continue to refer to it as Jackpot in this section. The language uses declarative patterns to define source code queries, these queries are used in conjunction with multiple rewrite definitions. This is used in the Apache Netbeans suite of tools to allow for declarative refactoring of code.
This is quite similar to the form of \DSL, as both language define som query by using similar structure, in Jackpot you define a \textit{pattern}, then every match of that pattern can be re-written to a \textit{fix-pattern}, each fix-pattern can have a condition attached to it. This is quite similar to the \textit{applicable to} and \textit{transform to} sections of \DSL. Jackpot also supports something similar to the wildcards in \DSL, as you can define variables in the \textit{pattern} definition and transfer them over to the \textit{fix-pattern} definition. This is closely related to the definition of wildcards in \DSL, though without type restrictions and notation for matching more than one AST node.
This is quite similar to the form of \DSL{}, as both language define som query by using similar structure, in Jackpot you define a \textit{pattern}, then every match of that pattern can be re-written to a \textit{fix-pattern}, each fix-pattern can have a condition attached to it. This is quite similar to the \textit{applicable to} and \textit{transform to} sections of \DSL{}. Jackpot also supports something similar to the wildcards in \DSL{}, as you can define variables in the \textit{pattern} definition and transfer them over to the \textit{fix-pattern} definition. This is closely related to the definition of wildcards in \DSL{}, though without type restrictions and notation for matching more than one AST node.
The example of a query and transformation below, will query the code for variable declarations with initial value of 1, and then change them into a declaration with initial value of 0.
\begin{lstlisting}
@ -83,7 +83,7 @@ When doing structural search in Jetbrains IntelliJ IDEA, templates are used to d
This tool is an interactive experience, where each match is showcased in the find tool, and the developer can decide which matches to apply the replace template to. This allows for error avoidance and a stricter search that is verified by humans. If the developer wants, they do not have to verify each match and just replace everything.
When comparing this tool to \DSL and its corresponding program, there are some similarities. They are both template based, which means a search uses a template to define query, both templates contain variables/wildcards to match against a free section, and the replacing structure is also a template based upon those same variables. A way of matching the variables/wildcards of structural search and replace also exists, one can define the amount of X node to match against, similar to the \texttt{+} operator used in \DSL. A core difference between \DSL and structural search and replace is the variable type system. When performing a match and transformation in \DSL the types are used extensively to limit the match against the wildcards, while this limitation is not possible in structural search and replace.
When comparing this tool to \DSL{} and its corresponding program, there are some similarities. They are both template based, which means a search uses a template to define query, both templates contain variables/wildcards to match against a free section, and the replacing structure is also a template based upon those same variables. A way of matching the variables/wildcards of structural search and replace also exists, one can define the amount of X node to match against, similar to the \texttt{+} operator used in \DSL{}. A core difference between \DSL{} and structural search and replace is the variable type system. When performing a match and transformation in \DSL{} the types are used extensively to limit the match against the wildcards, while this limitation is not possible in structural search and replace.
\section{Other JavaScript parsers}

2
generators/appendix-b.tex Normal file
View file

@ -0,0 +1,2 @@
\chapter{Examples of transformations performed in Evaluation}

6
generators/commands.sty
View file

@ -1,6 +1,6 @@
\newcommand{\DSL}{JSTQL }
\newcommand{\exProp}{\textbf{declare numerical literal}}
\newcommand{\DSL}{JSTQL}
\newcommand{\exProp}{\textbf{``numerical literals''}}
\newcommand{\discardBindings}{Discard Bindings }
\newcommand{\DSLSH}{JSTQL-SH }
\newcommand{\DSLSH}{JSTQL-SH}
\newcommand{\figFull}[1][missing]{Figure \ref{#1}}
\newcommand{\fig}[1][missing]{\ref{#1}}

37
generators/refs.bib
View file

@ -518,4 +518,41 @@
urldate = {2024-05-30},
note = {[Online; accessed 30. May 2024]},
url = {https://www.w3.org/TR/xslt20}
}
@book{Kleppe,
author = {Kleppe, Anneke G.},
title = {{Software Language Engineering: Creating Domain-Specific Languages Using Metamodels}},
year = {2008},
month = jan,
urldate = {2024-05-31},
isbn = {978-0-32155345-4},
publisher = {Addison-Wesley Professional},
address = {Boston, MA, USA}
}
@misc{PipelineHistory,
title = {{proposal-pipeline-operator/HISTORY.md at main {$\cdot$} tc39/proposal-pipeline-operator}},
journal = {GitHub},
year = {2024},
month = may,
urldate = {2024-05-31},
note = {[Online; accessed 31. May 2024]},
url = {https://github.com/tc39/proposal-pipeline-operator/blob/main/HISTORY.md}
}
@misc{ElixirPipe,
title = {{Enumerables and Streams {\ifmmode---\else\textemdash\fi} Elixir v1.16.3}},
year = {2024},
month = may,
urldate = {2024-06-01},
note = {[Online; accessed 1. Jun. 2024]},
url = {https://hexdocs.pm/elixir/1.16.3/enumerable-and-streams.html#the-pipe-operator}
}
@misc{BashPipeline,
title = {{Bash Reference Manual}},
year = {2024},
month = jun,
urldate = {2024-06-01},
note = {[Online; accessed 1. Jun. 2024]},
url = {https://www.gnu.org/software/bash/manual/bash.html#Pipelines}
}