\chapter{Implementation}

In this chapter, the implementation of the tool utilizing the \DSL and \DSLSH will be presented. It will describe the overall architecture of the tool, the flow of data throughout, and how the different stages of transforming user code are completed. 

\section{Architecture of the solution}

The architecture of the solution described in this thesis is illustrated in \figFull[fig:architecture]

In this tool, there exists two multiple ways to define a proposal, and each provide the same functionality, they only differ in syntax and writing-method. One can either write the definition in \DSL, or one can use the program API with \DSLSH, which is more friendly for programs to interact with. 

In the architecture diagram, circular nodes show data passed into the program sections, and square nodes is a specific section of the program. 

\begin{description}
    \item[\DSL Code] is the raw text definition of proposals
    \item[Self-Hosted Object] is the self-hosted version in \DSLSH format
    \item[1. Langium Parser] takes raw \DSL source code, and parses it into a DSL
    \item[2. Pre-parser] extracts the wildcards from the raw template definitions
    \item[1. Prelude-builder] translates JavaScript prelude into array of wildcard strings
    \item[3. Babel] parses the templates and the users source code into an AST
    \item[4. Custom Tree Builder] translates the Babel AST structure into our tree structure
    \item[5. Matcher] finds matches with \texttt{applicable to} template in user code
    \item[6. Transformer] performs transformation defined in \texttt{transform to} template to each match of the users AST
    \item[7. Generator] generates source code from the transformed user AST
\end{description}


\begin{figure}[H]
\begin{center}
\begin{tikzpicture}[
    roundnode/.style={ellipse, draw=red!60, fill=red!5, very thick, minimum size=7mm},
    squarednode/.style={rectangle, draw=red!60, fill=red!5, very thick, minimum size=5mm}
]
\node[squarednode] (preParser)                                              {2. Wildcard Extraction};
\node[squarednode] (preludebuilder)         [above right=of preParser]      {1. Prelude Builder};
\node[roundnode]   (selfhostedjsoninput)    [above=of preludebuilder]       {Self-Hosted Object};
\node[squarednode] (langium)                [above left=of preParser]       {1. Langium Parser};
\node[roundnode]   (jstqlcode)              [above=of langium]              {JSTQL Code};
\node[squarednode] (babel)                  [below=of preParser]            {3. Babel};
\node[roundnode]   (usercode)               [left=of babel]                 {User source code};
\node[squarednode] (treebuilder)            [below=of babel]                {4. Custom Tree builder};
\node[squarednode] (matcher)                [below=of treebuilder]          {5. Matcher};
\node[squarednode] (transformer)            [below=of matcher]              {6. Transformer};
\node[squarednode] (joiner)                 [below=of transformer]          {7. Generator};


\draw[->] (jstqlcode.south) -- (langium.north);
\draw[->] (langium.south) |- (preParser.west);
\draw[->] (preParser.south) |- (babel.north);
\draw[->] (babel.south) -- (treebuilder.north);
\draw[->] (treebuilder.south) -- (matcher.north);
\draw[->] (matcher.south) -- (transformer.north);
\draw[->] (transformer.south) -- (joiner.north);
\draw[->] (selfhostedjsoninput.south) -- (preludebuilder.north);
\draw[->] (preludebuilder.south) |- (preParser.east);
\draw[->] (usercode.east) -- (babel.west);
\end{tikzpicture}
\end{center}

\caption[Tool architecture]{Overview of tool architecture}
\label{fig:architecture}
\end{figure}


\section{Parsing \DSL using Langium}

In this section, the implementation of the parser for \DSL will be described. This section will outline the tool Langium, used as a parser-generator to create the AST used by the tool later to perform the transformations. 

\subsection{Langium}

Langium~\cite{Langium} is a language workbench primarily used to create parsers and Integrated Development Environments for domain specific languages. These kinds of parsers produce Abstract Syntax Trees that is later used to create interpreters or other tooling. In this project, we use Langium to generate an AST definition in the form of TypeScript Objects. These objects and their structure are used as definitions for the tool to do matching and transformation of user code. 

In order to generate this parser, Langium requires a definition of a grammar. A grammar is a specification that describes a valid program. The \DSL grammar describes the structure of \DSL, such as \texttt{proposals}, \texttt{cases}, \texttt{applicable to}, and \texttt{transform to}. A grammar in Langium starts by describing the \texttt{Model}. The model is the top entry of the grammar, this is where the description of all valid top level statements. 

Contained within the \texttt{Model} rule, is one or more proposals. Each proposal is defined with the rule \texttt{Proposals}, and starts with the keyword \texttt{proposal}, followed by a name, and a code block. This rule is designed to contain every definition of a transformation related to a specific proposal. To hold every transformation definition, a proposal definition contains one or more cases. 

The \texttt{Case} rule is created to contain a single transformation. Each case starts with the keyword \texttt{case}, followed by a name for the current case, then a block for that case's fields. Cases are designed in this way to separate different transformation definitions within a proposal. Each case contains a single definition used to match against user code, and a definition used to transform a match. 

The rule, \texttt{AplicableTo}, is designed to hold a single template used for matching. It starts with the keywords \texttt{applicable} and \texttt{to}, followed by a block designed to hold the matching template definition. The template is defined as the terminal \texttt{STRING}, and is parsed as a raw string for characters by Langium\cite{Langium}.

The rule, \texttt{TransformTo}, is created to contain a single template used for transforming a match. It starts with the keywords \texttt{transform} and \texttt{to}, followed by a block that holds the transformation definition. This transformation definition is declared with the terminal \texttt{STRING}, and is parser at a string of characters, same as the template in \texttt{applicable to}. 

In order to define exactly what characters/tokens are legal in a specific definition, Langium uses terminals defined using Regular Expressions, these allow for a very specific character-set to be legal in specific keys of the AST generated by the parser generated by Langium. In the definition of \texttt{Proposal} and \texttt{Pair} the terminal \texttt{ID} is used, this terminal is limited to allow for only words and can only begin with a character of the alphabet or an underscore. In \texttt{Section} the terminal \texttt{STRING} is used, this terminal is meant to allow any valid JavaScript code and the custom DSL language described in \ref{sec:DSL_DEF}. Both these terminals defined allows Langium to determine exactly what characters are legal in each location. 

\begin{lstlisting}[caption={Definition of \DSL in Langium.}, label={def:JSTQLLangium}]
grammar Jstql

entry Model:
    (proposals+=Proposal)*;

Proposal:
    'proposal' name=ID "{"
        (case+=Case)+
    "}";

Case:
    "case" name=ID "{"
        aplTo=ApplicableTo
        traTo=TransformTo
    "}";

ApplicableTo:
    "applicable" "to" "{"
        apl_to_code=STRING
    "}";
TransformTo:
    "transform" "to" "{"
        transform_to_code=STRING
    "}";
hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal STRING: /"[^"]*"|'[^']*'/;
\end{lstlisting}

In the case of \DSL, we are not implementing a programming language meant to be executed. We are using Langium in order to generate an AST that will be used as a markup language, similar to YAML, JSON or TOML\cite{TOML}. The main reason for using Langium in such an unconventional way is Langium provides support for Visual Studio Code integration, and it solves the issue of parsing the definition of each proposal manually. However with only the grammar we cannot actually verify the wildcards placed in \texttt{apl\_to\_code} and \texttt{transform\_to\_code} are correctly written. This is done by using a feature of Langium called \texttt{Validator}. 


\subsection*{Langium Validator}

A Langium validator allows for further checks DSL code, a validator allows for the implementation of specific checks on specific parts of the grammar. 

\DSL does not allow empty typed wildcard definitions in \texttt{applicable to} blocks, this means a wildcard cannot be untyped or allow any AST type to match against it. This is not possible to verify within the grammar, as inside the grammar the code is simply defined as a \texttt{STRING} terminal. This means further checks have to be implemented using code. In order to do this we have a specific \texttt{Validator} implemented on the \texttt{Case} definition of the grammar. This means every time anything contained within a \texttt{Case} is updated, the language server created with Langium will perform the validation step and report any errors.   

The validator uses \texttt{Case} as its entry point, as it allows for a checking of wildcards in both \texttt{applicable to} and \texttt{transform to}, allowing for a check for whether a wildcard identifier used in \texttt{transform to} exists in the definition of \texttt{applicable to}. 

\begin{lstlisting}[language={JavaScript}]
export class JstqlValidator {
    validateWildcardAplTo(pair: Pair, accept: ValidationAcceptor): void {
        try {
            if (validationResultAplTo.errors.length != 0) {
                accept("error", validationResultAplTo.errors.join("\n"), {
                    node: pair.aplTo,
                    property: "apl_to_code",
                });
            }
            if (validationResultTraTo.length != 0) {
                accept("error", validationResultTraTo.join("\n"), {
                    node: pair.traTo,
                    property: "transform_to_code",
                });
            }
        } catch (e) {}
    }
}

\end{lstlisting}

\subsection*{Using Langium as a parser}

Langium\cite{Langium} is designed to automatically generate extensive tool support for the language specified using its grammar. However, in our case we have to parse the \DSL definition using Langium, and then extract the Abstract syntax tree generated in order to use the information it contains. 

To use the parser generated by Langium, we created a custom function \texttt{parseDSLtoAST}, which takes a string as an input (the raw \DSL code), and outputs the pure AST using the format described in the grammar, see Listing \ref{sec:DSL_DEF}. This function is exposed as a custom API for our tool to interface with. This also means our tool is dependent on the implementation of the Langium parser to function with \DSL. The implementation of \DSLSH is entirely independent. 

When interfacing with the Langium parser to get the Langium generated AST, the exposed API function is imported into the tool, when this API is executed, the output is on the form of the Langium \texttt{Model}, which follows the same form as the grammar. This is then transformed into an internal object structure used by the tool, this structure is called \texttt{TransformRecipe}, and is then passed in to perform the actual transformation. 

\section{Wildcard extraction and parsing}

In order to refer to internal DSL variables defined in \texttt{applicable to} and \texttt{transform to} blocks of the transformation, we need to extract this information from the template definitions and pass that on to the matcher. 

\subsection*{Why not use Langium for wildcard parsing?}

Langium has support for creating a generator to output an artifact, which is some transformation applied to the AST built by the Langium parser. This suits the needs of \DSL quite well and could be used to extract the wildcards from each \texttt{pair} and create the \texttt{TransformRecipe}. This is the official way the developers of Langium want this kind of functionality to be implemented, however, the implementation would still be mostly the same, as the parsing of the wildcards still has to be done "manually" with code. Therefore, it was decided for this project to keep the parsing of the wildcards within the tool itself. If we were to use Langium generators to parse the wildcards, it would make \DSLSH not entirely independent, and the entire tool would rely on Langium. This is not preferred as that would mean both ways of defining a proposal are reliant of Langium. The reason for using our own extractor is to allow for an independent way to define transformations using our tool. 

\subsection*{Extracting wildcards from \DSL}

In order to allow the use of Babel\cite{Babel}, the wildcards present in the \texttt{applicable to} blocks and \texttt{transform to} blocks have to be parsed and replaced with some valid JavaScript. This is done by using a pre-parser that extracts the information from the wildcards and inserts an \texttt{Identifier} in their place. 

 To extract the wildcards from the template, we look at each character in the template. If a start token of a wildcard is discovered, which is denoted by \texttt{<<}, everything after that until the closing token, which is denoted by \texttt{>>}, is then treated as an internal DSL variable and will be stored by the tool. A variable \texttt{flag} is used (line 5,10 \ref{lst:extractWildcard}), when the value of flag is false, we know we are currently not inside a wildcard block, this allows us to just pass the character through to the variable \texttt{cleanedJS} (line 196 \ref{lst:extractWildcard}). When \texttt{flag} is true, we know we are currently inside a wildcard block and we collect every character of the wildcard block into \texttt{temp}. Once we hit the end of the wildcard block, when we have consumed the entirety of the wildcard, the contents of the \texttt{temp} variable is passed to a tokenizer, then the tokens are parsed by a recursive descent parser (line 10-21 \ref{lst:extractWildcard}).
 
Once the wildcard is parsed, and we know it is safely a valid wildcard, we insert an identifier into the JavaScript template where the wildcard would reside. This allows for easier identifications of wildcards when performing matching/transformation as we can identify whether or not an Identifier in the code is the same as the identifier for a wildcard. This however, does introduce the problem of collisions between the wildcard identifiers inserted and identifiers present in the users code. In order to avoid this, the tool adds \texttt{\_\-\-\_} at the beginning of every identifier inserted in place of a wildcard. This allows for easier identification of if an Identifier is a wildcard, and avoids collisions where a variable in the user code has the same name as a wildcard inserted into the template. This can be seen on line 187 of the example below.  

\begin{lstlisting}[language={JavaScript}, caption={Extracting wildcard from template.}, label={lst:extractWildcard}]
export function parseInternal(code: string): InternalParseResult {
     for (let i = 0; i < code.length; i++) {
        if (code[i] === "<" && code[i + 1] === "<") {
            // From now in we are inside of the DSL custom block
            flag = true;
            i += 1;
            continue;
        }

        if (flag && code[i] === ">" && code[i + 1] === ">") {
            // We encountered a closing tag
            flag = false;
            try{
                let wildcard = new WildcardParser(
                    new WildcardTokenizer(temp).tokenize()
                ).parse();
                cleanedJS += collisionAvoider(wildcard.identifier.name);

                prelude.push(wildcard);
                i += 1;
                temp = "";
                continue;
            }
            catch (e){
                // We probably encountered a bitshift operator, append temp to cleanedJS
            }

        }
        if (flag) {
            temp += code[i];
        } else {
            cleanedJS += code[i];
        }
    }
    return { prelude, cleanedJS };
}
\end{lstlisting}

\paragraph*{Parsing wildcard}

Once a wildcard has been extracted from  definitions inside \DSL, they have to be parsed into a simple Tree to be used when matching against the wildcard. This is accomplished by using a simple tokenizer and a~\cite{RecursiveDescent}{recursive descent parser}. 

Our tokenizer takes the raw stream of input characters extracted from the wildcard block within the template, and determines which part is what token. Due to the very simple nature of the type expressions, no ambiguity is present with the tokens, so determining what token is meant to come at what time is quite trivial. We use a switch case on the current token, if the token is of length one we accept it and move on to the next character. If the next character is an unexpected one it will produce an error. The tokenizer also groups tokens with a \textit{token type}, this allows for an simpler parsing of the tokens later. 

A recursive descent parser is created to closely mimic the grammar of the language the parser is implemented for, where we define functions for handling each of the non-terminals and ways to determine what non terminal each of the token-types result in. The type expression language is a very simple Boolean expression language, making parsing quite simple. 

\begin{lstlisting}[caption={Grammar of type expressions}, label={ex:grammarTypeExpr}]
Wildcard:
    Identifier ":" MultipleMatch

MultipleMatch:
    GroupExpr "*"
    | TypeExpr

TypeExpr:
    BinaryExpr
    | UnaryExpr 
   | PrimitiveExpr 

BinaryExpr: 
    TypeExpr { Operator TypeExpr }*

UnaryExpr:
    {UnaryOperator}? TypeExpr

PrimitiveExpr:
    GroupExpr | Identifier

GroupExpr:
    "(" TypeExpr ")"
\end{lstlisting}

The grammar of the type expressions used by the wildcards can be seen in \figFull[ex:grammarTypeExpr], the grammar is written in something similar to Extended Backus-Naur form, where we define the terminals and non-terminals in a way that makes the entire grammar \textit{solvable} by the Recursive Descent parser. 

Our recursive descent parser produces an~\cite{AST1,AST2}{AST} which is later used to determine when a wildcard can be matched against a specific AST node, the full definition of this AST can be seen in Appendix \ref{ex:typeExpressionTypes}. We use this AST by traversing it using a~\cite{VisitorPattern}{visitor pattern} and comparing each \texttt{Identifier} against the specific AST node we are currently checking, and evaluating all subsequent expressions and producing a boolean value, if this value is true, the node is matched against the wildcard, if not then we do not have a match. 



\paragraph*{Extracting wildcards from \DSLSH}

The self-hosted version \DSLSH also requires some form of pre-parsing in order to prepare the internal DSL environment. This is relatively minor and only parsing directly with no insertion compared to \DSL. 

In order to use JavaScript as the meta language, we define a \texttt{prelude} on the object used to define the transformation case. This prelude is required to consist of several \texttt{Variable declaration} statements, where the variable names are used as the internal DSL variables and right side expressions are strings that contain the type expression used to determine a match for that specific wildcard.

We use Babel to generate the AST of the \texttt{prelude} definition, this allows us to get a JavaScript object structure. Since the structure is very strictly defined, we can expect every \texttt{stmt} of \texttt{stmts} to be a variable declaration, otherwise throw an error for invalid prelude. Then the string value of each of the variable declarations is passed to the same parser used for \DSL wildcards. 

The reason this is preferred is it allows us to avoid having to extract the wildcards and inserting an Identifier. 

\section{Using Babel to parse}
\label{sec:BabelParse}

Allowing the tool to perform transformations of code requires the generation of an Abstract Syntax Tree from the users code, \texttt{applicable to} and \texttt{transform to}. This means parsing JavaScript into an AST, in order to do this we use a tool~\cite[Babel]{Babel}. 

The most important reason for choosing to use Babel for the purpose of generating the AST's used for transformation is due to the JavaScript community surrounding Babel. As this tool is dealing with proposals before they are part of JavaScript, a parser that supports early proposals for JavaScript is required. Babel works closely with TC39 to support experimental syntax\cite{BabelProposalSupport} through its plugin system, which allows the parsing of code not yet part of the language. 


\subsection*{Custom Tree Structure}

Performing matching and transformation on each of the sections inside a \texttt{case} definition, they have to be parsed into and AST in order to allow the tool to match and transform accordingly. To do this the tool uses the library~\cite[Babel]{Babel} to generate an AST data structure. However, this structure does not suit traversing multiple trees at the same time, this is a requirement for matching and transforming. Therefore we use this Babel AST and transform it into a simple custom tree structure to allow for simple traversal of the tree.

As can be seen in \figFull[def:TreeStructure] we use a recursive definition of a \texttt{TreeNode} where a nodes parent either exists or is null (it is top of tree), and a node can have any number of children elements. This definition allows for simple traversal both up and down the tree. Which means traversing two trees at the same time can be done in the matcher and transformer section of the tool. 


\begin{lstlisting}[language={JavaScript}, label={def:TreeStructure}, caption={Simple definition of a Tree structure in TypeScript}]
export class TreeNode<T> {
    public parent: TreeNode<T> | null;
    public element: T;
    public children: TreeNode<T>[] = [];

    constructor(parent: TreeNode<T> | null, element: T) {
        this.parent = parent;
        this.element = element;
        if (this.parent) this.parent.children.push(this);
    }
}
\end{lstlisting}

Placing the AST generated by Babel into this structure means utilizing the library~\cite{BabelTraverse}{Babel Traverse}. Babel Traverse uses the~\cite{VisitorPattern}{visitor pattern} to allow for traversal of the AST. While this method does not suit traversing multiple trees at the same time, it allows for very simple traversal of the tree in order to  place it into our simple tree structure. 

\cite{BabelTraverse}{Babel Traverse} uses the~\cite{VisitorPattern}{visitor pattern} to visit each node of the AST in a \textit{depth first} manner, the idea of this pattern is one implements a \textit{visitor} for each of the nodes in the AST and when a specific node is visited, that visitor is then used. In the case of transferring the AST into our simple tree structure we simply have to use the same visitor for all nodes, and place that node into the tree. 

Visiting a node using the \texttt{enter()} function means we went from the parent to that child node, and it should be added as a child node of the parent. The node is automatically added to its parent list of children nodes from the constructor of \texttt{TreeNode}. Whenever leaving a node the function \texttt{exit()} is called, this means we are moving back up into the tree, and we have to update what node was the \textit{last} in order to generate the correct tree structure. 

\begin{lstlisting}[language={JavaScript}]
traverse(ast, {
        enter(path: any) {
            let node: TreeNode<t.Node> = new TreeNode<t.Node>(
                last,
                path.node as t.Node
            );

            if (last == null) {
                first = node;
            }
            last = node;
        },
        exit(path: any) {
            if (last && last?.element?.type != "Program") {
                last = last.parent;
            }
        },
    });
    if (first != null) {
        return first;
    }

\end{lstlisting}

\section{Outline of transforming user code}

\begin{lstlisting}[language={JavaScript}, caption={Outline of the algorithm}, label={lst:outline}]
    let wildcards = ExtractWildcards();

    ParseWithBabel(); // Parse all JavaScript source code with Babel

    MakeTrees(); // Build the tree structure from Babel AST
   
    // Search the user code for matches to applicable to
    if(applicableTo.program.length > 1){
        let matches = multiStmtMatcher(codeTree, applicableTo, wildcards);
    }else{
        let matches = singleMatcher(codeTree, ApplicableTo, wildcards);
    }

    // Build the transformation with code matched to wildcards
    for(let match of matches){
        let transformed = buildTransform(match, transformTo, wildcards);
    }

    // Replace original matched sections with transformed section 
    for(let transform of transformed){
        traverse(codeAST){
            if(transform.node === codeAST.node){
                codeAST.replace(transform);
            }
        }
    }

    // Generate code from the AST
    return babel.generate(codeAST);
\end{lstlisting}

Each line in \ref{lst:outline} is a step in the full algorithm for transforming user code based on a proposal specification in our tool. These steps work as follows:
\begin{description}
    \item [Line 1:] Extract the wildcards from the template definitions and replace them with identifiers.
    \item [Line 3:] Parse all source code into a Babel AST using \texttt{@babel/parser}\cite{BabelParser}
    \item [Line 5:] Convert the Babel AST into our own tree structure for simpler traversal of multiple trees at the same time
    \item [Lines 7-12:] Based on the \texttt{applicable to} template, decide what matching function to use, and find all matching sections of the user code.
    \item [Lines 14-17:] Move all matched wildcard nodes into an instance of the \texttt{transform to} template.
    \item [Lines 20-26:] Insert all transformations from the previous step into the original user AST.
    \item [Line 29:] Generate source code from the user AST using \texttt{@babel/generate}\cite{BabelGenerate}. 
\end{description}

\section{Matching}

This section discusses how we find matches in the users code, this is the step described in lines 5-10 of Listing \ref{lst:outline}. Firstly, we will discuss how individual nodes are compared, then how the two traversal algorithms are implemented, and how matches are discovered using these algorithms.  



\paragraph*{Determining if AST nodes match.}

The initial problem we have to overcome is a way of comparing AST nodes from the template to AST nodes from the user code. This step also has to take into account comparing against wildcards and pass that information back to the AST matching algorithms.

When comparing two AST nodes in this tool, we use the function \texttt{checkCodeNode}, which will give the following values based on what kind of match these two nodes produce. 
\begin{description}
    \item[NoMatch:] The nodes do not match.
    \item[Matched:] The nodes are a match, and the node of \texttt{applicable to} is not a wildcard.
    \item[MatchedWithWildcard]: The node of the user AST produced a match against a wildcard.
    \item[MatchedWithPlussedWildcard]: The node of the user AST produced a match against a wildcard that can match one or more nodes against itself.   
\end{description}

When we are comparing two AST nodes, we have to perform an equality check. Due to this being a structural matching search, we can get away with just performing some preliminary checks, such as that names of identifiers, otherwise it is sufficient to just perform an equality check of the types of the nodes we are currently trying to match. If the types are the same, they can be validly matched against each other. This is sufficient because we are currently trying to determine if a single node can be a match, and not the entire template structure is a match. Therefore false positives that are not equivalent are highly unlikely due to the fact the entire structure has to be a false positive match. 

There is a special case when comparing two nodes, namely when encountering a wildcard. To know if we have encountered a wildcard, the current AST node of \texttt{applicable to} will be either an \texttt{Identifier} or a \texttt{ExpressionStatement} where the expression is an \texttt{Identifier}. The reason it might be an \texttt{ExpressionStatement} is due to the wildcard extraction step, where we replace the wildcard with an identifier of the same name. Due to this replacement, we might place an identifier as a statement, the identifier will then be wrapped inside an \texttt{ExpressionStatement} AST node. If the node of \texttt{applicable to} is of either of these types, we have to check if the name of the identifier is the same as a wildcard. If it is, we have to compare the type of the user AST node against the type expression of the wildcard. 

\begin{lstlisting}
if((aplToNode.type === "ExpressionStatement" && 
    aplToNode.expression.type === "Identifier") ||
    aplToNode.type === "Identifier"){

    // Check if aplToNode is a wildcard
}
\end{lstlisting}

When comparing an AST node type against a wildcard type expression, we pass the node type into a function \texttt{WildcardEvaluator}. This evaluator will traverse through the AST of the wildcard type expression. Every leaf of the tree is equality checked against the type, and the resulting boolean value is returned. Then we solve the expression, bubbling the values through the visitor until we have traversed the entire expression, and have a result. If the result of the evaluator is \texttt{false}, we return \texttt{NoMatch}. If the result of the evaluation is \texttt{true}, we know we can match the user's AST node against the wildcard. If the wildcard type expression contains a Kleene plus, the comparison returns \texttt{MatchedWithPlussedWildcard}, if not, we return \texttt{MatchedWithWildcard}. 

\subsection{Matching a single Expression/Statement template}
\label{sec:singleMatcher}

The larger and more complex the \texttt{applicable to} template is, the fewer matches it will produce, therefore using a single expression/statement as the matching template is preferred. This is because there will be a higher probability of discovering applicable code with a template that is as generic and simple as possible. A very complex matching template with many statements might result in a lower chance of finding matches in the users code. Therefore using simple, single root node matching templates provide the highest possibility of discovering a match within the users code. This section will cover line 11 of Listing \ref{lst:outline}. 

Determining if we are currently matching with a template that is only a single expression/statement, we have to verify that the program body of the template has the length of one, if it does we can use the single length traversal algorithm.

There is a special case for if the template is a single expression, as the first node of the AST generated by \texttt{@babel/generate}\cite{BabelGenerate} will be of type \texttt{ExpressionStatement}, the reason for this is Babel will treat free floating expressions as a statement. This will miss many applicable parts of the users code, because expressions within other statements are not wrapped in an \texttt{ExpressionStatement}. This will give a template that is incompatible with a lot of otherwise applicable expressions. This means the statement has to be removed, and the search has to be done with the expression as the top node of the template. If the node in the body of the template is a statement, no removal has to be done, as a statement can be used directly. 


\paragraph{Discovering Matches Recursively}

The matcher used against single expression/statement templates is based Depth-First Search to traverse the trees. The algorithm can be split into two steps. The initial step is to check if we are currently at the root of the \texttt{applicable to} AST, the second is to try to match the current nodes, and start a search on each of their child nodes. 


It is important we try to match against the template at all levels of the code AST, this is done by starting a new search one every child node of the code AST if the current node of the template AST is the root node. This ensures we have tried to perform a match at any level of the tree. This also ensures we have no partial matches, as we store it only if it returns a match when being called with the root node of \texttt{applicable to}.

\begin{lstlisting}[language={JavaScript}]
if(aplTo.element === this.aplToRoot){
    // Start a search from root of aplTo on all child nodes
    for(let codeChild of code.children){
        let childMatch = singleMatcher(codeChild, aplTo);

        // If it is a match, we know it is a full match and store it.
        if(childMatch){
            this.matches.push(childMatch);
        }
    }
}
\end{lstlisting}

We can now determine if we are currently exploring a match. This means the current code AST node is checked against the current node of \texttt{applicable to} AST. Based on what kind of result the comparison between these two nodes give, we have perform different steps.

\begin{description}
    \item[NoMatch:] If a comparison between the nodes return a \texttt{NoMatch} result, we perform an early return of undefined, as no match was discovered. We can safely discard this search, because we have started a search at all levels of the code AST.
    \item[Matched:] The current code node matches against the current node of the template, and we have to perform a search on each of the child nodes. 
    \item[MatchedWithWildcard:] When a comparison results in a wildcard match, we pair the current code node and the template wildcard, and do an early return. We can do this because if a wildcard matches, the nodes of the children does not matter and will be placed into the transformation.
    \item[MatchedWithPlussedWildcard:] this is a special case for a wildcard match. When a match against a wildcard that has the Kleene plus tied to it we also perform an early return. This result means some special traversal has to be done to the current nodes siblings, this is described below.      
\end{description}


A comparison result of \texttt{Matched} means the two nodes match, but the \texttt{applicable to} node is not a wildcard. With this case, we perform a search on each child nodes of \texttt{applicable to} AST and the user AST. This is performed in order, meaning the n-th child node of \texttt{applicable to} is checked against the n-th child node of the user AST. 

When checking the child nodes, we have to check for a special case when the comparison of the child nodes result in \texttt{MatchedWithPlussedWildcard}. If this result is encountered, we have to continue matching the same \texttt{applicable to} node against each subsequent sibling node of the code node. This is because, a wildcard with a Keene plus can match against multiple sibling nodes. This behavior can bee seen in line 17-31 of Listing \ref{lst:pseudocodeChildSearch}.

If all child nodes did not give the result of NoMatch, we have successfully matched every node of the \texttt{applicable to} AST. This does not yet mean we have a match, as there might be remaining nodes in the child node of the code AST. To check for this, we check whether or not \texttt{codeI} is equal to the length of \texttt{code.children}. If it is unequal, we have not matched all child nodes of the code AST and have to return \texttt{NoMatch}. This can be seen on lines 37-39 of Listing \ref{lst:pseudocodeChildSearch}.

\begin{lstlisting}[language={JavaScript}, caption={Pseudocode of child node matching}, label={lst:pseudocodeChildSearch}]
let codeI = 0;
let aplToI = 0;

while (aplToI < aplTo.children.length && codeI < code.children.length){
    let [pairedChild, childResult] = singleMatcher(code.children[codeI], aplTo.children[aplToI]);

    // If a child does not match, the entire match is discarded
    if(childResult === NoMatch){
        return [undefined, NoMatch];
    }

    // Add the match to the current Paired Tree structure
    pairedChild.parent = currentPair;
    currentPair.children.push(pairedChild);

    // Special case for Keene plus wildcard match
    if(childResult === MatchedWithPlussedWildcard){
        codeI += 1;
        while(codeI < code.children.length){
            let [nextChild, plusChildResult] = singleMatcher(code.children[codeI], aplTo.children[aplToI]);

            if(plusChildResult !== MatchedWithPlussedWildcard){
                i -= 1;
                break;
            }

            pairedChild.element.codeNode.push(...nextChild.element.codeNode);

            codeI += 1;
        }
    }

    codeI += 1;
    aplToi += 1;
}

if(codeI !== code.children.length){
    return [undefined, NoMatch]
}

return [currentPair, Match];
\end{lstlisting}

\subsection{Matching multiple Statements}

Using multiple statements in the template of \texttt{applicable to} means the tree of \texttt{applicable to} as multiple root nodes, to perform a match with this kind of template, we use a sliding window\cite{SlidingWindow} with size equal to the amount statements in the template. This window is applied at every \textit{BlockStatement} and \texttt{Program} of the code AST, as that is the only placement statements can reside in JavaScript~\cite[14]{ecma262}. 

The initial step of this algorithm is to search through the AST for ast nodes that contain a list of \textit{Statements}. Searching the tree is done by Depth-First search, at every level of the AST, we check the type of the node. Once a node of type \texttt{BlockStatement} or \texttt{Program} is discovered, we start the trying to match the statements. 

\begin{lstlisting}[language={JavaScript}]
multiStatementMatcher(code, aplTo) {
    if (
        code.element.type === "Program" ||
        code.element.type === "BlockStatement"
    ) {
        matchMultiHead(code.children, aplTo.children);
    }

    for (let code_child of code.children) {
        multiStatementMatcher(code_child, aplTo);
    }
}

\end{lstlisting}

\texttt{matchMultiHead} uses a sliding window~\cite{SlidingWindow}. The sliding window will try to match every statement of the code AST against its corresponding statement in the \texttt{applicable to} AST. For every statement, we perform a DFS recursion algorithm is applied, similar to algorithm used in Section \ref{sec:singleMatcher}, however this search is not applied to all levels, and if it matches it has to match fully and immediately. If a match is not found, the current iteration of the sliding window is discarded and we move on to the next iteration by moving the window one further. 

One important case here is we might not know the width of the sliding window, this is due to wildcards using the Keene plus, as they can match one or more nodes against the wildcard. These wildcards might match against \texttt{(Statement)+}. Therefore, we use a similar technique to the one described in Section \ref{sec:singleMatcher}, where we have two pointers and match each statement depending on each pointer. 

\subsection*{Output of the matcher}

The matches discovered have to be stored such that we can easily find all the nodes that were matched against wildcards and transfer them into the transformation later. To make this simpler, we make use an object  \texttt{PairedNodes}. This object allows us to easily find exactly what nodes were matched against each other. The matcher will place this object into the same tree structure described in \ref{sec:BabelParse}. This means the result of running the matcher on the user code is a list of \texttt{TreeNode<PairedNode>}.  
\begin{lstlisting}[language={JavaScript}]
interface PairedNode{
    codeNode: t.Node[],
    aplToNode: t.Node
}
\end{lstlisting}

Since a match might be multiple statements, we use an interface \texttt{Match}, that contains separate tree structures of \texttt{PairedNodes}. This allows storage of a match with multiple root nodes. 

\begin{lstlisting}[language={JavaScript}]
export interface Match {
    // Every matching Statement in order with each pair
    statements: TreeNode<PairedNodes>[];
}
\end{lstlisting}


\section{Transforming}

To perform the transformation and replacement on each of the matches, we take the resulting list of matches, the template from the \texttt{transform to} section of the current case of the proposal, and the AST version of original code parsed by Babel. All the transformations are then applied to the code and we use~\cite{BabelGenerate}{Babel generate} to generate JavaScript code from the transformed AST. 

An important discovery is to ensure we transform the leafs of the AST first, this is because if the transformation was applied from top to bottom, it might remove transformations done using a previous match. This means if we transform from top to bottom on the tree, we might end up with \texttt{a(b) |> c(\%)} in stead of \texttt{b |> a(\%) |> c(\%)} in the case of the pipeline proposal. This is quite easily solved in our case, as the matcher looks for matches from the top of the tree to the bottom of the tree, the matches it discovers are always in that order. Therefore when transforming, all that has to be done is reverse the list of matches, to get the ones closest to the leaves of the tree first.  

\subsubsection{Inserting wildcard into transformation template}

The transformations are performed by inserting the matched wildcards from the \texttt{applicable to} template into their respective locations in the \texttt{transform to} template. Then the entire transformed \texttt{transform to} template is placed into the original code AST where the root of the match was previously located. Doing this we are essentially doing a transformation that is a find and replace with context passed through the wildcards. 


First we have to extract every node that was matched against the wildcards in the match. To do this we recursively search through the match until we encounter an \texttt{Identifier} that shares a name with a wildcard. 

To insert all nodes matched against wildcards, we use \texttt{@babel/traverse}\cite{BabelTraverse}, and traverse the AST of the \texttt{transform to} template. We use custom visitors for \textit{Identifier} and \textit{ExpressionStatement} with an \texttt{Identifier} as expression. Each visitor checks if the identifier is a registered wildcard, if it is, we perform a replacement of the \texttt{Identifier} with the node/s the wildcard was matched with. 


\subsubsection*{Inserting the template into the AST}

Having a transformed version of the users code, it has to be inserted into the full AST definition of the users code, again we use~\cite{BabelTraverse}{babel/traverse} to traverse the entirety of the code AST using a visitor. This visitor does not apply to any node-type, as the matched section can be any type. Therefore we use a generic visitor, and use an equality check to find the exact part of the code this specific match comes from. Once we find where in the users code the match came from, we replace it with the transformed \texttt{transform to} nodes. This might be multiple statements, therefore the function \texttt{replaceWithMultiple} is used, to insert every Statement from the \texttt{transform to} body, and we are careful to remove any following sibling nodes that were part of the original match. This is done by removing the \textit{n-1} next siblings from where we inserted the transform to template.

\subsubsection*{Generating source code from transformed AST}

To generate JavaScript from the transformed AST created by this tool, we use a JavaScript library titled~\cite{BabelGenerate}{babel/generator}. This library is specifically designed for use with Babel to generate JavaScript from a Babel AST. The transformed AST definition of the users code is transformed, while being careful to apply all Babel plugins the current proposal might require.

