Shift Refactor
shift-refactor
is a suite of utility functions designed to analyze and modify JavaScript source files.
It originated as a tool to reverse engineer obfuscated JavaScript but is general-purpose enough for arbitrary transformations.
Who is this for?
Anyone who works with JavaScript ASTs (Abstract Syntax Trees). If you're not familiar with ASTs, here are a few use cases where they come in useful:
- Automatic refactoring, making sweeping changes to JavaScript source files (Developers, QA).
- Analyzing JavaScript for linting, complexity scoring, etc (Developers, QA).
- Extracting API details to auto-generate documentation or tests (Developers, QA).
- Scraping JavaScript for information or security vulnerabilities (Pen Testers, QA, Security Teams, Hacker types).
- Programmatically transforming malicious or obfuscated JavaScript (Reverse Engineers).
Status
Stable.
Installation
$ npm install shift-refactor
Usage
The script below finds and prints all literal strings in a script.
// Read 'example.js' as textconst fs = ;const src = fs; const refactor = ; // Create a refactor query objectconst $script = ; // Select all `LiteralStringExpression`sconst $stringNodes = // Turn the string AST nodes into real JS stringsconst strings = $stringNodes; // Output the strings to the consolestrings;
Advanced Example
This script takes the obfuscated source and turns it into something much more readable.
const refactor = ; // require('shift-refactor');const Shift = ; // Obfuscated sourceconst src = `var a=['\x74\x61\x72\x67\x65\x74','\x73\x65\x74\x54\x61\x72\x67\x65\x74','\x77\x6f\x72\x6c\x64','\x67\x72\x65\x65\x74','\x72\x65\x61\x64\x65\x72'];var b=function(c,d){c=c-0x0;var e=a[c];return e;};(function(){class c{constructor(d){this[b('0x0')]=d;}['\x67\x72\x65\x65\x74'](){console['\x6c\x6f\x67']('\x48\x65\x6c\x6c\x6f\x20'+this[b('0x0')]);}[b('0x1')](e){this['\x74\x61\x72\x67\x65\x74']=e;}}const f=new c(b('0x2'));f[b('0x3')]();f[b('0x1')](b('0x4'));f[b('0x3')]();}());`; const $script = ; const strings = ; const destringifyDeclarator = ; destringifyDeclarator; const destringifyOffset = destringifyDeclarator; const findIndex = c - destringifyOffsetvalue; ;; $script; console;
Query Syntax
The query syntax is from shift-query
(which is a port of esquery) and closely resemble CSS selector syntax.
The following selectors are supported:
- AST node type:
FunctionDeclaration
- wildcard:
*
- attribute existence:
[attr]
- attribute value:
[attr="foo"]
or[attr=123]
- attribute regex:
[attr=/foo.*/]
- attribute conditons:
[attr!="foo"]
,[attr>2]
,[attr<3]
,[attr>=2]
, or[attr<=3]
- nested attribute:
[attr.level2="foo"]
- field:
FunctionDeclaration > IdentifierExpression.name
- First or last child:
:first-child
or:last-child
- nth-child (no ax+b support):
:nth-child(2)
- nth-last-child (no ax+b support):
:nth-last-child(1)
- descendant:
ancestor descendant
- child:
parent > child
- following sibling:
node ~ sibling
- adjacent sibling:
node + adjacent
- negation:
:not(ExpressionStatement)
- matches-any:
:matches([attr] > :first-child, :last-child)
- subject indicator:
!IfStatement > [name="foo"]
- class of AST node:
:statement
,:expression
,:declaration
,:function
, or:target
Useful sites & tools
- Shift-query's online sandbox to test queries quickly.
- Shift-query CLI tool to query JavaScript on the command line.
- AST Explorer to explore JavaScript AST's visually (make sure to select "shift" on the top menu bar).
- Shift-AST.org - home of the Shift JavaScript tool suite.
API
refactor(string | Shift AST)
Create a refactor query object.
Note:
This function assumes that it is being passed complete JavaScript source or a root AST node (Script or Module) so that it can create and maintain global state.
Example
const refactor = ; const $script = ;
Refactor Query Object
The API is meant to look and feel like jQuery since – like jQuery – it works with CSS-style queries and regularly accesses nodes on a tree. Each query object is both a function and an instance of the internal RefactorSession
class.
Calling the query object as a function will produce a new query object, You can call a refactor query with a query to produce a new query object with the new nodes or you can call methods off the object to act on the nodes already selected. The examples prefix refactor query objects with a $
to indicate they are refactor query objects and not naked Nodes or other objects.
Example
const refactor = ; const $script = ;const $variableDecls = const $bindingIdentifiers = ;const names = $bindingIdentifiers;
Methods
.$(queryOrNodes)
.append(replacer)
.closest(closestSelector)
.codegen()
.declarations()
.delete()
.filter(iterator)
.find(iterator)
.findMatchingExpression(sampleSrc)
.findMatchingStatement(sampleSrc)
.findOne(selectorOrNode)
.first(selector)
.forEach(iterator)
.get(index)
.logOut()
.lookupVariable()
.lookupVariableByName(name)
.map(iterator)
.nameString()
.parents()
.prepend(replacer)
.print()
.query(selector)
.raw()
.references()
.rename(newName)
.replace(replacer)
.replaceAsync(replacer)
.replaceChildren(query, replacer)
.statements()
.toJSON()
.type()
.$(queryOrNodes)
Sub-query from selected nodes
Example
const refactor = ; const src = `let a = 1;function myFunction() { let b = 2, c = 3;}` $script = ; const funcDecl = ;const innerIdentifiers = funcDecl;// innerIdentifiers.nodes: myFunction, b, c (note: does not include a)
.append(replacer)
Inserts the result ofreplacer
after the selected statement.
Note:
Only works on Statement nodes.
Example
const refactor = ;const Shift = ; const src = `var message = "Hello";console.log(message);` $script = ; ;
.closest(closestSelector)
Finds the closest parent node that matches the passed selector.
Example
const refactor = ; const src = `function someFunction() { interestingFunction();}function otherFunction() { interestingFunction();}` $script = ; // finds all functions that call `interestingFunction`const fnDecls = ;
.codegen()
Generates JavaScript source for the first selected node.
Example
const refactor = ; const src = `for (var i=1; i < 101; i++){ if (i % 15 == 0) console.log("FizzBuzz"); else if (i % 3 == 0) console.log("Fizz"); else if (i % 5 == 0) console.log("Buzz"); else console.log(i);}` $script = ; const strings = console;
.declarations()
Finds the declaration for the selected Identifier nodes.
Note:
Returns a list of Declaration objects for each selected node, not a shift-refactor query object.
Example
const refactor = ; const src = `const myVariable = 2, otherVar = 3;console.log(myVariable, otherVar);` $script = ; // selects the parameters to console.log() and finds their declarationsconst decls = ;
.delete()
Delete nodes
Example
const refactor = ; $script = ; ;
.filter(iterator)
Filter selected nodes via passed iterator
Example
const refactor = ; const src = `let doc = window.document;function addListener(event, fn) { doc.addEventListener(event, fn);}` $script = ; const values = ;
.find(iterator)
Finds node via the passed iterator iterator
Example
const refactor = ; const src = `const myMessage = "He" + "llo" + " " + "World";` $script = ; ;
.findMatchingExpression(sampleSrc)
Finds an expression that closely matches the passed source.
Note:
Used for selecting nodes by source pattern instead of query. The passed source is parsed as a Script and the first statement is expected to be an ExpressionStatement.Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.
Example
const refactor = ; const src = `const a = someFunction(paramOther);const b = targetFunction(param1, param2);` $script = ; const targetCallExpression = $script;
.findMatchingStatement(sampleSrc)
Finds a statement that matches the passed source.
Note:
Used for selecting nodes by source pattern vs query. The passed source is parsed as a Script and the first statement alone is used as the statement to match. Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.
Example
const refactor = ; const src = `function someFunction(a,b) { var innerVariable = "Lots of stuff in here"; foo(a); bar(b);}` $script = ; const targetDeclaration = $script;
.findOne(selectorOrNode)
Finds and selects a single node, throwing an error if zero or more than one is found.
Note:
This is useful for when you want to target a single node but aren't sure how specific your query needs to be to target that node and only that node.
Example
const refactor = ; const src = `let outerVariable = 1;function someFunction(a,b) { let innerVariable = 2;}` $script = ; // This would throw, because there are multiple VariableDeclarators// $script.findOne('VariableDeclarator'); // This won't throw because there is only one within the only FunctionDeclaration.const innerVariableDecl = ;
.first(selector)
Returns the first selected node. Optionally takes a selector and returns the first node that matches the selector.
Example
const refactor = ; const src = `func1();func2();func3();` $script = ; const func1CallExpression = ;
.forEach(iterator)
Iterate over selected nodes
Example
const refactor = ; const src = `let a = [1,2,3,4];` $script = ; ;
.get(index)
Get selected node at index.
Example
const refactor = ; const src = `someFunction('first string', 'second string', 'third string');`$script = ; const thirdString = ;
.logOut()
console.log()
s the selected nodes. Useful for inserting into a chain to see what nodes you are working with.
Example
const refactor = ; const src = `let a = 1, b = 2;` $script = ; ;
.lookupVariable()
Looks up the Variable from the passed identifier node
Note:
ReturnsVariable
objects from shift-scope, that contain all the references and declarations for a program variable.
Example
const refactor = ; const src = `const someVariable = 2, other = 3;someVariable++;function thisIsAVariabletoo(same, as, these) {}` $script = ; // Finds all variables declared within a programconst variables = ;
.lookupVariableByName(name)
Looks up Variables by name.
Note:
There may be multiple across a program. Variable lookup operates on the global program state. This method ignores selected nodes.
Example
const refactor = ; const src = `const someVariable = 2, other = 3;` $script = ; const variables = $script;
.map(iterator)
Transform selected nodes via passed iterator
Example
const refactor = ; const src = `let doc = window.document;function addListener(event, fn) { doc.addEventListener(event, fn);}` $script = ; const values = ;
.nameString()
Retrieve the names of the first selected node. Returns undefined for nodes without names.
Example
const refactor = ; const src = `var first = 1, second = 2;` $script = ;const firstName = ;
.parents()
Retrieve parent node(s)
Example
const refactor = ; const src = `var a = 1, b = 2;` $script = ;const declarators = ;const declaration = declarators;
.prepend(replacer)
Inserts the result ofreplacer
before the selected statement.
Note:
Only works on Statement nodes.
Example
const refactor = ;const Shift = ; const src = `var message = "Hello";console.log(message);` $script = ; ;
.print()
Generates JavaScript source for the first selected node.
Example
const refactor = ;const Shift = ; const src = `window.addEventListener('load', () => { lotsOfWork();})` $script = ; ; console;
.query(selector)
Sub-query from selected nodes
Note:
synonym for .$()
.raw()
Returns the raw Shift node for the first selected node.
Example
const refactor = ; const src = `const a = 2;` $script = ; const declStatement = ;
.references()
Finds the references for the selected Identifier nodes.
Note:
Returns a list of Reference objects for each selected node, not a shift-refactor query object.
Example
const refactor = ; const src = `let myVar = 1;function someFunction(a,b) { myVar++; return myVar;}` $script = ; const refs = ;
.rename(newName)
Rename all references to the first selected node to the passed name.
Note:
Uses the selected node as the target, but affects the global state.
Example
const refactor = ; const src = `const myVariable = 2;myVariable++;const other = myVariable;function unrelated(myVariable) { return myVariable }`$script = ; ;
.replace(replacer)
Replace selected node with the result of the replacer parameter
Example
const refactor = ;const Shift = ; const src = `function sum(a,b) { return a + b }function difference(a,b) {return a - b}` $script = ;
.replaceAsync(replacer)
Async version of .replace() that supports asynchronous replacer functions
Example
const refactor = ; $script = ; { await }
.replaceChildren(query, replacer)
Recursively replaces child nodes until no nodes have been replaced.
Example
const refactor = ;const Shift = ; const src = `1 + 2 + 3` $script = ; $script;
.statements()
Returns the selects the statements for the selected nodes. Note: it will "uplevel" the inner statements of nodes with a.body
property.Does nothing for nodes that have no statements property.
Example
const refactor = ; const src = `console.log(1);console.log(2);` $script = ; const rootStatements = $script;
.toJSON()
JSON-ifies the current selected nodes.
Example
const refactor = ; const src = `(function(){ console.log("Hey")}())` $script = ; const json = $script;
.type()
Return the type of the first selected node
Example
const refactor = ;const Shift = ; const src = `myFunction();` $script = ; const type = type;