javascripthon 0.6

Introduction

JavaScripthon is a small and simple Python 3.5+ translator to JavaScript which aims to be able to translate most of the Python’s core semantics without providing a full python-in-js environment, as most existing translators do. It tries to emit code which is simple to read and check. It does so by switching to ES6 construct when possible/required. This allows to simplify the needs of polyfills for many of the expected Python behaviors.

It is designed to be the first step in a pipeline that translates your Pyhton code into something that a browser can understand. Usually it is used with tools like BabelJS and Webpack to prepare the final bundle that will be served to the browser. The steps from the source code to the bundle are the following:

1. JavaScripthon converts your Python 3.5+ code to ES6 JavaScript modules;

2. the BabelJS loader (configured inside Webpack or standalone) translates the ES6 JavaScript to ES5 so that the browser can understand it;

3. Webpack parses the resulting source code and packages your source code with its dependencies by analyzing import statements and emits a bundle.js ready to be served to the browser.

Along this process the corresponding source maps are read and integrated at every step, allowing you to place breakpoints on your original Python source files when working with the developer tools of your browser.

An example of such setup is provided in the examples directory.

In addition to that, you can choose to do most these steps without using external JS tools. It comes with an embedded js interpreter that loads a standalone version of BabelJS and converts your code to ES5 JavaScript without the need to install anything else. In fact most of the the test you can find in tests/test_evaljs.py use the embedded interpreter to dual evaluate the source code (one time in Python, one time in JavaScript) and simply check that the results are the same.

Thanks to that, JavaScripthon can also be used as a server-side library to translate single functions or classes that you want your browser to load and evaluate.

The interface with the JS world is completely flat, just import the modules or use the expected globals (window, document, etc…) as you would do in JavaScript.

Brief list of the supported Python semantics

The fact that JavaScripthon doesn’t reinvent the wheel by reimplementing in Python many of the features available with JavaScript translators/transpilers allows it to be lean while implementing quite a decent set of the core Python semanticts. These are, briefly:

• Misc

  • list slices;

  • list’s append();

  • dict’s copy(), update();

  • len();

  • print();

  • str();

  • type(instance);

  • yield and yield from;

  • async and await;

  • import and from...import to use any JS module (see import statements);

  • callable();

  • hasattr(), getattr(), setattr();

  • template literals with tmpl('a string with ${substitution}');

  • simple Python 3.6+ f-strings (see Strings);

  • template literals and tagged_templates (see Strings);

  • names starting with d_ and dd_ will have that part replaced with $ and $$, respectively;

  • names ending with an underscore will have it removed. Useful for example with the AVA ES6 test runner which has a check named is;

  • __instancecheck__ to [Symbol.hasInstance];

• Comparisons (see section Simple stuff for the details)

  • most of the basics;

  • isinstance() and issubclass();

  • element in container for use with lists, objects, strings and the new ES6 collections like Map, Set and so on;

  • identity checks: foo is bar;

  • chained comparisons like x < y <= z;

• Statements (see section Simple stuff and for statement for the details)

  • if...elif...else;

  • while loop;

  • for over list, over range, over plain js objects, over iterables (JS iterables);

  • try...except...finally with pythonesque behavior (see try…except…finally statement section for the details);

  • assert statement;

• Functions (see Functions section)

  • standard functions, generator functions, async functions;

  • parameters defaults;

  • keyword parameters;

  • parameters accumulators (*args and **kwargs), with some restrictions;

  • functions in methods are usually converted to “arrow functions” (the new ES6 syntax like (foo, bar) => foo * bar;) because they automatically keep this from the enclosing scope. Appending _fn to a function declaration will force the translation to a normal function;

• Classes (see Classes section)

  • single inheritance;

  • Exception classes for use with except statement;

  • class decorators and method decorators;

  • property descriptors;

  • special handling of property and classmethod descriptors;

  • async methods, generator methods;

  • non-function body members (i.e. member_of_class_Foo = bar);

Installation

Python 3.5 is required because Python’s AST has changed between 3.4 and 3.5 and as of now supporting multiple Python versions is not one of my priorities.

To install the package execute the following command:

$ pip install javascripthon

Usage

To compile or transpile a python source module, use the commandline:

$ python -m metapensiero.pj source.py

or:

$ python -m metapensiero.pj -5 source.py

to transpile.

A pj console script is also automatically installed:

$ pj --help
usage: pj [-h] [--disable-es6] [--disable-stage3] [-5] [--transform-runtime]
          [-o OUTPUT] [-d] [--pdb] [-s STRING] [-e]
          [file [file ...]]

A Python 3.5+ to ES6 JavaScript compiler

positional arguments:
  file                  Python source file(s) or directory(ies) to convert.
                        When it is a directory it will be converted
                        recursively

optional arguments:
  -h, --help            show this help message and exit
  --disable-es6         Disable ES6 features during conversion (Ignored if
                        --es5 is specified)
  --disable-stage3      Disable ES7 stage3 features during conversion
  -5, --es5             Also transpile to ES5 using BabelJS.
  --transform-runtime   Add trasform runtime as plugin during transpile
  -o OUTPUT, --output OUTPUT
                        Output file/directory where to save the generated code
  -d, --debug           Enable error reporting
  --pdb                 Enter post-mortem debug when an error occurs
  -s STRING, --string STRING
                        Convert a string, useful for small snippets. If the
                        string is '-' will be read from the standard input.
  -e, --eval            Evaluate the string supplied with the -s using the
                        embedded interpreter and return the last result. This
                        will convert the input string with all the extensions
                        enabled (comparable to adding the '-5' option) and so
                        it will take some time because of BabelJS load times.

This offers many ways to test the framework, both the string conversion and the evaluation using the embedded JavaScript interpreter are very handy. For example:

$ pj -s '"foo" if True else "bar"'
(true ? "foo" : "bar");

and evaluating the same statement:

$ pj -s '"foo" if True else "bar"' -e
foo

You can even try more fancy ES6 features, like destructuring assignment:

$ pj -s "a, b, c = (2, 3, 5) \na+b+c" -e
10

Conversions Rosetta Stone

Here is a brief list of examples of the conversions the tool applies, just some, but not all.

x < y <= z < 5

((x < y) && (y <= z) && (z < 5))

def foo():
    return [True, False, None, 1729,
            "foo", r"foo\bar", {}]

function foo() {
    return [true, false, null, 1729,
            "foo", "foo\\bar", {}];
}

while len(foo) > 0:
    print(foo.pop())

while ((foo.length > 0)) {
    console.log(foo.pop());
}

if foo > 0:
    ....
elif foo < 0:
    ....
else:
    ....

if ((foo > 0)) {
    ....
} else {
    if ((foo < 0)) {
        ....
    } else {
        ....
    }
}

str(x)

x.toString()

yield foo
yield from foo

yield foo
yield* foo

==

===

!=

!==

2**3

Math.pow(2, 3)

'docstring'

/* docstring */

self

this

len(...)

(...).length

print(...)

console.log(...)

isinstance(x, y)
isinstance(x, (y, z))

(x instanceof y)
(x instanceof y || x instanceof z)

typeof(x)

(typeof x)

type(x)

Object.getPrototypeOf(x)

FirstCharCapitalized(...)
new(any_function(...))

new FirstCharCapitalized(...)
new any_function(...)

foo in bar

var _pj;
function _pj_snippets(container) {
    function in_es6(left, right) {
        if (((right instanceof Array) || ((typeof right) === "string"))) {
            return (right.indexOf(left) > (- 1));
        } else {
            if (((right instanceof Map) || (right instanceof Set)
                || (right instanceof WeakMap)
                || (right instanceof WeakSet))) {
                return right.has(left);
            } else {
                return (left in right);
            }
        }
    }
    container["in_es6"] = in_es6;
    return container;
}
_pj = {};
_pj_snippets(_pj);
_pj.in_es6(foo, bar);

foo[3:]
foo[:3]

foo.slice(3);
foo.slice(0, 3);

list(foo).append(bar)

foo.push(bar);

dict(foo).update(bar)

Object.assign(foo, bar);

dict(foo).copy()

Object.assign({}, foo);

for el in dict(a_dict):
    print(el)

var _pj_a = a_dict;
for (var el in _pj_a) {
    if (_pj_a.hasOwnProperty(el)) {
        console.log(el);
    }
}

for el in an_array:
    print(el)

for (var el, _pj_c = 0, _pj_a = an_array, _pj_b = _pj_a.length;
      (_pj_c < _pj_b); _pj_c += 1) {
    el = _pj_a[_pj_c];
    console.log(el);
}

for i in range(5):
    print(i)

for (var i = 0, _pj_a = 5; (i < _pj_a); i += 1) {
    console.log(i);
}

for el in iterable(a_set):
    print(el)

var _pj_a = a_set;
for (var el of  _pj_a) {
    console.log(el);
}

def foo(a, b, c):
    pass

function foo(a, b, c) {
}

def foo(a, b, c):
    for i in range(a, b, c):
        yield i

for i in iterable(foo(0, 5, 2)):
    print(i)

function* foo(a, b, c) {
    for ... { // loop control omitted for brevity
        yield i;
    }
}

for (var i of foo(0, 5, 2)) {
    console.log(i);
}

async def foo(a, b, c):
    await some_promise_based_async

async function foo(a, b, c) {
    await some_promised_base_async;
}

def foo(a=2, b=3, *args):
    pass

function foo(a = 2, b = 3, ...args) {
}

def bar(c, d, *, zoo=2):
    pass

function bar(c, d, {zoo = 2}={}) {
}

foo(5, *a_list)

foo(5, ...a_list);

bar('a', 'b', zoo=5, another='c')

bar("a", "b", {zoo: 5, another: "c"});

def zoo(e, **kwargs):
    print(kwargs['bar'])

function zoo(e, kwargs = {}) {
    console.log(kwargs['bar'])
}

zoo(4, bar=6)

zoo(4, {bar: 6})

class Foo(bar):
    def __init__(self, zoo):
        super().__init__(zoo)


    def meth(self, zoo):
        super().meth(zoo)
        def cool(a, b, c):
            print(self.zoo)


    async def something(self, a_promise):
        result = await a_promise

    def generator_method(self):
        yield something

    @property
    def foo(self):
        return self._foo


    @foo.setter
    def foo(self, value):
        self._foo = value


    @classmethod
    def bar(self, val):
        do_something()


    def __len__(self):
        return 1


    def __str__(self):
        return 'Foo instance'

class Foo extends bar {
    constructor(zoo) {
        super(zoo);
    }

    meth(zoo) {
        super.meth(zoo);
        var cool;
        cool = (a, b, c) => {
            console.log(this.zoo);
        };
    }

    async something(a_promise) {
        var result;
        result = await a_promise;
    }

    * generator_method() {
        yield something;
    }

    get foo() {
        return this._foo;
    }

    set foo(value) {
        self._foo = value;
    }

    static bar(val) {
        do_something()
    }

    get length() {
        return 1;
    }

    toString() {
        return "Foo instance";
    }
}

class MyError(Exception):
    pass

raise MyError("An error occurred")

function MyError(message) {
    this.name = "MyError";
    this.message = (message || "Custom error MyError");
    if (((typeof Error.captureStackTrace) === "function")) {
        Error.captureStackTrace(this, this.constructor);
    } else {
        this.stack = new Error(message).stack;
    }
}
MyError.prototype = Object.create(Error.prototype);
MyError.prototype.constructor = MyError;
throw new MyError("An error occurred");

try:
    foo.bar()
except MyError:
    recover()
except MyOtherError:
    recover_bad()
finally:
    foo.on_end()

try {
    foo.bar();
} catch(e) {
    if ((e instanceof MyError)) {
        recover();
    } else {
        if ((e instanceof MyOtherError)) {
            recover_bad()
        } else {
            throw e;
        }
    }
} finally {
    foo.on_end();
}

import foo, bar
import foo.bar as b
from foo.bar import hello as h, bye as bb
from ..foo.zoo import bar
from . import foo
from .foo import bar

from foo__bar import zoo

from __foo.zoo import bar

from __globals__ import test_name

# this should not trigger variable definition
test_name = 2

# this instead should do it
test_foo = True

__all__ = ['test_name', 'test_foo']

var test_foo;

import * as foo from 'foo';
import * as bar from 'bar';
import * as b from 'foo/bar';
import {hello as h, bye as bb} from 'foo/bar';
import {bar} from '../foo/zoo';
import * as foo from './foo';
import {bar} from './foo';

import {zoo} from 'foo-bar';

import {bar} from '@foo/zoo';

test_name = 2;
test_foo = true;

export {test_name};
export {test_foo};

Strings

Javascripthon supports converting Python 3.6+ f-strings to ES6 template literals. The expression in the braces gets converted, but neither conversion nor format_spec are supported: f"Value of {a}" becomes `Value of ${a}` and f"Value of {self.foo}" becomes `Value of ${this.foo}`.

You can also write raw template literals by using the function tmpl() it does only a conversion of the string markers, from those used in Python’s literal string notation to template literal notation.

There is also the way to express tagged templates, template literals that are parsed using a provided function. This is done by using the function __. So for example:

__('A template ${string} with foo', bar)

gets translated to:

bar`A template ${string} with foo`

bar will be executed with the value of ${string} as a parameter, see the link for template literals for help.

Examples

Execute make inside the examples directory.

Testing

To run the tests you should run the following at the package root:

python setup.py test

How to contribute

So you like this project and want to contribute? Good!

These are the terse guidelines:

There are some TODO points in the readme, or even the issue #6 is
quite simple to fix. Feel free to pick what you like.

The guidelines are to follow PEP8 for coding where possible, so use
CamelCase for classes and snake_case for variables, functions and
members, and UPPERCASE for constants.

An exception to this rules are the function names inside
``metapensiero.pj.transformations`` subpackage. Those are matched
against names of the AST objects coming from the ``ast`` module in
standard lib, so they have to to match even in case.

Try to keep lines lengths under 79 chars, more or less ;-)

The workflow is to fork the project, do your stuff, maybe add a test
for it and then submit a pull request.

Have fun

Contributing

Any contribution is welcome, drop me a line or file a pull request.

Todo

This is a brief list of what needs to be done:

• refactor the comprehensions conversion to use the snippets facility;

• refactor snippets rendering to write them as a module and import them in the module when tree conversion is enabled;

• convert dict() calls to ES6 Map object creation;

• convert set literals to ES6 Set objects. Also, update “foo in bar” to use bar.has(foo) for sets;

Done

Stuff that was previously in the todo:

• translate import statements to ES6;

• translate __all__ definition to ES6 module exports;

• write a command line interface to expose the api;

• make try…except work again and implement try…finally;

• convert async and await to the same proposed features for js (see BabelJS documentation);

• convert argument defaults on functions to ES6;

• convert call keyword arguments;

• convert *iterable syntax to ES6 destructuring;

• use arrow functions for functions created in functions;

• properties to ES6 properties (getter and setter);

• take advantage of new duckpy features to use a JS execution context that lasts multiple calls. This way the BabelJS bootstrap affects only the initial execution;

• class and method decorators;

• implement yield, yield from and generator functions;

• update “foo in bar” to use bar.has(foo) for maps;

External documentation

A good documentation and explanation of ES6 features can be found on the book Exploring ES6 by Axel Rauschmayer (donate if you can).

An extensive documentation about Python’s AST objects, very handy.

Tools

Have a look at ECMAScript 6 Tools by Addy Osmani.

To debug source maps have a look at source-map-visualization and its package on npm.

Still i found these links to be helpful:

• BASE64 VLQ CODEC (COder/DECoder)

• Testing Source Maps

Here is an example of the latter tool showing code generated by JavaScripthon, have fun!

Notes

• A post about proposed solutions to use ES6 classes with Backbone. See also the bug open on github.

• A benchmark of ES6 features and discussion about it on hacker’s news.

• A compatibility table of ES6 features showing completeness of support feature by feature.

• A story about ES6 crazyest stuff… symbols

Changes