Using ES2015 with Electron — introducing electron-compile

As part of writing the Slack Desktop application, we created a new library / set of tools that will save other developers writing Electron applications a lot of time and effort. We call it electron-compile, and this post will describe how to use it and explain how it works.

Just what is Electron?

Electron is a platform for writing cross-platform Desktop applications using Web technologies — this means, instead of writing your desktop app in languages like C# or Objective C, and using frameworks like Cocoa or WPF, you write your app once in HTML, CSS, and JavaScript.

Traditionally, this approach has had some significant limitations — if your app wanted to do something that wasn’t implemented by the framework or the confines of what a web browser could do, your options were very limited (or even non-existent!). Electron is one of the first platforms that is extensible — adding new native features to the platform can be done via npm modules, either written in JavaScript, or via C++ native node.js modules. Having access to the entire npm ecosystem out of the gate means that Electron can do many things that other platforms got stuck on.

Electron works by combining the Chromium Content Module from the Chromium project, the codebase that powers the Google Chrome browser, and integrating the node.js runtime and standard library, as well as adding APIs of its own as built-in node modules. These modules allow you to interact with the operating system in ways that normal browsers don’t allow, such as being able to create Menu Bar menus or show OS File Open/Save dialogs.

Using ES2015, LESS, and other Web Languages

Out of the box, Electron understands the same languages that Chrome and node.js understand — HTML, CSS, and JavaScript ES5. Developers however, have created a number of higher-level languages that can be compiled into these base languages. Projects like Babel, TypeScript, and LESS allow developers to write code more quickly and correctly, and use features that have perhaps not been fully implemented by browsers. In the Slack Desktop app, we make significant use of ES2015 / ES2016 features, such as ES2015 modules and async/await.

However, we noticed when interacting with the community, that while people were excited with these new features, they also felt a lot of frustration trying to get started, because using these features required developers to spend a lot of time creating a “build pipeline” to compile their ES2015 code to something that Electron could understand. In particular, the popular React Framework requires an enormous amount of boilerplate build code in order to get it working with Electron.

What if, Electron could just understand all of these languages natively instead, without a build step?

To this end, we’ve worked with the Electron community to create electron-compile, a project that allows Electron to natively “Just In Time” compile a number of popular web languages:

• ES2015/2016 + React, via Babel
• TypeScript
• CoffeeScript
• LESS
• Jade / Pug

These languages work inside standalone files, and even work inside inline <script> elements.

Getting Started

Using electron-compile is very easy — in your package.json, replace your reference to electron-prebuilt and instead use electron-prebuilt-compile. Install, and you’ll have a version of Electron that automatically initializes the electron-compile library behind the scenes — even your opening file can be written in ES2015.

Similar to BabelJS, electron-compile is configured via a special dotfile, called .compilerc, which allows you to pass options to the various compiler libraries that electron-compile uses. Here’s an example from our app:

"env": {
    "development": {
      "application/javascript": {
        "presets": ["es2016-node5", "react"],
        "plugins": ["transform-class-properties", "transform-async-to-generator", "transform-runtime"],
        "sourceMaps": "inline"
      }
    },
    "production": {
      "application/javascript": {
        "presets": ["es2016-node5", "react"],
        "plugins": ["transform-class-properties", "transform-async-to-generator", "transform-runtime"],
        "sourceMaps": false
      }
    }
  }

Even if you don’t provide a .compilerc, electron-compile has fairly reasonable defaults which are probably well-suited for getting started.

How it works

In order to make Electron natively understand web languages, we need to first intercept the different ways that Electron loads web content. Intercepting JavaScript loaded via node.js’s require statement is fairly straightforward, via the mechanisms that the node module system provides. We can see this hook in electron-compile in require-hook.js

However, the trickier problem is intercepting content loaded by Chromium, which doesn’t use require to load content. In order to do this, we use Electron’s protocol module in a novel way — instead of registering a new protocol, we intercept the file: protocol in protocol-hook.js, so that when Chromium requests local content, we get a chance to instead return the compiled result.

Being able to intercept content gives us the ability to do some useful things — for example, in the original version of electron-compile, we required developers to initialize every process separately. However, we control all content! This means that we control the HTML that is loaded — automatically initializing electron-compile is as simple as silently inserting a script tag at the very top of any included HTML file.

Caching Compiled Results

Now that we’ve got a way to intercept content, we need a way to cache compiled results, from which we’ll take some inspiration from the Git version-control system. We want to create a table mapping the SHA1s of input contents, to the compiled result. We can then create a write-through cache that persists to disk, where the filenames are the SHA1s of the input file contents.

In order to do this efficiently, we store SHA1s of seen files in a separate cache (file-change-cache.js), as well as some file metadata about input files, such as whether this file has a Source Map or if it’s in node_modules, which is a good hint that we shouldn’t be compiling it.

Using the File Change cache means that subsequent fetches of a file’s SHA1 only requires a stat call to check the file size and mtime. This cache also comes in handy later when we build the app for production, which we’ll see later — it gets saved out as part of compiler-info.json.gz.

Initially looking at the directory structure of an electron-compile cache, it seems to be intentionally obfuscated — SHA1s everywhere! There is some reason to the rhyme however — let’s have a look:

└── compileCache_b0fc9be94022fb6fe0cb8f827f7540fc
 ├── 2ef605183e649031f74e57cedc1545c0d1b2ff9e
 │ ├── 1a0db394fddc0a8eef5874d46b02e797236b1f7a
 │ ├── 2f3aa26d0dfff495c4d62d87609c2c493be64257
 │ ├── 680354109deb5f5842b75bc330bf440f9286adfa
 │ ├── b587c4467a85c3db0d0ddd2c9f4185d1e490fabc
 │ └── e28783abd8a6c809856823a74694a71d4feeff72
 ├── 8da8cb376c733a9b2c1c5ff2b728832dede922fd
 │ ├── 07396d6dd8bbf6966b3c2dba9b83c8b5f9e9bdac
 │ ├── 074b68afbb01816f18bc7b6757e52ab933829498
 │ ├── 09dbc2eb575c717679def660e03c8ca13db25b56
 │ ├── 0b640b26fd5c4f1d0a623f3e48b2ebb1b62f7b87
 └── compiler-info.json.gz

The root directory’s SHA1 represents the path to the application — we don’t want two separate applications writing to the same cache. We see compiler-info.json.gz as the only human-readable filename, which contains the contents of the FileChangedCache, as well as the compiler options passed to every compiler, as well as the versions of the compilers.

Because different versions or compiler options generate different code, when this input changes, we need to forget about our already-compiled code. In order to do this, we simply calculate a SHA1 based on this compiler option + version information, and use it for the folder name.

Opening any of these files gets gibberish, but in fact, these files are simply gzipped — using gzcat reveals that they are simply JSON, which includes the compiled code as well as the MIME type of the result:

$ gzcat b587c4467a85c3db0d0ddd2c9f4185d1e490fabc

{“code”:”<head>n <link rel=”stylesheet” type=”text/less” href=”loading-screen.less”>n <link rel=”stylesheet” type=”text/less” href=”login-view.less”>n <link rel=”stylesheet” type=”text/less” href=”slack-app.less”>n <link rel=”stylesheet” type=”text/less” href=”team-view.less”>n <link rel=”stylesheet” type=”text/less” href=”teams-display.less”>n <link rel=”stylesheet” type=”text/less” href=”team-selector.less”>n <link rel=”stylesheet” type=”text/less” href=”team-selector-item.less”>n</head>n”,”mimeType”:”text/html”,”dependentFiles”:[]}

Combining these features of electron-compile, along with our policy to gzip everything, means that in production, electron-compile does less I/O work than even precompiling all assets.

Compiling for production

electron-compile is great for development, but we can also use the same framework for production mode — simply by compiling every file in our project, then copying the compile cache into our application. Because we definitely don’t want to be compiling anything in production, we configure the library to work in Read-Only Mode. An important facet of this mode is that it creates dummy versions of the compiler classes in read-only-compiler.js that return the same information as the original compilers, using the information stored in compiler-info.json.gz.

Since our FileChangedCache is already seeded with all of the files that are valid files to load along with their sizes, we can use this to determine what files to load, without even opening the original files. Unfortunately, while we’re this close to being able to ship just the cache and not the original source which would save space, the node.js module system will path.resolve anything it loads — the original files have to be there (though they can be zero-length).

From a developer-using-the-library perspective, electron-compile provides both a command-line as well as a full API to implement compiling for production, and just as electron-prebuilt-compile provides a wrapper for electron-prebuilt, it also provides a wrapper for the popular electron-packager project that will automatically do all the electron-compile work behind the scenes when packaging your project.

Learning More

We’ve used the electron-compile package quite a bit in both development and production on all of the major desktop operating systems, and we’ve found it to be really useful, especially for quickly prototyping ideas in React. Here’s where you can learn more:

• Example React App
• GitHub Repo