import myModule from "./my-module.torrent": requiring Node modules from BitTorrent

by
, posted

Recent versions of Node.js added a feature called “Customization Hooks”. This lets you change the way modules load. In other words, you can control how import works.

I wondered…could you import a JavaScript file from BitTorrent? Could you import a module from a .torrent file?

Introducing torrent-import

After a few days of work, I built torrent-import. It lets you import Node modules from .torrent files or magnet: URIs. Use it like this:

import { greet } from "./greet.js.torrent";
console.log(greet());

After installing the module and grabbing greet.js.torrent, you can run it like this:

node --import torrent-import my-app.js
# => Hello world!

It’s a proof of concept and has lots of problems, but you can check out the module for yourself. Keep reading to learn how it works.

Why BitTorrent?

I chose BitTorrent for two main reasons:

  1. To explore content-addressed modules.

    The current state-of-the-art has developers fetching modules by location. When you run npm install lodash, it resolves to something like https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz. You might also refer to a module based on its file system path.

    BitTorrent uses content-addressed storage. Instead of a URL, you’re effectively given a SHA-1 hash, and it’s up to BitTorrent to find the file somewhere out there.

    This offers some advantages:

    • Preservation. Microsoft owns the npm registry, and they wouldn’t be the first company to shutter an unprofitable piece of software. Content addressing can help preserve these modules in the unlikely event of npm’s closure, because you don’t need to install things from the registry (or any one specific place).
    • Built-in data integrity. When you download something from BitTorrent, you hash its contents to make sure things match; no need for lockfiles.1
    • De-duplication. If you download a package with a given hash, you should never need to download it again anywhere on your system.

    Content-addressable systems have disadvantages, too (for example, it’s a lot easier to remember “Lodash” than 679591c564c3bffaae8454cf0b3df370c3d6911c). And I don’t know if BitTorrent is the best content-addressable system for this. But I have a hunch that content-addressability is underrated in today’s package management and wanted to explore it.

  2. For fun!

    I think BitTorrent is interesting and enjoy learning about it. This was my primary reason.

So how did I cram BitTorrent into Node’s module system?

Overriding imports with Customization Hooks

Customization Hooks, a new Node feature, lets you change the way modules are loaded. They are at the core of how my project works. (As of this writing, it’s in the “release candidate” stage, so I expect it to be stabilized soon.)

You might imagine writing all kinds of hooks. Maybe a hook that imports files from HTTP. Maybe one that logs all imports but doesn’t change anything; could be useful for profiling the startup time of your app. Or a hook that breaks imports on weekends. Or something else!

It’s pretty easy to build your own hook. You basically need to do two things:

  1. Create a “hook file”. This is a JavaScript file that exports a function which gets called when you import something.
  2. Register the hooks.

Part 1: creating our hook

There are basically two kinds of hooks: load and resolve.

load lets you completely rewrite an import. For example, the Node docs show an example showing how to transpile CoffeeScript files. You can then do things like this:

import { foo } from "./bar.coffee";

The hook effectively reads bar.coffee, transpiles it with the CoffeeScript compiler, and returns the transpiled code as a string. (Check out Node’s docs for more details.)

resolve is a bit simpler: its job is basically to return the path that should get imported. This is what I do in my importer: I download the .js file and return its path.

Let’s make a dummy hook that rewrites .torrent imports to always import from /tmp/fake-torrent-import.js. We’ll create hook.js which exports the resolve function.

That function should:

  1. Check if we’re importing a .torrent file.
  2. If we are, point to /tmp/fake-torrent-import.js.
  3. Otherwise, defer to the existing import behavior as normal.

Here’s what that might look like:

// hook.js

// Node will call this function with:
//
// 1. The string you're importing.
// 2. A context, which we don't really need to worry about.
// 3. A function to continue onto the next hook.
export const resolve = async (specifier, context, nextLoad) => {
  // Are we importing a .torrent file?
  if (specifier.endsWith(".torrent")) {
    // Point to a fake file.
    return {
      format: "module",
      shortCircuit: true,
      url: "file:///tmp/fake-torrent-import.js",
    };
  } else {
    // Defer to the existing import behavior.
    return nextLoad(specifier, context);
  }
};

Let’s also create /tmp/fake-torrent-import.js, a silly module which will soon be irrelevant:

// /tmp/fake-torrent-import.js

export function foo() {
  return "bar";
}

Now let’s see how to register this hook.

Part 2: registering our hook

Let’s create a sample file that we’ll use to test our hook. Create app.js:

// app.js

import { foo } from "./foo.torrent";
console.log(foo());

If we run this with node app.js, we’ll see an error, because Node doesn’t know how to import .torrent files.

Let’s create another file, register-hooks.js. This will register our hook file by calling module.register:

// register-hooks.js

import { register } from "node:module";
register("./hook.js", import.meta.url);

One step remains: when we start Node, we need to tell it to run register-hooks.js first. Add --import ./register-hooks.js to our call:

node --import ./register-hooks.js app.js
# => bar

If you’ve done this correctly, you should see the correct output!

Setting up the hooks is the hard part. Downloading the torrents is pretty quick if you use a library.

Downloading torrents with WebTorrent

So far, we’re detecting .torrent files but not actually downloading them.

I imported WebTorrent to do this. Instead of returning a dummy path, I download the torrent data to disk, and then I return the path of the downloaded file. I don’t think it’s worth going into the nitty-gritty details of how I did this, but here’s a simplified version:

// This isn't *exactly* the real code, but
// it's pretty similar and should still work.

import { once } from "node:events";
import * as path from "node:path";
import WebTorrent from "webtorrent";

export const resolve = async (specifier, context, nextLoad) => {
  // Bail if we're not loading a torrent.
  const isTorrent =
    specifier.endsWith(".torrent") || specifier.startsWith("magnet:");
  if (!isTorrent) {
    return nextLoad(specifier, context);
  }

  // Download the torrent.
  const client = new WebTorrent();
  const torrent = client.add(specifier);
  await once(torrent, "done");

  // Grab the first .js file.
  const file = torrent.files.find((f) => f.path.endsWith(".js"));

  // Return its path.
  const filePath = path.join(torrent.path, file.path);
  const fileUrl = pathToFileURL(filePath).toString();
  return { format: "module", shortCircuit: true, url: fileUrl };
};

The real version is a lot like this, but does a better job with edge cases, error handling, debug logging, and cleanup.

WebTorrent has some rough edges (I even fixed a small bug while I was working) but I was pleasantly surprised how well it worked. It was pretty easy to get working for a proof of concept.

Once this was done, I needed an actual torrent to use!

Publishing code to the torrent-verse

Once this was done, I could run node --import ./register-hooks.js app.js, but I needed a real torrent!

I started with a very simple JavaScript module:

export const greet = () => "Hello world!";

I uploaded this to the Internet Archive, which lets you download items as torrents and it will seed them for you.2 Armed with that torrent file, I wrote a simple example that uses it:

import { greet } from "./greet.js.torrent";
console.log(greet());

When I ran node --import ./register-hooks.js app.js, I saw “Hello world!”. I cleaned up my messy code and my proof of concept was done!

Beyond a proof of concept?

My goal wasn’t to build a production-ready torrent importer, but a proof of concept. A lot holds this back from being production-ready:

There are probably other problems with this idea, but those are several I came up with.

I hope this post gets people thinking about the upcoming power of Customization Hooks in Node, as well as the potential of BitTorrent and similar technologies. If you have any feedback, let me know!

  1. BitTorrent doesn’t necessarily guarantee data integrity…more on this later. ↩︎

  2. I believe the Internet Archive doesn’t have “traditional” torrent peers of its own, and instead includes web seeds in their torrents. ↩︎