Slides from my talk, Webpack: It's Not Magic.
Recording from BuzzJS, January 2017: https://www.youtube.com/watch?v=_QEM9kdV-b0
Transcript in progress.
Slides available at: https://naomiajacobs.github.io/WebpackTalk.
Hi, my name is Naomi. I'm a software engineer at Mavenlink, we're a software company in San Francisco, and we make project management software for consulting firms. We have a pretty cool culture, we do pair programming, and yes, we are hiring! If you're interested, please come up and talk to me afterward. Currently, on my team I'm helping to transition away from Backbone by building our first feature in React. But today I'm here to talk to you about Webpack! Let's get started.
Why are we talking about Webpack today? Well, it's a tool for web developers that is * one: very popular, and 2: not well-explained, as far as I can tell.
How popular? This data is from npm, the JS package manager, and it's tracking downloads per week for Webpack and some similar tools, Gulp, and Grunt, and Browserify. Webpack is the red line, and we're tracking from 2014 to 2017. So as you can see, Webpack has gotten pretty popular. And if you're wondering what that giant dip is all the way at the end, that's the winter holidays, nobody codes on the holidays. The first stable version of Webpack was released in January of 2014, and as you can see, since then it's gotten more and more popular, and now it has over a million downloads per week. That means that there are a lot of people using Webpack, and as far as I know, a lot of people who are confused about how to use Webpack, so have any of you used Webpack before? Awesome! Cool, I'm glad to see that.
So, how did I learn Webpack? I used it for a couple of side projects, cause I wanted to use React, and when I learned it, I thought that its documentation and API were really confusing. It felt like pretty much all I was doing was copy-pasting from Stack Overflow into my config file until things worked, and I had no idea why. Have any of you guys done the same thing? Yeah, ok, good, I'm glad. I thought it was really confusing, and I sort of put it away, and I was like "Wow, it's cool, it seems like it can do a lot of things, but I'm gonna forget about it for now." * But then my company, Mavenlink, started using Webpack to compile its assets when we did the switch from Backbone to React, and suddenly I had to figure out what was going on, and I realized that learning how to use a tool is a lot harder when you don't know how it works. So I started trying to figure out, you know, what Webpack was doing under the hood, and I found a ton of stuff on how to set up my config file, but not really very much about what it was actually doing. So, I started doing some research, and today I want to share with you what I've learned. * I want to do this because I think that we're all better and happier developers when we understand our tools, so the idea here is going to be to demystify what Webpack is doing when it creates an asset, cause that's going to make it easier for you to understand what's going on when you're debugging through a Webpack asset, or trying to figure out why a build failed, and it's going to lay the foundation for being able to understand all of the fancy things Webpack can do.
So today, we're going three questions: What is Webpack? What problems does Webpack solve, and How does it work? Let's start with what it is.
So, as Webpack describes itself, it is a module bundler that takes modules with dependencies and emits static assets representing those modules. When I first heard that I had no idea what any of that meant, so the way that I like to think of it is that Webpack takes stuff that is nice for you to write and turns it into stuff that is nice for your browser to read. Let's talk about what that means.
So, easy for you to write means that it's easy for humans to read and write. We have pretty formatting, we have readable variable names, our code is separated into multiple files so that we can separate it logically. It might not even be in a language that the browser can understand. ES6, also known as ES2015, is the new specification for JS, but browsers haven't implemented a lot of the new features yet. Some people don't even want to write in JS, they want to use a language that transpiles to JS, like CoffeeScript, or TypeScript, or Clojurescript. * But what's nice for your browser to read is old-school JS, right? ES5, the standard stuff. We want to make fewer requests, which is great for bad internet connections, and we want to have smaller responses, right? We don't need whitespace or pretty variable names, the browsers don't care about that, only we do. And, as you'll be able to see, Webpack is going to help us do those transformations.
What problems does Webpack solve?
The first one is namespacing. In JS, variables that are declared outside of functions are global, and that means it can be really easy for our global namespace to get cluttered, and that means it can be difficult to keep a good separation of concerns.
The second problem is that files need a lot of work before they can go to the client. We have to do transpiling, we have to uglify our code, we have to minify our code, there's lots of stuff we want to do change it, so that by the time it gets to the browser, it's in the state that we want.
And the third problem is how to send the code to the browser efficiently. We have a ton of files, but if we had to put in one script tag per file, that would be insane. First of all, it's slow, and second, it's really hard to manage our order dependencies. We'd have to make sure our scripts are requested in the right order, and that can be really difficult, so it would be great if we didn't have to do that.
I want to note obviously, that Webpack did not solve these problems first! I'm not getting up here and saying, "Webpack does something that no one else has done before!" Gulp and Grunt and Browserfiy or some combination of those have been amazing. My company is a Rails app on the backend and we used the asset pipeline to do this, and that worked great when we had it - I'm not bashing those. But Webpack kinda does it all at once. I think it does it more elegantly than some of the other ones, and it turns out that its implementation is actually really simple under the hood - so simple that an entire Webpack asset will be able to fit on one side later, as you'll see, and I think that's really cool. (Thing I should have said: I oversimplified pretty much everything in this talk and lots of what Webpack does is not simple! This is not meant to trivialize the amazing work its maintaners have put into it. It is meant to show that the core Webpack function is not difficult to understand.)
So, the fun stuff: how does Webpack actually work? How does it do its magic? Before I tell you how it works, I want to talk about how I thought it worked when I was first starting out. I thought we have a bunch of files, represented by these little blocks,
And we have dependencies, * which we're going to represent with these little bars at the tops of our files. So our orange file depends on our yellow file, * and our purple file depends on our orange and green files, * our yellow file depends on our green file, and so on.
And as far as I knew, the Webpack wizard just kind of smooshed them all together into a single plain JS file that we could send to the browser. And if you had asked me, "How do you think it works?"
I would have said, "Well, it's probably doing all of these crazy calculations to figure out what order the files should go in based on their dependencies and then it's concatting them all together." That's not crazy, right? We have other tools that do that, Grunt and Gulp do something pretty similar.
It turns out that's not how Webpack works at all! So if you are thinking, "That's what I used to do, that's probably what this does," just forget about it so we can talk about how it actually works. Cool.
So to start out, Webpack takes the path to a single starting file. * If you've used Webpack at all, that's going to be your entry file, that's what it's called, and it's going to be the top of your app, the thing that's going to kick everything else off. And based on that starting file, Webpack is going to make an asset for you in four steps. * First it's going to find dependent files, it's going to apply loaders, it's going to implement our module system, and it's going to put it all together and make the final asset, and if you dont know what those mean yet, that's cool, we're going to go through them right now.
So let's talk about finding dependent files. * So the first thing Webpack is going to do is it's going to make an array that's going to keep track of our dependent files. * We're going to start at our entry file, * and we're going to look for dependency declarations. Those are going to be requires or imports. Webpack can support multiple kinds of module systems - AMD, CommonJS, ES6, and we'll talk about that a bit later, but for now all you have to know is that it looks for those dependency declarations. * And then for each dependency, we have to validate the path, we have to make sure it leads to an actual file or actual node module that exists, you know, you didn't mistype the path, and you've actually installed the node module. And if it leads to a valid path, * cool, we're going to add that file to our array. * And we're going to remember this array for later cause it's going to be really important.
The next thing we're going to do is apply our loaders. A loader is something that you've probably read about, there are lots of loaders for lots of different things, but at their heart they're really simple: a loader is just a function that takes in source code, makes changes, and returns the new code. That's it.
This is the JSON loader, it's available on NPM, and as you can see, all it does is take in a source, and it stringifies the code into JSON and returns it. This is a loader that people use, it's super simple.
More about loaders - where do they come from? They're on npm - anybody can write them. Webpack itself uses a lot of internal loaders and has published a bunch of external loaders that you can use, or you can write one to do whatever you want. * How are they used? Well, we tell Webpack which loaders to apply in our config file, and they can be applied selectively with regex. My company used to work in Backbone, and we also worked in CoffeeScript, so we have a ton of CoffeeScript files, and a ton of plain JS files. We don't want to transpile our plain JS into plain JS again, right? That'd be a waste of time. So we say, "Only apply the CoffeeScript loader to the files that end in .coffee." And you can chain loaders together in whatever order you want to get the result that you need. * And so all the things that we talked about before that we want to do - our transpiling, our uglifying, our minifying, all of those things happen through loaders. You can also add polyfills through loaders, you can do whatever you want, like if anyone uses eval, throw in a giant ascii art that says NO DON'T DO THAT GO REVERT THAT COMMIT. You can do anything.
And then Webpack is going to implement the module system. To talk about that, we're going to take a little detour and talk about CommonJS. Webpack supports multiple systems. They all kind of work the same way under the hood, but we're going to use CommonJS to get the idea of it. The principles are going to be the same, just the syntax will differ across different ones.
So, what is CommonJS? * CommonJS is a system of writing JS that allows files to declare dependencies on each other by importing and exporting modules from those files. In this system, we have one file per module, and if I say file or module, I'm talking about the same kind of thing. * So we import modules by requiring them by name or by path, * and we export modules by assigning them to a variable called module.exports. We don't have to define module.exports in our code, we just assume it's available to us, and we attach it. * The important thing to understand is that CommonJS is a standard, it's not a library - you don't import CommonJS code. It's a set of rules that says, "Hey if you write your code this way, things will how you expect," so we write our code following the CommonJS system, and Webpack makes that system work by implementing the standard.
Cool, so how does it actually do that implementation? Let's start with an example. Let's say we have two modules, square.js and hypotenuse.js. Hypotenuse is going to be our entry file, and it depends on the module square. In order to implement our module system, Webpack has to do two things to these files. First of all, * it has to wrap each file in a function. That function is going to take in two arguments, module and a function called webpackRequire that Webpack defines and that we'll talk about later. * Then, if you remember that array, we can note that hypotenuse is going to be at index 0, and it's going to be at index zero because it's our entry file, and our entry file is always going to be the module at the first index in the array. The other ones can be in sort of whatever order we want, and so square is going to be at index 1.