The npm Blog

Q: Hi! Can you state your name and what you do?

A: Ahoy, I’m Alistair Brown and I’m a lead front-end engineer at ShopKeep, primarily focusing on our BackOffice app, which enables more than 23,000 merchants the ability to manage their business operations from anywhere. With ShopKeep’s BackOffice, business owners manage everything from inventory management to accessing customized reporting specific to their business, so this a vital component to the ShopKeep product.

How’s your day going?

It’s going pretty well — my team is mostly front-end focused, so we use npm many times every day. We’re currently prepping some dependency upgrades to make sure we’re ready to jump on the newest version of React (v16) when it’s released. It’s important to us that we stay up to date, getting the benefits of optimization, bug fixes, and new tools.

What is your history with npm?

I’ve used npm in a few jobs to manage dependencies as well as publish some personal projects as modules on the npm registry. A few years ago, I was given an iKettle for Christmas and spent much of that holiday creating an npm module so I could boil water remotely using JavaScript — not a very popular module, but a lot of fun to build! More recently, I’m excited about the release of npm5. We’ve just rolled it out across our developer machines and onto the CI servers, and we’re really seeing the benefits.

What problem did you have that npm Orgs helped you fix?

The main problem we wanted to solve was being able to share code between upcoming projects. The npm Organization setup allowed us to create our own modules and keep control over who could access them. Having private packages within the organization has allowed us the freedom to create a versioned module, but without the fanfare of opening it up to the world.

Can you tell us a story about a specific package you wanted to make that private packages really enabled you to do?

At Node.js Interactive Europe last year, I’d been inspired by a talk by Aria Stewart, called “Radical Modularity.”  With the concept of “anything can be a package” in mind, we first started small with our brand colours (JSON, SASS, etc.) and configs. I explored pulling these components out of our code base into separate modules as part of a Code Smash (our internal hackathon). This allowed us to test the waters. As we mainly write in React and had created a number of generic components, there were lots of packages we wanted to extract. In the end, we started modularizing everything and have even extracted out our icon assets.

How’s the day to-day experience of using private packages and orgs?

It’s super easy. Day to day, there’s no difference from using any other package from npm. Once the code is out in a module, we get to treat it just like any other piece of third-party code. There had been a little bit of fear that the scope prefix would cause problems with existing tooling, but so far there have been no problems at all — a great feat!

Does your company do open source? How do you negotiate what you keep private and public?

We have several repositories of useful tools that we’ve open-sourced on GitHub, hoping these same tools could be useful for other developers. These range from shpkpr, a tool we use for managing applications on marathon and supporting zero-downtime deploys, to our air traffic controller slack bot, which helps us coordinate deployments to all of the different services we run. Open sourcing a project is an important undertaking and we always want to make sure that we have pride in what we release. Using private packages gives us that middle ground, allowing us to separate out reusable code but keep it internal until we’re ready to show it off.

To people who are unsure how they could use private packages,how would you explain the use case?

We started off wanting to get code reuse by sharing code as a package. Making private packages allowed us to be more confident about pulling the code out, knowing it wasn’t suddenly visible to the world. Our ESLint config is a nice example of a small reusable module we created, containing rules which enforce our internal code style. Splitting this out allowed us to apply the rules across multiple codebases by extending from this central config. Later, we added a new rule, and having immutable packages meant we could cut a new version and stagger the updates to dependent projects. Really, we get all the benefits that you’d expect from using a third-party package, while keeping control of updating and distribution.

How would you see the product improved or expanded in the future?

With the rapid development of the JavaScript ecosystem, it can be hard to keep up to date with new versions as they come out. The `outdated` command helps towards this, but anything that can be built to help developers stay on the latest and greatest would be really handy.

Would you recommend that other groups or companies use Orgs?

Definitely! It’s not just so you can use private packages, it’s also a great way to group your modules under a brand and avoid naming clashes. With the recent pricing change making organizations free, there really is no excuse for open source groups and companies not to publish their modules under an org.

What’s your favorite npm feature/hack?

I’m a huge fan of npm scripts. It’s allowed us to provide a single interface for useful commands and avoid forcing developers to install global dependencies. From building our application with gulp, upgrading tooling with a shell script, to publishing multiple modules with lerna, the developer experience stays the same by hiding the internals behind the simplicity of `npm run`.

What is the most important/interesting/relevant problem with the JavaScript package ecosystem right now? If you could magically solve it, how would you?

Building a package manager is a difficult problem to solve and it’s great to see so much engagement in this space. Deterministic installs is something that has been really important, so it’s good to see this in npm5 and yarn. I think the natural next step is a client-agnostic lock file. When there are multiple developers on a project, making sure that we can replicate a development environment across all dev machines and CI servers is very important — we use a shrinkwrap file (moving soon to a package-lock.json!), but those are npm-specific. Reducing that barrier between different packaging clients should allow for more experimentation on new approaches and optimisations.

Any cool npm stuff your company has done that you’d like to promote?

No — we’re just happy users!

This piece is a part of our Customer Convos series. We’re sharing stories of how people use npm at work. Want to share your thoughts? Drop us a line.

Q: Hi! Can you state your name and what you do?

A: Gregor, community manager at the open source project Hoodie, and co-founder at Neighbourhoodie, a consultancy; we do Greenkeeper.

How’s your day going?

I just arrived in Berlin and life is good!

What is your history with npm?

Love at first sight! I’m a big fan of all of what you do! npm is a big inspiration for how a company can self-sustain and be a vital part of the open source community at the same time.

What problems has npm helped you fix?

We love small modules. Because of the maintenance overhead that comes with small modules, we created semantic-release and eventually Greenkeeper. Here is an overview of all our modules.

The `@hoodie` scope allows us to signal that this is a module created by us, the core hoodie team, and that it’s part of the core architecture. I could imagine using the scope in the future for officially supported 3rd party plugins, too.

How’s it going? How’s the day to day experience?

Our release process is entirely automated via semantic-release so that we don’t use different npm accounts to release software. Technically, it’s all released using the https://www.npmjs.com/~hoodie account.

How would you see the product improved or expanded in the future?

Hmm I can’t think of anything… I’ll ask around.

How would you recommend that other groups or companies use npm?

I don’t see why companies would not use scopes. I think it’s a great way to signal an “official” package, in order to differentiate 3rd party modules from the community.

What’s your favorite npm feature/hack?

As a developer, I love that I can subscribe to the live changes of the entire npm registry. It allows us to build really cool tools, including Greenkeeper.

What is the most important/interesting/relevant problem with the JavaScript package ecosystem right now? If you could magically solve it, how would you?

For me, a big challenge is to fix a bug in a library like `hoodie` that is caused by dependency (or sub dependency, or sub sub…). It would be cool if there was a way to easily set up a development environment in which I could test the bug on the main module, but working on the dependency, until I have it resolved. That would make it simple to release a new fixed version of the dependency and the package.json of the main module to request the fixed version.

This is kind of related: let’s say I have a module A with a dependency of module B and B depends on module C, so it’s A=>B=>C. Now, if I fix C and release a new fix version, I cannot release a new version of A that enforces this new version, because it’s a sub-dependency. I’m not sure what the right approach to this problem is, but that’s one that’s bothering me related to npm.

Any cool npm stuff your company has done that you’d like to promote?

Greenkeeper.

It’s here!

Starting today, if you type `npm install npm@latest -g`, you’ll be updated to npm version 5. In addition, npm@5 is bundled in all new installations of Node.js 8, which has replaced Node.js 7 in the Node Project’s current release line.

Over the last year and a half, we’ve been working to address a huge number of pain points, some of which had existed since the registry was created. Today’s release is the biggest ever improvement to npm’s speed, consistency, and user experience.

The definitive list of what’s new and what’s changed is in our release notes,

but here are some highlights:

It’s fast

We’ve reworked package metadata, package download, and package caching, and this has sped things up significantly. In general, expect performance improvements of 20–100%; we’ve also seen some installations and version bumps that run 5x faster.

(Installing the npm website on our own dev environments went from 99 seconds using npm@4 to 27 seconds with npm@5. Now we spend less time jousting.)

Since npm was originally designed, developers have changed how they use npm. Not only is the npm ecosystem exponentially larger, but the number of dependencies in the average npm package has increased 250% since 2014. More devs now install useful tools like Babel, Webpack, and Tap locally, instead of globally. It’s a best practice, but it means that `npm install` does much more work.

Given the size of our community, any speed bump adds up to massive savings for millions of users, not to mention all of our Orgs and npm Enterprise customers. Making npm@5 fast was an obvious goal with awesome rewards.

It’s consistent

Default lockfiles

Shrinkwrap has been a part of npm for a long time, but npm@5 makes lockfiles the default, so all npm installs are now reproducible. The files you get when you install a given version of a package will be the same, every time you install it.

We’ve found countless common and time consuming problems can be tied to the “drift” that occurs when different developer environments utilize different package versions. With default lockfiles, this is no longer a problem. You won’t lose time trying to figure out a bug only to learn that it came from people running different versions of a library.

SHA-512 hashes

npm@5 adds support for any tarball hash function supported by Node.js, and it publishes with SHA-512 hashes. By checking all downloaded packages, you’re protected against data corruption and malicious attacks, and you can trust that the code you download from the registry is consistent and safe.

Self-healing cache

Our new caching is wicked fast, but it’s also more resilient. Multiple npm processes won’t corrupt a shared cache, and npm@5 will check data on both insertion and extraction to prevent installing corrupted data. If a cache entry fails an integrity check, npm@5 will automatically remove it and re-fetch.

It’s easier to use

With your feedback, we’ve improved the user experience with optimizations throughout npm@5. A big part of this is more informative and helpful output. The best example of this is that npm no longer shows you the entire tree on package install; instead, you’ll see a summary report of what was installed. We made this change because of the larger number of dependencies in the average package. A file-by-file readout turned out to be pretty unwieldy beyond a certain quantity.

It needs you

npm@5 is a huge step forward for both npm and our awesome community, and today’s release is just the beginning. A series of improvements in the pipeline will make using npm as frictionless as possible and faster than ever before.

But: npm exists because of its users, and our goal remains being open and flexible to help people build amazing things, so we depend on your feedback.

What works for you? What should we improve next? How much faster are your installs? Let us know. Don’t hesitate to find us on Twitter, and, if you run into any trouble, be sure to drop us a note.