Web APIs in Node.js Core: Past, Present, and Future by Joyee Cheung | JSConf EU 2019
Articles,  Blog

Web APIs in Node.js Core: Past, Present, and Future by Joyee Cheung | JSConf EU 2019


You may have opinion at JSConf Europe last
year. And this morning, there was a great introduction
to workers. It is exciting times again for animations
in serverside JavaScript run times. So, here I am to talk about our recent effort
to bring a new set of APIs into Node Core. This may be an opportunity for us to really
think about the JavaScript API service in the ecosystem and create a more universal
developer experience across the different platforms. So, let’s finish this that is right, I’m Joyee,
I live in Hangzhou. I worked on Gaia, and I’m a member of the
Node steering committee. On Twitter or GitHub, Joyee Cheung handle. Some of you may be aware of this effort of
bringing more web APIs into Node Core. But maybe some folks are just curious because
they hear things that are interesting. So, in case I am using some lingoes that only
a very few people understood. So, what does it mean when we talk about bringing
web APIs into Node Core? of well, we are basically talking about adding browser APIs to the Node.js
runtime as buildings. So, you can run them on the server side without
using npm install. So, in this talk I will try to be objective
and summarize how far we have come in this journey, where we are, and where we are going. On a teeny tiny side note, Node.js is silent. So, like many others, I will try to use the
correct pronunciation in this talk. Here is a mental picture of how JavaScript
in the wild came to be. In the beginning before Node came along there
were two kinds of APIs that came with the runtime for JavaScript developers. There were some JavaScript builtins like date,
regular submissions, errors that were part of the language. It was created by system 9 and implemented
by engines. There was a host API. And at that time hosts were basically just
browsers. These APIs were implemented by the browsers
and are more or less run by organizations like W3C. And then Node was created a started to take
off on the server side. Initially it included some of the web APIs
that were already present in the browsers. Or like at least some APIs that look like
what you’ll find in browsers. But Node also introduced several different
APIs for doing certain things on the server side like instead of even target in the browsers. Because the semantics of the equivalent APIs
on the browser side may not make sense for servers. Like, for example, there isn’t one DOM tree
in this execution context for events to bubble through on the server side. So, it does not really make sense to implement
part of the event target specification in Node. So, as time goes by, browsers have developed
and synthesized more APIs in the browser to empower web developers. While Node also developed several equivalent
APIs on the server side with a didn’t design. Because of the popularity of the Node.js runtime,
these home-grown APIs gradually became the defacto factors for the server side run times. But obviously no one likes to remember two
sets of APIs for doing basically the same thing. So, over the past few years people have been
sending feature requests to Node to bring the two platforms closer together. For now, it usually means making the runtime
more compatible with browsers. So, Node did evaluate and implement some of
the requests and it is looking into more APIs from the web. So, we currently have different types of web
APIs implemented in Node Core. So, some of them are more localize the with
the browsers. And, for instance, we have methods in the
browsers, and this is the living standard of the timers in the HTML specification. In Node we also have a set of timer APIs,
that’s similar, but the implementation does not strictly follow what the browsers do. For example, it returns an object while browsers
usually return a number instead. The other type of web APIs we have in Node
Core are the ones implemented with an existing specification in mind and behave mostly the
same as the APIs that you will find in browsers. For instance, the new URL implementation in
Node Core was developed specifically according to the URL standard. This means we did look at the spec test when
we implemented them. And we also have tests. So, how do we know how close our implementations
are to the ones in the browser? We run a subset of the web platform tests
which are tests which are on browsers and other implementations. So, to see the current status of the performance
in Node, you can look at the status files under tests, the status folder of the Node
project. There are a few pretty selfexplanatory JSON
files documenting the web standards implemented in Node. So, this one is not technically a web API
addition, but I wanted to mention that there is now another type of web standards implemented
in the core. We have collaborated with the what is the
HTML specification and the first implementation of JSON modules has landed in the master branch
as part of our experimental ECMAScript module. So, Node implemented first. So, here we have the web APIs implemented
in Node Core that can be alternatives to certain home-grown Node build teams. Legacy methods under the URL and the string
built in module can be replaced by the new URL and the URL search params classes. And searching and coding and decoding methods
of buffer and string decoder can be replaced by text decoder and text decoder as well. And there is the performance API which, for
example, can be used to replace certain timing methods in the process object. In addition, we also have some APIs that are
not exactly replacements for existing Node APIs. But are similar enough in many use cases that
it is reasonable to consider them as alternatives. Be sure to read the documentation if you want
to rewrite your code with these APIs. For example, cue microtask can be used to
cue a microtask which will be run asynchronously which will be used to replace the process
if you do not have strict requirements about the timing you want your task to be run. There is also web worker which spawns threads
in Node. And it may be used to replace child processes
if you are only looking for a way to upload and you don’t necessarily need processes. So, as of Node 12, we have several web API
implementations that are now stable and can be used in production. Those are listed on the left here. And these are available as globals. So, you don’t have to require a builtin to
get hold of them. These are also covered by the web platform
task in Node Core. There may be some additional extensions in
these APIs. There’s some minor unspecified behavior differences. But at least those are fairly limited, and
we are aware of them. We also have a few APIs that are still experimental
which means there may still be breaking changes in the future. These are currently placed under builtin modules
and not on the global object yet. The current experimental web APIs that we
have are workers and performance time being API. But they also differ significantly from what
you would get from the browsers. And we do not run web platform tests for them
yet. So, watch out. So, there is also an implementation of WebAssembly,
the JavaScript API of WebAssembly that we get for free from V8. But the web API of it has not been implemented
yet. Other than existing APIs, there are a bunch
of others that are hunting the issue tracker. Oops. There are still under active discussion. They’re web strings which are the foundation
of web specifications. But then we already have Node strings in Core
which are also the foundation of many existing Node Core APIs. In case you didn’t know, there are many different
types of strings in Node. There’s string number one. Which has several issues. So, they now introduce string number two. But then they also had some issues. And then actually introduce string number
three which still had issues. So, there’s now a new implementation of strings
called Bob which is under development for some time. And will hopefully solve all our problems. Anyway, this is not a string talk. You can read the documentation if you want
to learn more about them. So, with all the strings, you can imagine
how complicated it would be to bring yet another string into Node Core. And there is also Fetch. Which is probably the most requested web API
in Node Core. We just had a session about it this week at
the Open JS Coverage Summit in Berlin and there is now a new work in progress pull request
to bring it fresh into Node Core. Yay! [ Applause ]
So, we have to talk about the history of these web APIs in Node and what may be coming next. So, why exactly are we doing this? One obvious run is with a common API surface,
there is less cognitive burden for developers. We could their documentation, tutorials, tooling. Instead of developing and maintaining a separate
set of educational resources. This is especially important for just beginners. At the moment they have to choose between
Web APIs and Node APIs when they are just getting started. With a common API service, beginners can be
less distracted learning about basics. There are still differences between the two
platforms. But it will be less intimidating when they
already learned a bit more about these APIs. So, another reason for web APIs into Node
core, we have more containers compared to more npm modules. This is open source and it’s natural that
contributors come and go. In Node, even when the existing maintainers
of a specific builtin module shop less often, we have an open governance and an effective
nomination and onboarding process to bring new contributors into the team. Compared to regular npm modules, this kind
of maintenance story fits better with the Web APIs that are designed as builtins for
the host environment. So, if you have been paying some attention
to this topic, you may be aware that it takes an extraordinary amount of time for this API
additions to be accepted into Node Core. Here I will lay out some of the reasons why
there are several requests that keep showing up in the issue checker but have never really
gone anywhere. So, everything I’m going to talk about later
are in the context of the consensusseeking model of Node Core. So, Node Core is operated under the consensus
from over 100 core collaborators. These are contributors who have write access
to the repository. For every technical decision, any one of these
100 more than 100 Node Core collaborators creates an objection. If consensus cannot be reached within the
collaborators, it may come down to a vote among the members of the tech committee who
are a subset of the collaborators. But we usually try to avoid voting. We also take community feedback into account
even if it’s not from someone who has committed into Node before. So, here are the common arguments against
adding Web APIs into Node Core. There is still, to some extent, a small core
philosophy within Node Core. It’s about providing only the basics functionalities
in Core and empower users to implement user modules instead of building our own opinionated
APIs that may become a compatibility or maintenance burden. This philosophy has been broken several times
in the past. But at least as far as Web API goes, we are
still mostly just trying to expose the existing functionalities through a different API services. This is not exactly an idea that’s welcomed
by everybody either. Especially when the web API may also lack
features that do not make sense for browsers. Oh, yeah. And like sometimes they may be necessary for
servers. So, one alternative to adding these APIs in
Node is to release them as official modules. Theoretically for modules that are maintained
under the Node organization, it will be possible to have a maintenance story similar to the
one that Node query has. But some may also argue that it is easier
to optimize if it’s done in Core because it can use certain internal APIs. Or sometimes it may just not be technically
possible to implement something without access to internals. A part of the philosophical concerns, apart
from those, there are concerns about the design of the web APIs. They’re more than just a bunch of interfaces. Behind the Design of these APIs, there is
just a very different context. For instance, the browser has a very different
security model. When you fetch an API endpoint, for example,
at a script and you want to send credentials like cookies along with the request. Fetch as implemented according to the specifications
should follow the crossorigin resource protocol and check the access control allowed header
in the HTTP response before invoking that with the data. So, that scripts cannot untrusted scripts
cannot steal your cookies when the server is not aware of them. Somewhere in Node, there isn’t really a concept
of origins. At least for now. So, these are loaded from your local file
system and are just trusted by default within current security model of Node. If you perform the request using the existing
HTTP request method in Node, this security policy would have to be implemented by the
users. Well, if they do want them. It is not impossible to implement something
like this in Node. But this may just be confusing for most users
because then we’ll have two conflicting security models in Node. So, when we look at the Fetch specification,
the interface itself is just the tip of the iceberg. There are many implications under the surface
of the API like course origin, consent and security policy, caching interop with service
workers and like potential management. So, some of this may make sense for now. Some of them don’t. If we only implement part of the API that
we think makes sense for Node, we may confuse our users more because this will bring another
kind of platform compatibility headache to everyone. There is also another type of concern. The ecosystem has come to depend on a lot
of existing infrastructure in Node. And this may differ significantly from their
web equivalence. For example, we have different interfaces
to do stringing and to emit events. As usual, there are even more differences
in the underlying design of these infrastructures. When implementing web APIs in Node, we also
need to decide whether we want to introduce the web infrastructure into Node Core or base
the higher-level APIs on the existing Node infrastructure or just use some instruction
layer instead. We will also need to figure out the interop
between these abstractions for other existing modules. And this work just takes a lot of time. So, there are a lot of open questions to answer. A lot of decisions to be made. And this, you know, just takes time in the
current this is the model in Node Core. But how do new Web API actually ended up being
added to Node Core these days? It usually starts with a feature request opened
in the Node.js/node repository. To actually make progress on the request,
someone, or some group of people, need to step up and start a prototype. They do not need to be Node collaborators. It could be anyone who are willing to invest
their time in the work. Typically, they would create a fork. Either as a personal fork or as a fork under
the Node organization. And they will hack together an initial invitation
and then send a pull request back to the main repository against the master branch. At these stages there may be objections coming
from collaborators or the community. Sometimes the proposal just gets stalled and
closed. For example, this is the current status of
the feature request for web crypto. Sometimes, but rarely, there are no objections. Or these objections get resolved either through
discussion or through voting. Either way, someone must be interested enough
in this feature to get it through the consensusseeking model. Then the initial implementation may get merged
into the master branch. So, once the feature is merged in master,
we’ll start iterating on this. Fixing bugs, optimizing. At a certain point, depending how visible
it is, we may start with the subset of web platform tests with it and collaborate with
the web platform tests upstring as well as the authors to improve this spec and the test
suite. It is also typical to eventually expose these
interfaces to the global object. But this is depending on various complexities. At this phase, the feature is still in experimental
status. Depending on the release schedule, this feature
may be released to users while it is experimental. And it may get updated in the release branch
with patches back ported and the master branch. So, this is the current status of web workers
and the performance timing API. Eventually this feature would be, or it is
supposed to be, moved out of experiment and becomes a stable feature. This is the current status of the URL and
encoding implementation. So, here’s a quick summary. Node query APIs have diverged from the web
APIs because they were designed for very different use cases. But more and more web APIs have now been added
into Node Core. We have work would through the existing web
APIs in Node Core and their status. Then we looked into the challenges and the
workflow of bringing for APIs into Node Core in the future. So, finally, we are starting an open standards
initiative in Node to collaborate more with senders and other implementations. As we said earlier, there are many questions
to answer and it takes a lot of energy to find the answers. If you are interested, please get involved. Thank you. [ Applause ]

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *