Special treatment for package.json resolution and exports?

@ljharb

📗 API Reference Docs Problem

Version: 14.2.0
Platform: any
Subsystem: esm

Location

Section of the site where the content exists

Affected URL(s):

https://nodejs.org/dist/latest-v14.x/docs/api/esm.html

Problem description

Concise explanation of what you found to be problematic

With the introduction of pkg.exports a module only exports the paths explicitly listed in pkg.exports, any other path can no longer be required. Let's have a look at an example:

Node 12.16.3:

> require.resolve('uuid/dist/v1.js');
'/example-project/node_modules/uuid/dist/v1.js'

Node 14.2.0:

> require.resolve('uuid/dist/v1.js');
Uncaught:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/v1.js' is not defined by "exports" in /example-project/node_modules/uuid/package.json

So far, so good. The docs describe this behavior (although not super prominently):

Now only the defined subpath in "exports" can be imported by a consumer:
…
While other subpaths will error:
…

While this meets the expectations set out by the docs I stumbled upon package.json no longer being exported:

> require.resolve('uuid/package.json');
Uncaught:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './package.json' is not defined by "exports" in /example-project/node_modules/uuid/package.json

For whatever reason I wasn't assuming the documented rules to apply to package.json itself since I considered it package metadata, not package entrypoints whose visibility a package author would be able to control.

This new behavior creates a couple of issues with tools/bundlers that rely on meta information from package.json.

So right now, packages that do contain information to be consumed externally (e.g. by bundlers) in their package.json have to add the package.json to the pkg.exports field.
On the other hand bundlers/etc should then handle the case where a dependency doesn't export the package.json gracefully, since it might very well be the fact that a given package doesn't need to export any bundler meta information (and otherwise almost all packages on npm that could ever be used in a react-native project would have to add package.json to their exports).
If bundlers however handle a missing package.json exports gracefully, I see the risk that many modules that currently rely on their package.json simply being externally consumable without additional effort might suddenly behave in odd ways when the meta information from package.json is no longer readable by bundlers (and no error is thrown).

Examples where this issue already surfaced:

Now the question is how to move forward with this?

One option would be to keep the current behavior and improve the documentation to explicitly warn about the fact, that package.json can no longer be resolved unless added to exports. EDIT: Already done in 1ffd182 / Node.js v14.3.0
Another option would be to consider adding an exception for package.json and always export it.

I had some discussion on slack with @ljharb and @wesleytodd but we didn't come to an ultimate conclusion yet 🤷‍♂️ .

I would like to work on this issue and submit a pull request.

cc @nodejs/modules-active-members

other option would be to consider adding an exception for package.json and always export it.

To me this seems like a great solution. Having the package metadata is an awesome ergonomic feature of the current setup, and having module authors explicitly have to opt in would be a huge burden across the community. To me it seems like we would need a concrete reason not to have this exception. Can anyone think of a reason to make this case?

@wesleytodd

@wesleytodd I think it just comes down to what is public/private still. People putting public data into a a package.json for things like tools to consume isn't really an issue. People putting configuration data like secrets is more the concern. I imagine it would still be able to be censored if people re-wrote the import still.

However, I'm unclear on the privacy model here since the benefit seems largely to be around bundlers which wouldn't run with the same constraints since they are ahead of time tools and general thought to access things in a more permissive manner than 2 independent modules with mutual distrust. It seems the problem is in part that these tools are using APIs that make them act as the same level of trust as any other module and other packages when they upgrade are removing permissions to view the package.json data (even if by accident). I think the concrete discussion here is if people should have to opt-out of package.json to avoid an accident prone workflow which is the inverse of all other resources in the package.

A different option since there is a specific use case that seems to need this is to have a flag of some kind for these ahead of time tools. Either to require.resolve which looks to be the cause of issues above, to node via CLI/ENV, or something else. I do think providing an exception would make things just work, but somewhat go against the privacy intent of "exports".

My main concern with exposing it via exports is that there’s two options:

Make it non-configurable. ./package.json always has to map to ./package.json.
Tools that use require or import to load metadata may get custom files because a package decided to remap ./package.json.

The first option would force every package to treat its metadata file as a public API. Some users eslint config reuses what I put into my package.json#eslint section? Well, it’s exported so how can I blame them.

The second option means that tools may not actually get the metadata when they think they’re loading the metadata. It could be argued that it only affects “weird” packages but given a sufficiently large number of users that know about this “trick”, I can totally see people use it.

I think bundlers shouldn’t use require (“load code for this environment”) to load metadata. So I’d rather have a new API to load the package.json that belongs to a specifier/referrer combination, exposing logic we already have in the loader. That API could be used by bundlers etc to cleanly get the metadata without having to hijack require.

I think the primary issue is that there's no way besides require.resolve to resolve the package.json for a package. Even path.join(require.resolve(pkg), 'package.json') won't work because some packages might have their "main" resolve to a subdirectory.

Here's the recommended way to do this with ES modules:

import { readFileSync } from 'fs';
(async () => {
  const pkgPath = await import.meta.resolve('pkg/')
  console.log(pkgPath);
  console.log(readFileSync(new URL('package.json', pkgPath)).toString());
})();

The above also simplifies with TLA of course.

Currently the above only executes with --experimental-import-meta-resolve.

@nodejs/modules-active-members I think we should discuss unflagging this feature.

I also want to clarify that the problems I have seen in the wild were always just about require.resolve('pkg/package.json') in order to then load that file from the filesystem. I didn't see anybody trying to directly require('pkg/package.json') to really load the json data as a module.

@guybedford

@guybedford import.meta.resolve('pkg/') would fail if the package didn't have a main/dot, wouldn't it? or, if the ./ was mapped to a different directory, like ./src?

@ljharb

@ljharb no, the trailing / is specially specified to allow resolving package boundaries.

@guybedford

@guybedford require.resolve('es-get-iterator/') in latest node throws Package exports for '$PWD/node_modules/es-get-iterator' do not define a './' subpath. I would expect import.meta.resolve to behave identically, so the same capabilities exist in both CJS and ESM.

iow, import.meta.resolve only solves for ESM, not CJS, so it's not a solution to this problem.

@ljharb

@ljharb yes, because trailing slashes in CommonJS still apply extension searching, which the ESM resolver does not, which is a fundamental difference between the module systems.

@bmeck

People putting configuration data like secrets is more the concern (@bmeck)

This is a not a concern. Using exports to hide secrets is not ever a reasonable solution anyway.

Make it non-configurable. ./package.json always has to map to ./package.json. (@jkrems)

This is what I was thinking as well. Seems perfectly reasonable to enforce this constraint.

The first option would force every package to treat its metadata file as a public API. (@jkrems)

It already is. This is not a change in the ecosystem as it is today. Every file is part of the public api, and needs to be treated as such. If projects choose not to strictly follow semver, that is a different issue.

Here's the recommended way to do this with ES modules: (@guybedford)

We can also use hacks around require for this. The point is that the most ergonomic way is also popularly in use, so should be added as an exception to the exports spec, even if there are other ways around it.

I didn't see anybody trying to directly require('pkg/package.json') to really load the json data as a module. (@ctavan)

I have seen this many places. Although I am not going to spend the time now collecting references since I don't think this should be the primary focus of the discussion, if it become a key point I am happy to dig for them.

I think allowing censorship is necessarily good and wouldn't feel comfortable with ./package.json always mapping to ./package.json would not seem to allow that. In particular, people do set environment variables in their ./package.json when deploying things in various environments that support it. Environment variables might be able to be removed at runtime via process.env but if the deployment does not have a writable filesystem they could not censor their package.json. I am not really here to judge if this workflow is a good idea, just to note that it does present a concern for myself.

I think allowing censorship is necessarily good and wouldn't feel comfortable with ./package.json always mapping to ./package.json would not seem to allow that.

Making it more complicated for everyone seems to strongly outweigh this. I know the point of engines is to help package authors have more explicit contracts with their users, but if this just breaks everyone's tooling it is a net negative to the community, especially for the package authors who now have to deal with this added complexity.

I am not really here to judge if this workflow is a good idea, just to note that it does present a concern for myself.

Do you have examples of this type of workflow? The app developer use case is not what I was considering at first, so maybe there is some common practice I have not seen like this. If so we would not want to break it. That said, I feel like the current state before exports had no restrictions on this, so I am not sure how we would be making anything worse.

refactor: exports for types and package.json

@GeoffreyBooth

I'm not sure what the use-case is (if it exists) for restricting access to package.json via Node.js' module importing facilities. But the ship has sailed.

That said, to avoid breaking things, it would be helpful for tooling authors if Node.js would expose an API like:

// node:util

getPackageJson(resolvable: string | URL): Promise<unknown>;

getPackageJsonSync(resolvable: string | URL): unknown;

...which would fulfill with the contents of the "closest" (sibling or ancestor) package.json to resolvable (for a rough userland equivalent, see find-pkg-up).

A simpler and easier-to-implement API would expect a resolvable package name or path/URL to package directory, and resolve with the contents of its package.json--without otherwise crawling the directory structure (rough userland equivalent: read-pkg).

Should probably throw/reject if the operation fails.

Right now, adding package.json to exports is just boilerplate that library authors need to remember to add. Which is crap, of course. 😄

cc @GeoffreyBooth @wesleytodd @ljharb

Yes, a new API could work. Up above I suggested getPackageMetadata, as this could be added without any breaking changes to "exports" and has the additional benefit of being able to use the existing internal cache that Node keeps of all the package.json files it’s already read for all modules.

Here's the recommended way to do this with ES modules:
import { readFileSync } from 'fs';
(async () => {
  const pkgPath = await import.meta.resolve('pkg/')
  console.log(pkgPath);
  console.log(readFileSync(new URL('package.json', pkgPath)).toString());
})();
The above also simplifies with TLA of course.

Currently the above only executes with --experimental-import-meta-resolve.

@nodejs/modules-active-members I think we should discuss unflagging this feature.

// Nodejs v22.6.0

import.meta.resolve('@babel/runtime/')
// Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './' is not defined by "exports" in /Users/.../node_modules/@babel/runtime/package.json imported from ...

🥲

@GeoffreyBooth

@GeoffreyBooth Should we open a new issue for adding getPackageMeta? Or is there no point unless someone is going to send a PR?

fix: export package.json required by react-native and bundlers (uuidj…

Navigation Menu

Repository navigation

Special treatment for package.json resolution and exports? #33460

📗 API Reference Docs Problem

Location

Problem description

113 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Special treatment for package.json resolution and exports? #33460

Description

📗 API Reference Docs Problem

Location

Problem description

Activity

ljharb commented on May 18, 2020

wesleytodd commented on May 18, 2020

bmeck commented on May 18, 2020

hybrist commented on May 18, 2020

ljharb commented on May 18, 2020

guybedford commented on May 18, 2020

ctavan commented on May 18, 2020

ljharb commented on May 18, 2020

guybedford commented on May 18, 2020

ljharb commented on May 18, 2020

guybedford commented on May 18, 2020

wesleytodd commented on May 18, 2020

bmeck commented on May 18, 2020

wesleytodd commented on May 19, 2020

113 remaining items

boneskull commented on Jun 14, 2024

GeoffreyBooth commented on Jun 15, 2024

yangmingshan commented on Aug 21, 2024

boneskull commented on Sep 11, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions