HowToPublishAPackage

How to publish a package

This tutorial describes how to share an Open Pioneer Trails package with other developers. You can skip this document if you’re just working in the context of a single application.

Until now, this manual has only been concerned with building an application, possibly including a set of local packages. The result of building an application is always a set of static files (.i.e. .html, .js, etc.) that can be interpreted by a browser. This output is ideal for distributing a final application, but it is not suitable for sharing reusable parts of your application.

One of the key advantages of the Open Pioneer Trails client framework is the ability to scale up application development by sharing reusable application components as packages. Shared Trails packages are similar to “plain” node packages (in fact, they are node packages), but they come with the additional feature set you have learned already: services of a package are automatically started when needed, styles are automatically applied, etc.

There are some things to consider and a few steps to follow when indenting to share a package with other developers. We will first discuss the underlying concepts and then work through a specific example.

Concepts

Package Compilation

To prepare a package for publishing, it must be built first. Building a package produces a dist directory in that package’s source directory containing a compiled version of that package. We use the build-pioneer-package CLI tool to build a package.

This step is also called separate (package) compilation since packages are compiled independently before being linked into an application at a later point in time.
Publishing

We use our node package manager to publish the contents of the dist directory. Ultimately this means executing pnpm publish (opens in a new tab) or something equivalent.

The package manager will upload the compiled version of our package to the registry, which is the public registry of npmjs.com (opens in a new tab) by default (private registries such as Nexus (opens in a new tab) or Verdaccio (opens in a new tab) can also be used).
Package installation

After a package has been published, it can simply be installed in any project, e.g. by executing pnpm add. Our build tools automatically detect Trails packages in an app’s dependency graph. A package installed this way should work the same way as if it were developed locally.

Package compilation

Our CLI tool build-pioneer-package implements package compilation. It can be installed by adding @open-pioneer/build-package-cli (opens in a new tab) as a (dev) dependency in your project’s root package.json:

1
$ pnpm add -w -D @open-pioneer/build-package-cli

It is designed to be invoked from the source directory of a Trails package and reads its configuration from that package’s package.json and build.config.mjs (see Reference). After successful execution, it will have assembled a compiled version of that package in the dist directory (in that package’s source directory).

build-pioneer-package also performs some validation to enforce best practices and to catch common errors. By default, this validation is strict: all validation errors are fatal. This can be changed by configuring publishConfig.strict: false in your package’s build.config.mjs, which is convenient when first preparing an existing package for publishing. strict should be true in production!

For a properly configured package, the output of build-pioneer-package can look like this:

1
$ pnpm exec build-pioneer-package
2
Building package at <...>
3
Building JavaScript...
4
Generating TypeScript declaration files...
5
Building styles...
6
Copying i18n files...
7
Copying assets...
8
Writing package metadata...
9
Copying auxiliary files...
10
Success

As an example, this is how the dist directory of the layout-sidebar (opens in a new tab) package looks at the time of this writing:

1
dist
2
├── i18n
3
│   ├── de.yaml
4
│   └── en.yaml
5
├── _virtual
6
│   ├── _virtual-pioneer-module_react-hooks.js
7
│   └── _virtual-pioneer-module_react-hooks.js.map
8
├── CHANGELOG.md
9
├── index.d.ts
10
├── index.js
11
├── index.js.map
12
├── LICENSE
13
├── package.json
14
├── README.md
15
├── Sidebar.d.ts
16
├── Sidebar.js
17
├── Sidebar.js.map
18
├── styles.css
19
└── styles.css.map

Package compilation involves the following steps:

Compile all required TypeScript/JavaScript files to JavaScript.

We compile everything down to JavaScript for a stable distribution format. This way we do not depend on a certain version of TypeScript or specifics in the JSX syntax. This also gives us the (future) opportunity to apply optimizations (e.g. minify code).

Files that are not needed will be skipped (such as *.test.*). To decide which files are actually needed, your must configure your package’s entryPoints in the build.config.mjs. All modules used by one of your entry points (transitively) will be included in the compiled package; other modules will not.

Most packages will simply have a single entry point (e.g. index), or none at all. If your package defines services, your services module (usually ./services) does not need to be listed: it is always automatically an entry point.

This compilation step also ensures that packages used by your code are listed in the package.json as dependencies.
Create type declaration files (.d.ts) for TypeScript consumers.

This way users of your package can benefit from type hints if they’re using TypeScript. This step is enabled by default if your package contains a TypeScript source file. You can also force enable or disable this step by configuring publishConfig.types in your build.config.mjs.
Compile styles to CSS.

We compile all styles (CSS or SCSS) into a single CSS file. This step is skipped if your package does not define any styles.
Copy i18n files.

I18N files for supported locales (e.g. i18n/en.yaml) are copied to dist.
Copy assets.

Assets are “normal” files that are distributed alongside your code, for example images or fonts. These can be configured via publishConfig.assets. Files matching any of the glob patterns in publishConfig.assets are simply copied to the dist directory. By default, all files in your package’s assets directory are copied (if it exists).
Generate a package.json file.

build-pioneer-package generates a custom package.json file for your compiled package. Most properties from your package’s source package.json will simply be copied into the generated package.json (such as name, version and many more).

When generating the package.json, build-pioneer-package also takes into account your package’s metadata from the build.config.mjs: it will make entry points public (via exports) and include information about services and other Open Pioneer Trails features using the openPioneerFramework property.
Copy auxiliary files.

By default, a package must contain the following text files:

README, README.md or README.txt
LICENSE, LICENSE.md or LICENSE.txt
CHANGELOG, CHANGELOG.md or CHANGELOG.txt

Additionally, a NOTICE, NOTICE.md or NOTICE.txt may be present. All these files are copied into the compiled package.

You can configure publishConfig.validation to opt out of these required files.

Publishing

Publishing a npm package has a few requirements:

The package must not be set to private.
The package must have a valid version. Note that, when using the public registry at https://npmjs.com (opens in a new tab), you cannot overwrite an existing version. You must always publish a new version instead.
The package should have a valid license and a LICENSE file. It should also contain a README and ideally a CHANGELOG.

build-pioneer-package will by default check the last item, and pnpm will by itself refuse to publish a private package or a package with an invalid version.

We’re also using a pnpm-specific property called publishConfig.directory (opens in a new tab) for our packages. This option tells the package manager that it shall publish the contents of the configured directory, and not the source directory from where it is invoked:

1
{
2
    "name": "@open-pioneer/experimental-ol-map",
3
    "version": "1.2.3",
4
    "license": "Apache-2.0",
5
    // ...
6
    "publishConfig": {
7
        // Tells pnpm to publish these contents instead
8
        // https://pnpm.io/package_json#publishconfigdirectory
9
        "directory": "dist",
10
        // Recommended as default value to avoid confusion, see
11
        // https://pnpm.io/package_json#publishconfiglinkdirectory
12
        "linkDirectory": false
13
    }
14
}

After making the necessary changes to your package.json, build the package via build-pioneer-package. When that has completed successfully, the contents of the dist directory can be published:

1
# requires being logged in with the registry
2
$ pnpm publish
3
# or to explicitly make the package public (off by default for scoped packages):
4
$ pnpm publish --access=public

pnpm publish also accepts a --dry-run option to simulate publishing. You can also execute pnpm pack instead to generate a <PKG_NAME>.tgz in the dist directory; this allows you to inspect what would be published to the registry.

Using a different registry

npm (and pnpm) will by default publish to the public registry at https://npmjs.com (opens in a new tab). It can sometimes be an advantage to use another registry, for example for performance reasons (e.g. for better caching) or to keep the code private within an organization.

There are multiple options:

Using a different global registry.

All your packages will be fetched from this registry. The registry must therefore also act as a proxy for https://npmjs.com (opens in a new tab), since you still need access to all public packages.

This is the recommended operation mode by the developers of Verdaccio (opens in a new tab).

See here (opens in a new tab) for setup instructions (which also apply to pnpm).
Using a different registry for one or more specific scopes.

You will still use the default registry for all other packages, but packages from your scope (e.g. @my-company/*) would be loaded from a custom registry.

See here (opens in a new tab) for setup instructions.

Depending on your chosen scenario and the software you’re using you will also have to consider authentication and permission management, but that is out of scope for this document.

Neither option should have any impact on the framework itself, as long as all required packages can be downloaded.

Walkthrough

We will create a simple package, prepare it for publishing and then build it. We skip the actual publish step as that cannot be undone (we use --dry-run instead).

Step 1: Create the test package

We create a tiny package that provides a trivial Greeter class. For the sake of simplicity, we do not use any services or other advanced features in our package.

We create two TypeScript files and the necessary package.json and build.config.mjs in src/packages/test-package.

First, the (only) entry point. This file is intended to be used from the outside.

1
import { LOGGER } from "./logger";
2

3
export class Greeter {
4
    constructor() {
5
        LOGGER.debug("Greeter constructed");
6
    }
7

8
    greet(name: string) {
9
        return `Hello ${name}!`;
10
    }
11
}

An internal utility module:

1
import { createLogger } from "@open-pioneer/core";
2
import { sourceId } from "open-pioneer:source-info";
3

4
export const LOGGER = createLogger(sourceId);

The initial package.json:

1
{
2
    "name": "test-package",
3
    "main": "index.ts",
4
    "scripts": {
5
        "build": "build-pioneer-package"
6
    }
7
}

And the initial build.config.mjs

1
import { defineBuildConfig } from "@open-pioneer/build-support";
2

3
export default defineBuildConfig({
4
    publishConfig: {
5
        // Disable strict mode during initial setup to see more problems at the same time.
6
        // This should be set to `true` when we're done.
7
        strict: false
8
    }
9
});

Step 2: Getting the package to build

You can use the build-pioneer-package CLI to validate your package, it will notify you about problems with your package.

To execute the command, point your terminal to the package’s directory src/packages/test-package and then execute one of the two following commands:

1
$ pnpm build-pioneer-package # the manual way
2
$ pnpm build                 # the prepared 'build' script in package's package.json

Because we have disabled strict validation in the build.config.mjs, build-pioneer-package will show a lot of warnings but it will also “succeed”:

1
$ pnpm build
2

3
> test-package@ build /home/michael/projects/pioneer/starter/src/packages/test-package
4
> build-pioneer-package
5

6
Building package at /home/michael/projects/pioneer/starter/src/packages/test-package
7
/home/michael/projects/pioneer/starter/src/packages/test-package/build.config.mjs must define the 'entryPoints' property in order to be built.
8
Generating TypeScript declaration files...
9
Copying assets...
10
Writing package metadata...
11
/home/michael/projects/pioneer/starter/src/packages/test-package/package.json should define a version.
12
/home/michael/projects/pioneer/starter/src/packages/test-package/package.json should define a license.
13
/home/michael/projects/pioneer/starter/src/packages/test-package/package.json should define 'publishConfig.directory' to point to the 'dist' directory (see https://pnpm.io/package_json#publishconfigdirectory).
14
Copying auxiliary files...
15
Failed to find LICENSE in /home/michael/projects/pioneer/starter/src/packages/test-package (attempted exact match and extensions .md, .txt).
16
Failed to find README in /home/michael/projects/pioneer/starter/src/packages/test-package (attempted exact match and extensions .md, .txt).
17
Failed to find CHANGELOG in /home/michael/projects/pioneer/starter/src/packages/test-package (attempted exact match and extensions .md, .txt).
18
Success

Configuring entry points

Only entry point modules and modules imported by them will be included in the build package (under dist). You will notice that right now, the package is empty aside from its package.json, because we have not configured any entry points yet.

Here is how to add them:

1
import { defineBuildConfig } from "@open-pioneer/build-support";
2

3
export default defineBuildConfig({
4
   entryPoints: ["index"],
5
    publishConfig: {
6
        strict: false
7
    }
8
});

However, executing build now gives us a different problem:

1
$ pnpm build
2

3
> test-package@ build /home/michael/projects/pioneer/starter/src/packages/test-package
4
> build-pioneer-package
5

6
Building package at /home/michael/projects/pioneer/starter/src/packages/test-package
7
Building JavaScript...
8
[plugin check-imports] Failed to import '@open-pioneer/core', the package '@open-pioneer/core' must be configured either as a dependency or as a peerDependency in /home/michael/projects/pioneer/starter/src/packages/test-package/package.json
9
... in logger.ts
10

11
... SNIP ...

We’re using createLogger from @open-pioneer/core at runtime but forgot to add a dependency. We will fix that in the next step, together with the other problems in the package.json.

Updating package.json

We’re missing a license, a version and the correct dependencies. We have also been told to configure publishConfig.directory:

1
{
2
    "name": "test-package",
3
    "main": "index.ts",
4
   "version": "1.0.0",
5
   "license": "Apache-2.0",
6
    "scripts": {
7
        "build": "build-pioneer-package"
8
    },
9
   "dependencies": {
10
       "@open-pioneer/core": "<omitted from example>"
11
   },
12
   "publishConfig": {
13
       "directory": "dist",
14
       "linkDirectory": false
15
   }
16
}

We also added linkDirectory: false, see Publishing.

Adding required files

By default, build-pioneer-package requires LICENSE, README and CHANGELOG. These requirements can be disabled by configuring publishConfig.validation in your package’s build.config.mjs.

For this example, we just add three minimal files:

README.md:

1
# test-package
2

3
Here would be your README.

LICENSE:

1
... Copy of the Apache-2.0 license ...

CHANGELOG.md:

1
# Changelog test-package
2

3
## 1.0.0
4

5
- Initial release

Finally, we remove strict: false from our build.config.mjs:

1
import { defineBuildConfig } from "@open-pioneer/build-support";
2

3
export default defineBuildConfig({
4
    entryPoints: ["index"],
5
    publishConfig: {
6
       strict: false
7
    }
8
});

Rebuilding the package

Building the package after all these changes should now succeed:

1
$ pnpm build
2

3
> test-package@1.0.0 build /home/michael/projects/pioneer/starter/src/packages/test-package
4
> build-pioneer-package
5

6
Building package at /home/michael/projects/pioneer/starter/src/packages/test-package
7
...
8
Success

Step 3: Publishing the package

Still from your package’s source directory, execute:

1
$ pnpm publish --access=public --dry-run --no-git-checks
2
npm notice
3
npm notice 📦  test-package@1.0.0
4
npm notice === Tarball Contents ===
5
npm notice 56B    CHANGELOG.md
6
npm notice 11.4kB LICENSE
7
npm notice 43B    README.md
8
npm notice 120B   index.d.ts
9
npm notice 224B   index.js
10
npm notice 610B   index.js.map
11
npm notice 102B   logger.d.ts
12
npm notice 152B   logger.js
13
npm notice 347B   logger.js.map
14
npm notice 494B   package.json
15
npm notice === Tarball Details ===
16
npm notice name:          test-package
17
npm notice version:       1.0.0
18
npm notice filename:      test-package-1.0.0.tgz
19
npm notice package size:  5.3 kB
20
npm notice unpacked size: 13.8 kB
21
npm notice shasum:        38047b90f3983ef3414ce45de578b992efff8f9d
22
npm notice integrity:     sha512-eLI0mBJ106sgE[...]5BNhucoTnwJYA==
23
npm notice total files:   12
24
npm notice
25
npm notice Publishing to https://registry.npmjs.org/ (dry-run)
26
+ test-package@1.0.0

--dry-run makes (p-)npm stop just before uploading it to the actual registry. --no-git-checks allows us to use the command with a dirty working directory (this should not be used in production).

Finally, when you’re ready to publish, commit and tag your release and leave out the --dry-run flag.

Checklist

Automation

We recommend automating the process of releasing your packages (and your app) as much as possible. This has multiple advantages:

It reduces the possibility of human errors. Releases are usually complex and involve lots of small steps. A script cannot forget to perform those steps.
Automated releases are faster, making it easier to release often.
If a release process has been automated well, any team member can trigger a release as it requires no expert knowledge anymore.

For automatic management of your package’s changelogs and versions we can recommend Changesets (opens in a new tab). You can use their CLI as parts of your release script. If you’re using GitHub, they also provide an Action (opens in a new tab).

pnpm’s website has a section about changesets (opens in a new tab) and we’re using their action successfully in some of our repositories, see here (opens in a new tab) for example.