Monterey infrastructure | Aurelia import

Initial state

Content of this article

1. Specification by Rob Eisenberg
2. Flowcharts
3. Relevant portion of Pacman definition
4. Mapping between Pacman and Specification
5. Plan to evolve this project from items 2-4.

1. Specification by Rob Eisenberg

Aurelia Import

At present, installing libraries into an Aurelia CLI application is tedious and error-prone. We want to remove this pain by providing a new CLI command named au import.

The idea is not to be a package manager, but a package-to-project importer. We want to let NPM, Yarn, Bower and future technologies handle the local installation and management of dependencies. We want our tool to manage the application import side of this, based on specific knowledge of Aurelia's project and module loader.

Here's how we would want to accomplish installing aurelia-dialog, for example:

npm install aurelia-dialog --save
au import aurelia-dialog

Eventually, we could combine these into a single command:

au install aurelia-dialog

The above command would automate NPM (or whatever package manager you have configured) to install the package and then use Aurelia's import command to configure the project.

How would the "import" command configure the project?

For most libraries, executing the import command would simply cause the CLI to open the aurelia.json file and add some configuration data. In some special cases, a library may choose to write a custom importer which would allow that library to do all sorts of things. Some examples might be:

• Generate code directly into the project.
• Configure new CLI commands
• Generate/Override parts of the library itself, based on user feedback
• Provide a tutorial/help/walkthrough to the developer
• Install custom tools

How should the CLI's "import" command function internally?

When you run au import library the CLI will take the following steps:

• Look into the node_modules folder for the library, find its package.json and see if that library defines a custom importer. If it does, run the custom importer and exit.
• If no custom importer is found, look into the node_modules folder for the library, find its package.json and see if a custom aurelia project configuration is provided. If it is, use it and exit.
• If no custom configuration is found, look into the CLI's known registry of libraries and see if it has a default configuration for the imported library. If it does, use it and exit.
• If no information is found in the registry, look back into the node_modules folder for the library, find its package.json and inspect it to see if it has a JSPM section. If so, determine if we can use the JSPM configuration information. If so, use it and exit.
• If no JSPM section is found, look for a browser configuration in the package.json and use it if available.
• If no browser spec configuration is found, use the standard package main field. Load the main module with amodro trace and see if it has any dependencies. If it does, configure the package as a commonjs package format. If it doesn't use a standard configuration. (I'm not sure if we can do this successfully with amodro. If not, we can just default to a standard configuration).
• Inform the user of which mechansim was used to locate the configuration.
• Inform the user how to import the module in their code.

One thing the default (and custom) importers should be able to do is ask the end user which bundle they want to configure the library within. The behavior should be as follows:

• If there is only one bundle defined, don't ask the user. Just add it to the existing bundle.
• If there are two bundles defined, ask the user but provide a quick default for whichever bundle has the most 3rd party dependencies.

If the user is coding in TypeScript, we may want to take additional actions such as checking for d.ts files and potentially attempting to install those. We can add this in a phase 2.

How should it be architected?

The base functionality is the "importer". That API needs to be defined first. All the fallback behaviors after the first option above should simply be implemented as the default importer. In fact, another way of stating the above would be:

• Check to see if the library has a customer importer. If it does, use it.
• If no custom importer is found, use the default importer.

Importers should receive a set of services from the CLI so that they can do input/output, module resolution, code generation, etc. through the CLI's standard abstractions. This would enable importers to work when run from other contexts and to work correctly for whatever project they are being run for, based on its configuration. For example, if the VS Code plugin automates the CLI, then an importer that uses the CLI's input/output abstraction will autmatically display its UI in VS Code.

2. Flowcharts

Comparison between desired and currently implemented flows.

2.1 Aurelia-CLI Flow

2.2 Pacman's current Flow

3. Pacman definition

Terms:

• CLI: aurelia-cli build system
• Pacman: aurelia-cli-pacman extension package

Main idea behind Pacman

As we all know it, configuring external plugins with CLI requires a lot of manual work.

Pacman has been created to extend CLI's functionality with package configuration/import features. It is built on top of CLI infrastructure, uses that as much as possible to avoid any duplication. Pacman avoids duplicating any existing feature present in CLI.

Metadata file

How could pacman possibly know how to configure a package?

It uses pre-defined metadata files provided by plugin creators or by pacman itself. These are json-formatted instructions for pacman.

Full-featured example:

{
    "patches": [
        { "op": "replace", "path": "/build/loader/plugins/0/stub", "value": false}
    ],
    "dependencies": [],
    "bundles": [
        ...
    ],
    "tasks": [
        "prepare-materialize"
    ],
    "scripts": {
        "install": [
            "au prepare-materialize",
            "node node_modules/requirejs/bin/r.js -o tools/rbuild.js"
        ],
        "uninstall": [

        ]
    }
}

What features it allows, by sections:

[patches]: used by applyPatches

Apply patches to aurelia.json using RFC6902 standard.

[dependencies]: used by registerDependencies

Append new dependency configurations to the default/specified bundle.dependencies array.

[bundles] used by registerBundles

Add new bundles or even override an entire, existing bundle.

[tasks] used by installTasks

Copy custom CLI-tasks from package folder into aurelia_project/tasks folder.

[scripts] used by executeScripts

You can execute any number or type of node scripts as needed.

Bonus: since it runs after installTasks, previously installed CLI-tasks can also be executed here.

Architecture

Important note: it has been designed to run in synchronous fashion.

Although it uses mostly Promise-based methods, those are chained together and being executed one-by-one, in a precise order. This means that the execution order matters and it provides the possibility for the execution to be canceled or even roll-backed in some cases.

Layers:

• au pacman
  • [Optional] Provider (NPM)
  • Analyzer
  • ImportEngine
    • ImportBase
    • Custom Import hooks

Installed au pacman task

Pacman uses CLI's awesome feature, namely that it can execute custom tasks created by developers. It holds no business logic, as it uses a default implementation of above classes, but its parts can be replaced or overridden on demand.

Providers (Package Management)

There is currently one integration with NPM. It's disabled by default, can be activated by removing comments in pacman custom task. All it does that it installs the package before configuration steps.

Analyzer

Gathers basic information, such as aurelia.json, command line parameters, auto-discovers custom hooks placed within the installed package.

ImportEngine

Configures and executes all available custom importer steps according to information defined by <metadata.json>.

Example flow:

• Steps: ['applyPatches', 'registerDependencies']

• Importers: [ImportBase, MyCustomImport, SecondCustomImport]

• Step: applyPatches

  • Order:
    1. ImportBase.applyPatches()
    2. MyCustomImport <-- applyPatches() cannot be found, skipping
    3. SecondCustomImport.applyPatches()

• Step: registerDependencies

  • Order:
    1. ImportBase.applyPatches()
    2. MyCustomImport.registerDependencies()
    3. SecondCustomImport <-- registerDependencies() cannot be found, skipping

ImportBase (Custom workflows)

Example implementation for metadata file processing. This runs first by default, and you can add any number of your own implementations as well. I've created an autodiscovery feature, which is capable of detecting hooks placed into the package itself (<node_modules>/<package>/install/import-hooks.js).

Built-in steps in execution order (it follows the format of metadata file.)

(0. register)

1. applyPatches
2. registerDependencies
3. registerBundles
4. saveProject
5. installTasks
6. executeScripts

Example for custom hooks:

module.exports = class {

    constructor() { }

    register(engine, options) {
        this.engine = engine;
        this.cliParams = options;

        this.engine.availableSteps.pop('runBeforePatches');
    }

    // newly added custom step
    runBeforePatches() {
        // my customized operations

        if (this.cliParams.quiet) {
            console.log('I can display my own messages too ;)');
        }
    }

    registerBundles() {
        // additional todos after ImportBase.registerBundles() finished
    }

4. Mapping between Pacman and Specification

Pacman differences:

• By default, the built-in base logic runs as first tier (ImportBase)
• Allows partial extension/override of base logic
  • (complete override without having to run base logic is also possible)
• Base class is able to process instructions defined in a metadata.json file
• There are no dynamic strategies to discover custom importers. I'm looking for custom importer using a fixed path: <package_dir>/install/import-hooks.js.

5. Plan to evolve this project from items 2-4.

Required modifications by specification:

• Base logic should run only when custom importer haven't been found/specified.
• Question: should we provide an opt-in possibility to execute parts of base logic in a custom importer if desired?
  • Example: patching aurelia.json could be a public feature available to anyone, so duplication could be avoided.
• Question: Pacman has its metadata format, which can instruct base logic. Is this a feature worth keeping?

Custom Importer API for plugin creators/package maintainers

Simple API with an execute() method.

When importer strategy is turned out to be a custom importer, control is being handled over to built-in ImportEngine, which is responsible to initialize found CustomImporter class, pass CLI-services to it, and calls its execute method.

Flow:

1. ImportEngine
   • initiate strategy discovery
2. StrategyLoader
   • find custom importer,
   • return metadata to ImportEngine
3. ImportEngine
   • notify user about chosen strategy
   • create an instance of custom importer,
   • inject CLI-services (UI, project info, etc..)
   • call customImporter.execute()
4. Custom Importer
   • Plugin creators/maintainers custom logic executes within CLI-infrastructure
   • return back to ImportEngine
5. ImportEngine
   • handle errors, successful execution
   • cleanup (if needed)
   • terminate

Injecting services by ImportEngine:

Inject a helper property into Custom importer, so it could call generic tasks (json patching, instruction metadata-processing, run a built-in strategy instead).

Possible solutions for injection:

1. With Aurelia Container, (UI, Resolver, ImportEngine, etc..)

Defining to be injected properties would be a task for importer creator (not okay)

2. Pass services to custom a property before execution starts.

Example:

ImportEngine:

...
customImporter.cliServices = {
  ui: ConsoleUI,
  etc: ETC,
  importEngine: this,
};

customImporter.execute();

Custom Importer class:

execute() {
  let ui = this.cliServices.ui;
  ui.question('...')
    .then(answer => do_something());
}

Importer strategies:

If no custom importer is found: provide an extensible feature to determinate appropriate import strategy. Currently, there are 6 strategies defined in Section 1.

• API: StrategyLoader + custom strategy classes with numbered filenames

StrategyLoader:

class StrategyLoader {

  constructor() {
  }

  /**
   * search for strategies and create an array of instances,
   * array contents will be ordered by filename
   * 
   * @void
   */
  init() {
  }

  /**
   * Loop through found strategy instances and check for its eligibility 
   *
   * @return CustomStrategyBase|boolean
   */
  load() {
    for (let strategy of this.strategies) {
      let chosen = strategy.determinate();
      if (chosen === true) {
        return strategy;
      }
    }

    return false;
  }
};

Custom Strategy:

class CustomImporterStrategy {

  constructor(...toBeInjected) { }

  /**
   * Determinate whether this particular strategy should be used or not
   * @returns {boolean}
   */
  determinate(...argsToBeDefined) {
    return true;
  }

  /**
   * Main entry point called by ImportEngine
   */
  execute(...argsToBeDefined) {

  }

  /**
   * Short description display in CLI output
   * @returns {string}
   */
  get name() {
    return 'Custom Importer Strategy';
  }
}

Proposed file structure within aurelia-cli

Package managers part involves a little refactor by moving aurelia-cli/lib/npm.js file to aurelia-cli/lib/package-managers/npm.js path.

CLI Feature suggestion:

Extending au new wizard steps with "Choose default package manager" option:

• save package manager configuration into aurelia.json
• cli parameter option for override (--with) [npm, yarn, bower]

aurelia.json section:

"installer": {
  "default": "yarn",
  "fallbacks": ["bower", "etc"]
}

This would allow for an au install command to know, which package manager should be used by default.

Example: default is set to yarn: