Golem

Introduction

Contributions are very welcome!

Do not hesitate to create a PR with no change to start the conversation about something you’d be interested in developing.

What to do?

If you have no specific idea in mind, you can have a look at the Roadmap or the Known issues.

Of course, much remains to be done to make Golem the best build system!

Among many things, Golem needs more tests.

Pull requests

Do not hesitate to create a PR with no change to start the conversation about something you’d be interested in developing.

That said, avoid PRs addressing differents issues.

For example, renaming a lot of variables for a cleaner code and fixing a bug are 2 different matters, and therefore requires independant PRs.

But refactoring of functions and renaming variables pursue the same objective and can be submitted under the same PR.

The point is to be able to focus the discussion on one matter at a time.

How is it designed?

Golem is powered by Waf, but provides a completely different API. It’s a sophisticated frontend to Waf that adds many features and simplifies for the users how to define their project.

Quick overview of how it works

Flow of execution

Calling a command on golem calls a wrapper script (main.py).

This script adapts the arguments needed to call the waflib scripting entry point. Mainly because by default Waf wouldn’t leave the project directory untouched. Golem fixes this behavior by forcing Waf to work from the build directory.

When the waflib scripting entry point gets called, the execution flow continues in a templated wscript file and gets redirected into context.py

Calling golem configure calls def configure(self): in builder.py, which prepares a Context object and calls def configure(self): on it in context.py

From here, Golem reads the project file, the environment variables, the options to properly set how to call waflib primitives. That is, detecting Qt, determining the cache configuration, etc.

Waf is designed around a task system. To build a program, tasks are generated by Golem with all the needed compilation and linking options and Waf runs them.

Remains some cases where Golem isn’t using yet the task system (but should), such as when resolving, cloning, configuring, building dependencies. In these cases, Golem calls Git directly, or calls itself directly.

Resolving a dependency

Resolving a dependency (golem resolve) consists of 5 steps:

  1. Resolving the version (commit hash) of the dependency the project refers to
  2. Cloning the dependency in the right cache directory (if doesn’t exist yet)
  3. Configuring the dependency before build
  4. Resolving the dependencies of the dependency if any (recursion)

Resolving the version consists of interpreting version in the dependency definition found in the project file. This version can be a commit hash (no resolution needed), a branch name, a tag, or a Node SemVer to search a SemVer tag.

To do so, Golem uses git ls-remote --tags and git ls-remote --heads to search for a corresponding commit hash. See dependency.py.

Cloning the dependency in the cache consists of:

  1. Generating a recipe ID corresponding to the dependency and adding to it the short commit hash (8 first characters). See def generate_recipe_id(url): and def make_dep_base(dep): in helpers.py.
  2. Determining where the dependency must be cached. See def find_dep_cache_dir(self, dep, cache_conf): in context.py.
  3. Cloning the repository in the cache. See def make_repo_ready(self, dep, cache_dir, should_clean=False): in context.py.

Example of a cloned repository in a cached dependency: json@com.github.nlohmann+65ee6845\repository

  • json@com.github.nlohmann : recipe ID
  • 65ee6845 : commit hash

Configuring the dependency consists of:

  1. Determining the build location, which is a directory named after a condensed concatenation of characters designating the platform, architecture, compiler, runtime, linking and variant information. See def build_path(self, dep=None): in context.py.
  2. Running golem configure with all the needed options on the dependency. See def run_dep_command(self, dep, cache_dir, command): in context.py.

When configuring the dependency, the build directory is setup.

Preparation of the build directory by golem configure consists of:

  • Determining if the project has a project file or a recipe (it not, error) See def load_recipe(self): in context.py.
  • Creating an artifact directory with a specific slug to avoid conflicts between projects asking for different build options. See def make_dependencies_slug(self, dependencies): in context.py.
  • Creating a configuration file to hold how a project can use the dependency.
  • Creating a dependencies.json file to hold the list of dependencies the dependency relies on.

Example of an artifact directory in a cached dependency: json@com.github.nlohmann+65ee6845\w64mshshd\bin-da39a3ee

  • w64mshshd : Windows, x64, msvc, shared runtime, shared linking, debug
  • da39a3ee : dependency slug to isolate the artifacts

Example of a configuration file: json@com.github.nlohmann+65ee6845\w64mshshd\conf\json@json@com.github.nlohmann.json

  • json@json@com.github.nlohmann.json the name follows the format <target>@<recipe_id>
{
    "configuration": {
        "artifacts": [
            {
                "location": "${GOLEM_CACHE_DIR}\\json@com.github.nlohmann+65ee6845\\repository",
                "path": "LICENSE.MIT",
                "repository": "https://github.com/nlohmann/json.git",
                "resolved_version": "\"v3.12.0\"",
                "type": "license"
            }
        ],
        "header_only": true,
        "isystem": [
            "${GOLEM_CACHE_DIR}\\json@com.github.nlohmann+65ee6845\\include"
        ],
        "licenses": [
            "${GOLEM_CACHE_DIR}\\json@com.github.nlohmann+65ee6845\\repository\\LICENSE.MIT"
        ],
        "targets": [
            "json"
        ]
    },
    "dependencies": []
}

This file contains all of what’s needed when linking against the target json of the dependency json@com.github.nlohmann.

Once resolved, the dependencies can be built with golem dependencies. An include directory will be added in the dependency’s cache directory to contain the headers meant to be used by the calling project. The artifacts will be built and stored in the artifact directory.

Prev
Environment Variables