Reference

Workspace

Whatever you build with Polycrate happens in your workspace. The workspace contains configuration and lifecycle artifacts. It's a directory on your filesystem (the so called workspace directory; can be specified using --workspace/-w) that can be synced and collaborated on via git or other tooling.

The workspace can be assembled using the workspace configuration.

Polycrate container

Most of the Polycrate magic happens inside of a Docker container running on the system that invokes the polycrate command.

The container will be started whenever you run an action. It is based on a public image (cargo.ayedo.cloud/library/polycrate) provided by ayedo (Dockerfile) and contains most of the best-practice tooling of cloud-native development and operations.

The container gives you access to a state-of-the-art DevOps runtime. Polycrate exports a snapshot of the workspace in various formats (yaml, json, environment vars, hcl, ...) and makes it available to the tooling inside the container so you can start building right away.

Note

You can run actions locally instead of in a container using the --local flag.

The main purpose for this would be:

• Use the Polycrate CLI inside its own container in CI
• Run local actions like setting up developer environments

This is currently EXPERIMENTAL and not well tested.

Dockerfile

Polycrate looks for a Dockerfile in your workspace (defaults to Dockerfile.poly, can be configured using --dockerfile). If it finds one, it will build it and run the container based on this image instead of the default image.

This can be used to persist changes to the workspace, like installing additional tools or libraries.

By convention, the Dockerfile should be built upon the official Polycrate image:

FROM cargo.ayedo.cloud/library/polycrate:latest

RUN pip install hcloud==1.16.0

Using loglevel 2 you can see and debug the build process:

DEBU[0000] Found Dockerfile.poly in Workspace           
DEBU[0000] Building image 'polycrate-demo:latest', --build=true 
WARN[0000] Building custom image polycrate-demo:latest  
DEBU[0000] Assembling docker context                    
DEBU[0000] Building image                               
DEBU[0001] Step 1/2 : FROM cargo.ayedo.cloud/library/polycrate:latest
DEBU[0001]  ---> 67237198f4a5                           
DEBU[0001] Step 2/2 : RUN pip install hcloud==1.16.0    
DEBU[0001]  ---> Using cache                            
DEBU[0001]  ---> 92a78743b4f4                           
DEBU[0001] Successfully built 92a78743b4f4              
DEBU[0001] Successfully tagged polycrate-demo:latest

Workspace configuration

The workspace configuration (default: workspace.poly) holds the configuration for a workspace and must be located inside the workspace directory.

Note

You can specify a custom workspace configuration file by using --workspace-config YOUR/CUSTOM/workspace-config.yml. This can be especially helpful when using Polycrate in CI.

# workspace.poly
name: polycrate-demo
blocks:
  - name: custom-block
    config:
      foo: bar

Note

The workspace name is limited to certain characters: ^[a-zA-Z]+([-/_]?[a-zA-Z0-9_]+)+$.

This constraint applies to ALL name stanzas in Polycrate.

Blocks

A Polycrate workspace is a modular system built out of so called blocks. Blocks are dedicated pieces of code/functionality that can be configured using the config stanza in the block configuration (default: block.poly) or the workspace configuration (default: workspace.poly). Blocks expose actions that can be executed using polycrate run $BLOCK_NAME $ACTION_NAME.

Polycrate looks for blocks inside the blocks root (defaults to blocks). Nested directories (e.g. blocks/foo/bar/baz) are allowed.

Note

If a block's name contains one or multiple slashes (/) and is installed through the registry, it will be saved to a nested directory structure: the block ayedo/k8s/harbor will be saved to blocks/ayedo/k8s/harbor. This also applies to the block's artifact directory.

Inheritance

Blocks can be inherited from other blocks which merges configuration of the parent block (typically a block you installed or created inside the blocks directory) into its child and changes the workdir to the parent block's workdir whenever an action runs. Such blocks that are based on other blocks are called dynamic blocks.

Note

When merging a parent block's config into a child block, existing config in the child block will not be overriden. The most common scenario where this is relevant is when you define defaults inside a block.poly file and overwrite them in your workspace.poly file.

...
blocks:
  - name: harbor
    from: ayedo/k8s/harbor
...

Dynamic blocks

Blocks can be created dynamically by defining their configuration in the workspace configuration directly. These blocks do not use custom code but only rely on the available tooling inside the Polycrate container.

Dynamic blocks can also inherit their default configuration and workdir from blocks that already exist in the blocks root by using the from: stanza in the block definition.

Dependencies

Polycrate supports workspace-level block dependencies by using the dependencies stanza:

...
dependencies:
  - ayedo/hcloud/inventory:0.0.1
  - ayedo/hcloud/k8s-infra:0.0.2
  - ayedo/hcloud/k8s:0.0.9
  - ayedo/k8s/nginx:0.0.2
  - ayedo/k8s/portainer:0.0.7
  - ayedo/k8s/cert-manager:0.0.3
  - ayedo/k8s/external-dns:0.0.21
  - cargo.ayedo.cloud/ayedo/k8s/harbor:0.0.1
...

Polycrate checks the configured dependencies against the installed blocks in the workspace with every invocation of the polycrate command. If a dependency is missing, it will be downloaded.

Pull blocks

To dynamically add blocks to the workspace from the registry, you can run polycrate pull $BLOCK_NAME:$BLOCK_VERSION. If $BLOCK_VERSION is not defined, latest will be assumed.

Blocks can be uninstalled from the workspace by running polycrate block uninstall BLOCK1 or simply deleting the block's directory.

If you want to use a custom registry, reference it when pulling blocks or adding them to the workspace dependencies: polycrate block pull index.docker.io/my-block

Push blocks

To push a block to an OCI-compatible registry, run the following command: polycrate block push $BLOCK_NAME.

If the block name doesn't contain a valid registry url, a default registry will be used. A custom registry can be used as target by giving the block an appropriate name, like: my.registry.com/block/name. Polycrate will resolve the registry url from the block's name.

Note

When pushing a block, omit the tag (what's coming after the :) as Polycrate will automatically add the version defined in the block's block.poly to the image.

Block directory

The block directory is the directory that contains custom code and the block.poly file of a block underneath the blocks root.

Note

By convention, the name of the block directory should be the same you defined in the name stanza of that Block.

Block configuration

The block configuration (defaults to block.poly) holds the configuration for a single block and must be located in the block directory.

# block.poly
name: custom-block
config:
  baz:
    foo: bar
actions:
  install:
    script:
      - echo "Install"
  uninstall:
    script:
      - echo "Uninstall"

The config stanza of the block configuration is free form and not typed. You can use it to define the configuration structure of your block according to your needs.

Note

Block names are limited to certain characters: ^[a-zA-Z]+([-/_]?[a-zA-Z0-9_]+)+$.

This constraint applies to ALL name stanzas in Polycrate.

Actions

actions:
  - name: install
    playbook: install.yml

A block can expose an arbitrary amount of actions. Actions are used to implement the actual functionality of a block. Examples would be install or uninstall, but also status or init.

To execute a certain playbook, specify its name in the playbook stanza of an action. Polycrate will create the respective ansible-playbook command and make the workspace snapshot available as extra-vars to the playbook so they can be used even in the host stanza of a playbook.

Actions also support a script stanza which contains a list of commands that will be merged into a Bash script and executed inside the Polycrate container (or locally if you specifiy --local) when you run the action. The script stanza is mutually exclusive with the playbook stanza.

Note

You can use Go Templates in your action scripts.

[...]
actions:
  - name: template
      script:
        - echo This is Action {{ .Action.Name }} in Block {{ .Block.Name }}
        - echo Running in Workspace {{ .Workspace.Name }}
[...]

Note

Action names are limited to certain characters: ^[a-zA-Z]+([-/_]?[a-zA-Z0-9_]+)+$.

This constraint applies to ALL name stanzas in Polycrate.

Note

Polycrate does not persist data between runs apart from changes made to the workspace directory (mounted at /workspace inside the execution container).

Note

It's fine to write data to the workspace directory. However it's best-practice to use Artifacts to persist custom data.

Artifacts

Artifacts can be stored in the artifacts root inside your workspace (which is configurable using --artifacts-root and defaults to artifacts).

By default, Polycrate looks for Ansible Inventories and Kubeconfigs in the Artifacts Directory of a Block.

For each block in the workspace a directory will automatically be created underneath the artifacts root (e.g. artifacts/block/).

Ansible Inventory

Polycrate can consume yaml-formated Ansible inventory files inside the artifacts directory of a block. Polycrate looks for a file named inventory.yml by default - this can be overridden using the inventory.filename stanza in the block configuration.

An inventory file can be created automatically by a block or provided manually (useful for existing infrastructure).

The inventories can be consumed by the owning block itself or by other blocks using the inventory stanza in the block configuration:

# workspace.poly
blocks:
  - namename: block-a
    inventory:
      from: block-b
      filename: inventory.yml

This will add an environment variable (ANSIBLE_INVENTORY=path/to/inventory/of/block-b) to the container that points Ansible to the right inventory to work with.

Kubeconfig

Polycrate is integrated with Kubernetes and can connect to a cluster using a kubeconfig file. By default, Polycrate looks for kubeconfig files named kubeconfig.yml inside the artifacts directory of a block. This can be overridden using the kubeconfig.filename stanza in the block configuration.

A kubeconfig file can be created automatically by a block or provided manually (useful for existing infrastructure).

The kubeconfig file can be consumed by the owning block itself or by other blocks using the kubeconfig stanza in the block configuration:

# block.poly
name: block-a
  kubeconfig:
    from: block-b
    filename: kubeconfig.yml

This will add an environment variable (KUBECONFIG=path/to/kubeconfig/of/block-b) to the container that points kubectl, etc to the right kubeconfig to work with.

Workspace snapshot

Whenever you run an action with Polycrate, a workspace snapshot will be captured. This snapshot contains the computed workspace configuration and will be exported in the following formats:

yaml

Polycrate exports the workspace snapshot to a yaml file that will be saved locally and mounted into the container. The path of the file will be exported to the POLYCRATE_WORKSPACE_SNAPSHOT_YAML environment variable.

environment vars

Polycrate converts the workspace snapshot to a flattened map and exports it to the environment.

[...]
ACTION_BLOCK: custom-block
ACTION_NAME: hello
ACTION_SCRIPT_0: echo Hello
ACTION_SCRIPT_1: echo World
BLOCK_ACTIONS_0_BLOCK: custom-block
BLOCK_ACTIONS_0_NAME: hello
BLOCK_ACTIONS_0_SCRIPT_0: echo Hello
BLOCK_ACTIONS_0_SCRIPT_1: echo World
BLOCK_ARTIFACTS_CONTAINERPATH: /workspace/artifacts/blocks/custom-block
BLOCK_ARTIFACTS_LOCALPATH: $HOME/.polycrate/workspaces/polycrate-demo/artifacts/blocks/custom-block
BLOCK_CONFIG_FOO: bar
BLOCK_NAME: custom-block
BLOCK_VERSION: ""
BLOCK_WORKDIR_CONTAINERPATH: /workspace/blocks/custom-block
BLOCK_WORKDIR_LOCALPATH: $HOME/.polycrate/workspaces/polycrate-demo/blocks/custom-block
[...]

You can use the --snapshot flag when invoking polycrate run. This will prevent any action from running and instead dumps the workspace snapshot. Also, polycrate workspace snapshot has the same effect, but doesn't contain data about the current action and block.

Workflows

Polycrate supports workflows, i.e. the ordered execution of block actions.

# workspace.poly
name: workflow-workspace
blocks:
  - name: block-1
    actions:
      - name: action-1
        script:
          - echo "block 1 action 1"
  - name: block-2
    actions:
      - name: action-1
        script:
          - echo "block 2 action 1"
workflows:
  - name: workflow-1
    steps:
      - name: block-1-action-1
        block: block-1
        action: action-1
      - name: block-2-action-1
        block: block-2
        action: action-1
        prompt: "Do you really want to run this action?"

You can use polycrate workflows run workflow-1 (or for short polycrate run workflow-1) to execute this workflow.

If a step has the prompt stanza an it's not empty, Polycrate will interrupt the workflow and lets the user confirm the execution of the current step.

Loglevel

Polycrate supports 3 loglevel:

• Loglevel 1: The default. Will only print logs of type INFO or above
• Loglevel 2: Debug-Level. You know ...
• Loglevel 3: Trace level.

The loglevel will be mapped to the respective Ansible verbosity meaning --loglevel 3 will result in Ansible executing as if you used -vvv.

Sync

Polycrate can be configured to automatically sync with a git repository whenever it is invoked. Polycrate then writes a history log to history.log in the workspace directory and commits and pushes before and after invoking an action, saving the command that has been issued as well as the exit code the command resulted in.

Sync must be enabled in the workspace configuration:

sync:
  enabled: true
  auto: true
  remote:
    url: ssh://git@gitlab.ayedo.de:10022/devops/workspaces/starkiller.git

If sync.auto is false, you need to manually push to the connected git repository.

With polycrate log $MESSAGE you can write and sync arbitrary history logs.

Registry

Polycrate blocks can be pushed and pulled to and from a OCI-compatible registry. Polycrate uses cargo.ayedo.cloud as the default registry to obtain blocks from or push them to.

You can define your own registry by using --registry-url and pointing it to your OCI-compatible registry or simply include the registry URL in the block's name, just like with Docker.

Polycrate does not implement authentication with the registry but instead makes use of your local Docker credential helper. To authenticate against a registry, run docker login $REGISTRY_URL before pushing or pulling blocks.

Ansible

Polycrate provides a special integration with Ansible. The workspace snapshot that is being exported to yaml format and mounted to the Polycrate container will be consumed by Ansible automagically. As a result, the snapshot is available directly as top-level variables in Ansible which can be used in playbooks and templates.

The following example shows:

• the default configuration (block.poly) of a block called traefik
• the user-provided configuration for the block in workspace.poly
• the Ansible playbook using the exposed variables ( block.config...)
• an Ansible template using the exposed variables (templates/docker-compose.yml.j2)
• the resulting file /polycrate/docker-compose.yml that is templated to a remote host

Note

Note how in block.poly the configured image is traefik:2.6 but in workspace.poly it's traefik:2.7. In the resulting docker-compose.yml, the image is traefik:2.7 as defaults in block.poly will be overridden by user-provided configuration in workspace.poly.

The block variable contains the configuration of the current block invoked by polycrate run traefik install. Additionally, there's a variable workspace available, that contains the fully compiled workspace including additional blocks that are available in the workspace.

Polycrate makes use of a special Ansible Vars Plugin to read in the Yaml-Snapshot and expose it as top-level variables to the Ansible facts.

block.polyworkspace.polyinstall.ymltemplates/docker-compose.yml.j2/polycrate/traefik/docker-compose.yml

name: traefik
config:
  image: "traefik:v2.6"
  letsencrypt:
    email: ""
    resolver: letsencrypt
actions:
  - name: install
    script:
      - ansible-playbook install.yml
  - name: uninstall
    script:
      - ansible-playbook uninstall.yml
  - name: prune
    script:
      - ansible-playbook prune.yml

name: ansible-traefik-demo
blocks:
  - name: traefik
    inventory:
      from: inventory-block
    config:
      letsencrypt:
        email: info@example.com
      image: traefik:2.7

- name: "install"
  hosts: all
  gather_facts: yes
  tasks:
    - name: Create remote block directory
      ansible.builtin.file:
        path: "{{ item }}"
        state: directory
        mode: '0755'
      with_items:
        - "/polycrate/{{ block.name }}"

    - name: Copy compose file
      ansible.builtin.template:
        src: docker-compose.yml.j2
        dest: "/polycrate/{{ block.name }}/docker-compose.yml"

    - name: Deploy compose stack
      docker_compose:
        project_src: "/polycrate/{{ block.name }}"
        remove_orphans: true
        files:
          - docker-compose.yml

version: "3.9"

services:
  traefik:
    image: "{{ block.config.image }}"
    container_name: "traefik"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.{{ block.config.letsencrypt.resolver }}.acme.email={{ block.config.letsencrypt.email }}"
      - "--certificatesresolvers.{{ block.config.letsencrypt.resolver }}.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.{{ block.config.letsencrypt.resolver }}.acme.tlschallenge=true"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "traefik-letsencrypt:/letsencrypt"
    networks:
      - traefik

networks:
  traefik:
    name: traefik

volumes:
  traefik-letsencrypt:

version: "3.9"

services:
  traefik:
    image: "traefik:2.7"
    container_name: "traefik"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.email=info@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "traefik-letsencrypt:/letsencrypt"
    networks:
      - traefik

networks:
  traefik:
    name: traefik

volumes:
  traefik-letsencrypt:

---

Last update: January 29, 2023