Managing package versions with Poetry

Table of Contents

0. The problem

Handling project version numbers is painful and can be tedious if not automated. To make matters worse, we often need to update the version in multiple files.

This post will explain how to simplify this process and how to automate it. It only focuses on how to manage it using poetry since it’s the tool I recommend for handling virtual environments. If you need to set up Poetry, see: Poetry python package manager

1. poetry-bumpversion

With Poetry-bumpversion, we can easily update the package version.

It assumes you follow Semver for the versioning of the project.

In short with semver the version consists of three numbers (M.m.p) where:

• M: major version. Represents breaking changes
• m: minor version. Represents relevant changes
• p: patch version. For fixes without new functionallity

1.1. Installing poetry-bumpversion

You can do it with:

# If you don't have Poetry installed, first do:
pip install poetry

# Add poetry-bumpversion
poetry self add poetry-bumpversion

# Make sure to install the poetry dependencies again with:
poetry install

1.2. Updating the version in pyproject.toml

You can do it by running:

poetry version major
poetry version minor
poetry version patch

1.3. Updating other files

Frequently you will have other files that declare the version. For example, you could have dbt_project.yml if you are using DBT. In that case, you will need to declare in pyproject.toml the files you want to update along with the format it should look for. As an example for DBT:

/pyproject.toml

[[tool.poetry_bumpversion.replacements]]
files = ["dbt_project.yml"]
search = "version: '{current_version}'"
replace = "version: '{new_version}'"

2. Automatically validate versions

The idea is to create a GitHub action that automatically validates that the version is updated. This way you can decide if you want to increase the major, minor, or patch digit.

The first thing you need is a simple script that reads the package version from pyproject.toml:

/.github/scripts/get_version.py

import click
import toml
from loguru import logger as log

from utils import set_output

PYPROJECT_FILE = "pyproject.toml"


@click.command()
@click.option("--name")
def get_version(name):
    version = toml.load(PYPROJECT_FILE)["tool"]["poetry"]["version"]
    log.info(f"'{name}' branch {version=}")

    set_output(f"VERSION_{name.upper()}", version)


if __name__ == "__main__":
    get_version()

This scripts uses the set_output function to export the version as an environment variable in GitHub action. Since this is used in another python script, it is declared in a utils.py file:

/.github/scripts/utils.py

import os

from loguru import logger as log


def set_output(name, value):
    log.info(f"Setting {name=} {value=}")
    with open(os.environ["GITHUB_ENV"], "a") as fh:
        print(f"{name}={value}", file=fh)

Then you will need a script that checks if the version needs to be updated. It will compare the version from main to the one in the current PR.

/.github/scripts/check_if_update_needed.py

import click

from packaging import version
from loguru import logger as log

@click.command()
@click.option("--version_current")
@click.option("--version_main")
def compare_versions(version_current, version_main):
    log.info(f"Running with {version_current=}, {version_main=}")

    version_current = version.parse(version_current)
    version_main = version.parse(version_main)
    
    if version_current <= version_main:
        logger.error("Version needs to be updated")
        exit(1)

    logger.success(f"Version is correctly updated")
    exit(0)


if __name__ == "__main__":
    compare_versions()

And the last step is to create a GitHub action that validates the version:

/.github/workflow/fix_version.yaml

name: Fix Version

on:
  pull_request:
    paths:
      - dbt_northius/**
      - poetry.lock
      - pyproject.toml

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  fix:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4

      # Install requirements. Should match Dockerfile versions
      - name: Install requirements
        run: pip install poetry==1.6.1 poetry-bumpversion==0.3.1 toml loguru click

      # Get version from main
      - name: Checkout main
        uses: actions/checkout@v3
        with:
          ref: main
      - name: Get main version
        run: python .github/scripts/get_version.py --name=main

      # Get version from the current branch
      - name: Checkout current branch
        uses: actions/checkout@v3
        with:
          # Those are needed because of https://github.com/EndBug/add-and-commit#working-with-prs
          repository: ${{ github.event.pull_request.head.repo.full_name }}
          ref: ${{ github.event.pull_request.head.ref }}
      - name: Get current version
        run: python .github/scripts/get_version.py --name=current

      # Checks if version needs updating
      - name: Check if version needs to be updated
        run: python .github/scripts/check_if_update_needed.py --version_current=$VERSION_CURRENT --version_main=$VERSION_MAIN

      # Update only when needed
      - name: Update version
        if: env.NEEDS_UPDATE == 'true'
        run: poetry version minor

      # Commit changes and force GitHub_status to be updated
      - name: Commit new version
        if: env.NEEDS_UPDATE == 'true'
        uses: EndBug/add-and-commit@v9
        with:
          default_author: github_actions
          message: "Poetry minor version update"

The way this works is by:

1. Extracting the version in main branch
2. Extracting the version from the current branch in the pull request
3. If current_version <= main_version, then fail

3. Automatically tag versions

In order to keep better tracking of the package version, we will be tagging all commits to main with their version. We can do that with the following GitHub action:

/.github/workflow/tag_commits_on_main.yaml

name: Tag

on:
  push:
    branches:
      - main
    paths:
      - pyproject.toml

jobs:
  tag_with_version:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4

      - name: Install requirements
        run: pip install toml loguru click

      - name: Checkout current branch
        uses: actions/checkout@v3

      - name: Get current version
        run: python .github/scripts/get_version.py --name=current

      - name: Tag commit with current version
        uses: actions/github-script@v5
        with:
          script: |
            github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: `refs/tags/${{env.VERSION_CURRENT}}`,
              sha: context.sha
            })

4. Environment variables in github actions

One of the ways of passing information from one step in a GitHub action to another is by using environment variables.

4.1. Storing data in an environment variable

There is a couple of ways of storing data in an environment variable. This first one would be with an echo like:

- name: Get main version
  run: echo "VERSION_CURRENT=$(python scripts/get_version.py)" >> $GITHUB_ENV

In this example you would need scripts/get_version.py to output the version.

The second option is to do it directly with python with:

def set_output(name, value):
    """
    Args:
        name:  name of the environment variable
        value: value to store in the environment variable
    with open(os.environ["GITHUB_ENV"], "a") as fh:
        print(f"{name}={value}", file=fh)

4.2. Retrevient data from an environment variable

To retrieve the environment variable, simply use $ENV_VAR_NAME like:

- name: Check if version needs to be updated
  run: python .github/scripts/check_if_update_needed.py --version_current=$VERSION_CURRENT --version_main=$VERSION_MAIN

Notice that in the tag_commits_on_main.yaml GitHub action, we are using the value directly within a string. This is done with:

`text ${{env.ENV_VAR_NAME}}`

Like we saw in the previous snippet:

- name: Tag commit with current version
  uses: actions/github-script@v5
  with:
    script: |
      github.rest.git.createRef({
        owner: context.repo.owner,
        repo: context.repo.repo,
        ref: `refs/tags/${{env.VERSION_CURRENT}}`,
        sha: context.sha
      })