Managing package versions with Poetry

0. The problem

Handling project version numbers is painful and can be tedious if not automated. To make matters worse, we often need to update the version in multiple files.

This post will explain how to simplify this process and how to automate it. It only focuses on how to manage it using poetry since it’s the tool I recommend for handling virtual environments. If you need to set up Poetry, see: Poetry python package manager

1. poetry-bumpversion

With Poetry-bumpversion, we can easily update the package version.

It assumes you follow Semver for the versioning of the project.

In short with semver the version consists of three numbers (M.m.p) where:

• M: major version. Represents breaking changes
• m: minor version. Represents relevant changes
• p: patch version. For fixes without new functionallity

# If you don't have Poetry installed, first do:
pip install poetry

# Add poetry-bumpversion
poetry self add poetry-bumpversion

# Make sure to install the poetry dependencies again with:
poetry install

poetry version major
poetry version minor
poetry version patch

/pyproject.toml

[[tool.poetry_bumpversion.replacements]]
files = ["dbt_project.yml"]
search = "version: '{current_version}'"
replace = "version: '{new_version}'"

2. Automatically validate versions

The idea is to create a GitHub action that automatically validates that the version is updated. This way you can decide if you want to increase the major, minor, or patch digit.

The first thing you need is a simple script that reads the package version from pyproject.toml:

/.github/scripts/get_version.py

import click
import toml
from loguru import logger as log

from utils import set_output

PYPROJECT_FILE = "pyproject.toml"


@click.command()
@click.option("--name")
def get_version(name):
    version = toml.load(PYPROJECT_FILE)["tool"]["poetry"]["version"]
    log.info(f"'{name}' branch {version=}")

    set_output(f"VERSION_{name.upper()}", version)


if __name__ == "__main__":
    get_version()

This scripts uses the set_output function to export the version as an environment variable in GitHub action. Since this is used in another python script, it is declared in a utils.py file:

/.github/scripts/utils.py

import os

from loguru import logger as log


def set_output(name, value):
    log.info(f"Setting {name=} {value=}")
    with open(os.environ["GITHUB_ENV"], "a") as fh:
        print(f"{name}={value}", file=fh)

Then you will need a script that checks if the version needs to be updated. It will compare the version from main to the one in the current PR.

/.github/scripts/check_if_update_needed.py

import click

from packaging import version
from loguru import logger as log

@click.command()
@click.option("--version_current")
@click.option("--version_main")
def compare_versions(version_current, version_main):
    log.info(f"Running with {version_current=}, {version_main=}")

    version_current = version.parse(version_current)
    version_main = version.parse(version_main)
    
    if version_current <= version_main:
        logger.error("Version needs to be updated")
        exit(1)

    logger.success(f"Version is correctly updated")
    exit(0)


if __name__ == "__main__":
    compare_versions()

And the last step is to create a GitHub action that validates the version:

/.github/workflow/fix_version.yaml

name: Fix Version

on:
  pull_request:
    paths:
      - dbt_northius/**
      - poetry.lock
      - pyproject.toml

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  fix:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4

      # Install requirements. Should match Dockerfile versions
      - name: Install requirements
        run: pip install poetry==1.6.1 poetry-bumpversion==0.3.1 toml loguru click

      # Get version from main
      - name: Checkout main
        uses: actions/checkout@v3
        with:
          ref: main
      - name: Get main version
        run: python .github/scripts/get_version.py --name=main

      # Get version from the current branch
      - name: Checkout current branch
        uses: actions/checkout@v3
        with:
          # Those are needed because of https://github.com/EndBug/add-and-commit#working-with-prs
          repository: ${{ github.event.pull_request.head.repo.full_name }}
          ref: ${{ github.event.pull_request.head.ref }}
      - name: Get current version
        run: python .github/scripts/get_version.py --name=current

      # Checks if version needs updating
      - name: Check if version needs to be updated
        run: python .github/scripts/check_if_update_needed.py --version_current=$VERSION_CURRENT --version_main=$VERSION_MAIN

      # Update only when needed
      - name: Update version
        if: env.NEEDS_UPDATE == 'true'
        run: poetry version minor

      # Commit changes and force GitHub_status to be updated
      - name: Commit new version
        if: env.NEEDS_UPDATE == 'true'
        uses: EndBug/add-and-commit@v9
        with:
          default_author: github_actions
          message: "Poetry minor version update"

The way this works is by:

1. Extracting the version in main branch
2. Extracting the version from the current branch in the pull request
3. If current_version <= main_version, then fail

3. Automatically tag versions

In order to keep better tracking of the package version, we will be tagging all commits to main with their version. We can do that with the following GitHub action:

/.github/workflow/tag_commits_on_main.yaml

name: Tag

on:
  push:
    branches:
      - main
    paths:
      - pyproject.toml

jobs:
  tag_with_version:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4

      - name: Install requirements
        run: pip install toml loguru click

      - name: Checkout current branch
        uses: actions/checkout@v3

      - name: Get current version
        run: python .github/scripts/get_version.py --name=current

      - name: Tag commit with current version
        uses: actions/github-script@v5
        with:
          script: |
            github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: `refs/tags/${{env.VERSION_CURRENT}}`,
              sha: context.sha
            })

4. Environment variables in github actions

One of the ways of passing information from one step in a GitHub action to another is by using environment variables.

- name: Get main version
  run: echo "VERSION_CURRENT=$(python scripts/get_version.py)" >> $GITHUB_ENV

def set_output(name, value):
    """
    Args:
        name:  name of the environment variable
        value: value to store in the environment variable
    with open(os.environ["GITHUB_ENV"], "a") as fh:
        print(f"{name}={value}", file=fh)

- name: Check if version needs to be updated
  run: python .github/scripts/check_if_update_needed.py --version_current=$VERSION_CURRENT --version_main=$VERSION_MAIN

`text ${{env.ENV_VAR_NAME}}`

- name: Tag commit with current version
  uses: actions/github-script@v5
  with:
    script: |
      github.rest.git.createRef({
        owner: context.repo.owner,
        repo: context.repo.repo,
        ref: `refs/tags/${{env.VERSION_CURRENT}}`,
        sha: context.sha
      })