Improving Athena performance by reducing Glue versions

1.1. Glue versions in dbt

On the dbt side, the impact was smaller — but not zero.

We’re mostly using materialized = table, which means dbt recreates the table entirely on each run. This avoids version accumulation because Glue treats each run as a clean slate.

However, views are a different story. Since they’re not dropped and recreated, each run updates the definition, which adds a new Glue version every time.

dbt does support version cleanup using the expire_glue_table_versions macro, and you can configure how many versions to keep using the versions_to_keep parameter (see DBT Athena config | High availability for more info).

This is especially useful when working with incremental Iceberg models or when using Hive tables with high availability (HA) — the two cases where Glue version bloat can accumulate rapidly. By default, versions_to_keep is set to 4, but this can be customized.

In our case, it’s not needed because we use materialized = table, so our tables are fully recreated on each run, avoiding accumulation.

So while dbt wasn’t the main contributor to version bloat, it still affected our total version count through view models.

2.1. Getting database and tables info

from datetime import datetime, timedelta, timezone
import awswrangler as wr
import boto3
import botocore
from loguru import logger

DB_DEFAULT = "default"
MIN_VERSIONS = 5
RETENTION_DAYS = 14

def get_databases(limit=1000):
    logger.info(f"Getting all databases ({limit=})")
    df = wr.catalog.databases(limit=limit)
    logger.info(f"There are {df.shape[0]} databases")
    return df["Database"].to_list()

def get_tables(database, limit=1000):
    logger.info(f"Getting all tables in {database=} ({limit=})")
    df = wr.catalog.tables(database=database, limit=limit)
    logger.info(f"There are {df.shape[0]} tables in {database=}")
    return df["table"].to_list()

def get_versions(database, table, client=None):
    client = client or boto3.client("glue")
    paginator = client.get_paginator("get_table_versions")
    out = []
    for data in paginator.paginate(DatabaseName=database, TableName=table):
        out += data["TableVersions"]
    return out

2.2. Cleaning versions of a table with batches

def _chunked(iterable, size):
    """Yield successive chunks from iterable of given size."""
    for i in range(0, len(iterable), size):
        yield iterable[i : i + size]

def clean_old_glue_table_versions(database, table, min_versions, retention_days):
    cutoff_date = datetime.now(timezone.utc) - timedelta(days=retention_days)
    prefix = f"Table '{database}.{table}'"
    client = boto3.client("glue")
    versions = get_versions(database, table, client=client)
    versions = sorted(versions, key=lambda v: v["Table"]["UpdateTime"])

    if len(versions) <= min_versions:
        logger.debug(f"{prefix} has {len(versions)} versions < {min_versions=}, nothing to do")
        return 0

    logger.debug(f"{prefix} has {len(versions)} versions")

    old_versions = []
    for v in versions:
        created = v.get("Table", {}).get("UpdateTime")
        if created and created < cutoff_date:
            old_versions.append(v)

    if len(old_versions) <= min_versions:
        logger.debug(
            f"{prefix} has {len(old_versions)} versions older "
            f"than {retention_days=} < {min_versions=}, nothing to do"
        )
        return 0

    delete_versions = old_versions[:-min_versions]

    logger.info(
        f"{prefix} has {len(old_versions)} versions older "
        f"than {retention_days=}, deleting {len(delete_versions)}"
    )

    version_ids = [x["VersionId"] for x in delete_versions]

    for chunk in _chunked(version_ids, 100):
        client.batch_delete_table_version(
            DatabaseName=database, TableName=table, VersionIds=chunk
        )
    return len(version_ids)

2.3. Cleaning the whole catalog

def clean_one_database(database, min_versions, retention_days):
    tables = get_tables(database)
    logger.info(f"Processing {database=} with {len(tables)=}")
    n_deletes = 0
    for table in tables:
        try:
            n_deletes += clean_old_glue_table_versions(
                database, table, min_versions, retention_days,
            )
        except botocore.exceptions.ClientError as e:
            if e.response["Error"]["Code"] == "EntityNotFoundException":
                logger.warning(f"Table '{database}.{table}' not found, skipping it")
            else:
                raise # Re-raise unknown errors
    logger.info(f"{database=} done with {n_deletes=}")
    return n_deletes

def process(min_versions=MIN_VERSIONS, retention_days=RETENTION_DAYS):
    databases = get_databases()
    total_deletes = 0
    for i, database in enumerate(databases):
        logger.info(f"[Database {i + 1}/{len(databases)}] Processing {database=}")
        total_deletes = clean_one_database(database)
    logger.info(f"{len(databases)} databases proccessed with {total_deletes=}")