Villoro's personal websitehttps://villoro.com/enhttps://villoro.com/blog/be-nice-to-data-team/https://villoro.com/blog/be-nice-to-data-team/A practical guide for product, engineering, ops, and business teams on how to design source systems and frame requests so the data team can deliver faster and more reliably.Tue, 05 May 2026 00:00:00 GMTData EngineeringBest PracticesAdminhttps://villoro.com/blog/marimo-notebooks/https://villoro.com/blog/marimo-notebooks/Why marimo is an interesting notebook alternative for Python projects.Wed, 15 Apr 2026 00:00:00 GMTPythonToolsAWSDEAdminhttps://villoro.com/blog/recover-s3-files-delete-markers/https://villoro.com/blog/recover-s3-files-delete-markers/Learn how to recover accidentally deleted S3 files using versioning and delete markers — including listing deleted objects, filtering by time window, and restoring them with boto3 and pandas.Tue, 17 Mar 2026 00:00:00 GMTDEAWSS3PythonBest PracticesVersioningAdminhttps://villoro.com/blog/protecting-prod-tables-dbt/https://villoro.com/blog/protecting-prod-tables-dbt/Learn three proven strategies to safeguard production tables in dbt: protecting against bad data, preventing unexpected changes, and securing sensitive data with seeds.Tue, 17 Feb 2026 00:00:00 GMTdbtSQLDEAdminhttps://villoro.com/blog/github-hooks/https://villoro.com/blog/github-hooks/Learn how to build scalable GitHub Actions pipelines using reusable hooks, matrix jobs, gate checks, and concurrency for cleaner, faster, and more maintainable CI/CD workflows.Wed, 14 Jan 2026 00:00:00 GMTCI/CDGitHubPythonBest PracticesMonorepoAdminhttps://villoro.com/blog/monorepo-multimage/https://villoro.com/blog/monorepo-multimage/Learn how to evolve a Python project from a single workspace to a scalable, multi-package monorepo with isolated builds, targeted CI, and lightweight ECS deployments using uv.Tue, 09 Dec 2025 00:00:00 GMTAWSECSDockeruvDEFargateMonorepoAdminhttps://villoro.com/blog/fast-python-ecs-with-uv/https://villoro.com/blog/fast-python-ecs-with-uv/Speed up ECS tasks with lightweight Docker images and versioned Python environments using uv. Learn how to build once, run anywhere with zero rebuilds and instant cold starts.Tue, 25 Nov 2025 00:00:00 GMTAWSECSDockeruvDEFargateAdminhttps://villoro.com/blog/querying-apis-best-practices/https://villoro.com/blog/querying-apis-best-practices/Learn proven best practices for building resilient Python API clients: structure, safety, error handling, and advanced patterns like streaming and incremental loads.Tue, 14 Oct 2025 00:00:00 GMTPythonAPIDEBest PracticesAdminhttps://villoro.com/blog/duckdb-fast-analytics/https://villoro.com/blog/duckdb-fast-analytics/DuckDB is the “SQLite for analytics”: fast, modern SQL, and capable of billions of rows on a laptop. Learn its SQL superpowers, explore practical use cases, and discover the promise of DuckLake.Mon, 15 Sep 2025 00:00:00 GMTPythonSQLDEAdminhttps://villoro.com/blog/vpn-tailscale/https://villoro.com/blog/vpn-tailscale/Explore the differences between commercial VPNs for privacy and security and private network VPNs for secure remote access. Learn how Tailscale simplifies private VPN setups and how exit nodes can help bypass restrictions.Fri, 08 Aug 2025 00:00:00 GMTToolsSetupSecurityNetworkingAdminhttps://villoro.com/blog/handling-pipeline-failures/https://villoro.com/blog/handling-pipeline-failures/Learn how to make your data pipelines resilient by design using the ETU (Extract, Transform, Upload) approach. This guide covers failure handling strategies in dbt, Prefect, and more.Thu, 12 Jun 2025 00:00:00 GMTSQLDEBest PracticesdbtPrefectAdminhttps://villoro.com/blog/smoothing-time-series/https://villoro.com/blog/smoothing-time-series/A practical guide to smoothing time series data using Python and SQL. Explore moving averages, Gaussian and Lowess smoothers, SQL medians, quantiles, and their trade-offs through side-by-side visual comparisons.Thu, 08 May 2025 00:00:00 GMTDEPythonSQLBest PracticesAdminhttps://villoro.com/blog/improving-aws-athena-performance-by-reducing-glue-versions/https://villoro.com/blog/improving-aws-athena-performance-by-reducing-glue-versions/Learn how AWS Glue schema version limits can silently break Athena queries and dbt runs—and how cleaning up old versions can improve performance by up to 4×.Thu, 10 Apr 2025 00:00:00 GMTDEAWSBest PracticesPerformancedbtAdminhttps://villoro.com/blog/astral-tools-uv-and-ruff/https://villoro.com/blog/astral-tools-uv-and-ruff/Discover how Astral's Rust-powered tools, uv and ruff, provide a blazing-fast alternative for Python package management and linting. Learn how to set them up, integrate them into CI/CD workflows, and boost your development speed.Fri, 14 Mar 2025 00:00:00 GMTPythonToolsPerformanceBenchmarkSetupBest PracticesAdminhttps://villoro.com/blog/manage-s3-costs/https://villoro.com/blog/manage-s3-costs/Discover strategies to optimize Amazon S3 costs by leveraging inventory exports and lifecycle policies for automated storage management.Fri, 28 Feb 2025 00:00:00 GMTAWSS3Best PracticesAdminhttps://villoro.com/blog/megasync-terramaster-docker-setup/https://villoro.com/blog/megasync-terramaster-docker-setup/A step-by-step guide to installing and configuring MEGASync on a TerraMaster NAS using Docker for seamless MEGA cloud synchronization.Wed, 15 Jan 2025 00:00:00 GMTNASHardwareTerramasterDockerAdminhttps://villoro.com/blog/setting-up-terramaster-nas/https://villoro.com/blog/setting-up-terramaster-nas/A comprehensive guide on setting up and configuring the Terramaster F2-424 NAS, including hardware selection, RAID setup, storage configuration, network optimization, and essential app installations.Sat, 21 Dec 2024 00:00:00 GMTNASHardwareTerramasterAdminhttps://villoro.com/blog/control-amd-cpu-temperatures/https://villoro.com/blog/control-amd-cpu-temperatures/Learn how to control your AMD CPU's temperatures by configuring TDP limits to apply automatically at startup using Task Scheduler in Windows.Fri, 29 Nov 2024 00:00:00 GMTTutorialSetupPerformanceWindowsAdminhttps://villoro.com/blog/solving-problem-small-files-data-lake/https://villoro.com/blog/solving-problem-small-files-data-lake/This post addresses the common problem of small files in data lakes, which can lead to significant performance degradation and increased costs. It provides an in-depth guide on understanding the issues caused by small files, determining optimal file sizes, and effectively managing file sizes using tools like Apache Spark, AWS Athena, Delta Lake, and Apache Iceberg. The post also covers strategies for tracking file sizes and partitioning tables to optimize data processing and storage efficiency.Wed, 23 Oct 2024 00:00:00 GMTDEPerformanceSparkdbtAdminhttps://villoro.com/blog/async-openai-calls-rate-limiter/https://villoro.com/blog/async-openai-calls-rate-limiter/Learn how to implement multiple concurrent async calls to the OpenAI API while managing quota limits effectively using a rate limiter in Python. This guide covers best practices for API management in data engineering.Wed, 25 Sep 2024 00:00:00 GMTAPIBest PracticesDEOpenAIPythonAdminhttps://villoro.com/blog/transcribe-audios-whisper-extract-structured-data-chatgpt/https://villoro.com/blog/transcribe-audios-whisper-extract-structured-data-chatgpt/Learn how to extract and analyze call data using OpenAI models, with a focus on structured outputs for clear and consistent results. This guide covers transcribing calls with Whisper, extracting insights with ChatGPT, handling invalid calls, and processing multiple files efficiently. Explore how to use Pydantic models and OpenAI's API for structured data extraction.Mon, 02 Sep 2024 00:00:00 GMTPythonDEAITutorialOpenAIAdminhttps://villoro.com/blog/clean-windows-install/https://villoro.com/blog/clean-windows-install/Learn how to download, install, and set up a clean Windows installation without bloatware. This comprehensive guide covers creating a bootable USB, setting up a local account, and configuring essential settings for optimal performance.Wed, 14 Aug 2024 00:00:00 GMTTutorialSetupWindowsAdminhttps://villoro.com/blog/extracting-data-salesforce-python/https://villoro.com/blog/extracting-data-salesforce-python/The post serves as a comprehensive guide to extracting data from Salesforce using Python, focusing on the Simple Salesforce library. It covers various querying options, strategies to avoid timeouts and memory errors, and insights into different types of fields within Salesforce. Additionally, it provides practical code examples and recommendations for effective data extraction.Tue, 16 Jul 2024 00:00:00 GMTPythonDETutorialAdminhttps://villoro.com/blog/dbt-testing-duckdb/https://villoro.com/blog/dbt-testing-duckdb/Learn how to test dbt projects using DuckDB to ensure high-quality, consistent SQL code. This guide covers setting up a SQL linter with Sqlfluff, automating testing with pre-commit hooks, and creating a streamlined CI pipeline for efficient testing.Wed, 19 Jun 2024 00:00:00 GMTSQLDEBest PracticesdbtAdminhttps://villoro.com/blog/running-dbt-with-aws-ecs-fargate/https://villoro.com/blog/running-dbt-with-aws-ecs-fargate/Guide to setting up and running dbt on AWS ECS (Fargate), covering Dockerization, package handling, Docker images, integration with Prefect and exporting results.Fri, 07 Jun 2024 00:00:00 GMTSQLDEAWSSetupTutorialdbtAdminhttps://villoro.com/blog/self-healing-pipelines/https://villoro.com/blog/self-healing-pipelines/This post explores self-healing pipelines, which automate backfilling missing data to reduce manual interventions. It discusses challenges in traditional batch processing and introduces techniques like partitioning, deduplication, and schema change handling. Detailed Python and SQL code snippets illustrate implementation, emphasizing the importance of proper setup in dbt.Thu, 23 May 2024 00:00:00 GMTDEBest PracticesAdminhttps://villoro.com/blog/effortless-emr-guide-running-pyspark/https://villoro.com/blog/effortless-emr-guide-running-pyspark/Tired of complex setups for PySpark on EMR? This guide offers a simpler approach. From running scripts to handling parameters and using Docker, we cover it all. With practical examples and clear explanations, simplify your PySpark workflow on EMR and make big data processing a breeze! Tue, 07 May 2024 00:00:00 GMTAWSTutorialAdminhttps://villoro.com/blog/prefect-server-setup-configuration-deployment/https://villoro.com/blog/prefect-server-setup-configuration-deployment/Learn how to set up and deploy Prefect Server for orchestrating workflows seamlessly. Understand the steps involved, including server configuration, creating deployments, and connecting to the serverThu, 11 Apr 2024 00:00:00 GMTToolsOrchestrationTutorialSetupAdminhttps://villoro.com/blog/prefect-essentials-setup-and-migration/https://villoro.com/blog/prefect-essentials-setup-and-migration/Discover Prefect's simplicity and flexibility. Learn the basics, setup Prefect Cloud, and smoothly migrate from other orchestrators.Mon, 08 Apr 2024 00:00:00 GMTToolsOrchestrationIntroSetupAdminhttps://villoro.com/blog/clean-repo-rewrite-git-history/https://villoro.com/blog/clean-repo-rewrite-git-history/Discover how to clean up your Git history effortlessly with our step-by-step guide. Learn to eliminate unnecessary commits, maintain versioning consistency, and automate the process with Python scripts. Streamline your repository and enhance collaboration today!Sun, 17 Mar 2024 00:00:00 GMTGITAdminhttps://villoro.com/blog/managing-package-versions-with-poetry/https://villoro.com/blog/managing-package-versions-with-poetry/This post explains how to automate project version management using Poetry and poetry-bumpversion. It follows semantic versioning and recommends tools for updating versions. GitHub actions are set up to automatically update the version, commit changes when necessary, and tag commits on the main branch. Tue, 07 Nov 2023 00:00:00 GMTPythonToolsPoetryVersioningAdminhttps://villoro.com/blog/reading-from-redshift-with-pyarrow/https://villoro.com/blog/reading-from-redshift-with-pyarrow/This post discusses efficient ways to consume data from Amazon Redshift as pandas dataframes. It outlines the challenges of scaling Redshift performance, especially with bots consuming data, and provides solutions for more efficient data access. The recommended approach is unloading data to Parquet files, and the post explains various methods for reading Parquet files into pandas dataframes using pandas, pyarrow, and pyarrow.dataset. The performance comparisons demonstrate the advantages of the new pyarrow.dataset introduced in pyarrow 3.0.0 for handling partitioned data efficiently. Tue, 08 Jun 2021 00:00:00 GMTPythonDESQLAWSAdminhttps://villoro.com/blog/regex-python/https://villoro.com/blog/regex-python/This post provides a detailed guide on regular expressions (regex), covering their basics, character classes, quantifiers, groups, boundaries, flags, and more. It also explains how to use regex in Python, Pandas, and Amazon Redshift, offering practical examples and functions for working with text data. This comprehensive resource helps users understand and apply regex in various programming contexts. Mon, 05 Apr 2021 00:00:00 GMTPythonAdminhttps://villoro.com/blog/ing-api-python/https://villoro.com/blog/ing-api-python/Learn how to interact with ING's Open Banking API using Python. This guide covers setting up certificates, making authenticated requests, and integrating with the ING API to perform basic operations.Wed, 10 Mar 2021 00:00:00 GMTPythonDEAPIAdminhttps://villoro.com/blog/java-udf-pyspark/https://villoro.com/blog/java-udf-pyspark/Learn how to create and use Java UDFs with PySpark for improved performance. This guide covers setting up Java functions, compiling them, and integrating them into your PySpark workflows, with a performance comparison to Python UDFs.Tue, 03 Nov 2020 00:00:00 GMTPythonSparkBenchmarkAdminhttps://villoro.com/blog/python-environment-configuration/https://villoro.com/blog/python-environment-configuration/Learn how to properly set up a python development environment with all the useful tools you can use as well as some good tips to work as a pro.Wed, 20 May 2020 00:00:00 GMTPythonToolsAdminhttps://villoro.com/blog/pre-commit/https://villoro.com/blog/pre-commit/With pre-commit you can automate some repetitive tasks before doing a commit on a git repository. It is useful for identifying simple issues before submission to code review. Thu, 14 May 2020 00:00:00 GMTToolsAdminhttps://villoro.com/blog/poetry-python-package-manager/https://villoro.com/blog/poetry-python-package-manager/Poetry is a python packaging and dependency manager. It makes it really easy to manage packages while using environments under the hood. It also allows build and publishing packages to PyPI (or other sites). Fri, 08 May 2020 00:00:00 GMTPythonToolsAdminhttps://villoro.com/blog/cmder-terminal-windows/https://villoro.com/blog/cmder-terminal-windows/Cmder is a really good terminal for windows that can replace the default one.Sat, 29 Feb 2020 00:00:00 GMTToolsAdminhttps://villoro.com/blog/luigi-orchestrator/https://villoro.com/blog/luigi-orchestrator/Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more. Mon, 06 Jan 2020 00:00:00 GMTToolsOrchestrationAdminhttps://villoro.com/blog/pyspark-example/https://villoro.com/blog/pyspark-example/This is an example of how to work with spark dataframes using as an example a CSV with 22 GB. This should be an example that you probably won't be able to work with pandas. It will give some best practices for working with data of this size. Tue, 24 Dec 2019 00:00:00 GMTPythonSparkAdminhttps://villoro.com/blog/databricks-intro/https://villoro.com/blog/databricks-intro/Databricks provides a Unified Analytics Platform that accelerates innovation by unifying data science, engineering and business. Learn how to set databricks up and what it can offer you. Mon, 23 Dec 2019 00:00:00 GMTToolsSparkAdminhttps://villoro.com/blog/h2o-automl/https://villoro.com/blog/h2o-automl/H2O is a fully open source, distributed in-memory machine learning platform with linear scalability. In this post you will see how to use AutoML to let h2o train multiple Machine Learning algorithms automatically. Sun, 22 Dec 2019 00:00:00 GMTPythonSparkMLIntroAdminhttps://villoro.com/blog/pyspark-intro/https://villoro.com/blog/pyspark-intro/Apache Spark is a very fast unified analytics engine for big data and machine learning. This is a beginner tutorial of how to use spark to work with dataframes.Sat, 21 Dec 2019 00:00:00 GMTPythonSparkIntroAdminhttps://villoro.com/blog/pandas-intro/https://villoro.com/blog/pandas-intro/Learn how to use python pandas library to work with tabular data.Fri, 01 Nov 2019 00:00:00 GMTPythonPandasIntroAdminhttps://villoro.com/blog/dropbox-python/https://villoro.com/blog/dropbox-python/Learn how to read, write and delete all kind of files using the dropbox library for python.Sat, 14 Sep 2019 00:00:00 GMTPythonPandasDEAdminhttps://villoro.com/blog/google-analytics-etl-python/https://villoro.com/blog/google-analytics-etl-python/Google Analytics is really useful but sometimes you want to retrieve all the data and perform your own analysis and create your reports. In this post you will see how to do that with python.Thu, 04 Jul 2019 00:00:00 GMTAPIPythonPandasDETutorialAdminhttps://villoro.com/blog/git-with-ssh/https://villoro.com/blog/git-with-ssh/Learn how to work securely with git by using SSH authentication with github.Sat, 15 Jun 2019 00:00:00 GMTGITSecuritySetupAdminhttps://villoro.com/blog/using-gitflow/https://villoro.com/blog/using-gitflow/Development good practices: how to using gitflow as code control.Tue, 04 Jun 2019 00:00:00 GMTGITBest PracticesAdminhttps://villoro.com/blog/improve-python-performance-with-numba/https://villoro.com/blog/improve-python-performance-with-numba/Numba translates Python functions to optimized machine code at runtime. Numba-compiled numerical algorithms in Python can approach the speeds of C or FORTRAN.Tue, 21 May 2019 00:00:00 GMTPythonBenchmarkPerformanceAdminhttps://villoro.com/blog/python-logging/https://villoro.com/blog/python-logging/An overview of my own custom logging library that handles console and file output.Thu, 16 May 2019 00:00:00 GMTPythonToolsAdminhttps://villoro.com/blog/encrypting-secrets-python/https://villoro.com/blog/encrypting-secrets-python/Learn how a team can work securely with password by storing them encrypted in a dictionary-like file.Fri, 10 May 2019 00:00:00 GMTPythonSecurityAdminhttps://villoro.com/blog/creating-python-packages/https://villoro.com/blog/creating-python-packages/Learn how to create a python package that can be installed with pip.Thu, 02 May 2019 00:00:00 GMTPythonToolsPyPIAdminhttps://villoro.com/blog/aws-lambdas-to-control-ec2/https://villoro.com/blog/aws-lambdas-to-control-ec2/Learn how to use AWS Lambdas to start/stop EC2 and RDS instances using an API.Tue, 23 Apr 2019 00:00:00 GMTPythonAWSAdminhttps://villoro.com/blog/python-decorators/https://villoro.com/blog/python-decorators/Decorators are a really useful tool for python development but not a lot of people know what they are and how they work. In this post you are going to master them so that you can use them and create your own.Tue, 16 Apr 2019 00:00:00 GMTPythonAdminhttps://villoro.com/blog/intro-machine-learning/https://villoro.com/blog/intro-machine-learning/An introduction to what Machine Learning is and how it works using a real example.Fri, 12 Apr 2019 00:00:00 GMTPythonMLIntroTutorialAdminhttps://villoro.com/blog/storing-tables-efficiently-pandas/https://villoro.com/blog/storing-tables-efficiently-pandas/When working with pandas people usually need to store one or more tables. There are a lot of different formats to do that. In this post I am going to compare the performance between them.Wed, 03 Apr 2019 00:00:00 GMTPythonBenchmarkPerformanceAdminhttps://villoro.com/blog/serving-python-app-nginx-gunicorn/https://villoro.com/blog/serving-python-app-nginx-gunicorn/How to serve Python apps with nginx and gunicorn.Thu, 21 Mar 2019 00:00:00 GMTPythonWebAdminhttps://villoro.com/blog/setting-up-airflow/https://villoro.com/blog/setting-up-airflow/How to set up Apache Airflow, the platform to programmatically author, schedule and monitor workflows.Tue, 19 Mar 2019 00:00:00 GMTToolsOrchestrationTutorialSetupAdminhttps://villoro.com/blog/creating-ec2-aws/https://villoro.com/blog/creating-ec2-aws/How to create an EC2 AWS instance. Detailed instructions on how to do it in some minutes.Mon, 18 Mar 2019 00:00:00 GMTAWSTutorialSetupAdminhttps://villoro.com/blog/create-static-website/https://villoro.com/blog/create-static-website/See how I created a static webpage in some minutes. And how you can do it too.Tue, 12 Mar 2019 00:00:00 GMTPythonWebTutorialSetupAdminhttps://villoro.com/blog/basic-tools-for-blogs/https://villoro.com/blog/basic-tools-for-blogs/View how to set up a mailing list (Mailchimp), add a comments section (Disqus) on posts and track visits (Google Analytics).Sat, 02 Mar 2019 00:00:00 GMTWebToolsSetupAdminhttps://villoro.com/blog/singleton-python/https://villoro.com/blog/singleton-python/The singleton pattern will help you have only one instance and allow you to call only one sync function.Sun, 24 Feb 2019 00:00:00 GMTPythonAdminhttps://villoro.com/blog/string-best-practices-with-python/https://villoro.com/blog/string-best-practices-with-python/There are a lot of ways to work with strings in Python. And there are some cool tricks I want to share that will make it easier to deal with strings.Sat, 16 Feb 2019 00:00:00 GMTPythonBest PracticesAdminhttps://villoro.com/blog/material-colors-and-palettes/https://villoro.com/blog/material-colors-and-palettes/Choosing between colors is always a difficult task. But with the material design palette it is possible to solve most of color related problems easily.Fri, 15 Feb 2019 00:00:00 GMTPythonWebBest PracticesAdminhttps://villoro.com/blog/sql-alchemy-python/https://villoro.com/blog/sql-alchemy-python/SQL is one of the most relevant languages for databases. So at one point you will need to interact to get, store or modify some data. Fortunately you can do it in python using SQL Alchemy.Fri, 08 Feb 2019 00:00:00 GMTPythonSQLIntroTutorialAdminhttps://villoro.com/blog/black-code-formatting/https://villoro.com/blog/black-code-formatting/Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. Thu, 07 Feb 2019 00:00:00 GMTPythonToolsBest PracticesAdminhttps://villoro.com/blog/personal-webpage/https://villoro.com/blog/personal-webpage/I used to have a personal webpage made with WordPress. Since I have been working as a developer for some years I decided that I could create something that better suited my needs.Thu, 27 Dec 2018 00:00:00 GMTWebPythonSetupAdmin