Jobbyo.ai
|Python Data Engineer
Remote, Spain
Summary
Engineered and maintained scalable, high-throughput job ingestion pipelines, developed real-time analytics dashboards, and implemented CI/CD automation for robust data processing and enrichment.
Highlights
Engineered and maintained scalable, high-throughput job ingestion pipelines using Python, Selenium, HTTP, and API collectors, optimizing with multithreading and retry logic to reliably process large volumes of diverse job data.
Developed real-time analytics and data quality dashboards in Metabase, enhancing visibility into job coverage, throughput, and deduplication rates to enable faster issue detection and data-driven decisions.
Implemented CI/CD automation using GitHub Actions for Dockerized scraping and processing services, enabling centralized container log streaming and runtime visibility via Dozzle for robust production operations.
Integrated Sentry for structured logging, error tracking, and alerting, significantly improving observability across distributed pipelines and reducing diagnostic and resolution times for production issues.
Built robust data processing and enrichment workflows in Python on GCP with PostgreSQL, leveraging OpenAI for AI-assisted extraction and geocoding to ensure consistent, high-quality structured data.