Skip to content
a. skou-larsen

projects/data-fetching-pipeline

Data Fetching Pipeline

Built a pipeline framework for Novonesis that retrieves data from public APIs and saves it to Databricks Unity Catalog.

2026

Summary

A pipeline framework, built during a data engineering internship at Novonesis, that fetches data from public APIs and stores it in Databricks Unity Catalog. Without a shared framework, every team that needs API data for analytics and business intelligence repeats the same groundwork. With it, adding or changing a data source is simple, data management is already handled, and the resulting data is easy to access for analysis.

Stack

  • Python
  • PySpark
  • Databricks and Unity Catalog
  • GitHub Actions (CI/CD)

My role

Sole developer: designed and built the framework end to end during a data engineering internship in the Novonesis Data Office. Colleagues in the Data Office provided guidance.

  • Reliability: error handling, logging, tests and automated scheduling.
  • Delivery: CI/CD with GitHub Actions, and documentation so the team can maintain and extend the framework.

Results

Shipped the framework to production, along with a monitoring dashboard that tracks pipeline health and data freshness at a glance, and set up automatic email alerts whenever a pipeline run fails.