projects/data-fetching-pipeline
Data Fetching Pipeline
Built a pipeline framework for Novonesis that retrieves data from public APIs and saves it to Databricks Unity Catalog.
2026
Summary
A pipeline framework, built during a data engineering internship at Novonesis, that fetches data from public APIs and stores it in Databricks Unity Catalog. Without a shared framework, every team that needs API data for analytics and business intelligence repeats the same groundwork. With it, adding or changing a data source is simple, data management is already handled, and the resulting data is easy to access for analysis.
Stack
- Python
- PySpark
- Databricks and Unity Catalog
- GitHub Actions (CI/CD)
My role
Sole developer: designed and built the framework end to end during a data engineering internship in the Novonesis Data Office. Colleagues in the Data Office provided guidance.
- Reliability: error handling, logging, tests and automated scheduling.
- Delivery: CI/CD with GitHub Actions, and documentation so the team can maintain and extend the framework.
Results
Shipped the framework to production, along with a monitoring dashboard that tracks pipeline health and data freshness at a glance, and set up automatic email alerts whenever a pipeline run fails.
Links
No public links, since it is proprietary software owned by Novonesis.