Juan – Python, Databricks, GCP

The project was developed for the biggest hard discount retail company in Colombia. The main challenge was to utilize Lakehouse, unify all the company's data in a single source of truth, and replace a legacy on-premises data warehouse that processed more than 200 million transactions daily.

Juan's successes include the following achievements:

• Designed the architecture for the whole solution;
• Implemented the data ingestion processes from the different data sources;
• Build the data pipelines to transform the data;
• Modeled the data warehouse layer;
• Data Governance.

The project was developed for a popular pet e-commerce in Colombia. The main idea was to unify the company's data in a single source of truth, allowing the company to have a curated repository ready to use by BI tools in almost real-time.

This solution helped the company to democratize their data, allowing all the different business areas to access its data without any intermediates.

Juan achieved the following:

• Designed the architecture for the whole solution;
• Implemented the data ingestion processes from the different data sources;
• Build the data pipelines to transform the data;
• Modeled the data warehouse layer;
• Helped create several dashboards to track important KPIs;
• Data Governance.

This project intended automating the creation of different reports that tracked the performance KPIS from the slot machines deployed in several casinos across the USA. All the data was ingested into BigQuery, allowing the team to automate the different transformations needed for the reports, which allowed the company to have a single source of truth and replace man-made Excel files with dashboards in Tableau.

Among others, Juan managed such responsibilities as:

• Developed several data pipelines using BigQuery Stored procedures;
• Migrated all the business logic from Excel files to SQL queries;
• Helped build different dashboards tracking important KPIs.

Migration of the historical data from COBOL-indexed files to the Hadoop ecosystem for the largest credit bureau in the country. Additionally, reimplementation of all the business rules in Apache Spark and Scala.

Juan has successfully implemented the following tasks:

• Ingested all the historical data to the company HDFS;
• Recreated the business rules using Apche Spark;
• Deployed the new pipelines to a production cluster.