Thosan – Python, SQL, Apache Spark

The client was a company that specializes in providing chat commerce solutions, aiming to enhance the customer experience by leveraging conversational AI to facilitate seamless and engaging interactions between businesses and their customers. The main challenge was to architect the event-driven data lakehouse, harvesting insights from hundreds of streaming sources to optimize recommendations, conversion, and retention rates for customer-facing companies worldwide.

Thosan managed the following tasks:

• developed an Airflow-based data ingestion using Dataproc batch;
• built Data Lakehouse in event-based architecture on BQ-Iceberg using Spark;
• handled muti-tenant analytics page migration using GoodData and DBT.

This client was an Indonesian agritech company that provides technology solutions for fish and shrimp farming. Their products included automatic feeders and data-driven platforms to optimize feeding, reduce waste, and improve yields. The aim of the project was to build a modern, high-quality data pipeline, data warehouse, and data analytics platform to help hundreds of data users and thousands of aquaculture farmers across Indonesia.

Among others, Thosan carried out such tasks:

• kickstarted and designed data quality initiatives using multiple custom Great Expectations scripts and Trino, as a query engine;
• built a data observability platform using Datahub, including a custom data lineage error tracing mechanism;
• setup & migrated data warehouse modeling platform using DBT.

The customer was a leading Indonesian e-commerce platform, providing a marketplace for small and medium-sized enterprises (SMEs) to sell their products online. They offered a wide range of goods and services, including electronics, fashion, and household items, catering to millions of users across Indonesia. The project aimed to empower local businesses by facilitating digital transformation and improving access to the online market.

• optimized GCP Data Architecture to handle hundreds of queries per minute;
• assessing multiple data architecture performances between GCP, Azure, and Databrick;
• spearheaded the development of a customized data modeling platform (DBT-like), tailored to the client's specific requirements;
• maintained multi-instance airflow for data modeling and pipelining;
• ensuring a high BigQuery success rate (>90%) by optimizing DS queries and data modeling while decreasing its operational cost;
• built a cost anomalies detection for multiple GCP Projects;
• built a custom-curated dashboard website using Looker, ReactJS, and Spring for C-level executives as a daily company overview;
• mentored and guided junior data engineers for a better onboarding process.

The client was the government agency responsible for collecting, analyzing, and disseminating statistical data in Indonesia. Established to support national development, they provided comprehensive data on various sectors, including population, economy, agriculture, and industry. The agency aimed to deliver accurate and timely statistics to inform policy-making, development planning, and public awareness.

Thosan proficiently implemented the following:

• setup automated web crawlers;
• built pipeline for data collection, data wrangling, and data visualization;
• analyzed data to get actionable insights from it, and presented it to B-level executives as a policy recommendation for the president;
• conducted multiple researches on calculating poverty rate based on nighttime light satellite imagery and approximating the occupancy rate using multiple OTA public API data.