Ashwin – AWS, Python, Airflow

Led the modernization of an enterprise data platform by migrating legacy Spark SQL ETL pipelines to Snowflake. The project focused on establishing a scalable and secure cloud data architecture, implementing data governance and medallion architecture best practices, and ensuring data consistency throughout the migration. It also included building observability dashboards, optimizing query performance and cloud resource usage, and developing a custom MCP server to streamline production support and log analysis.

• Owned the data architecture for migrating legacy Spark SQL ETL pipelines to Snowflake;
• Defined the end-to-end migration strategy, including assessment, code conversion, validation, data quality checks, and reconciliation to ensure consistency with legacy outputs;
• Established a data governance framework for development and production Snowflake environments, including warehouses, databases, schemas, roles, and role-based access control (RBAC);
• Led the data migration initiative in collaboration with Snowflake consultants and provided technical direction to engineers;
• Defined enterprise data management best practices and established a medallion architecture for the migrated data platform;
• Built Streamlit dashboards to monitor Snowflake credit consumption, analyze usage patterns, and identify query optimization opportunities;
• Optimized high-credit Snowflake queries and adjusted Amazon EMR cluster sizing to reduce Snowflake and AWS infrastructure costs;
• Led the development of a custom MCP server to support use cases such as faster production troubleshooting and CloudWatch log analysis.

Led the development and integration of a data analytics platform, expanding its capabilities to support multiple data sources beyond Twitter (X). Collaborated with business stakeholders to define product requirements and led the engineering team in delivering new integrations, modernizing AWS infrastructure, building data ingestion pipelines, and developing application features.

• Led a diverse team of 8 members across Europe, overseeing project execution and ensuring alignment with organizational goals;
• Partnered with senior business stakeholders to define the data strategy and roadmap, aligning technology initiatives with business objectives and securing buy-in for new data engineering projects;
• Identified and implemented AWS cost-saving measures, reducing monthly cloud expenditure by $5,000;
• Collaborated with development and operations teams to implement AWS best practices and build a cost-efficient cloud architecture;
• Designed and automated CI/CD pipelines using CircleCI and the Serverless Framework, enabling seamless and efficient releases across development, staging, and production environments;
• Collaborated with third-party data providers to integrate external data sources and APIs into the application, negotiate data access agreements, and ensure compliance with data usage policies;
• Designed and developed ETL pipelines to integrate new data sources using Apache Spark (Spark SQL), dbt, Airflow, and Amazon S3.

Provided data engineering consulting services to insurance clients, focusing on ETL pipeline migrations and optimization of claims processing workflows to improve data integration, processing efficiency, and reliability.

• Led data migration from on-prem Oracle Data Warehouse to Snowflake on AWS;
• Optimized ETL processes and data pipelines with Apache Spark and Airflow;
• Re-engineered critical ETL pipelines, increasing data processing efficiency and reducing latency;
• Converted ML models from R to Python and deployed them using CI/CD pipelines;
• Developed dynamic user interfaces using Plotly Dash to provide actionable insights to stakeholders;
• Mentored junior data engineers across the US, Canada, and India on data engineering best practices.

Contributed to the development of a new lending strategy for a small business banking unit, focusing on streamlining loan application and approval processes. The project involved implementing near-real-time credit scoring capabilities to support faster, data-driven lending decisions.

• Developed APIs with Spring Boot to interface with external information sources and publish data into OneLake;
• Automated data quality checks, removed duplicate records, and performed balance aggregates using Apache Spark;
• Created CI/CD pipeline integration for applications using Jenkins and Git hooks;
• Developed a Python package to pull analytical data using DBT and Jinja templating with data quality, lineage, and audit trail capabilities.

A digital platform for automotive dealerships, integrating third-party data from multiple automotive marketplaces and vehicle valuation providers. The project focused on consolidating external data sources to enhance product functionality and provide dealerships with more comprehensive vehicle information and business tools.

• Implemented a lambda architecture using MongoDB and Vertica to support real-time and historical querying for client-facing applications;
• Generated scatter plots and bar charts with zoom components using React and D3.js to help clients understand inventory;
• Improved DevOps through Terraform-based infrastructure automation and Jenkins pipeline integration;
• Optimized Vertica tables and created faster projections for quicker query responses;
• Created data pipelines using AWS Step Functions and Athena to replace long-running EMR clusters for processing data from Amazon S3 and Kinesis;
• Reduced AWS costs by using EKS with AWS Step Functions and Lambdas instead of long-running EMR clusters;
• Applied ML algorithms in SageMaker to predict vehicle scores for sales predictions;
• Designed and developed a real-time aggregation framework using Python, Apache Spark, Redis, DynamoDB, and Elasticsearch to generate KPIs.
• Developed backend application features for "Connections," an app that helped dealers get insights into shopper behavior using real-time data feeds;
• Migrated AWS resources to new regions;
• Developed microservices-based applications.

Development of a real-time A/B testing dashboard for automotive dealerships, enabling dealers to monitor experiment results, compare performance metrics, and evaluate the effectiveness of different strategies to optimize their online presence and business performance.

• Designed and developed a data pipeline framework for aggregating KPIs such as ad click rates, bounce rates, TTI, and TTFB;
• Developed applications using Apache Spark to process around 13 million logs per day in real time using AWS;
• Optimized data pipeline architecture to reduce AWS costs;
• Automated infrastructure development using Terraform;
• Applied machine learning algorithms to forecast TTI and TTFB values;
• Analyzed data and created data models for storing data in Elasticsearch indexes and Hive tables;
• Developed a monitoring and alerting system for anomaly detection using forecasted values.