Vadym – Python, Apache Spark, Snowflake

A greenfield development of a Databricks-based Data Platform for an e-commerce business, leveraging an AI SDLC. The platform powers a B2B spare-parts procurement business serving the automotive market. The entire development lifecycle runs on AI SDLC with AI agents covering all stages, using tools, MCPs, skills, commands, and memory

• Led the greenfield development of a Databricks-based data platform for an e-commerce business using an AI-driven SDLC.
• Created and tuned agents and skills for the full development cycle, including architecture design and validation, implementation, testing, and reviewing, using Claude Code.
• Designed the core data platform architecture.
• Conducted stakeholder workshops, gathered requirements, and cross-collaborated with infrastructure teams for resource provisioning.
• Authored ADRs to justify decisions, including ingestion patterns, CI/CD pipelines, deployment strategies, and data model design.
• Implemented the solution on Databricks utilizing Lakeflow SDP and Declarative Automation Bundles.

An enterprise data warehouse for the biggest student transportation company in North America. The company has a fleet of around 50000 vehicles and more than 60000 workers.

• Led the release of enterprise-wide data models, driving 30 reports and 1 data product.
• Aligned data engineering efforts with the concurrent development of the underlying source systems.
• Reduced Snowflake cost by up to 40% through auto-suspend tuning, warehouse right-sizing, scaling policy tuning, and other adjustments.
• Implemented CI pipelines to run dbt unit and data tests, decreasing bugs in UAT and production environments by about 30%.
• Gathered requirements to fulfill the needs of business users.
• Designed data models using Kimball methodology with dbt and Snowflake.
• Developed more than 100 DBT models processing tables up to 6 TB of data.
• Developed models for finance, operations, and telemetric domains.

A migration from SAP IQ to Redshift and a proprietary orchestration tool to Airflow.

• Designed and developed Airflow pipelines that extracted and loaded data across Redshift, S3, Postgres, HBase, Kafka, and REST API-based systems.
• Migrated SQL stored procedures to dbt.
• Optimized Redshift performance for dbt models from 5 hours to 1 hour on the same cluster configuration.
• Designed and implemented CI/CD pipelines for dbt and MWAA.

A data infrastructure for analytics from scratch as the first data engineer on the project.

• Designed a data lake on AWS S3.
• Developed ETL jobs to populate the data lake using Glue with PySpark.
• Designed a data lakehouse on AWS using S3, Athena, and Iceberg.
• Used dbt for data modeling, documentation, data quality, and data lineage.

An entertainment media that gets visitors from North America and Europe.

• Developed ingestion jobs in Python to extract data from third-party APIs in different data formats.
• Developed ETL jobs using Glue with PySpark.
• Orchestrated ETL jobs with Airflow.
• Improved bottlenecks in ETL jobs by decreasing run time.
• Designed DWH schemas and optimized SQL scripts.
• Developed a Flask application to work with the Telegram Bot API for automating routine tasks.