Eric – BigQuery, Apache Spark, Snowflake

A daily ad-analytics data product that ingests about 1 TB per day of ad attribution and campaign data from a third-party measurement vendor and turns it into canonical analytical datasets powering a partner analytics team's executive dashboards. I led the data engineering side of the partnership end to end: ingestion architecture, data modeling, metric definitions, and delivery.

• Architected the multi-stage daily ingestion flow (vendor S3 drops into GCS, converted to Parquet, exposed as BigQuery external tables) normalizing irregular CSV/TSV vendor exports into clean data models;
• Authored dbt-based production data models and complex multi-hundred-line SQL transformations turning ambiguous stakeholder requirements into reliable analytical surfaces;
• Owned column-level decisions, metric definitions, and data-integrity trade-offs across millions of daily attribution events;
• Built a 7-step Airflow (Cloud Composer) delivery DAG exporting curated datasets from BigQuery through GCS to partner S3 with row-count integrity validation as a delivery gate;
• Diagnosed and fixed a partner-reported impossible conversion funnel, a partition-key dedup bug spanning 9 SQL locations, and validated the fix against raw attribution data;
• Pioneered the team's Airflow adoption, architecting the integration layer to the centrally hosted deployment that 5+ services now extend;
• Established a configuration-driven BigQuery backup DAG using dynamic task mapping where protecting a new table is a config-only change.

A contextual advertising intelligence platform that analyzes the content of millions of publisher web pages to power privacy-safe ad targeting without cookies. The system crawls publisher URLs at internet scale, generates LLM embeddings for every page, scores them against an advertising taxonomy with cosine similarity, and serves the resulting segments to demand-side and supply-side ad platforms. I owned the end-to-end data pipeline, from crawl to embedding to the TB-scale analytical tables that partners consume.

• Designed and shipped a 5-stage event-driven Kafka pipeline of FastAPI microservices (orchestration, site discovery, page fetching, HTML parsing, content-mapping ETL), replacing a legacy cron and queue fan-out;
• Achieved 99%+ storage reduction (1.15 TB down to 9.88 GB) and 500x+ query speedup on a multi-TB BigQuery table via custom NTILE-based partitioning and nested STRUCT schemas;
• Parallelized embedding and cosine-similarity scoring over a 10-15 TB dataset using Dask distributed dataframes, with PySpark jobs for the heaviest re-scoring runs;
• Backfilled OpenAI embeddings for about 13.47 million URLs in a single coordinated operation with deduplicating MERGE logic;
• Authored a tiktoken pre-flight and adaptive-batch retry pattern for the OpenAI API plus SHA-256 content-hash caching, delivering 40%+ ETL speedup on re-runs;
• Operated a high-concurrency async web crawler (aiohttp) with robots.txt compliance and differentiated retry strategies for transient vs permanent errors;
• Built a shared Python utility library (internal PyPI) used by every service: Kafka wrappers with DLQ and back-pressure, BigQuery MERGE upsert helpers, Vault secrets client;
• Deployed all services on Kubernetes with Docker; built GCP Monitoring and PagerDuty alerting from scratch with real-time log-based data-integrity alerts;
• Cut new-supply onboarding latency from 3+ weeks to a few days via parallelized async processing and Kafka retry/DLQ architecture.

An enterprise FinOps and IT-service analytics platform unifying cloud spend across Azure, AWS, and GCP with ServiceNow ITSM operational data into one governed dimensional model. It gave 15+ business units tailored cost visibility and turned over $1M of cloud spend into tracked, remediated savings.

• Engineered multi-cloud (Azure, AWS, GCP) FinOps dimensional models in Power BI with row-level security, visualizing $1M+ cloud spend with per-resource cost-saving recommendations;
• Unified analytics across 10+ departments and 50K+ ITSM tickets through a centralized dimensional model orchestrated with Apache Airflow DAGs, cutting report duplication by 40%;
• Built an automated dashboard integrating ServiceNow ticket data to track remediation progress, visualizing $100K+ in captured savings;
• Defined and enforced cross-cloud tagging standards as the data-quality contract for cost attribution across 3 platforms;
• Built a Python-based data quality framework enforcing schema integrity, type validation, and null-rate thresholds across 10,000+ records feeding ML input pipelines;
• Applied SQL and Python text analytics over ITSM ticket exports to surface recurring root causes, improving visibility by 30%.

A secure, multi-tenant analytics platform for a private equity investment firm: positions, transactions, and portfolio data flowing from legacy systems into a modern Azure lakehouse serving OLAP analytics under SOC 2 requirements. I owned the transformation layer and the platform evaluation for the modernization.

• Built PySpark transformation jobs on Azure Databricks processing multi-GB datasets from Azure Data Lake into curated star-schema layers organized as Delta Lake bronze/silver/gold medallion tables, cutting nightly batch windows by roughly 60%;
• Tuned Databricks job clusters for skewed joins and partition pruning (broadcast hints, Z-ORDER, autoscaling); used Delta time travel to reproduce and debug bad loads;
• Led a Snowflake vs Microsoft Fabric platform evaluation, building proof-of-concept ingestion pipelines and authoring complex Snowflake SQL (window functions, VARIANT handling, clustering keys) to benchmark cost and performance;
• Modularized the dimensional transformation layer as dbt models with source-freshness and schema tests, deployed through Azure DevOps CI/CD;
• Designed star schemas with slowly-changing-dimension patterns across 150+ database fields, enforcing SOC 2-compliant tenant isolation;
• Architected Azure Data Factory ETL/ELT pipelines into Azure SQL DB and Data Lake, reducing data latency by 50% with idempotent, schema-evolution-safe design;
• Built infrastructure as code with ARM templates and Terraform modules deploying the full data stack (Data Lake, dimensional SQL DBs, Cosmos DB, Databricks workspaces) across isolated dev/QA/prod.

The data backbone for a 1,000+ user, 20+ location enterprise migration program: consolidating user, machine, and workspace records from more than 20 source systems into a single validated dataset that drove scheduling, remediation, and leadership reporting.

• Built automated Python (pandas) and SQL ETL pipelines cleaning, validating, and consolidating 50K+ records from 20+ source systems, achieving 95% data readiness;
• Designed SQL Server staging schemas and SSIS-based ingestion flows with reusable unique-identifier systems and data-quality rules (dedup, referential checks, exception queues);
• Built Tableau dashboards over the consolidated dataset tracking engagement and readiness metrics, surfacing risks early across 7 deployment scenarios and 15+ scheduling holds;
• Led migration data strategy across 20+ site IT leads, standardizing collection templates and integrity practices program-wide.