Faisal – AWS, Apache Spark, Python

To build a deep learning model for semantic segmentation of tissue images (identifying tumors, normal cells, and transitional regions), Faisal automated and optimized the ML training pipeline. He addressed the inefficiencies of Google Colab by implementing a cost-effective, on-demand EC2 instance via GitHub Actions, enabling background training, better collaboration, and centralized logging.

• Set up GitHub Action Workflow Dispatch to run the Deep Learning Training on demand;
• Set up on-demand EC2 instance as a GitHub Actions runner;
• Set up instance termination post training to avoid idle instance cost;
• Conducted research to find the best instance type (based on the CPU, Memory and GPU types) and AMI with the most compatible CUDA and Pytorch version.

As a Senior MLOps Engineer, developed an LLM-powered news feed to support investment decisions. The system on Google Cloud Platform (GCP) automates the ingestion of financial documents, uses LLMs (including OpenAI and Groq) to extract relevant companies and generate summaries with market-impact metrics, and then ranks the insights based on their relevance to a user's investment portfolio.

• Cut 50% of OpenAI token consumption cost by migrating all async calls to the OpenAI API to the OpenAI Batch API;
• Standardize the pipeline using multiple reusable Kubeflow components so all team members can productionize their Pipeline seamlessly;
• Set up CI/CD Pipeline to test and deploy from GitHub to the GCP ecosystem;
• Set up GitHub Actions Workflow Dispatch to submit and schedule Vertex AI Pipelines;
• Provisioned and managed GCP infrastructure using IaC tools like Pulumi;
• Refactored experimental codes from data scientists to meet production standards and deployment;
• Integrated different services and components built by other team members so the system can run smoothly and efficiently;
• Write and maintain technical documentation for the system;
• Provide technical support and guidance to the team;
• Participate in the design and architecture of the system.

The project involved deploying a Deep Learning model on a serverless AWS architecture designed to handle high-resolution image processing at scale. To manage the lifecycle and consistency of this infrastructure, Faisal utilized Terraform to implement Infrastructure as Code (IaC), ensuring reproducible environments across the development lifecycle.

Architectural Implementation:

• Preprocessing & Orchestration: The workflow began with DZI Tiling to break down large-scale input images into manageable tiles, aligning the inference data with the model's training format. To drive high throughput, Faisal implemented a decoupled architecture where tile S3 URIs were pushed to Amazon SQS.
• Compute & Scaling: These messages were consumed by a distributed fleet of Kubernetes (EKS) pods. By leveraging Kubernetes, he was able to manage container orchestration effectively, ensuring that the model inference tasks scaled dynamically based on queue depth.
• State Management: Faisal utilized Amazon DynamoDB as a distributed state store to track the processing status of individual tiles, providing a real-time indicator of overall inference completion.

Observability & Monitoring To ensure system health and performance, Faisal integrated Datadog for full-stack observability. This included:

• APM & Tracing: Monitoring the latency of inference tasks across the Kubernetes clusters.
• Infrastructure Metrics: Tracking SQS visibility timeouts and DynamoDB RCU/WCU utilization.
• Log Management: Centralizing logs to quickly troubleshoot bottlenecks in the tiling or inference stages.

• Infrastructure as Code: Deployed deep learning inference on AWS EKS and Fargate using Terraform for reproducible infrastructure.
• Parallelization & Scaling: Engineered high-throughput tile processing using SQS to decouple ingestion from model inference.
• Observability: Integrated Datadog for real-time monitoring of queue depth, pod health, and inference latency.
• State Management: Utilized DynamoDB to track tile processing status and ensure idempotent execution of long-running jobs.
• Data Synthesis: Developed post-inference scripts to aggregate tile-level predictions into consolidated result summaries.

An AI data platform/startup for educators. It's a predictive, generative AI data platform that delivers meaningful insights to educators by re-imagining analytics for K-12 education. I was assigned to migrate their data pipeline to a modern data orchestration tool. The project involves moving an existing data pipeline from an API source to BigQuery and a new data source from Snowflake to BigQuery.

• Migrated legacy data pipeline, which doesn't have orchestration and monitoring to Mage;
• Modified the ingestion from using NDJSON to using Parquet, which improves the loading time by ~3 times;
• Implemented ingestion from Snowflake to BigQuery and tackled huge data ingestion issues, especially for backfill purposes;
• Implemented multi-region job executions to avoid quota exhaustion.

Led the development of an internal BI data platform on AWS, implementing a medallion architecture. The platform ingests data from diverse sources (databases, APIs, S3 files) into a Snowflake data warehouse. Orchestrated with Dagster, pipelines use dbt to transform raw data through bronze, silver, and gold layers, with a Cube.js semantic layer for governed data marts.

• Deployed Dasgter on EKS;
• Implemented medallion architecture on Snowflake with dbt to handle the transformation;
• Implemented a semantic layer using Cube.dev to act as data marts and allow centralized governance;
• Optimized query and storage on Snowflake.

Deploy data quality validation pipeline to automate the validation on data.

• Reduced extra steps to validate data manually before sending data to the client.

Collect data related to property prices based on location and size.

• Collected POI (Points of Interest) data using the Google Maps API to enhance property value estimation;
• Integrated with government-provided property valuation data based on location and tax valuation APIs;
• Deployed the integration flow to the production environment for operational use.

The project involved developing a data pipeline from various sources to Data Lake on S3 and then connecting it to Athena for BI use cases.

• Implemented complex data transformations using Spark for efficient processing;
• Enabled streaming functionality to support real-time data processing for the use case.

The project involved the implementation of Partitions and Clustering to the BigQuery Data warehouse tables.

• Reduced analytics costs by 300 times, from approximately 20k USD per month to around 60 to 70 USD per month;
• Enhanced query performance by 600%.

The project involved utilizing Migration Analytics to transfer data from a MySQL replication database to BigQuery.

• Improved data and report retrieval speed by 180 times;
• Decreased database replication workload and infrastructure-related issues by 80%;
• Collaborated with the BI team to design data models.

Batch processing data pipeline from various sources in AWS databases and some 3rd party tools to BigQuery Data Warehouse.

• Ingested data from various sources, including RDBMS, NoSQL databases, 3rd party APIs, and Google Sheets;
• Preprocessed data on the go using batch and stream processing frameworks;
• Managed storage to optimize cost and performance;
• Designed and maintained data warehouse models in BigQuery.