Logo
Rahul – AWS, Apache Spark, Python, experts in Lemon.io

Rahul

From Canada

flag
Data EngineerSenior
Hire developer
15 years of commercial experience
Banking
Data analytics
Retail
Lemon.io stats

Rahul – AWS, Apache Spark, Python

Rahul is a Senior Data Engineer. He is skilled in ETL design, development, and deployment and has a solid architectural mindset for data-driven solutions. He can incorporate business needs along with technical decision-making. His way of reviewing code is quite unique. He can come up with a code smell and find a way to make the code more maintainable.

Main technologies
AWS
3 years
Apache Spark
7 years
Python
7 years
Tableau
1 year
Additional skills
Ready to start
ASAP
Direct hire
Potentially possible

Experience Highlights

Principal Data Engineer
Mar 2022 - Feb 20241 year 11 months
Project Overview

It's an international management consulting and technology company and a valued partner of many of the world’s largest financial services providers.

Skeleton
Skeleton
Skeleton
Responsibilities:
  • Enhanced data quality and compliance through robust quality checks and strict adherence to IT security standards;
  • Built data ingestion pipelines utilizing Python libraries (Pandas, SQLAlchemy) and Snowflake connectors, ensuring scalability and robustness for large-scale data processing;
  • Streamlined deployment processes with automated CICD pipelines, reducing deployment time and errors;
  • Implemented comprehensive data quality checks and logging mechanisms to actively detect, track, and resolve data issues, maintaining adherence to SLAs;
  • Provided reliable production support, ensuring minimal downtime and quick resolution of data issues;
  • Provided ongoing BAU support for production job monitoring, issue resolution, and bug fixes, with proactive monitoring and alerting for job failures and data quality issues.
Project Tech stack:
PySpark
Python
Databricks
Terraform
Git
GitHub
GitHub Actions
Amazon S3
Senior Technical Lead
May 2021 - Mar 202210 months
Project Overview

It's a central data repository that helps to address data silo issues. A data lake stores vast amounts of raw data in its native – or original – format.

Skeleton
Skeleton
Skeleton
Responsibilities:

• Introduced PySpark, Unix, and CICD pipelines to enhance data management processes within the ADA platform; • Designed and implemented comprehensive ETL pipelines, managing data flow for migration initiatives; • Transferred data from SAS and Teradata to the ADA platform, ensuring data integrity and quality; • Created Unix shell scripts to facilitate the migration of data from SAS and Teradata servers to ADA; • Integrated Apache Kafka for real-time data streaming and processing; • Implemented Grafana for monitoring and alerting on data pipeline performance; • Optimized data management processes by integrating PySpark and Unix; • Successfully migrated large volumes of data from SAS and Teradata to ADA, improving data accessibility and processing efficiency; • Streamlined ETL processes, enhancing overall data pipeline performance and reliability; • Enhanced real-time data processing capabilities with Apache Kafka; • Improved monitoring and alerting mechanisms with Grafana, ensuring better oversight of data pipelines.

Project Tech stack:
PySpark
AWS Lambda
Amazon S3
SASS
Amazon RDS
Senior application developer
May 2019 - May 20212 years
Project Overview

The company offers financial solutions to the people and businesses within and connecting with ASEAN. Through data and relationship-led insights, it creates solutions tailored to unique needs. Its comprehensive regional network and one-bank approach connect the business to new opportunities in ASEAN.

Skeleton
Skeleton
Skeleton
Responsibilities:
  • Led the development of ETL pipelines for the Tookitaki AML suite, ensuring compliance with regulatory requirements;
  • Implemented end-to-end Big Data processing solutions within the Hadoop ecosystem, optimizing performance for efficient data processing;
  • Guided the development team, providing technical direction and ensuring project milestones were met;
  • Utilized AWS and Azure for scalable and efficient cloud-based data processing solutions;
  • Successfully implemented ETL pipelines for the Tookitaki AML suite, enhancing compliance and data processing efficiency;
  • Optimized Big Data processing techniques, significantly improving performance within the Hadoop ecosystem;
  • Led a high-performing development team, ensuring timely delivery of project goals;
  • Leveraged cloud platforms for scalable and efficient data processing.
Project Tech stack:
Machine learning
PySpark
Git
Senior Technical Consultant (ETL Developer)
Mar 2013 - May 20196 years 2 months
Project Overview

A global bank—an institution connecting millions of people across hundreds of countries and cities. The bank provides financial services that enable growth and economic progress. The core activities are safeguarding assets, lending money, making payments, and accessing the capital markets on behalf of their clients.

Skeleton
Skeleton
Skeleton
Responsibilities:
  • Supported and developed VNG applications on Actimize Employee Fraud Solution and RCM, leveraging Big Data processing and analytics;
  • Led end-to-end code deployment processes in production environments, ensuring the reliability and scalability of data solutions;
  • Provided technical expertise in Big Data processing and analytics, ensuring optimal performance and data integrity;
  • Successfully supported and developed VNG applications, enhancing fraud detection and compliance;
  • Led reliable and scalable code deployments, ensuring minimal downtime and high availability;
  • Demonstrated strong expertise in Big Data processing and analytics, contributing to the overall success of the Actimize platform.
Project Tech stack:
Apache Hadoop
Scala
PySpark
Oracle
MySQL
SQL
PL
SQL
Software engineer
Apr 2012 - Apr 20131 year
Project Overview

Project Overview: Developed and maintained ETL processes for various banking applications, ensuring data integrity and performance optimization.

Skeleton
Skeleton
Skeleton
Responsibilities:

Key Responsibilities: • ETL Development: Designed and implemented ETL processes for banking applications, focusing on data integrity and performance. • Data Management: Managed data extraction, transformation, and loading operations, ensuring seamless data flow and accuracy. • Performance Optimization: Optimized ETL processes for improved performance and efficiency. Technologies Used: • Programming Languages: Python, SQL • Data Tools: Informatica, Oracle • Version Control: Git Achievements: • Successfully developed ETL processes that ensured high data integrity and performance. • Optimized data management practices, enhancing overall data processing efficiency.

Project Tech stack:
ETL
Unix
PL
SQL
Software Engineer (ETL Developer)
Oct 2011 - Apr 20126 months
Project Overview
Skeleton
Skeleton
Skeleton
Responsibilities:

Project Overview: Developed ETL processes for financial applications, focusing on data accuracy and performance. Key Responsibilities: • ETL Development: Designed and implemented ETL processes for financial applications, ensuring data accuracy and performance. • Data Integration: Managed data integration operations, ensuring seamless data flow and consistency. Technologies Used: • Programming Languages: Informatica, SQL • Data Tools: Informatica, Oracle • Version Control: Git

Project Tech stack:
ETL
Unix
PL
SQL
Oracle
Software Engineer (ETL Developer)
Jan 2010 - Dec 201011 months
Project Overview

Project Overview: Implemented ETL solutions for banking applications, focusing on data transformation and performance optimization.

Skeleton
Skeleton
Skeleton
Responsibilities:

Key Responsibilities: • ETL Development: Developed ETL solutions for banking applications, ensuring efficient data transformation and loading. • Data Transformation: Managed data transformation operations, ensuring accurate and timely data delivery. Technologies Used: • Programming Languages: SQL • Data Tools: Informatica, Oracle • Version Control: Git Achievements: • Successfully implemented ETL solutions, enhancing data transformation and loading efficiency. • Improved data accuracy and performance through optimized ETL processes.

Project Tech stack:
ETL

Copyright © 2024 lemon.io. All rights reserved.