Christopher – FastAPI, Django, Python

The Data-hub asynchronous data ingestion project for Net2Grid aims to measure instantaneous power from national data hubs. The project consists of a highly scalable API that provides Webhooks callbacks to the datahub. The datahub can use these webhooks to send different events or instantaneous power measurements for a user.

• Reduced scope of work and clarified requirements from the client;
• Designed the architecture using AWS services;
• Created milestones and other features to ensure agile project implementation spread in time;
• Managed the performance of a team of 4 developers;
• Participated in the implementation and code reviews;
• Integrated with 3 different third-party data hubs. Main achievement:
• The architecture proved highly scalable in the production environment, allowing more than 35000 customers to be onboarded in 2 months.

Net2Grid collects instantaneous power measurements for different users worldwide and provides energy insights into consumption and cost per appliance. Hundreds of devices provide rapid power measurements that report multiple KBs every second. The Platform Data Warehouse project was initiated to scale active devices from 20,000 to more than 600,000. It consists of refactoring how data are ingested, transformed, aggregated, and stored in Net2Grid's cloud platform.

• Increased data ingestion throughput in iterations that included RabbitMQ sharding, replacing RabbitMQ with Kafka, consuming Kafka with Go and Python consumers, and using AWS Kinesis for real-time streaming and easier auto-scaling of shards;
• Migrated most of the data from RDS to DynamoDB and S3 while making use of Memcached Elastic Cache;
• Benchmarked various techs, including Kafka with Faust, InfluxDB, and Cassandra;
• Created cost analysis and load-tested the new solutions;
• Participated in integrating all Net2Grid's microservices with the new architecture to abstract data ingestion, consumption from queues, stream processing, and data storage;
• Designed metrics for platform monitoring and acted proactively to ensure platform reliability and fault tolerance.

This project for Net2Grid offers a unique integration solution to third-party electricity distributor companies. Account Management, Tariffs Importers, and Measurements Importers can enjoy using it.

• Designed cloud architecture for the services;
• Created serverless microservices with FastAPI & Dependency-injection;
• Implemented a solution for large file imports using S3, SQS, and AWS Lambda functions that scaled to more than 10GB per file and more than 10K concurrent file uploads;
• Created Design Documents and Jira tickets delegating the work to other developers;
• Implemented real-time measurement imports using AWS Kinesis and a custom lambda for auto-scaling;
• Created a highly scalable API for Account Management, following the Event-driven architecture;
• Used the Amazon RDS Proxy to scale multiple DB connections.

The project for the Hellenic army aims to provide a solution for monitoring activities on several PCs connected to a private network. Python producer is installed in each PC and reports several metrics regarding network traffic, USB connections, and file modification. The Server not only writes the metrics to the MongoDB cluster and another queue for real-time data processing.

• Implemented Socket.IO for bidirectional communication and wrote unit tests for it;
• Configured a Spark cluster for batch processing;
• Configured Nginx as a reverse proxy and load balancer with sticky sessions.

The Electronic Patient Record Service consists of multiple microservices that pair up to provide a frontend (SPA) with the appropriate tool for secretaries, doctors, and managers to perform actions for managing health information of their patient's medical history, treatment plans, and storing files.

• Developed pages for patients' history using React.js & Redux;
• Implemented a solution for file upload to S3 through one-time link generation without overwhelming the backend;
• Containerized backend services and improved local development using docker-compose;
• Implemented several backend features regarding patients' data storage, initially storing all data to RDS, later moving to DynamoDB;
• Configured backups and disaster recovery for DynamoDB;
• Improved policies for Load Balancing;
• Implemented bidirectional communication between frontend and backend for real-time notifications using gRPC.