Jason – AI agent development, Python, AWS

A generative AI-powered platform that automates calls to health insurance providers to collect and deliver patient benefits information. The project focused on backend systems, infrastructure, VoIP integrations, and agentic AI workflows.

• Took ownership of a production Kubernetes cluster, improving backup reliability, implementing auto-scaling, and separating live call traffic into a dedicated deployment;
• Improved AI transcription workflows by introducing heuristic-guided agents, reducing ambiguous cases and increasing the success rate by ~10%;
• Optimized pod lifecycle management to keep active calls running during deployments, reducing average cost per call by 15%;
• Improved real-time AI response latency by migrating to a faster LLM and optimizing heuristics, reducing response times from up to 2s to 400–1000ms;
• Reduced latency and infrastructure costs by implementing audio caching for frequently used phrases;
• Improved AI call acceptance rates by making voice interactions more natural through realistic voices, conversational behaviors, and call-center background noise.

Developed a Python-based prompt testing and evaluation harness for a generative AI platform handling live voice calls. The project focused on systematically testing prompts and identifying the root causes of prompt failures to improve reliability and performance.

• Designed and developed a Python-based testing harness for debugging and validating generative AI behavior in live voice conversations;
• Introduced a breakpoint-based testing approach to identify where conversations deviated from expected behavior and prevent regressions when changing prompts, models, or heuristics;
• Built automated test execution that dynamically reconstructed conversation context, ran the next conversation turn, and evaluated the output against expected responses using another AI model;
• Created reusable test suites from real conversation cases, enabling faster validation of new features and prompt/heuristic changes without running costly live calls;
• Extended the framework to data extraction agents, enabling exact JSON comparison and objective quality measurement through edit-distance scoring;
• Shared the testing framework with the engineering team, improving development efficiency and reducing the time required to troubleshoot and validate AI workflows.

An LLM-powered data extraction pipeline designed to reliably extract structured information from noisy and poorly transcribed text, improving the usability and accuracy of data generated from unstructured sources.

• Improved LLM-based data extraction from noisy and poorly transcribed conversations, increasing successful extraction rates from ~50% to ~65%;
• Designed a proto-agentic extraction workflow by splitting transcripts into smaller chunks to reduce context complexity and improve model performance;
• Implemented a multi-stage approach where the LLM first identified and tagged relevant information, which was then processed by a dedicated extraction stage;
• Reduced the number of successfully completed calls requiring costly reprocessing or manual handling.

A fully serverless mobile platform for a digital health program focused on chronic pain recovery. The solution used React Native for the mobile application and AWS serverless services for the backend, with data streamed through Kinesis and stored in DynamoDB and Redshift. The project also involved platform engineering and DevOps responsibilities.

• Developed a SAML/OpenID SSO flow integrated with AWS Cognito. The identity provider used a custom, non-standard IdP-initiated flow, requiring both technical expertise and close collaboration with the identity provider;
• Improved observability by developing error-handling tools that captured Lambda failures and sent detailed error information to Datadog;
• Designed and implemented a middleware architecture that simplified and streamlined verbose application functions;
• Introduced shared libraries to improve code reusability and readability across the application;
• Became the sole DevOps engineer during a period of layoffs, supporting the infrastructure needs of the entire company for approximately three months, despite having limited prior experience with Terraform.