Principal Application Support Engineering – AI Products & Agents

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

About Tech@Lilly

At Lilly, everything we do starts with patients. We unite caring with discovery to make life better for people around the world. Headquartered in Indianapolis, Indiana, our global team of over 50,000 employees work with urgency and purpose to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. We bring our best to this work because people depend on it. If you're driven by purpose and determined to make a meaningful difference for patients, we invite you to bring your skill and your commitment to Lilly.

At Lilly, technology is not a support function. It is how a global medicine company operates, innovates, and delivers. Lilly in Hyderabad builds the capabilities that make this possible, cloud platforms, AI systems, and automation at enterprise scale, all in service of a purpose that makes this technology work genuinely distinctive, from advancing drug discovery to enabling connected clinical trials to keeping a global medicine company running at the standard patients deserve.

Principal Engineer – Operations, Reliability & AI Platforms (R3)

About the Role

As a Principal Engineer – Operations, Reliability & AI Platforms , you will provide technical leadership for enterprise-scale production systems, cloud platforms, and AI-powered applications. You will drive reliability, operational excellence, observability, automation, and resiliency across multiple products and platforms while serving as a trusted engineering leader for critical business services.

This role combines deep technical expertise, architectural leadership, operational ownership, and strategic thinking. You will partner with Engineering, Product, Platform, Security, and Operations teams to establish reliability standards, eliminate operational toil through automation, improve system resilience, and ensure mission-critical services operate securely and efficiently at scale.

What You'll Be Doing

Technical Leadership & Strategy

• Define and drive the operational engineering strategy for enterprise applications, cloud-native platforms, and AI-enabled systems.

• Establish standards for reliability, scalability, availability, performance, security, and operational excellence.

• Lead architecture reviews with a focus on supportability, observability, resiliency, disaster recovery, and long-term sustainability.

• Provide technical leadership during major incidents, service disruptions, and high-severity production events.

• Influence product and engineering roadmaps to proactively address technical debt, operational risks, and reliability improvements.

Production Operations & Reliability Engineering

• Own reliability outcomes through Service Level Objectives (SLOs), Service Level Indicators (SLIs), error budgets, and operational health metrics.

• Drive adoption of Site Reliability Engineering (SRE) and operational excellence practices across teams.

• Lead complex root cause investigations involving distributed systems, cloud platforms, integrations, databases, and AI services.

• Establish operational readiness reviews, production acceptance criteria, release governance, and supportability standards.

• Define and implement resiliency engineering practices including failover validation, disaster recovery testing, capacity planning, and business continuity programs.

• Drive continuous improvement through incident trend analysis, reliability metrics, and operational insights.

AI Platform & Agent Operations

• Provide technical leadership for AI-powered applications, copilots, virtual assistants, chatbots, and agent-based systems operating in production environments.

• Establish operational standards for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI agents, vector databases, MCP ecosystems, and orchestration platforms.

• Lead investigations involving AI model failures, retrieval issues, prompt orchestration, tool-calling workflows, agent interactions, and integration failures.

• Define monitoring strategies for AI quality, token consumption, latency, throughput, reliability, and operational health.

• Drive AI governance, responsible AI practices, security, compliance, and operational risk management.

Automation & Operational Excellence

• Lead initiatives to eliminate manual operational effort through automation, intelligent workflows, self-healing capabilities, and autonomous remediation.

• Design and implement operational automation using scripting, Infrastructure as Code, CI/CD pipelines, observability platforms, and AIOps solutions.

• Establish enterprise runbooks, playbooks, operational standards, and knowledge management frameworks.

• Drive adoption of platform engineering principles to improve reliability, developer productivity, and operational efficiency.

Collaboration & Mentorship

• Partner with Engineering, Product, Platform, Security, Compliance, and Operations teams to improve service quality and customer outcomes.

• Mentor engineers and technical leaders in reliability engineering, distributed systems, cloud operations, and AI platform support.

• Communicate technical risks, operational health, and strategic recommendations to both technical and executive stakeholders.

• Lead cross-functional initiatives that improve reliability, resilience, supportability, and operational maturity across the organisation.

How You Will Succeed

• Demonstrate exceptional analytical, troubleshooting, and systems-thinking capabilities across complex distributed environments.

• Drive measurable improvements in service reliability, availability, performance, and operational efficiency.

• Build scalable observability, monitoring, tracing, and automation capabilities that proactively detect and resolve issues.

• Operate effectively across application, platform, infrastructure, cloud, and AI technology stacks.

• Influence engineering organisations through technical leadership, collaboration, and data-driven decision making.

• Balance business priorities, operational risk, engineering investment, and long-term sustainability.

• Establish measurable operational KPIs, SLOs, SLIs, and reliability programmes.

• Become the escalation point for the most complex technical and operational challenges.

What You Should Bring

Experience

• 10+ years of experience in Software Engineering, Site Reliability Engineering, DevOps, Platform Engineering, Cloud Engineering, Production Operations, or related disciplines.

• Proven experience operating and supporting large-scale enterprise production environments.

• Experience leading reliability, operational excellence, and platform transformation initiatives.

• Demonstrated success driving operational improvements across multiple products, systems, and teams.

• Experience operating within regulated industries such as Life Sciences, Healthcare, Financial Services, or similar environments.

Technical Expertise

• Deep expertise in distributed systems, microservices, APIs, event-driven architectures, and cloud-native platforms.

• Strong understanding of software architecture, performance engineering, resiliency, security, and operational excellence.

• Advanced troubleshooting capabilities across applications, integrations, databases, networking