Sr Analyst, Data Integration & Workflows

About the Role

Grade Level (for internal use)

11

The Team

The Senior Analyst, Data Integration & Workflows plays a critical role within the Data AI & Enablement organization, serving as a senior technical leader responsible for designing, implementing, and operationalizing production-grade data pipelines and workflow automation that power SPDJI's index and analytical solutions. This role combines hands-on technical expertise with leadership capabilities to drive delivery excellence, mentor technical talent, and ensure all data integration solutions meet enterprise standards for quality, reliability, and maintainability.

Responsibilities and Impact:

Technical Leadership & Solution Delivery

• Lead complex data integration initiatives from design through production deployment, ensuring solutions are scalable, observable, and aligned with enterprise architecture standards

• Design and implement production-grade data pipelines (batch and streaming) that transform raw inputs into trusted curated outputs, incorporating robust error handling, validation, and reconciliation controls

• Establish and evangelize engineering best practices for ETL/ELT patterns, workflow orchestration, data quality controls, and operational observability across the team and value streams

• Drive technical decision-making for pipeline architecture, technology selection, and design patterns, balancing business requirements with technical feasibility and long-term maintainability

• Partner with PPD on technical planning and feasibility, providing realistic estimates, identifying technical dependencies, and shaping scope to ensure achievable delivery commitments

Enablement & Co-Development

• Lead hands-on enablement with value stream SMEs through pair programming, structured guidance, and co-development sessions—adapting approach based on SME technical capability

• Assess SME technical readiness and recommend appropriate engagement models (SME-led with review, co-development, or led build with validation)

• Build reusable automation components and templates (frameworks for ingestion, validation, transformation, publishing, backfills) that accelerate consistent delivery across domains

• Develop SME technical capabilities through targeted coaching, code reviews, and knowledge transfer, fostering a culture of engineering excellence and continuous learning

• Create and maintain technical documentation, including reference architectures, design patterns, coding standards, and implementation guides

Quality Assurance & Production Readiness

• Conduct comprehensive code reviews for SME-built and team-developed pipelines, ensuring adherence to standards for maintainability, testing, logging, data validation, and documentation

• Implement data reliability controls including validation rules, reconciliation checks, anomaly detection, and completeness/timeliness monitoring that protect downstream index processes

• Engineer observability and monitoring solutions by implementing logging standards, metrics, alerts, and runbooks that enable effective production support

• Prepare IT-ready handover artifacts including technical documentation, test evidence, operational procedures, and clear support boundaries

• Partner with IT during QA and deployment, resolving issues quickly and ensuring solutions meet enterprise standards for security, supportability, and operational excellence

Operational Excellence & Continuous Improvement

• Provide L3 support for production business-logic issues, collaborating with value stream SMEs to drive root-cause analysis and implement permanent fixes for recurring failures

• Optimize pipeline performance and cost through appropriate partitioning strategies, caching, incremental processing patterns, and compute resource tuning

• Implement workflow orchestration patterns (scheduling, dependency management, retries, idempotency, parameterization) ensuring pipelines are resilient to upstream variability

• Capture and share lessons learned, updating engineering playbooks, patterns, and standards based on production outcomes and emerging best practices

• Monitor operational metrics related to pipeline reliability, data quality, performance, and cost efficiency; drive continuous improvement initiatives

Collaboration & Stakeholder Management

• Collaborate with Data Integration Lead to shape team strategy, prioritize initiatives, and align technical approaches with organizational goals

• Partner effectively with AI Solutions and Data Governance teams on cross-cutting concerns including data quality standards, AI pipeline requirements, and compliance

• Engage with Data Value Streams to understand business requirements, validate technical solutions, and ensure alignment with domain expertise

• Work with Data Services & Strategy teams (Vendor Governance, Catalog) to establish scalable integration patterns and ensure proper metadata and lineage tracking

• Build strong relationships with IT and PPD teams to ensure infrastructure readiness, smooth deployments, and operational excellence

Shared Accountabilities

• With Data Integration Lead: Execute on team strategy; provide technical leadership on complex initiatives; mentor junior team members; contribute to standards and capability development

• With PPD: Collaborate on technical feasibility assessments and planning; provide realistic estimates; align integration efforts with platform capabilities and roadmap

• With IT: Ensure infrastructure readiness; coordinate handover processes; support production gateway requirements; partner on operational excellence

• With Data Value Streams: Co-develop solutions with SMEs; validate business logic alignment; assess and develop SME technical capabilities

• With Data Services & Strategy: Establish scalable integration patterns; ensure proper metadata and lineage tracking; align with vendor governance requirements

• With AI Solutions & Data Governance: Coordinate on data quality standards, AI data pipeline requirements, and governance compliance

Ownership

• Complex Technical Initiatives: Own the end-to-end delivery of high-complexity data integration and workflow automation projects

• Engineering Standards Implementation: Responsible for implementing and enforcing technical standards, patterns, and best practices within assigned domain or value streams

• SME Technical Development: Own the hands-on enablement and capability development of assigned value stream SMEs in data engineering practices

• Production Solution Quality: Accountable for ensuring all solutions meet production readiness criteria before IT handover

Parameters for Success

• Deliver Production-Ready Solutions: Consistently deliver high-quality, production-ready data pipelines that meet business requirements and enterprise standards

• Build SME Capability: Demonstrably improve technical capabilities of value stream SMEs through effective enablement and mentorship

• Drive Reusability: Create and promote adoption of reusable components and standardized patterns that accelerate delivery

• Ensure Operational Excellence: Implement robust observability, monitoring, and support frameworks that minimize production incidents and enable rapid issue resolution

• Foster Technical Excellence: Contribute to a culture of craftsmanship, continuous improvement, and engineering best practices

Key Performance Indicators (KPIs)

• Solution Delivery Quality: Percentage of delivered pipelines that pass IT QA on first submission; production incident rate for delivered solutions

• SME Capability Development: Measurable improvement in technical skills of mentored SMEs through assessments, code review quality progression, and feedback

• Operational Reliability: Pipeline uptime and reliability metrics; mean time to resolution for production issues; data quality incident rates

What We’re Looking For

Basic Required Qualifications

Experience

• 8+ years of experience in data engineering,