Associate Director, Data Platforms — Technical Lead
Description
About the Role
Grade Level (for internal use)
12
Key Responsibilities
Data Pipeline Transition and Platform Delivery
-
Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target data platform.
-
Drive the implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.
-
Ensure pipelines are designed and managed in a way that supports long-term platform consistency, reliability, observability, and ease of support.
-
Guide the design and operation of cloud-native data pipelines using AWS services such as:
-
Amazon S3 for durable data lake storage
-
AWS Glue for integration, cataloging, and processing
-
AWS Lambda for event-driven processing
-
Amazon Kinesis for streaming use cases
-
AWS Lake Formation for governed data lake controls
-
Promote the use of AWS IAM, encryption, environment-level controls, and platform guardrails to enforce secure access to platform resources and data products.
-
Support practical application of lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns.
Data Onboarding and Asset-Agnostic Enablement
-
Define and operationalize onboarding patterns that support a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
-
Work with platform, data engineering, architecture, governance, and business-aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform.
-
Create or contribute to reusable technical assets such as design patterns, reference implementations, onboarding templates, pipeline frameworks, technical documentation, and operational runbooks.
-
Support asset-agnostic onboarding by ensuring data pipelines are configurable, metadata-driven, scalable, and aligned with enterprise data platform standards.
Data Mastering Platform Integration
-
Support integration of platform pipelines and datasets with the enterprise data mastering platform.
-
Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.
-
Help establish data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.
-
Contribute to issue resolution and continuous improvement related to mastering-related ingestion and distribution workflows.
-
Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including: Data acquisition, Cleansing, Enrichment, Mastering, Reconciliation, Golden copy generation & Downstream distribution of trusted data products.
Technical Leadership and Engineering Excellence
-
Serve as a senior technical individual contributor for data platform engineering, providing expertise across pipeline migration, lakehouse architecture, AWS-native data engineering, governance, and mastering integrations.
-
Influence technical direction without direct people management responsibility.
-
Contribute to architecture discussions, design reviews, implementation planning, code reviews, technical standards, and production readiness reviews.
-
Translate broader architectural direction into actionable engineering patterns, implementation plans, and technical deliverables.
-
Promote engineering best practices including: Version control, Automated testing, CI/CD, Release automation, Monitoring and alerting, Incident response, Documentation.
-
Help establish cloud engineering standards for infrastructure automation, release management, and environment promotion using tools and services such as AWS CodePipeline, AWS CodeBuild, and infrastructure automation frameworks.
-
Drive operational rigor across production data pipelines, including observability, logging, telemetry, support models, service ownership, and incident management.
Collaboration, Governance, and Platform Standards
-
Partner effectively with global platform, architecture, governance, security, Databricks-aligned, and data mastering teams to ensure delivery aligns with enterprise standards.
-
Act as a technical bridge between platform strategy and engineering execution.
-
Support governance requirements through appropriate controls around:
-
Data lineage
-
Schema consistency
-
Data quality
-
Metadata
-
Retention
-
Access control
-
Encryption
-
Auditability
-
Secure data distribution
-
Ensure monitoring and operational health practices are in place using logging, alerting, telemetry, dashboards, and AWS-native operational tooling where appropriate.
Required Qualifications
-
8+ years of experience in data engineering, data platforms, cloud data architecture, or related engineering domains.
-
Proven ability to drive technical initiatives and influence engineering outcomes in a complex, execution-focused environment.
-
Experience partnering with global or distributed teams to deliver platform and pipeline initiatives across time zones.
Technical Expertise
-
Strong hands-on experience with modern data engineering and pipeline development, including batch and/or streaming data workflows.
-
Experience working with Databricks-based data platforms and supporting migration or transition of pipelines into a lakehouse-oriented architecture.
-
Strong familiarity with AWS cloud-native data engineering, including services such as Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, AWS IAM, and AWS-native monitoring, logging, and security capabilities.
-
Working knowledge of technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data lake governance, lakehouse architecture, and catalog-driven processing.
-
Experience supporting data integration patterns involving mastering, MDM, reference data, market data, investment data, risk data, or trusted data distribution workflows.
-
Solid understanding of data quality, schema management, lineage, metadata, access control, encryption, and operational support for production data pipelines.
-
Experience with AWS IAM, data lake governance, policy-based access controls, and secure platform operations.
-
Familiarity with engineering best practices such as version control, automated testing, CI/CD, infrastructure automation, release management, and monitoring.
-
Experience designing or implementing data platform observability and reliability practices, including alerting, monitoring, telemetry, operational dashboards, and production support procedures.
AI Tooling and LLM-Enabled Engineering
-
Practical experience using AI tooling such as Claude, large language models, or similar AI-assisted development platforms in an engineering context.
-
Experience applying LLMs to data engineering or software engineering workflows, including code generation, refactoring, test creation, documentation, debugging, troubleshooting, and pipeline analysis.
-
Ability to use AI-assisted workflows to support Databricks, AWS data engineering, SQL, PySpark, pipeline migration, data quality analysis, and operational support activities.
-
Ability to evaluate, validate, and refine AI-generated code or recommendations before applying them to production-grade engineering work.
Collaboration and Execution
-
Strong communication and stakeholder management skills, with the ability to work effectively across US-based, offshore, and global teams.
-
Ability to translate broader architectural direction into actionable technical deliverables, implementation plans, and reusable engineering patterns.
-
Comfortable operating in a role that blends technical strategy, architectural influence, hands-on engineering, and delivery enablement.