Databricks Senior Architect
Description
Job Overview
Role Overview
As a Databricks Senior Architect , you will lead the architecture and technical delivery of Primoris’ Databricks-based data platform and use cases across data engineering, analytics, and AI/ML. You will step into an in-flight program, rapidly absorb context, and provide the architectural backbone needed to reliably deliver value from Primoris’ data.
You will act as a trusted technical advisor and delivery owner , working closely with Primoris’ data, IT, and business teams (and implementation partners) to establish a clear architectural vision, define and prioritize use cases, and guide them into production on the Databricks Lakehouse. While this role is primarily focused on architecture, design, and technical leadership , you will also contribute hands-on where needed to accelerate critical workloads and mentor the wider engineering team.
This role is ideal for someone who wants to grow as a platform and solution leader , enjoys solving complex data and AI problems, and can connect technical decisions to measurable business outcomes (e.g., project profitability, safety and quality insights, forecasting, and operational efficiency).
Key Responsibilities
Platform & Architecture
-
Define and maintain the end-to-end architecture for Primoris’ Databricks Lakehouse, including Medallion layer patterns, ingestion, transformation, serving, and AI/ML/GenAI workloads.
-
Design and implement scalable, secure Databricks patterns (including serverless where appropriate) that power batch, streaming, ML, AI Gat eway and GenAI workloads, optimizing for performance, cost, security, and reliability .
-
Architect and implement solutions using the Databricks Medallion architecture , Unity Catalog, and Delta Lake to support both analytical and operational use cases.
-
Partner with Primoris’ enterprise / solution architecture, security, infrastructure, and application teams to align Databricks with broader platform and integration standards .
Delivery & Operations
-
Assist in architecting, developing, deploying, and migrating priority use cases and workloads into production on the Databricks Lakehouse.
-
Serve as the first point of contact for technical challenges or questions related to development and production workloads (triage, root cause analysis, coordination of escalation as needed).
-
Provide technical leadership and delivery oversight across hybrid teams, including internal employees, consulting partners, and offshore resources, ensuring seamless coordination, clear accountability, and consistent outcomes.
-
Establish and champion engineering standards (CI/CD, automated testing, observability, monitoring/alerting, incident runbooks) to ensure production-grade delivery and operations.
-
Design and review workflows, jobs, and streaming pipelines (e.g., Auto Loader, DLT, structured streaming) for robustness, recoverability, and SLA adherence.
-
Implement Databricks product innovations, private previews, and upgrades following change management and release best practices, in coordination with Primoris IT and partners.
Data Engineering
-
Build and review multi-language notebooks (SQL, Python / PySpark) with a primary focus on data engineering and data science / AI/ML workflows.
-
Design and guide implementation of streaming and batch data pipelines (Auto Loader, DLT, Structured Streaming) handling structured, semi-structured, and unstructured data.
Governance, Security & Compliance
-
Implement and oversee Row-Level Security (RLS), column masking, data encryption/decryption , and other data protection controls using Unity Catalog and cloud-native capabilities.
-
Architect and govern enterprise GenAI platforms on Databricks, including Unity Catalog, AI Gateway, Model Serving, and Vector Search, ensuring secure, compliant, observable, and cost-effective AI solutions across the organization.
-
Work with security and compliance stakeholders to define and implement data governance models , access patterns, data classification, auditing, and regulatory/contractual controls as required.
-
Ensure all architectures and implementations align with Primoris’ security, networking, and compliance standards for Azure and Databricks.
Stakeholder Management & Communication
-
Participate in discovery workshops with business and technical stakeholders to understand current-state architecture, data flows, pain points, and priority use cases.
-
Translate findings into strategic roadmaps , architecture options, and phased delivery plans with clear trade-offs, milestones, and ownership.
-
Provide regular internal and external updates on progress, risks, and blockers to leadership, with clear options, mitigations, and recommendations.
Coaching, Standards & Reuse
-
Define and evangelize pragmatic engineering and platform standards (coding patterns, data modeling, observability, cost optimization, workspace standards).
-
Mentor engineers through design and code reviews , knowledge-sharing sessions, and pairing on complex tasks.
-
Develop and maintain reusable assets (reference architectures, templates, starter notebooks, Terraform/IaC patterns, runbooks) to accelerate adoption and improve consistency.
-
Demonstrate proactive ownership : surface risks early, drive decisions, and remove blockers to maintain delivery momentum and stakeholder confidence.
Qualifications
-
Bachelor’s degree in Computer Science, Information Systems, Engineering, Data Science , or equivalent practical experience.
-
8+ years of hands-on experience in Data, Analytics, or AI roles (e.g., data engineering, data architecture, analytics platform engineering), with 4+ years accountable for technical delivery and architecture/tech-lead responsibilities .
-
Proven experience architecting and implementing Databricks solutions using the Medallion architecture , including ingestion, transformation, serving, and AI/ML workloads.
-
Hands-on expertise implementing Databricks on Microsoft Azure (required), including familiarity with Azure security, networking, identity, and storage concepts.
-
Experience designing and developing large-scale distributed systems and big data solutions using technologies such as Apache Spark™ , Hadoop, or Cassandra.
-
Proficiency in Python (including PySpark) and SQL , including performance tuning and best practices for large-scale data processing.
-
Proficiency in designing and implementing Auto Loader , Delta Live Tables (DLT) , and streaming solutions in Databricks to handle structured, semi-structured, and unstructured data.
-
Experience with Unity Catalog and related capabilities (data governance, RBAC, lineage, auditing, and secure data sharing) or similar enterprise data governance tools.
-
Demonstrated ability to attribute business value and outcomes to specific project deliverables and technical KPIs.
-
Experience conducting discovery workshops , creating strategic roadmaps , performing business/process analysis , and managing or influencing the delivery of complex data/AI programs or projects .
-
Expertise implementing end-to-end data security controls in Databricks (RLS, column masking, encryption, key management), in alignment with enterprise security policies.
-
Evidence of continuous learning and staying current with the fast-evolving Databricks Lakehouse platform, Azure services, and modern data/AI patterns.
-
Nice to have:
-
One or more Databricks certifications (e.g., Databricks Certified Data Engineer Professional , Databricks Certified Data Architect Associate/Professional ).
-
Deep hands-on knowledge of Big Data Engineering or cloud DWH technologies (e.g., Synapse, Snowflake, BigQuery, Redshift).
Skills and Competencies
-
Technical leadership : Ability to set architectural direction, make trade-off decisions, and guid