Sr Data Engineer

Job Description Summary The Senior Data Engineer designs, builds, and operates enterprise-scale data solutions on Healthfirst’s AWS Lakehouse platform. They implement batch and streaming data ingestion, ELT/ETL, data quality, and transformation patterns that produce curated, trusted, operations/analytics ready data products.

The role partners with product owners, data modelers, domain stakeholders, and platform teams to translate business requirements into scalable pipelines, reusable frameworks, and clear standards for storing, processing, and moving data. The Senior Data Engineer practices DataOps — CI/CD/CT, automated validation, observability, and Agile delivery — and provides technical leadership through design reviews, mentoring, and engineering best practices.

They own reliable datasets that enable analytics and reporting — not ad-hoc analysis—and ensure pipelines support lifecycle management, resiliency, and governed access across the Lakehouse (raw → refine → publish).

Job Description

Key Responsibilities

• Design and implement ELT/ETL solutions for batch and streaming ingestion, integration, refinement, and publish patterns on the Lakehouse. • Develop reusable data processing frameworks and configuration-driven pipelines using Python and PySpark (EMR, Glue, or comparable Spark runtimes). • Build and maintain scalable orchestration workflows (e.g., Airflow) for production data delivery, including retries, historical loads, and operational runbooks. • Implement…