Data Engineer (Real-Time Data)

Описание: The independent technology consultancy works across data, cloud and technology transformation, helping clients modernise their platforms and make better use of their data.

Задачи:

• Design, build and maintain scalable ELT/ETL data pipelines within Snowflake. • Implement data ingestion from multiple sources, including real-time streams using Kafka. • Develop and deploy KSQL and KStreams pipelines for real-time data processing. • Develop efficient and scalable data models within Snowflake for analytics and business intelligence. • Ensure data quality, integrity and reliability throughout the data lifecycle. • Optimise data pipelines and Snowflake workloads for performance and scalability. • Collaborate with architects, analysts and data scientists to understand requirements and deliver effective data solutions. • Contribute to data engineering standards, best practices and continuous improvement across the platform.

Требования:

• Solid commercial experience as a Data Engineer delivering cloud-based data solutions. • Previous experience within insurance or financial services. • Strong knowledge of Snowflake, including data modelling and performance optimisation. • Hands-on experience with Kafka for data ingestion and stream processing. • Practical experience developing and deploying KStreams and KSQL pipelines. • Strong SQL skills and experience with Python or another scripting language. • Experience with data orchestration tools such as Airflow. • Experience with…