Lead Data Engineer

Newcastle, United Kingdom

Job Reference 1248018

Location Newcastle, United Kingdom

Industry IT - Data & Analytics

Function IT & Telecoms

Job Type Contract

Date Added August 30, 2024

Lead Data Engineer

 

Location: Remote (Must be in Europe and right to work in country of residence)

Contract: Initially until the end of the year.

Rate: 400 euros a day gross

Start: ASAP

 

Key Responsibilities:

  • Implement the best data engineering tools and practices suited to the customer’s on-premise environment.
  • Deliver robust, reusable, and maintainable ETL/ELT data pipelines with stable performance.
  • Improve the operational functionality of existing data pipelines.
  • Lead and participate in regular code reviews to ensure high-quality standards.
  • Engage in ongoing knowledge sharing and onboarding sessions.

Required Skills and Experience:

  • Strong proficiency in Python and extensive experience in ETL/ELT development, orchestration, and maintenance.
  • Hands-on experience with unit testing of Airflow DAGs and data processing applications.
  • Expertise in cutting-edge Big Data processing technologies, primarily batch processing with some streaming.
  • Proven track record in deploying Machine Learning solutions into production.
  • Deep understanding of data architecture and ability to choose the right tools and solutions for specific needs.
  • Familiarity with fundamental principles of data visualization.
  • Proficiency in Apache Airflow, PostgreSQL, S3 object storage, Kafka, and some data visualization tools (Grafana, Superset), and Kubernetes.

Nice to Have Skills:

  • Knowledge of performance metrics for evaluating ML models.
  • Familiarity with S3-compatible storage (MinIO), Kafka, Spark, MongoDB or other NoSQL databases, ELK stack, Loki, Apache Superset, and Grafana.
  • Experience in the telecommunications (Telco) sector is a plus.

Don’t see a job for you? Register for our candidate portal

Sign in or register for Airswift Digital to stay updated with recommended jobs.

Visit our candidate portal