Expert Data Engineer

Barry Callebaut Digital (BC Digital) is on a mission to lead the digital revolution in the chocolate industry, and we're looking for an Expert Data Engineer for the Data Pipelines, to help us build a foundation for actionable insights, based on robust, scalable, and efficient data pipelines. Reporting to the Head of Data Engineering, your focus will lie on ensuring data availability for analytical needs, through the transformation and provisioning of data from various source systems and domains across Barry Callebaut. As part of the central Data Engineering unit, you will be responsible for managing and coordinating external and distributed engineers, while providing community support, training, and governance to drive best practices across the engineering community within BC.

MAIN RESPONSIBILITIES & SCOPE

The candidate will:

• Design and build robust and scalable data pipelines to load and transform data from various source systems
  • Lead the design, development, and deployment of scalable data solutions using Microsoft Fabric and other Azure Data Services.
  • Build and maintain data pipelines to integrate structured and unstructured data for advanced analytics and business intelligence.
  • Collaborate with architects and business stakeholders to create logical and physical data models.
• Design and build robust and scalable platform capabilities
  • Leverage Microsoft Fabric capabilities for seamless data integration, transformation, and visualization.
  • Ensure efficient use of Fabric’s unified analytics platform, connecting OneLake, Synapse, and Power BI across the organization.
  • Design and manage ETL/ELT pipelines for large-scale data ingestion using MS Fabric Dataflows.
  • Implement transformations using Notebooks, or Spark-based solutions
  • Optimize and manage data lakes and data warehouses in Azure Data Lake Storage (ADLS) and Microsoft Fabric’s Lakehouse.
  • Implement partitioning, caching, and indexing strategies to ensure optimal data performance and query execution.
• Design and build performance monitoring and troubleshooting
  • Monitor and improve pipeline performance, identifying and resolving bottlenecks
  • Implement validation processes to ensure data is accurate, consistent, and accessible
  • Build automations for repetitive tasks related to data analytics
  • Perform root cause analysis and implement corrective measures
  • Identify opportunities to optimize pipelines for performance and cost efficiency

• Collaborate across stakeholders
  • Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and build the right solutions
  • Document data engineering processes, systems, and procedures
  • Manage and coordinate BC’s external and distributed engineers, ensuring effective collaboration and integration
  • Provide community support, training, and governance to drive best practices across the data engineering community
  • Lead the development and implementation of global frameworks and patterns for data ingestion and pipelines, working closely with all other internal and external Data Engineer roles

EDUCATION, LANGUAGE, SKILLS & QUALIFICATIONS

• Degree in IT/technology, data analytics, or a comparable field
• Proficiency in English

ADDITIONAL CONSIDERATIONS OR COMMENTS

• N/A  

ESSENTIAL EXPERIENCE & KNOWLEDGE / TECHNICAL OR FUNCTIONAL COMPETENCIES

• Advanced knowledge of data engineering processes, with a minimum of 7 years of experience (senior)
• Comprehensive understanding of data engineering patterns and best practices for pipeline orchestration
• Extensive experience in developing data analytics solutions on Azure
• Strong background in designing and building efficient, reliable, and automated data pipelines, ETL workflows, data warehousing, and Big Data processing, with experience in respective technologies like Airflow, dbt, etc.
• Proficiency in Spark and Databricks Lakehouse technologies
• Hands-on expertise with Python or PySpark
• In-depth knowledge of Azure Stack components like Azure Databricks, Azure Data Factory, Azure Synapse, and ADLS Gen2 from a data engineering perspective
• Familiarity with utilizing DevOps and DataOps methodologies to enhance development and deployment practices (e.g., CI/CD)
• Experience in managing and coordinating data engineers, providing community support, training, and governance to drive best practices
• Expertise in leading the development and implementation of global frameworks and patterns
• Excellent problem-solving skills and structured way of working

LEADERSHIP COMPETENCIES & PERSONAL STYLE

The ideal candidate…

• Collaborates well across diverse and globally distributed teams, with the ability to build and maintain positive relationships across different levels and functions of the organization
• Is a true team player, supporting their colleagues by sharing knowledge and experience and commits to the teams’ joint success
• Is capable of effective communication, conveying complex problems in a clear and persuasive manner to internal and external stakeholders
• Is open to try and learn new technologies and skills
• Has an independent and self-driven personality, taking responsibility and owning tasks
• Independent and self-driven personality with strong analytical skills and techniques
• Is capable of guiding diverse teams and driving change in a collaborative way on a global scale