Responsibilities:

• Develop and maintain data ingestion, transformation, and publishing processes using Databricks and PySpark.
• Build and optimize ETL/ELT pipelines for large volumes of data.
• Implement Delta Lake structures and data models for operational and analytical consumption.
• Develop incremental loading, historization, and CDC (Change Data Capture) mechanisms.
• Integrate data from multiple sources, including Oracle, SQL Server, APIs, files, banking applications, etc.
• Participate in defining the Data Lakehouse architecture.
• Propose scalable and high-performance data models.
• Contribute to the development of development standards and platform architecture.
• Ensure data traceability and consistency throughout the entire processing flow.
• Implement automated Data Quality controls.
• Develop data reconciliation and validation processes.
• Configure monitoring, alerting, and process logging.
• Analyze and resolve production incidents.
• Manage data reloads, reruns, corrections, and recovery processes.
• Optimize PySpark and SQL processes for cost efficiency and performance.
• Identify and eliminate processing bottlenecks.
• Optimize Databricks resource consumption.
• Contribute to the automation of development and deployment processes.

Qualifications:

• Minimum 5 years of experience in Data Warehouse development, Big Data, or enterprise data platforms.
• Minimum 2–3 years of hands-on experience with Databricks.
• Proven experience working on complex Data Warehouse, Data Mart, or Data Lake projects.
• Experience in developing end-to-end data pipelines.
• Strong knowledge of Data Lakehouse architecture.
• Experience implementing historization, CDC, data reconciliation, and Data Quality processes.
• Ability to take full ownership of technical deliveries and provide technical leadership to the team.
• Technical skills: Azure Databricks / PySpark, Advanced SQL, Delta Lake, Git & Jira, Data Modelling, ETL / ELT,
Data Quality Frameworks, Pipeline Orchestration.