Accountabilities

The role involves leading the reconstruction, optimization, and validation of data pipelines while ensuring alignment with modern Lakehouse architecture standards. Key responsibilities include:

• Rebuild legacy data warehouse pipelines using Databricks, PySpark, and Spark SQL based on specifications created through reverse engineering.
• Implement bronze, silver, and gold data layers following the Medallion architecture, ingestion standards, and project reconstruction guidelines.
• Execute migration waves by business domain while maintaining coexistence between legacy systems and the new platform until final cutover.
• Develop business logic transformations and implement automated testing across data pipelines.
• Perform data reconciliation and validate parity between legacy data warehouse outputs and new Lakehouse implementations.
• Optimize pipeline performance and cloud costs through techniques such as partitioning, OPTIMIZE/Z-ORDER strategies, and job sizing improvements.
• Contribute to technical documentation, migrated business rules, and prioritization of future migration activities.
• Collaborate with engineering teams to improve data quality, reliability, and delivery processes.

Requirements

The ideal candidate is an experienced Data Engineer with strong knowledge of scalable data platforms, cloud environments, and data migration initiatives. Required qualifications include:

• 5+ years of experience in Data Engineering roles.
• Advanced proficiency in PySpark and Python, including large-scale batch pipeline development, testing practices, and engineering standards.
• Strong SQL expertise and experience with complex transformations, performance tuning, and relational or dimensional data modeling.
• Hands-on experience with Databricks and Delta Lake, including production workflows, jobs, Medallion architecture, Delta Live Tables, Lakeflow, and Asset Bundles.
• Experience migrating or rebuilding ETL pipelines by translating business rules from legacy systems into Spark-based solutions with CDC and batch ingestion patterns.
• Knowledge of AWS data services such as S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions.
• Experience with Azure Synapse environments, including SQL pools and pipelines, to support reverse engineering activities.
• Strong understanding of data quality practices, automated testing, data quality gates, and legacy versus new platform reconciliation.
• Experience using Git and CI/CD practices for data pipeline development.
• Ability to communicate effectively, collaborate with technical teams, and work independently in complex transformation projects.

Preferred qualifications:

• Experience analyzing DataStage jobs for reverse engineering purposes.
• Previous involvement in Data Warehouse migration programs toward Lakehouse architectures.
• Knowledge of Unity Catalog, permissions management, data lineage, and data contracts.
• Experience in financial services or credit-related environments.
• Databricks certifications such as Data Engineer Associate or Professional.

Benefits

• Opportunity to work on a large-scale data modernization and Lakehouse transformation initiative.
• Remote work flexibility within Brazil.
• Exposure to advanced technologies including Databricks, Delta Lake, Spark, AWS, and Azure platforms.
• Opportunity to collaborate with experienced engineering teams on high-impact data projects.
• Professional growth through access to emerging AI, cloud, and data engineering practices.
• Participation in a technology-driven environment focused on innovation and continuous learning.