Lead Data Engineer

About the role:

As a Data Engineer, you’ll design and scale the infrastructure that powers modern data pipelines, build scalable models, and turn massive datasets into actionable insights. Beyond coding, your work will influence product, engineering, and business decisions, helping HackerRank leverage data to accelerate innovation worldwide. You’ll collaborate with a tight-knit, high-ownership team in a supportive, low-noise environment.

What you’ll do

• Design and maintain scalable streaming and batch data pipelines using AWS-native and open-source technologies.
• Architect and evolve our modern data lakehouse, ensuring reliability, security, and performance.
• Develop data models that power analytics and enable self-service reporting across teams.
• Evaluate emerging technologies, run POCs, and implement innovative solutions for complex data challenges.
• Partner with engineering, analytics, and product teams to deliver data systems that drive real business impact.
• Mentor junior engineers, guiding them on best practices in data design, quality, and infrastructure.

Who you are

• 5+ years of experience building data engineering or BI solutions at scale.
• Strong in data modeling and ETL pipeline design using tools like Apache Airflow or MageAI.
• Hands-on with Spark (PySpark or Scala) and comfortable optimizing queries on Redshift, Trino, or similar.
• Fluent in SQL and experienced with performance tuning for large datasets.
• Comfortable working with microservices-based architectures and familiar with deployment processes (CI/CD, containers, cloud environments).
• Skilled at solving problems of scalability, reliability, and performance across distributed systems.
• Collaborative communicator who thrives in fast-moving, cross-functional teams.

Even better if you have

• Experience with Kafka or other real-time data streaming tools.
• Exposure to BI platforms such as Redash, Looker, or Metabase
• Familiarity with data governance tools like Ranger, observability tools like Grafana, or cost-optimization on AWS.

You will thrive in this role if

• You love designing systems that scale elegantly and perform flawlessly.
• Enjoy turning complex data into clear, actionable insights.
• Take pride in building with precision, ensuring reliability and maintainability at every layer.
• Thrive in an environment that values ownership, curiosity, and craftsmanship over noise.