This role is for one of the Weekday's clients
Salary range: Rs 1900000 - Rs 2800000 (ie INR 19-28 LPA)
Min Experience: 4 years
Location: Remote (India)
JobType: full-time
We are looking for a highly skilled Big Data Engineer – Streaming to design, build, and maintain large-scale real-time data processing systems. This role is focused exclusively on streaming data pipelines and requires deep expertise in Java-based backend development along with hands-on experience in Apache Spark Streaming or Hadoop streaming ecosystems. The ideal candidate thrives in high-performance environments and has a strong passion for building scalable, low-latency data platforms.
Requirements
Key Responsibilities
- Design, develop, and optimize real-time streaming data pipelines for large-scale, distributed systems.
- Build and maintain robust data ingestion and real-time processing frameworks using Apache Spark Streaming and Hadoop-based streaming technologies.
- Develop high-performance, production-grade solutions using Java as the core programming language.
- Ensure streaming systems meet requirements for scalability, fault tolerance, reliability, and low latency.
- Integrate streaming pipelines with downstream applications, platforms, and analytics systems.
- Monitor, troubleshoot, and optimize performance bottlenecks across streaming data workflows.
- Collaborate closely with platform, infrastructure, and application teams to deliver end-to-end streaming solutions.
- Support CI/CD practices for continuous improvement and reliable deployment of data pipelines.
Technical Skills
- Strong expertise in real-time data processing and streaming architectures.
- Hands-on experience with Apache Spark Streaming and/or Hadoop streaming ecosystems (no batch processing).
- Advanced proficiency in Java for large-scale, distributed systems development.
- Experience with CI/CD pipelines and production-grade deployment practices.
- Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Solid understanding of system performance tuning, fault tolerance, and distributed computing concepts.
Skills
Big Data Engineering
Real-Time Streaming
Apache Spark Streaming
Hadoop Streaming
Java
Cloud Platforms (AWS / Azure / GCP)
CI/CD
