At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective Industries. if your passion for data analytics solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering! 🔮🌐
Key Responsibilities
- Design, develop, and optimize scalable ETL/ELT pipelines for enterprise data platforms
- Build and maintain robust data solutions using Snowflake, Python, and AWS services
- Develop advanced SQL queries using CTEs, window functions, query tuning, and performance optimization techniques
- Work with Snowflake warehouses, tasks, streams, and data pipelines for efficient data processing
- Create and maintain data transformation workflows using dbt (data build tool)
- Develop reporting datasets and support dashboarding solutions using Amazon QuickSight
- Perform data validation, data integrity checks, and root cause analysis to ensure data accuracy and reliability
- Integrate and manage data across platforms including DynamoDB, S3, AWS Glue, Lambda, and Rockset
- Collaborate with cross-functional teams including analytics, product, and clinical stakeholders to support reporting requirements
- Monitor pipeline performance and troubleshoot production data issues
Required Skills
- Strong experience in SQL including:
- CTEs
- Window Functions
- Query Optimization
- Performance Tuning
- Hands-on experience with Snowflake:
- Warehouses
- Streams
- Tasks
- Data Loading and Optimization
- Experience in Python for data processing and pipeline development
- Experience with AWS services:
- AWS Glue
- Lambda
- S3
- Good knowledge of dbt (Data Build Tool)
- Experience in creating dashboards and reports using Amazon QuickSight
- Knowledge of Rockset analytical engine
- Experience with DynamoDB and NoSQL databases
- Basic to intermediate knowledge of JavaScript
