Senior Data & Automation Engineer (Databricks)

Fluent is building the next generation advertising network, Partner Monetize & Advertiser Acquisition. Our vision is to build an ML/AI first network of advertisers and publishers to achieve a common objective, elevating relevancy in E-commerce for everyday shoppers. 

As a Senior Data & Automation Engineer, you will leverage your Databricks and Spark expertise to execute on building enterprise-grade data products that power Fluent’s business lines. These products serve as the foundation for sophisticated representations of customer journeys and marketplace activity across our ecosystem.

You will partner with Data Architects, Data Scientists, and Product Managers to transform Enterprise Data Models into optimized physical data models and real-time pipelines. You will elevate standards across the team in code quality, observability, and architecture design—while actively contributing as a hands-on engineer.

This role is fully remote in Ontario, with occasional travel to NYC or Toronto offices.    

What You’ll Do  

• Design, build, and support scalable real-time and batch data pipelines using PySpark and Spark Structured Streaming on Databricks.
• Implement process automation and end-to-end workflows following Bronze → Silver → Gold architecture using Delta Lake best practices.
• Handle event-driven ingestion with Kafka and integrate into automated pipelines.
• Orchestrate workflows using Databricks Workflows/Jobs and CI/CD automation.
• Implement strong monitoring, observability, and alerting for reliability and performance (Databricks metrics, dashboards).
• Collaborate cross-functionally in agile sprints with Product, Analytics, and Data Science teams.
• Translate enterprise logical data models into optimized physical and performance-tuned implementations.
• Write modular, version-controlled code in Git; contribute to code reviews and enforce quality standards.
• Implement robust logging, error handling, and data quality validation across automation layers.
• Utilize relevant AWS services (S3, IAM, Secrets Manager) and DevOps practices.
• Promote best practices through documentation, knowledge sharing, tech talks, and training.