Principal Data Architect — Databricks Enablement
•Caylent· 44m
💻 Engineering⏰ Full-time🎯 Lead / Principal🇲🇽 Mexico
44m ago
Swiggy·about 16 hours ago
Job Profile: Engineering Manager (Data Platform)
Location: Bangalore | Karnataka
Years of Experience: 8–10
About the Team
Swiggy's Data Platform organization sits at the intersection of streaming infrastructure, data warehousing, lakehouse architecture, and data governance. The team owns the foundational systems that power real-time and batch data movement across the company — spanning ingestion pipelines, CDC connectors, orchestration, storage, and access control — enabling every downstream team, from analytics to ML to product engineering, to work with trusted, timely, and well-governed data.
The team is expected to go beyond keeping pipelines running and build durable platform capabilities into how data is ingested, processed, stored, secured, and consumed at scale — across systems like Kafka, Databricks, Snowflake, and AWS. This includes driving reliability and performance in streaming and batch workloads, enforcing fine-grained access control and data protection standards, and ensuring the platform scales cost-effectively as data volumes and use cases grow across the business.
The Role
As a Data Platform Manager, you will lead a team responsible for building and scaling Swiggy's core data infrastructure — spanning streaming pipelines, CDC connectors, lakehouse storage, access control, and cost observability across Kafka, Databricks, Snowflake, and AWS. This is not a generic data engineering role. It requires a leader who can combine deep systems expertise, architectural judgment, and strong execution to turn recurring reliability, scale, and governance challenges into durable platform capabilities.
You will operate across multiple modes: driving day-to-day pipeline reliability and incident response, improving the performance and cost-efficiency of streaming and batch workloads, shaping access control and data protection standards across the org, and building automation that prevents classes of data quality, schema, and scaling issues from recurring in production.
What You Will Work On
Driving streaming and batch data platform reliability, including pipeline health, schema compatibility enforcement, and stronger production-readiness gates for critical data flows.
Improving CDC and connector management by moving from manual debugging toward automated monitoring, alerting, and self-healing for Kafka-to-Snowflake and other sink/source connectors.
Strengthening data access governance across Unity Catalog, including row-level security, attribute-based access control, and secure data-sharing patterns for sensitive datasets.
Building better platform tooling and observability frameworks for cost visibility, capacity planning, and infrastructure/pipeline health with low operational overhead.
Leading incident and escalation response for P0/P1 data platform issues, including war-room coordination, stakeholder communication, and closure discipline.
Improving the quality of internal data quality validation so downstream analytics and ML issues are caught at the platform layer, not discovered by consuming teams.
Partnering with platform, data engineering, and product teams on standards across ingestion, streaming architecture (Spark Structured Streaming, Kafka), warehousing (Snowflake, Databricks), and secure data delivery workflows.
Managing and mentoring data platform engineers, setting operating rhythm, and ensuring the team scales through clarity, ownership, and repeatable execution.
Own the execution and evolution of Swiggy's data streaming, warehousing, and platform governance programs.
Build a high-trust operating model for pipeline reliability, incident response, escalations, and stakeholder communication.
Ensure critical data issues are tracked with clear severity, explicit affected scope, accountable owners, and closure criteria.
Drive platform reliability into the data lifecycle through automation such as schema validation, connector health checks, cost monitoring, and reusable detection workflows.
Define secure and scalable patterns for data handling, especially where customer PII, cross-system CDC, or sensitive datasets are involved.
Partner with engineering leaders to prioritize high-impact gaps and convert ad hoc fixes into platform capabilities and team-wide standards.
Lead vendor/tool evaluations pragmatically, with clear success criteria around reliability, operational effort, performance impact, and cost.
Manage, mentor, and grow a data platform engineering team with strong standards for ownership, technical quality, and execution excellence.
What We Are Looking For
Strong hands-on depth in streaming infrastructure, data warehousing/lakehouse architecture, and platform-level system design.
Good judgment on schema management, CDC/connector reliability, access control, data protection, and cost-performance trade-offs.
Experience leading data platform engineers and operating across incident response, stakeholder management, and cross-functional execution.
Ability to distinguish between a one-off fix and a durable platform capability that prevents repeat issues.
Comfort working closely with data engineering, infrastructure, analytics/ML, and product engineering teams.
Strong written and verbal communication, especially during escalations, reviews, and platform program discussions.
Practical orientation toward automation, observability, and measurable reliability/cost improvements rather than checklist-driven operations.
Experience with Kafka, Spark Structured Streaming, or large-scale CDC pipeline design and operations.
Experience integrating platform reliability controls into CI/CD, schema registries, connector governance, or data tooling.
Familiarity with cloud/infrastructure cost management, Unity Catalog access control, or data observability tooling.
Experience operating in high-scale product and platform environments with multiple engineering teams and fast release cycles.
Why This Role Matters
Swiggy needs data platform leadership that can move the organization from fragmented pipelines and reactive firefighting toward proactive, embedded, and automatable platform controls. The role matters because the recurring problems are no longer just about fixing individual pipeline breaks; they are about preventing schema and connector issues from cascading downstream, reducing time-to-response on critical data incidents, protecting sensitive customer data, and creating platform systems that scale cost-effectively with engineering and business velocity.
If you want a leader who can combine data platform engineering depth, team leadership, systems thinking and execution rigor, this role should be framed exactly around that charter.
Visit our tech blogs to learn more about some of the challenging problem statements Swiggy works on:
We are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, disability status, or any other characteristic protected by law.
We are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, disability status, or any other characteristic protected by the law.