About the Role
You’ll build, operate, and improve the “engine room” of our data platform—reusable ingestion frameworks, reliability systems, and pipelines across structured, semi-structured, and unstructured sources. You’ll own ETL/ELT workflows (Fivetran and AWS Glue) and develop stable, observable, and cost-efficient pipelines that power analytics and AI. Your work prevents incidents before they happen through great design, guardrails, and monitoring—translating technical reliability into a seamless experience for downstream customers.
Who you are
- Pragmatic builder who writes clear SQL/Python and leaves systems more reliable than you found them.
- Infrastructure-minded engineer comfortable with Python, IaC, orchestration, and Snowflake administration.
- Customer-centric and fundamentals-first; you translate reliability into a delightful data consumer experience.
- Velocity-oriented: you deliver “good today” increments, measure impact, and iterate toward excellence.
- Owner mindset: you proactively drive outcomes, communicate trade-offs, and follow through on commitments.
- Intellectually honest: you share clear, candid updates and invite feedback to improve systems.
- Security-first with sound judgment around PII/PHI, least privilege, and secret management.
- Collaborative partner who can explain technical topics to both engineers and non-technical stakeholders.
- Naturally curious and thrive in ambiguity, seeking to solve business problems with pragmatic solutions.
- A self-starter who takes ownership of outcomes and iterates quickly to add value fast.
- Always balancing excellence with velocity—knowing when good enough today beats perfect next week.
What you will do
- Develop reusable ingestion frameworks (Python/Airflow/AWS Glue) for APIs and unstructured sources beyond Fivetran, handling various data formats (JSON, Parquet, etc.).
- Own the end-to-end Medallion (bronze/silver/gold) architecture for core domains, ensuring robust lineage and metadata across diverse data sources.
- Implement data observability (native tests, alerts, lineage hooks); lead incident management and root-cause analysis (RCA) for data.
- Help standardize reusable “paved-road” patterns (e.g. CI templates, ingestion operators) to improve developer productivity.
- Prepare datasets for AI/LLM use cases (feature stores, embeddings/RAG prep).
Must Haves
- 3–5+ years of data engineering with strong Python and SQL; hands-on Spark/PySpark (ideally via AWS Glue).
- Deep experience in AWS (S3, IAM, Lambda, CloudWatch) running secure, observable data workloads.
- Proficiency operating Snowflake (warehouse sizing, RBAC, resource monitors, clustering/partitioning).
- Proven governance/security patterns: masking policies, row-level security, and auditability.
- Orchestration experience (Airflow/MWAA) and event/file/API ingestion beyond managed connectors.
- CI/CD for data with GitHub Actions; test/promotion workflows; secrets and PII handling.
- Solid grasp of Medallion architecture, dimensional modeling (star schema), and data quality frameworks.
- Ownership of incident management and RCA with measurable reduction in MTTR.
Nice to Have
- Familiarity with BI tools (Sigma, Looker, Tableau) for downstream troubleshooting and enablement.
- Experience with iPaaS/automation (e.g., Workato) and reverse ETL patterns.
- Data observability tools (e.g., Monte Carlo) and open standards like OpenLineage.
- IaC for data infrastructure (Terraform) and environment provisioning.
- Experience with Parquet/S3/Iceberg lakehouse patterns and event/data contracts.
- Fivetran administration and ELT operations.
- Experience contributing to paved-road standards (templates, operators, codegen).
- Exposure to feature stores or embeddings/RAG pipelines supporting AI products.
#LI-EM4
Abnormal AI is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status or other characteristics protected by law. For our EEO policy statement please click here. If you would like more information on your EEO rights under the law, please click here.
