Staff Site Reliability Engineer - Linux, Python, Splunk, Observability tools, GenAI

Visa’s Observability ecosystem includes over 2,000 platform nodes, utilizing approximately 15 different tools for logging, monitoring, and tracing, alongside 80,000 client agents. The system handles daily log ingestion exceeding 100TB and oversees hundreds of critical applications, supporting vital alerts, dashboards, and reports. To maintain this high level of performance and reliability, we need a Staff Site Reliability Engineer (SRE) with comprehensive knowledge and practical experience. This position requires an I6-level engineer who can operate independently with minimal supervision.

About Visa’s PRE Observability Team:

Visa’s Product Reliability Engineering (PRE) Observability team partners with Product Development as well as Operations & Infrastructure teams to build and manage innovative, reliable, scalable, secure, and cost-effective observability platform solutions. We are looking for talented Staff Site Reliability Engineers to join our driven team, with a focus on maximizing system availability, performance, security, and reliability. This dynamic role requires technical leadership, strong problem-solving skills, and expertise in coding, testing, and debugging.

We are seeking an experienced Staff Site Reliability Engineer (SRE) to join our dynamic Observability team. In this role, you will play a critical part in building and maintaining reliable, secure, and cost-effective observability solutions that support our global infrastructure.

As a Staff SRE, you will act as a key liaison between development and operations, ensuring our systems and services are efficient, scalable, and highly available. You will work closely with software engineers, system administrators, and other technical stakeholders to automate workflows, enhance system performance, and streamline deployments.

You will take ownership of your solutions, prioritizing system stability, performance, and security. You will design, implement, and improve monitoring, logging, and alerting systems to proactively identify and resolve issues. You will also participate in incident response, troubleshooting live issues, performing root cause analysis, and contributing to preventive measures.

In this position, you will mentor and support junior SREs and collaborate within Agile Scrum teams, leveraging your technical expertise in coding, scripting, testing, and debugging to deliver high-quality results. You will help drive continuous improvement in processes, tools, and automation across the organization.

Responsibilities:

• Design, deploy, and optimize instrumentation for comprehensive monitoring and logging across distributed systems.
• Architect and implement automation to drive operational efficiency, reduce toil, and streamline system integration.
• Lead the support and management of observability platforms and tools, including Splunk, ClickHouse, Grafana, Prometheus, M3DB, OpenTelemetry, Fluent Bit, ElasticSearch, OpenSearch, and CloudWatch.
• Collaborate with development, operations, and security teams to design scalable monitoring solutions and assist in building robust development, staging, and production environments.
• Ensure system security and compliance by proactively applying hotfixes, patches, and hardening measures.
• Champion DevOps and SRE best practices across engineering teams, fostering a culture of reliability and continuous improvement.
• Design, build, and maintain CI CD pipelines, enabling rapid, automated, and reliable deployments.
• Oversee and optimize cloud infrastructure (AWS, GCP) to ensure high availability, scalability, performance, and security.
• Develop and manage infrastructure as code using tools such as Terraform, Ansible, or CloudFormation.
• Continuously monitor system health, analyze performance metrics, and implement solutions to improve reliability, efficiency, and scalability.
• Deploy, manage, and optimize containerization and orchestration technologies, including Docker and Kubernetes.
• Conduct root cause analysis on major incidents, lead postmortems, and drive implementation of preventative and corrective measures.
• Maintain and enhance comprehensive documentation for infrastructure, processes, and operational procedures.
• Provide advanced technical support and mentorship to resolve complex infrastructure and deployment issues.
• Identify and automate manual processes to further improve operational workflows and reduce human error.
• Lead incident response efforts, including on-call rotations, and drive improvements to incident management processes.
• Evaluate, recommend, and implement new tools and technologies that advance the organization’s reliability and observability capabilities.
• Collaborate with cross-functional teams on disaster recovery planning, business continuity, and capacity planning initiatives.
• Present technical findings, recommendations, and status updates to leadership and stakeholders.
• Foster knowledge sharing and best practice adoption through training, code reviews, and documentation initiatives.

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.

Basic Qualifications:

• 5+ years of relevant work experience with a Bachelor’s Degree or at least 4 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD, OR 8+ years of relevant work experience.

Preferred Qualifications:

• 5+ years of relevant work experience with a Bachelor’s Degree or at least 2 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD, OR 8+ years of relevant work experience.
• Proven experience performing upgrades and maintenance for observability tools such as Splunk Enterprise and Universal Forwarder, Fluent Bit, Prometheus container images, Grafana, ElasticSearch, OpenSearch, Kibana, and ClickHouse.
• Demonstrated expertise in configuring and deploying exporters (e.g., Node Exporter, Cert Exporter, and others) for metrics collection.
• Hands-on experience in GenAI skills required.
• Extensive hands-on experience with containerization and orchestration platforms, including Docker and Kubernetes.
• Deep proficiency with a range of observability tools such as Splunk, ClickHouse, Grafana, Prometheus, M3DB, OpenTelemetry, Fluent Bit, ElasticSearch, OpenSearch, and CloudWatch.
• Strong experience managing CI CD pipelines using tools like GitHub and Ansible.
• Proficiency with Infrastructure as Code tools (e.g., Terraform) and configuration management practices (e.g., GitOps).
• Advanced scripting skills in Python and or Shell scripting.
• Working knowledge of query languages such as PromQL, MS SQL, or Splunk SPL is highly desirable.
• Solid background working in Linux environments.
• GCP or AWS cloud certifications are a strong plus.
• Excellent analytical skills, with the ability to break down complex technical problems and communicate solutions at varying levels of detail for different stakeholders.
• Outstanding communication, collaboration, and leadership abilities, with a track record of mentoring and guiding technical teams.

Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.