Data Engineer (1 - 2 years of experience in Big Data, Spark, Hadoop, Shell scripting)

At Visa, the Corporate Information Technology, Billing & Incentives Platforms team, enables Visa's revenue growth through flexible pricing engines and global revenue platforms built on next-generation technologies. This includes managing system requirements, evaluating cutting-edge technologies, design, development, integration, quality assurance, implementation and maintenance of corporate revenue applications. The team works closely with business owners of these services to deliver customer developed solutions, as well as implement industry leading packaged software. This team has embarked on a major transformational journey to build and implement best of breed revenue and billing applications to transform our business as well as technology.  

The candidate should enjoy working on diverse technologies and should be excited to take initiatives to solve complex business problems and get the job done while taking on new challenges. You should thrive in team-oriented and fast-paced environments where each team-member is vital to the overall success of the projects.

Key Responsibilities

• Design, develop, and maintain automated test frameworks and scripts using PySpark for large-scale Big Data applications.

  Collaborate closely with data engineering, development, and DevOps teams to understand data pipelines, workflows, and transformation logic.

  Build and execute robust tests to validate data ingestion, processing, and transformation in Hadoop and Spark ecosystems.

  Perform comprehensive data validation, data integrity, and performance testing across Spark, Hive, and RDBMS data layers.

  Apply deep understanding of Spark concepts (RDDs, DataFrames, Datasets, Spark SQL) to create reusable test automation components.

  Leverage Shell scripting and Unix-based tools for automation, log analysis, and environment configuration.

  Integrate test automation into CI/CD pipelines to ensure continuous testing and high-quality data delivery across environments.

  Identify, document, and track defects, working proactively with developers to ensure quick resolution. Knowledge of Jira Standards.

  Optimize and tune Spark jobs and query performance for scalability and efficiency.

  Maintain detailed documentation, including test plans, test cases, and results, aligned with evolving business requirements.

  Conduct manual and functional testing for critical payment and financial data applications to ensure compliance and accuracy.

  Operate independently in a fast-paced, distributed environment, balancing multiple priorities to meet service-level goals.

  Collaborate effectively across teams and time zones, fostering transparent communication, trust, and long-term partnerships with business stakeholders.

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.

Basic Qualifications:
Bachelor's degree, OR 3+ years of relevant work experience

Preferred Qualifications:

• Bachelor’s degree in computer science, Information Technology, or related field.
• Relevant certifications in Big Data, Spark, QA.
• Experience with on premise infrastructure
• Knowledge of ETL processes and tools.

Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.