About Smart Working
At Smart Working, we believe your job should not only look right on paper but also feel right every day. This isn’t just another remote opportunity - it’s about finding where you truly belong, no matter where you are. From day one, you’re welcomed into a genuine community that values your growth and well-being.
Our mission is simple: to break down geographic barriers and connect skilled professionals with outstanding global teams and products for full-time, long-term roles. We help you discover meaningful work with teams that invest in your success, where you’re empowered to grow personally and professionally.
Join one of the highest-rated workplaces on Glassdoor and experience what it means to thrive in a truly remote-first world.
About the Role
We are hiring a highly experienced Anti-Bot Engineer to design, build, and operate large-scale, resilient web scraping systems that extract high-quality data from heavily protected and constantly evolving web environments.
This is a senior-level, hands-on engineering role focused on building production-grade systems that operate at scale under adversarial conditions (anti-bot systems, dynamic rendering, frequent structural changes, and high-volume data extraction).
You will be responsible for architecting and maintaining scraping infrastructure that must remain reliable, observable, and adaptable in real time. The role requires deep technical maturity, strong systems thinking, and the ability to operate independently in complex, ambiguous environments.
This position is best suited to engineers who have already built and operated large-scale distributed systems in production for several years and are comfortable owning mission-critical data pipelines end-to-end.
Responsibilities
- Design, build, and maintain large-scale Python-based scraping systems targeting highly protected websites (including Google-like environments).
- Architect resilient extraction systems capable of handling dynamic, JavaScript-heavy pages using browser automation and hybrid approaches.
- Continuously adapt systems to frequent changes in page structures, request flows, and anti-bot mechanisms.
- Build robust, production-grade data extraction pipelines with strong emphasis on accuracy, observability, and fault tolerance.
- Implement advanced strategies such as proxy rotation, fingerprinting, session management, and request routing to ensure stability at scale.
- Monitor system health, proactively detect anomalies, and debug complex production failures across distributed systems.
- Optimise scraping infrastructure for performance, cost efficiency, and reliability at scale.
- Collaborate with data engineering and product teams to ensure scraped data is structured, validated, and trusted.
- Operate and improve systems running continuously in cloud environments.
- Document system architecture, scraping logic, and operational procedures for long-term maintainability.
- Contribute to improving resilience, automation, and adaptability in adversarial environments.
Requirements
- 7+ years of professional software engineering experience, with a strong focus on backend systems, data engineering, or distributed systems.
- Proven experience building and operating large-scale production web scraping systems.
- Deep hands-on experience scraping Google or similarly complex, heavily protected / anti-bot environments.
- Strong expertise in Python (or comparable production languages such as Go, Rust, or JavaScript).
- Strong understanding of HTTP internals: headers, cookies, TLS, redirects, sessions, and browser networking behaviour.
- Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalent.
- Strong knowledge of HTML parsing, DOM traversal, and high-performance data extraction techniques.
- Proven experience handling anti-bot systems, including rate limiting, CAPTCHAs, IP rotation, and fingerprinting.
- Experience designing asynchronous and concurrent systems for high-throughput workloads.
- Strong debugging skills across distributed, failure-prone production systems.
- Experience running cloud-based systems at scale with strong operational ownership.
- Comfortable working in highly adversarial and fast-changing technical environments.
Nice to Have
- Experience with Docker and Kubernetes in production environments.
- Exposure to distributed task queues or large-scale job orchestration systems.
- Experience with monitoring, anomaly detection, or data quality validation systems.
- Background in search intelligence, advertising tech, or competitive intelligence platforms.
- Experience building systems that operate against high-defence web environments at scale.
- Familiarity with observability tooling and production-grade system monitoring.
- Exposure to AI-assisted development workflows or agentic coding tools.
Benefits
- Fixed Shifts: 12:00 PM - 9:30 PM IST (Summer) | 1:00 PM - 10:30 PM IST (Winter)
- No Weekend Work: Real work-life balance, not just words
- Day 1 Benefits: Laptop and full medical insurance provided
- Support That Matters: Mentorship, community, and forums where ideas are shared
- True Belonging: A long-term career where your contributions are valued
At Smart Working, you’ll never be just another remote hire.
Be a Smart Worker - valued, empowered, and part of a culture that celebrates integrity, excellence, and ambition.
If that sounds like your kind of place, we’d love to hear your story.
