Data Engineer II

About the team:

Our data team is driven by a clear mission to democratize data within HackerRank, making it accessible and actionable for everyone.

Recent Achievements:

The team recently launched an exports service that has revolutionized performance, achieving a remarkable 10x improvement. This milestone underscores our commitment to delivering impactful, scalable solutions.

Collaboration Style:

Collaboration is at the heart of how we work. We seamlessly balance synchronous and asynchronous methods, enabling us to work cohesively as a team while respecting individual workflows. This approach fosters efficiency and inclusivity in tackling tasks together.

About the role:

This role is all about building modern, scalable data systems that power real products. You’ll work across the full stack from designing robust data pipelines to supporting search and analytics platforms using tools like Airflow, dbt, and Spark. It’s a hands-on role with real ownership, where you’ll take projects from idea to production. You’ll work with cloud platforms (like AWS or GCP), open-source tools, and contribute to shaping a flexible, future-ready data foundation. If you enjoy solving complex problems, learning new technologies, and working both independently and with a collaborative team, this role is a great fit.

What you’ll do:

• Evaluate technologies, develop POCs, solve technical challenges and propose innovative solutions for our technical and business problems
• Delight our stakeholders, customers and partners by building high-quality, well-tested, scalable and reliable business applications.
• Design, build and maintain streaming and batch data pipelines that can scale.
• Architect, develop and maintain our Modern lake house Platform using AWS native infrastructure
• Designing Complex Data Models to deliver insights and enable self-service
• Take ownership of scaling, performance, security, and reliability of our data infrastructure
• Hiring, guiding and mentoring junior engineers
• Work in an agile development environment, participate in code reviews
• Collaborate with remote development teams and cross-functional teams

You will thrive in this role if:

• You love solving tough challenges that create real-world impact and are excited to dive into uncharted territories.
• You enjoy fast-paced, dynamic environments where collaboration isn’t just encouraged - it’s essential
• You care about understanding product challenges and finding creative solutions, beyond just coding tasks.
• You value shipping solutions quickly while refining and enhancing them as you go.
• You’re willing to break boundaries and contribute wherever needed, even if it’s outside your usual responsibilities.

What you bring:

• 2+ years of experience with designing, developing and maintaining data engineering & BI solutions.
• Experience with Data Modeling for Big Data Solutions.
• Experience with Spark, Spark Structured Streaming (Scala Spark)
• Experience with database technologies like Redshift or Trino
• Experience querying massive datasets using Languages like SQL, Hive, Spark, and Trino
• Experience with performance tuning complex data warehouses and queries.
• Able to solve problems of scale, performance, security, and reliability
• Self-driven, initiative taker with good communication skills, ability to lead and mentor junior engineers, work with cross-functional teams, and drive architecture decisions.
• Bonus: Experience with ETL Design & Orchestration using platforms like Apache Airflow, MageAI