Overview In this role you lead the Spark Structured Streaming initiative within Databricks, shaping a high-performance stream processing engine for OSS and the Databricks Data Intelligence Platform. You drive architectural decisions, talent development, and cross-functional alignment to deliver a robust, scalable streaming solution. You will push advanced state management and new operators while improving latency and throughput. This is a mission-critical role to broaden Stream Processing adoption across the product portfolio and impact customers at scale.
Compensation / Benefits- pay range transparency
- annual performance bonus
- equity
- comprehensive benefits
Responsibilities- Lead the Spark Structured Streaming engineering effort across OSS and Databricks platform
- Oversee recruitment and upskilling of team members
- Implement processes to translate product vision into roadmap and delivery
- Build high-quality, easy-to-operate software
- Drive Stream Processing adoption across the Databricks product portfolio
- Manage technical debt and long-term architecture decisions
Key requirements- 5+ years in a related system (big-data, Spark or database internals)
- passion for database, storage, distributed systems, language design, or performance optimization
- ability to deliver high-quality and reliable infrastructure services with testing, quality, and SLAs
- experience building and leading teams in complex technical domains (distributed data systems or database internals)
- ability to attract, hire, and coach engineers and grow leaders
- experience managing distributed teams (preferred)
- comfort working cross-functionally with product management and customers
- leadership
- talent development
- cross-functional collaboration
- Apache Spark
- Spark Structured Streaming
- distributed systems
