Overview In this role you will manage and optimize Hadoop infrastructure within a globally distributed delivery model. You will work with cross-functional teams to maintain clusters, ensure security and availability, and drive performance improvements. You will apply administration, scripting, and open-source tooling to support data delivery and analytics workloads. This role offers hands-on impact in upgrading ecosystems and collaborating with multiple stakeholders to ensure high data quality and uptime.
Responsibilities- Maintain and scale Hadoop ecosystems and clusters, including node lifecycle management
- Administer Cloudera Manager or Ambari, and monitoring tools like Ganglia and Nagios
- Configure and manage Hive, Pig, MapReduce environments and related jobs
- Set up Linux users, groups, Kerberos principals and authentication
- Collaborate with systems, network, DB, and BI teams to ensure hardware and software readiness
- Install, patch, upgrade software, and coordinate version updates
- Tune performance of Hadoop clusters and MapReduce routines
- Monitor cluster availability, security, and job performance
- Plan and support backup, high availability, and disaster recovery
- Support data delivery teams in setting up development environments for Hadoop apps
- Maintain Hadoop logs, file systems, and disk usage monitoring
- Work in a diverse, globally distributed delivery environment
Key requirements- At least 2 years in Hadoop implementation and administration
- At least 2 years in project life cycle activities on development and maintenance projects
- Operational troubleshooting expertise across memory, CPU, OS, storage, and networks
- Knowledge of Hadoop components: MapReduce, HBase, Hive, Pig, Mahout
- Experience with Cloudera Manager or Ambari, Ganglia, Nagios
- Experience with Hadoop schedulers (FIFO, Fair Scheduler, Capacity Scheduler)
- Experience with job schedulers like Oozie or Control-M/Tivoli
- Good knowledge of Linux
- Exposure to Ad/LDAP/Kerberos authentication
- Familiarity with Puppet or Chef and Linux scripting; Autosys
- Shell and Perl scripting; exposure to Python
- Knowledge of Core Java applications troubleshooting is a plus
- Exposure to Spark, Storm, Kafka
- Version control tools: Subversion, ClearCase, CVS, Github
- Experience with service management ticketing tools: ServiceNow, Remedy, Service Manager
- Experience with real-time execution engines like Spark, Storm, Kafka
- Data backup and recovery concepts, HDFS support
- Hadoop log management and data quality focus
- Experience with backup/HA/DR planning
- Ability to work with cross-functional teams in a global delivery environment
- Team collaboration
- Communication across diverse stakeholders
- Problem-solving and troubleshooting mindset
- Hadoop ecosystem (Hadoop, MapReduce, HBase, Hive, Pig, Mahout)
- Cloudera Manager or Ambari
- Ganglia, Nagios
