Overview In this role you will design, implement, and administer Hadoop-based infrastructure to support enterprise data processing. You will work within IT and engineering teams to manage development and maintenance lifecycles, optimize system performance, and troubleshoot capacity and reliability issues. You will leverage open-source tools and scheduling/automation platforms to keep data workflows running smoothly at scale. This position offers the opportunity to influence data engineering practices and contribute to a mature, service-oriented IT environment.
Responsibilities- Implement and administer Hadoop infrastructure and related components
- Lead project lifecycle activities on development and maintenance projects
- Troubleshoot system performance, capacity, memory, CPU, OS, storage, and network bottlenecks
- Manage Hadoop ecosystem tools (Hadoop, MapReduce, HBase, Hive, Pig, Mahout) and administration interfaces
- Utilize schedulers and job management (Oozie, Control-M, Tivoli) for workflow orchestration
- Work with Linux-based environments and authentication models (LDAP/Kerberos)
- Apply configuration management and scripting (Puppet, Chef, Linux scripting) and autosys
- Develop scripts (Shell, Perl, Python) and expose to real-time engines (Spark, Kafka)
- Maintain version control and ticketing integrations (Subversion/Github, ServiceNow/Remedy)
- Collaborate with cross-functional teams to support operational excellence
Key requirements- Hadoop ecosystem expertise (Hadoop, MapReduce, HBase, Hive, Pig, Mahout)
- Experience with Cloudera Manager or Ambari; Ganglia; Nagios
- Experience with Hadoop schedulers and job orchestration (Oozie, Control-M, Tivoli)
- Strong Linux knowledge; LDAP/Kerberos authentication exposure
- Scripting and automation with Shell, Perl, Python; configuration management tools (Puppet, Chef)
- Real-time engines exposure (Spark, Storm, Kafka)
- Version control (GitHub, SVN) and ticketing tools (ServiceNow, Remedy)
- Bachelor's degree or foreign equivalent; 4+ years IT experience
- Hadoop
- MapReduce
- HBase
- Hive
- Oozie
- Cloudera Manager
