Overview In this role you will design and deliver Databricks-based data solutions in a Boston-based, onsite setup. You will drive lakehouse architecture with Delta Lake, and enable integration with systems like Salesforce, MDM, and Col l ibra, while aligning with ETL/ELT and data warehousing best practices. You'll lead DAGs and orchestration in Airflow, leveraging AWS services to scale data pipelines and governance. This position combines hands-on engineering with architecture to shape our data platform.
Responsibilities- Architect end-to-end Databricks solutions and data pipelines
- Design integration patterns with Salesforce, MDM and Collibra
- Develop and optimize Airflow DAGs for orchestration and monitoring
- Leverage Delta Lake and Lakehouse concepts for data modeling
- Implement ETL/ELT processes and data warehousing best practices
- Work with AWS services (S3, EC2, Lambda, IAM) to support data workflows
- Collaborate with cross-functional teams to translate requirements into scalable data architectures
Key requirements- Proficiency in PySpark and Databricks (Delta Lake, clusters, jobs)
- Experience in architectural design for integrating Databricks with external systems
- Hands-on with Apache Airflow (DAG design and monitoring)
- Strong SQL and Python for transformations and orchestration
- Knowledge of Lakehouse architecture (Delta Lake) and data modeling
- Experience in ETL/ELT and data warehousing best practices
- Strong experience with AWS services (S3, EC2, Lambda, IAM)
- Databricks
- PySpark
- Delta Lake
- Airflow
- AWS (S3, EC2, Lambda, IAM)
- SQL
