Software Engineer - Data processing / Data Integration
Role details
Job location
Tech stack
Job description
- Data Integration: Integrate data from multiple sources and systems, including databases, APIs, log files, streaming platforms, and external data providers. Handle data ingestion, transformation, and consolidation to create a unified and reliable data foundation for analysis and reporting
- Data Transformation and Processing: Develop data transformation routines to clean, normalize, and aggregate data. Apply data processing techniques to handle complex data structures, handle missing or inconsistent data, and prepare the data for analysis, reporting, or machine learning tasks
- Contribute to common frameworks and best practices in code development, deployment, and automation/orchestration of data pipelines
- Implement data governance and data de-identification framework in line with company standards
- Partner with Data Analytics and Product leaders to design best practices and standards for developing and productionalizing analytic pipelines
- Partner with Infrastructure leaders on architecture approaches to advance the data and analytics platform, including exploring new tools and techniques that leverage the cloud environment (Azure, Snowflake, others)
- Monitoring and Support: Monitor data pipelines and data systems to detect and resolve issues promptly. Develop monitoring tools, alerts, and automated error handling mechanisms to ensure data integrity and system reliability
- Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
You will be rewarded and recognized for your performance in an environment that will challenge you and give you clear direction on what it takes to succeed in your role, as well as providing development for other roles you may be interested in.
Requirements
- Bachelor's degree in CS or IT related field
- 5+ years of experience in Azure Data Processing skills like Azure data Factory and Azure data Bricks
- 5+ years of experience in building data pipelines using ADF
- 5+ years of experience in SQL and complex queries
- 5+ years of experience with programming languages such as Python, PySpark, * Ability to learn and adapt to new data technologies
- Knowledge/Experience with Containerization - Docker, Kubernetes
- Knowledge/Experience with Bigdata/Hadoop ecosystem - Spark, Hive, HBase, Sqoop etc
- Build / Deployment Automation - Jenkins
- Knowledge/Experience using Microsoft Visio, Power Point
- Collaborate with team, architects, and product stakeholders to understand the scope and design of a deliverable
- Participate in product support activities as needed by the team
- Understand product architecture, features being built and come up with product improvement ideas and POC's
- Individual contributor for Data Engineering - Data pipelines, Data modelling and Data warehouse
Benefits & conditions
Pay is based on several factors including but not limited to local labor markets, education, work experience, certifications, etc. In addition to your salary, we offer benefits such as, a comprehensive benefits package, incentive and recognition programs, equity stock purchase and 401k contribution (all benefits are subject to eligibility requirements). No matter where or when you begin a career with us, you'll find a far-reaching choice of benefits and incentives. The salary for this role will range from $72,800 to $130,000 annually based on full-time employment. We comply with all minimum wage laws as applicable.
Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.