Data Engineer

Apptad Inc.

New York, United States of America

1 month ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Languages

English

Experience level

Senior

Compensation

$ 148K

Job location

New York, United States of America

Tech stack

API

Artificial Intelligence

Amazon Web Services (AWS)

Big Data

Cloud Computing

Information Engineering

Data Governance

Data Integrity

ETL

Data Structures

Data Visualization

Python

Pattern Recognition

Power BI

Standard Sql

Systems Integration

Tableau

Data Processing

Freeform SQL

Application Enhancement Tool

Sql Optimization

Large Language Models

Prompt Engineering

Spark

Caching

Data Lake

PySpark

Data Analytics

Machine Learning Operations

Databricks

Job description

Create data products in Databricks while leveraging AI-powered tools to enhance workflow automation, perform pattern recognition, analyze and generate content.

Requirements

Top Skills High proficiency in Python and SQL
ETL experience with Databricks using PySpark
Data processing and analytics/AI in Databricks
Strong understanding of algorithms, data structures, statistics, and mathematics
GenAI, LLM deployment and monitoring, * Collaborate with business users and engineers to align data analytics solutions with business goals.
Seeking an experienced Data Analytics/AI Engineer to join our team to:
Required Technical skills Databricks Proficiency: Hands-on experience with AWS Databricks platform; knowledge of Spark performance and cost-optimization technics (partitioning, clustering, caching).
Advanced SQL and Spark Skills: Proficiency in writing complex SQL queries and Spark code (Python/PySpark) for data manipulation, transformation, aggregation, and analysis tasks within Databricks notebooks.
GenAI and LLMs: Prompt engineering, RAG, experience leveraging Large Language Models (LLMs); familiarity with tools such as Amazon SageMaker and Bedrock.
Data Engineering & Processing: Experience building ETL pipelines and working with big data frameworks.
System Integration and Deployment: Deploy AI models and integrate them with existing systems via APIs.
Data Visualization: Using data visualization tools (Tableau, Power BI) to create interactive dashboards and communicate insights.
Cloud Computing and MLOps: Proficiency in deploying models using AWS; familiarity with Domino.
Data Lab, GitLab CI/CD Data Governance and Security: Understanding of data governance principles and implementing security measures to ensure data integrity, confidentiality, and compliance within the centralized data lake environment.