> Markdown version of [/jobs/ext/739448-databricks-data-engineer-remote](https://www.wearedevelopers.com/jobs/ext/739448-databricks-data-engineer-remote). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Data Engineer - Remote - **Company:** PALMETTO TECHNOLOGIES, LLC - **Location:** United States (Remote available) - **Experience:** Experienced - **Salary:** $135,200.0 - $141,440.0 - **Contract:** Permanent contract - **Skills:** Unity 3d, Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Audit Trail, Microsoft Azure, Software as a Service, Cloud Storage, Databases, Continuous Integration, Data Dictionary, Information Engineering, Data Governance, Data Integration, Relational Databases, Software Debugging, Github, Python (Programming Language), Key Management, Machine Learning, Open Source Technology, Operational Databases, Performance Tuning, Cloud Services, Search Technologies, Software Engineering, SQL Databases, Data Streaming, Unstructured Data, Enterprise Data Management, Azure Service Bus, Google Cloud, Enterprise Software Applications, Retrieval-Augmented Generation, Delivery Pipeline, Large Language Models, Prompt Engineering, Apache Spark, Generative AI, Data Layers, Data Lakes, Pyspark, Deployment Automation, HuggingFace, Real Time Data, Apache Kafka, Machine Learning Operations, Virtual Agents, Restful APIs, Terraform, Data Pipelines, Jenkins, Databricks - **Published:** June 29, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=d57ef47d055269a5 ## About the Role Do you have experience in Testing and evaluation?, * 7 or more years of experience in data engineering, software engineering, or enterprise data-platform development. * 4 or more years of hands-on experience with the Databricks platform. * 5 or more years of experience developing solutions using Python, PySpark, Apache Spark, and SQL. * Experience designing and implementing Databricks Lakehouse architectures. * Experience with Delta Lake, Delta Live Tables, Databricks Workflows, Databricks SQL, and Unity Catalog. * Experience developing batch and streaming data pipelines. * Experience designing data models and medallion architectures. * Hands-on experience developing generative AI, retrieval-augmented generation, or agentic AI solutions. * Experience building AI agents that use tools, APIs, enterprise data, and multi-step reasoning workflows. * Experience with at least one agent-development framework, such as: * Databricks Mosaic AI Agent Framework * LangChain * LangGraph * LlamaIndex * AutoGen * Semantic Kernel * Experience integrating large language models from platforms such as Azure OpenAI, OpenAI, Anthropic, AWS Bedrock, Hugging Face, or open-source model providers. * Experience with vector embeddings, vector search, semantic retrieval, document chunking, reranking, and prompt engineering. * Experience with MLflow, model registration, model serving, and AI application evaluation. * Experience integrating data from relational databases, REST APIs, cloud storage, event streams, SaaS applications, and document repositories. * Experience with cloud services in at least one of the following: * Microsoft Azure * Amazon Web Services * Google Cloud Platform * Experience implementing automated deployments and CI/CD pipelines. * Experience with data governance, security, lineage, access controls, and personally identifiable information protection. * Strong knowledge of data-quality testing, monitoring, debugging, and performance optimization. * Strong written and verbal communication skills., * Excellent communication skills - Very important ## Description * Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL. * Implement enterprise lakehouse architectures using Delta Lake, Delta Live Tables, Unity Catalog, and Databricks SQL. * Build batch and real-time data-processing solutions using technologies such as Structured Streaming, Kafka, Event Hubs, or Kinesis. * Develop reusable ingestion and transformation frameworks for structured, semi-structured, and unstructured data. * Design and implement medallion architectures using Bronze, Silver, and Gold data layers. * Develop agentic AI applications capable of planning tasks, retrieving information, calling tools, executing workflows, and validating results. * Build AI agents using frameworks such as Databricks Mosaic AI Agent Framework, LangChain, LangGraph, LlamaIndex, AutoGen, or Semantic Kernel. * Develop retrieval-augmented generation solutions using vector embeddings, semantic search, metadata filtering, and enterprise knowledge repositories. * Implement vector-search solutions using Databricks Vector Search or comparable vector databases. * Integrate large language models with enterprise applications, APIs, databases, document repositories, and business workflows. * Implement model and agent evaluation processes covering accuracy, groundedness, relevance, hallucination, safety, latency, and cost. * Develop agent memory, prompt-management, context-management, tool-calling, guardrail, and human-in-the-loop capabilities. * Use MLflow for experiment tracking, model registration, evaluation, deployment, and lifecycle management. * Build and support machine-learning and generative-AI deployment pipelines using Databricks Model Serving. * Implement CI/CD processes for notebooks, pipelines, infrastructure, models, prompts, and AI agents. * Automate Databricks deployments using Databricks Asset Bundles, Terraform, Azure DevOps, GitHub Actions, or Jenkins. * Establish data quality controls, reconciliation procedures, lineage, observability, monitoring, and alerting. * Implement security and governance using Unity Catalog, role-based access controls, row-level security, column masking, secrets management, and audit logging. * Optimize Spark workloads, cluster configurations, SQL queries, data layouts, and storage costs. * Troubleshoot production data pipelines, AI-agent workflows, integrations, model endpoints, and performance issues. * Collaborate with data architects, data scientists, machine-learning engineers, application developers, cybersecurity teams, and business stakeholders. * Create technical designs, architecture diagrams, data mappings, data dictionaries, runbooks, and operational documentation. * Mentor team members and establish engineering standards for Databricks, data pipelines, and agentic AI development. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [The Road to MLOps: How Verivox Transitioned to AWS](https://www.wearedevelopers.com/videos/1050-the-road-to-mlops-how-verivox-transitioned-to-aws) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)