> Markdown version of [/jobs/ext/3762465-data-engineer-azure-databricks](https://www.wearedevelopers.com/jobs/ext/3762465-data-engineer-azure-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer Azure Databricks - **Company:** TEKSYSTEMS INC. - **Location:** New York, NY, United States (Remote available) - **Experience:** Experienced - **Salary:** $93,600.0 - $114,400.0 - **Contract:** Temporary contract - **Skills:** Microsoft Windows, Artificial Intelligence, Business Analytics Applications, Data Analysis, Microsoft Azure, Cyber Security, Databases, Data Architecture, Information Engineering, Data Governance, Data Integration, Extract Transform Load (ETL), Data Transformation, Data Warehousing, Database Development, Apache Hive, Identity and Access Management, Python (Programming Language), Knowledge Management, PostgreSQL, Machine Learning, Metadata, Meta-Data Management, Language Modeling, Power BI, Search Technologies, Security Software, SQL Databases, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Git, Data Layers, Data Lakes, Pyspark, Information Technology, Data Lineage, Tools for Reporting, Splunk, Software Version Control, Data Pipelines, Docker, ChromaDB, Servicenow, Databricks - **Published:** October 15, 2026 - **Apply:** https://www.careercircle.com/jobs/all/all/usa/ny/new-york/8f819565-8b1d-4e38-8863-275296ebfa2b ## About the Role Work Authorization Requirement: Candidates must be legally authorized to work in the United States without current or future sponsorship. Additional eligibility requirements may apply based on the nature of the clients work., * Approximately three or more years of hands-on data engineering experience. * Strong Python and SQL development skills. * Hands-on experience building data pipelines in Azure Databricks or a comparable Databricks environment. * Experience with PySpark, Spark SQL, Databricks notebooks, and Delta Lake. * Experience developing ETL or ELT pipelines and automated data workflows. * Understanding of medallion architecture and Bronze, Silver, and Gold data layers. * Experience with data modeling, data warehousing, schema design, and analytics-ready datasets. * Experience building data-quality validations, monitoring, and error-handling processes. * Ability to explain previous Databricks projects and individual technical contributions. * Must be legally authorized to work in the United States without current or future sponsorship and meet any additional client eligibility requirements. Preferred Qualifications * Experience working with Microsoft Azure or Azure Government. * Experience integrating data from Microsoft 365, ServiceNow, Splunk, identity platforms, endpoint systems, or cybersecurity tools. * Experience with data governance, data lineage, metadata management, or semantic data catalogs. * Exposure to retrieval-augmented generation, vector databases, embeddings, or semantic search. * Experience with ChromaDB or another vector database. * Understanding of cosine similarity, clustering, metadata-filtered retrieval, or related matching techniques. * Exposure to large language model orchestration, prompt engineering, or structured LLM outputs. * Experience with Power BI, PostgreSQL, Git, or Docker. * Databricks Data Engineer or Machine Learning certification. * Bachelors or advanced degree in Data Engineering, Data Science, Computer Science, or a related field., Individual compensation offered for this position within this range will depend on many factors, including qualifications, skills, relevant experience, job knowledge, geographic location, internal equity, and other pertinent job-related factors. ## Description PySpark Power BI Metadata Operations Automation Governance ServiceNow Databricks PostgreSQL Scalability Data Layers Data Quality Data Science Data Lineage Data Modeling Microsoft 365 Cyber Security Data Pipelines Microsoft Azure Data Governance Semantic Search Vector Database Computer Science Azure Databricks Machine Learning Data Warehousing Data Engineering Cosine Similarity Business Valuation Exception Handling Prompt Engineering Workflow Management Metadata Management Service Improvement Reporting Solutions Full Stack Development Cyber Security Systems Large Language Modeling Business Transformation Authorization (Computing) SQL (Programming Language) Git (Version Control System) Extract Transform Load (ETL) Service Improvement Planning Python (Programming Language) Technology Adoption Lifecycle Retrieval Augmented Generation Frontline Decision-Making Autonomy, A leading global aerospace and defense technology organization is seeking a Data Engineer to support its Enterprise Services Infrastructure team. This role will focus on designing and maintaining enterprise data pipelines in Azure Databricks. The Data Engineer will integrate data from Microsoft 365, ServiceNow, Splunk, identity platforms, endpoint technologies, and cybersecurity systems into analytics-ready Delta Lake tables. The position will also support data governance, lineage, semantic search, natural-language analytics, and applied AI initiatives. This is an excellent opportunity for a developing Data Engineer who wants hands-on experience with enterprise-scale data, modern Lakehouse architecture, and emerging AI technologies., * Design, build, and maintain enterprise data pipelines using Azure Databricks. * Integrate ServiceNow, Splunk, Microsoft 365, identity, endpoint, infrastructure, and cybersecurity data. * Use Python, PySpark, and Spark SQL to transform raw data into analytics-ready Delta Lake tables. * Develop scalable ETL and ELT workflows using Delta Lake and medallion architecture. * Build Bronze, Silver, and Gold data layers to support analytics, reporting, and machine learning use cases. * Create and maintain data models that support Power BI reporting, operational analytics, and business decision-making. * Build automated data-quality checks, validation processes, monitoring, and error handling. * Develop data-lineage and governance solutions to identify upstream dependencies, downstream writes, undocumented transformations, stale datasets, ownership gaps, and redundant data products. * Help create semantic data catalogs that allow users to identify authoritative datasets and understand their business context. * Support natural-language analytics solutions that help users find relevant data and generate grounded analytical queries. * Contribute to retrieval-augmented generation solutions that connect current ServiceNow incidents with relevant historical resolutions. * Use vector embeddings, semantic search, metadata filtering, similarity matching, and clustering to improve enterprise knowledge management. * Help identify stale, duplicate, underused, missing, or low-quality knowledge articles. * Collaborate with infrastructure, identity, service desk, governance, cybersecurity, and analytics teams. * Translate operational challenges into scalable data products, reporting solutions, automation, and applied AI use cases. * Document pipelines, data models, transformations, dependencies, and engineering processes.