Sr. Data Engineer with Databricks, SQL, Lakehouse
Role details
Job location
Tech stack
Job description
Technical Engineering Services this team operates in an Azure + Databricks Lakehouse. The role requires the following technical skill set, Approximate Percent of Time* Leadership & Relationship Skills this team is part of a broader, cross-functional organization with a variety of role types and experience levels. This individual needs to be able to: Collaborate with technical teams and utilizes system expertise to deliver technical solutions. Continuously learns and teaches others existing and new technologies. Contributes to the development of others through mentoring, in house workshops, and learning sessions. Drive engineering projects by developing data solutions; conducting tests and inspections; preparing reports and calculations. Expected to supervise base and associate level engineers as needed. Understand system protocols, how systems operate and data flows. Aware of current technology benefits. Expected to independently develop a full software stack. Understands the building blocks, interactions, dependencies, and tools required to complete software and automation work. Independent study of current technology is expected. Interact with system engineers to define system requirements and/or necessary requirements for automation. Communication: This team leads the enterprise in platform and process. Communication of architecture, process, and implementation is expected to elevate all teams. This individual needs to be able to: Write basic documentation on how technology works. Create clear documentation for new code and systems used. Documenting systems designs, presentations, and business requirements for consumption and consideration at the manager and director level. Technology Strategy Contributes to emerging technologies to deliver business goals and understands current technology that supports business goals. Understands system protocols, how systems operate and data flows. Expected to independently develop a full software stack. Understands the building blocks, interactions, dependencies, and tools required to complete software and automation work. Aware of current technology benefits. Identify and design solutions to potential data processes that can be improved with automation or optimized with current technical best practices. Identify use cases and design technical solutions to include Agentic AI application MAIN RESPONSIBILITIES: Innovation this group leverages industry leading platforms and processes. This individual is expected to: Deal with the ambiguity involved in developing new systems/platforms Design and implement new ideas which utilize novel frameworks to improve an existing or new system/process/service. Review existing designs and processes to highlight more efficient ways to complete existing workload more effectively through industry perspectives. Contribute to new and existing technology options that support business goals and understand how the current technology supports existing/future business goals. Status: Open, Role Overview As a Backend Engineer, you will play a crucial role in shaping the future of AI by designing, developing, and maintaining robust server-side solutions for complex sof…
- 3 days ago
- Apply easily, Hybrid - Senior .NET Software Engineer with deep expertise in C#, .NET Core, Azure, Blazor and microservices. This Jobot Consulting Job is hosted by: Liz Valdez Are you a fit? E…
- 12 days ago
Requirements
Cloud data engineering experience, including: data orchestration with tools such as Azure Data Factory data storage in cloud storage such as Azure Data Lake SDLC + CI/CD through systems such as Azure DevOps SQL proficiency (TSQL, PostGresSQL, ) and familiarity with Spark, including the ability to develop ETL processes using PySpark and SparkSQL Data streaming experience (Kafka, event hub, etc) Experience with parquet and other data exchange/storage format Experience with agile development and standard DevOps practices. Should understand principles of indexes and partitioning Should have experience in optimizing data engineering code for performance Proficiency with parquet and other data exchange/storage formats Experience with agile development and standard DevOps practices. Should have a deep understanding of indexes and partitioning Should be proficient optimizing data engineering code for performance (able to read a DAG, determine where CBO is using most resources) Should be proficient in common distributed cloud computing concepts like fault tolerance, idempotency, SOA, etc.