Data Automation Engineer

We are seeking a highly skilled Databricks Engineer to support U.S. Coast Guard data modernization initiatives. The ideal candidate will design, develop, and maintain scalable data engineering solutions using the Databricks Lakehouse Platform to enable advanced analytics, reporting, and mission-critical decision-making. This role requires expertise in cloud-based data platforms, data integration, ETL/ELT processes, and secure handling of government data., Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and cloud-native services. Build and optimize ETL/ELT workflows to ingest, transform, and load structured and unstructured data from multiple sources. Implement and manage Delta Lake architectures to support reliable and high-performance data processing. Collaborate with data architects, analysts, and business stakeholders to define data requirements and technical solutions. Develop and maintain data models, data marts, and Lakehouse solutions that support Coast Guard operational and analytical needs. Monitor, troubleshoot, and optimize Databricks workloads for performance, scalability, and cost efficiency. Implement data quality, governance, lineage, and security controls in compliance with federal regulations and Coast Guard policies. Automate deployment processes using Infrastructure as Code (IaC) and CI/CD pipelines. Support cloud migration initiatives and modernization efforts involving legacy data systems. Create technical documentation, standard operating procedures, and architecture diagrams. Participate in code reviews, testing, and release management activities. Provide operational support and resolve production issues in a timely manner. Ensure compliance with cybersecurity requirements, including NIST, FISMA, and federal data protection standards. Support data integration with enterprise systems, APIs, and external data sources.

Bachelor’s degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field. 5+ years of experience in data engineering, data integration, or data platform development. 3+ years of hands-on experience with Databricks and Apache Spark. Strong proficiency in Python, PySpark, and SQL. Experience building and managing data pipelines in cloud environments such as Azure, AWS, or Google Cloud Platform. Experience with Delta Lake, Lakehouse architecture, and large-scale data processing. Knowledge of data warehousing concepts, dimensional modeling, and database optimization. Experience with source control systems such as Git. Familiarity with CI/CD tools and DevOps practices. Strong analytical, troubleshooting, and problem-solving skills. Excellent written and verbal communication skills. Ability to obtain and maintain a Public Trust or Secret clearance, as required., Databricks Certified Data Engineer Associate or Professional certification. Experience supporting federal government or Department of Homeland Security (DHS) programs. Experience with Azure Data Factory, AWS Glue, or equivalent data integration tools. Knowledge of data governance frameworks and metadata management tools. Experience with Power BI, Tableau, or other business intelligence platforms. Familiarity with Terraform, ARM templates, or other Infrastructure as Code tools. Experience implementing data security controls in regulated environments. Understanding of machine learning workflows and MLOps within Databricks. Active security clearance preferred.

Technical Skills

Databricks Lakehouse Platform Apache Spark / PySpark Python SQL Delta Lake Azure Databricks / AWS Databricks Data Warehousing ETL/ELT Development Git/GitHub CI/CD Pipelines REST APIs Data Governance & Security Cloud Platforms (Azure, AWS, GCP)

Big Impact Tech (BIT) is a Small Business providing IT and business management consulting to federal and commercial clients. We deliver mission-focused solutions in data, cloud, cybersecurity, and program management.