Data Engineer

Remote Hiring Remotely in United States Mid level Remote Hiring Remotely in United States Mid level Designs and optimizes scalable data platforms, architectures, and production pipelines supporting AI and machine learning. Responsibilities include Databricks Lakehouse development, batch and real-time processing, data governance, security, monitoring, performance tuning, infrastructure planning, and collaboration with DevOps, MLOps, and cross-functional teams. The role serves as the AI team’s data engineering subject matter expert and ensures reliable, secure, cost-effective data solutions. The summary above was generated by AI, The Senior Data Engineer plays a key role in advancing the company’s AI strategy by designing, developing, and optimizing scalable data platforms. This role focuses on data engineering as part of the AI team, contributing to technical development, data architecture, pipeline optimization, and actionable intelligence. The ideal candidate will have expert-level experience with Databricks, strong software engineering capabilities, and a proven ability to build robust, secure, and scalable data systems. The Senior Data Engineer will work closely with cross-functional teams to deliver high-quality data solutions, optimize production pipelines, and support AI and machine learning capabilities.

Responsibilities

• Design, develop, refine, and implement high-quality data architecture solutions for large-scale, complex data platforms.
• Establish best practices for writing modular, reusable, maintainable, and scalable data engineering code.
• Develop and maintain greenfield data engineering projects and production-level data pipelines.
• Design data-driven solutions to address complex cybersecurity challenges.
• Architect end-to-end data platforms by evaluating technical approaches, trade-offs, risks, and performance requirements.
• Work hands-on with engineers to implement data architecture and pipeline solutions.
• Collaborate with DevOps and MLOps teams to monitor and optimize data pipelines and machine learning workflows.
• Improve model and pipeline performance through monitoring, troubleshooting, and tuning to meet latency and cost requirements.
• Evaluate third-party AI tools against internally developed solutions to maximize business value and return on investment.
• Establish data governance, data quality, and security standards to ensure the ethical and compliant use of sensitive information in AI models.
• Develop and implement incremental development strategies and production release plans aligned with the long-term product vision.
• Serve as the subject matter expert in data engineering for the AI team.
• Ensure data pipelines are reliable, scalable, secure, and optimized for production workloads., The Senior Data Engineer will optimize data ingestion pipelines, manage PostgreSQL databases, design ETL processes, and ensure system reliability in a remote-first, collaborative environment., Own the strategy, vision, roadmap, and execution of Rapid7’s Managed Detection and Response product portfolio. Lead and develop a global product management team, partner with SOC and engineering leaders, translate customer and market insights into product decisions, define platform capabilities, and drive product OKRs. Serve as a senior customer-facing representative while collaborating with sales, marketing, and customer success on positioning, launches, adoption, retention, and expansion. Top Skills: CybersecurityDetection And Response PlatformsIncident ResponseManaged Detection And Response (Mdr)Saas PlatformsSecurity Operations Center (Soc)

What you need to know about the Colorado Tech Scene

With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.

Key Facts About Colorado Tech

• Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
• Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
• Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
• Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
• Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
• Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute

• Bachelor’s degree in Engineering, Computer Science, or a related technical field.
• Advanced degree in Engineering, Computer Science, Machine Learning, or Artificial Intelligence is preferred.
• Strong experience working as an individual contributor with a hands-on engineering mindset and a focus on delivery.
• At least 4 years of experience working with data platform solutions across data and analytics throughout the software development life cycle (SDLC).
• At least 2 years of experience designing and mapping requirements into Databricks data pipeline capabilities on Google Cloud Platform (GCP).
• Hands-on experience with Databricks Lakehouse architecture, including Databricks Datalake and Data Catalog capabilities.
• Experience designing Databricks Medallion Architecture, including data structure, data access, and data quality decisions.
• Strong experience with batch, mini-batch, and real-time data processing, including trade-offs and best practices.
• Hands-on experience with ELT/ETL processes, data pipelines, and orchestration tools.
• Experience processing large volumes of unlabeled data with complex schemas.
• Strong understanding of data modeling techniques and modern data platform trends, including AI applications in data engineering.
• Proficiency in Python, Pandas, PySpark, and related data processing frameworks.
• Experience with AI and machine learning tools such as TensorFlow, PyTorch, Keras, LangChain, LangSmith, and Infrastructure as Code (IaC).
• Experience working with structured and unstructured data formats, including Parquet, PostgreSQL, DuckDB, JSON, and PDF.
• Experience with infrastructure sizing and estimation for data platform solutions., * Experience in cybersecurity, automotive, energy, or healthcare industries.
• Experience working in both large enterprises (more than 10,000 employees) and small companies (fewer than 200 employees).
• Familiarity with supervised and unsupervised machine learning models.
• Experience with AI-driven data engineering solutions and integration of AI capabilities into data platforms.
• Knowledge of data governance, security, and compliance best practices.

Technical Skills

• Data Engineering: Data architecture, ETL/ELT, pipeline development, orchestration, data modeling, and data quality.
• Databricks: Lakehouse architecture, Delta Lake concepts, data cataloging, Medallion Architecture, and production pipeline optimization.
• Cloud Platform: Google Cloud Platform (GCP) and cloud-based data platform design.
• Programming: Python, PySpark, Pandas, and SQL.
• Data Processing: Batch, mini-batch, and real-time processing.
• Data Formats and Databases: Parquet, PostgreSQL, DuckDB, JSON, and PDF.
• AI and Machine Learning: TensorFlow, PyTorch, Keras, LangChain, and LangSmith.
• Infrastructure: Infrastructure as Code, infrastructure sizing, performance optimization, and cost management.
• Production Engineering: Monitoring, troubleshooting, release planning, scalability, and reliability.

Soft Skills & Core Competencies

• Excellent verbal and written communication skills.
• Strong collaboration skills and ability to work effectively with Product, Engineering, Field, and other cross-functional teams.
• Strong analytical thinking and problem-solving abilities, with the capacity to analyze complex issues and develop innovative solutions.
• Strong self-management skills and ability to prioritize tasks and manage time effectively.
• A proactive approach to work, with the ability to take ownership and initiative.
• Ability to work independently while contributing effectively to team objectives.

Remote or Hybrid United States 145K-196K Annually Senior level 145K-196K Annually Senior level Artificial Intelligence * Cloud * Information Technology * Sales * Security * Software * Cybersecurity Manage and mentor Product Managers while guiding product strategy, prioritization, roadmap execution, and lifecycle management for Rapid7’s cybersecurity platform. Partner with Product, Engineering, UX, executives, customers, and cloud ecosystem partners to define requirements, launch features, analyze usage and feedback, and drive market impact. The role also promotes AI-enabled product management practices, customer-centric decision-making, cross-functional alignment, and development of high-performing global teams., Remote or Hybrid CA, USA 259K-351K Annually Expert/Leader 259K-351K Annually Expert/Leader