Data Engineer Cloud Data & Cybersecurity Analytics

Propertyvalue Quantum Technologies Llc

Charlotte, NC, United States

22 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Compensation

$174,720.0

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Application Programming Interfaces (APIs) Amazon S3 Authentication Protocols Big Data Cloud Computing Cloud Database Code Review Cyber Security Computer Programming Continuous Integration Data Governance Extract Transform Load (ETL)

+34 more

Data Transformation Data Security Data Warehousing Database Development Software Debugging Electronic Data Interchange (EDI) Identity and Access Management JSON Python (Programming Language) OAuth Operational Databases Performance Tuning Query Optimization Cloud Services SQL Databases SQL Server Integration Services Automatic Programming Data Processing Data Storage Technologies Data Ingestion Snowflake Apache Spark Software Troubleshooting Backend Data Lakes Infrastructure Automation Frameworks Deployment Automation AWS Glue Apache Kafka Front End Software Development Api Design Restful APIs Terraform Data Pipelines

Job description

Develop and maintain AWS Glue ETL pipelines to process, transform, and integrate cybersecurity-related data.
Design, implement, and optimize Data Warehouse and Data Lake architectures.
Work with AWS S3 and Snowflake for large-scale data storage and analytics.
Develop and manage Terraform Infrastructure-as-Code (IaC) deployments.
Build and optimize Snowflake data models, queries, and data processing workflows.
Perform Snowflake performance tuning and optimization for large datasets.
Develop and manage APIs for data ingestion and data exchange between front-end applications, backend systems, and external platforms.
Work with RESTful APIs, JSON, OAuth, API keys, and other authentication mechanisms.
Develop, maintain, and troubleshoot SSIS packages.
Monitor production data pipelines and batch jobs.
Troubleshoot ETL failures and data processing issues.
Work with Apache Kafka for data ingestion and ETL processing.
Implement data transformation and processing solutions using Python, SQL, and Spark.
Support data governance, security controls, encryption, access management, and compliance requirements.
Participate in code reviews, security audits, architecture discussions, and technical design sessions.
Identify opportunities to improve data pipeline reliability, scalability, performance, and operational efficiency.

Requirements

5+ years of experience in cloud-based data engineering and big data processing.
Strong hands-on experience with AWS Glue for ETL development and job orchestration.
Strong expertise with Terraform and cloud Infrastructure-as-Code automation.
Strong understanding of Data Warehouse and Data Lake architectures.
Hands-on experience with AWS S3 and Snowflake.
Strong Snowflake experience, including:
Data storage and modeling
SQL development
Query optimization
Performance tuning
Large-scale data processing
Experience developing and managing API-based data ingestion from front-end applications and external systems.
Hands-on experience creating and debugging SSIS packages.
Strong knowledge of REST APIs, JSON, OAuth, API keys, and authentication mechanisms.
Strong programming and data transformation skills using Python, SQL, and Spark.
Hands-on experience with Apache Kafka, particularly in data ingestion and ETL processes.
Understanding of cloud data security best practices, including encryption, access controls, and secure data handling.
Strong troubleshooting and problem-solving skills., * Experience working with cybersecurity or security analytics data.
Experience with enterprise-scale cloud data platforms.
Knowledge of data governance and compliance frameworks.
Experience with CI/CD and automated deployment pipelines.
Experience optimizing high-volume streaming and batch data pipelines.