Sr. Data Engineer (Remote)

• Manage and enhance the enterprise data lake and data warehouse, ensuring the secure and efficient flow of quality data.
• Design, implement, and maintain scalable data pipelines on AWS using services like S3, DMS, Glue, Lambda, Step Functions/MWAA, and Redshift.
• Develop robust ETL/ELT workflows for data ingestion, cleansing, transformation, and loading from various sources using Python and PySpark.
• Design mechanisms for incremental changes, ensuring restartability, idempotency, duplicate handling, and recovery.
• Implement automated controls for data accuracy, reconciliation, schema changes, and lineage.
• Optimize workloads on Glue/Spark, Athena, Redshift, and S3 through partitioning, columnar formats, query tuning, and efficient storage/compute design.
• Create near real-time/event-driven pipelines using Kinesis/Kafka, ensuring ordering, retry, idempotency, and failure recovery.
• Implement AWS data security measures, access controls, data classification, and governance protocols.
• Monitor pipelines with CloudWatch, troubleshoot failures, and resolve production data incidents.
• Enforce version control, code review, automated testing, and CI/CD practices.
• Collaborate with product owners, architects, and business stakeholders to translate requirements into scalable data solutions.
• Document pipelines and operational procedures.

Education/Experience Required: Bachelor’s degree in a computer-related field from an accredited college or university and five (5) or more years of experience in data engineering, building scalable and distributed ETL data pipelines in enterprise environments.Job Description & Responsibilities, * Experience building and operating scalable AWS-based data platforms and pipelines using services like Lambda, Glue, Athena, S3, Redshift, DMS, MWAA (Airflow), and Step Functions.

• Advanced proficiency in Python, SQL, and PySpark with hands-on experience in developing reusable ETL/ELT frameworks, data warehouses, data marts, and integrations.
• Experience implementing data quality, governance, and optimization best practices, including automated validation frameworks and cost optimization across AWS data services.
• Strong communication skills, with the ability to translate complex data concepts for business stakeholders. Experience in regulated environments like healthcare or life sciences is preferred.
• Experience with metadata management, data lineage, data observability, master data management, or enterprise data catalog solutions.
• Knowledge of data modeling, analytical data models, schema design, schema evolution, and data structures optimized for reporting and analytics.
• Familiarity with data lake and data warehouse architecture, including data partitioning and columnar storage formats like Parquet.
• Relevant AWS certification, such as AWS Certified Data Engineer - Associate, or equivalent cloud or data engineering certification.

Job DescriptionPaladin Consulting is currently hiring a Sr. Data Engineer to join our team working remotely.

We work with companies that offer environments for our employees to contribute, learn, and advance their career. We treat you like you are part of the family.