AWS Data Engineer
IT America
Richmond, VA, United States
3 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Job source
Tech stack
Agile Methodology
Amazon Web Services
Big Data
Cloud Computing
Information Engineering
Software Debugging
Distributed Computing Environment
Apache Hadoop
Python (Programming Language)
SQL Databases
Data Logging
Data Processing
+8 more
Data Ingestion
Apache Spark
Pyspark
Information Technology
AWS Glue
AWS Data Analytics
Data Pipelines
Serverless Computing
Job description
- Develop and optimize scalable data pipelines using Python and PySpark, with a primary focus on leveraging AWS services like Glue, EMR, and Step Functions.
- Design, implement, and maintain data processing systems utilizing the capabilities of AWS Glue for data ingestion, transformation, and orchestration.
- Utilize EMR clusters for distributed data processing and analytics, ensuring efficient resource utilization and performance.
- Implement serverless workflows using AWS Step Functions to orchestrate complex data processing tasks and workflows.
- Collaborate closely with cross-functional teams to understand data requirements and translate them into scalable and efficient solutions.
- Ensure data quality, governance, and security standards are adhered to throughout the data pipeline.
- Troubleshoot and debug data-related issues in production environments, leveraging AWS monitoring and logging tools.
- Stay abreast of the latest advancements in big data technologies, AWS services, and best practices in data engineering.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or related field.
- Demonstrated expertise in developing data pipelines with Python and PySpark, with a strong emphasis on utilizing AWS Glue, EMR, and Step Functions.
- Proficiency in SQL for data manipulation and querying.
- Experience with distributed computing frameworks like Hadoop and Spark.
- Strong understanding of cloud computing principles and experience with AWS services.
- Excellent problem-solving and communication skills, with the ability to work collaboratively in a team environment.
- Familiarity with Agile development methodologies is advantageous.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…