PySpark Developer

Inizio Partners Corp
Hartford, CT, United States
3 days ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours

Tech stack

Clean Code Principles Airflow Amazon Web Services Amazon S3 Data Analysis Microsoft Azure Big Data BigQuery Cloud Computing Software Quality Computer Programming Databases
+36 more
Continuous Integration Information Engineering Data Files Data Governance Extract Transform Load (ETL) Data Mining Data Warehousing Relational Databases DevOps Distributed Computing Environment Distributed Systems Data Flow Control Apache Hadoop Python (Programming Language) PostgreSQL MySQL NoSQL NumPy Oracle (Applications) Cloud Services Standard Sql Software Engineering Unstructured Data Workflow Management Systems Data Processing Scripting Google Cloud Apache Spark Pandas Data Lakes Pyspark Information Technology Data Management Machine Learning Operations Data Pipelines Databricks

Job description

We are seeking a highly skilled and experienced Python and PySpark Developer to join our team. The ideal candidate will be responsible for designing, developing, and optimizing big data pipelines and solutions using Python, PySpark, and distributed computing frameworks. This role involves working closely with data engineers, data scientists, and business stakeholders to process, analyze, and derive insights from large-scale datasets., Data Engineering & Development:

  • Design and implement scalable data pipelines using PySpark and other big data frameworks.
  • Develop reusable and efficient code for data extraction, transformation, and loading (ETL).
  • Optimize data workflows for performance and cost efficiency.

Data Analysis & Processing:

  • Process and analyze structured and unstructured datasets.
  • Build and maintain data lakes, data warehouses, and other storage solutions.

Collaboration & Problem Solving:

  • Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
  • Troubleshoot and resolve performance bottlenecks in big data pipelines.

Code Quality & Documentation:

  • Write clean, maintainable, and well-documented code.
  • Ensure compliance with data governance and security policies.

Requirements

Software Engineering, Distributed Computing, Data Sets, Python Programming/Scripting Language, Analysis Skills, Data Management, Big Data, Data Science, Business Processes, Database Extract Transform and Load (ETL), Programming Skills:

  • Proficient in Python with experience in data processing libraries like Pandas and NumPy.
  • Strong experience with PySpark and Apache Spark.

Big Data & Cloud:

  • Hands-on experience with big data platforms such as Hadoop, Databricks, or similar.
  • Familiarity with cloud services like AWS (EMR, S3), Azure (Data Lake, Synapse), or Google Cloud (BigQuery, Dataflow).

Database Expertise:

  • Strong knowledge of SQL and NoSQL databases.
  • Experience working with relational databases like PostgreSQL, MySQL, or Oracle.

Data Workflow Tools:

  • Experience with workflow orchestration tools like Apache Airflow or similar.

Problem Solving & Communication:

  • Ability to solve complex data engineering problems efficiently.
  • Strong communication skills to work effectively in a collaborative environment., * Knowledge of data Lakehouse architectures and frameworks.
  • Familiarity with machine learning pipelines and integration.
  • Experience in CI/CD tools and DevOps practices for data workflows.
  • Certification in Spark, Python, or cloud platforms is a plus.

Education:

  • Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…