Data Engineer

This role involves designing and developing data solutions that are strategic for the business, utilizing the latest cloud data technologies such as Azure Databricks, Python, PySpark, SparkSQL, Azure functions, Delta Lake, and Azure DevOps CI/CD. The Data Engineer will work in a fast-paced environment supporting the enterprise data needs., * Design, architect, and develop solutions leveraging cloud big data technology to ingest, process, and analyze large, disparate data sets to meet business requirements.

• Create data management and persistence solutions for application use cases via relational and non-relational databases, enhancing data processing capabilities.
• Develop proofs of concept to influence platform architecture, product management, and software engineering teams, and facilitate migration to modern technology platforms.
• Design data lake solutions for structured and unstructured data, providing technical guidance for migration and platform modernization.
• Contribute to CI/CD processes, adhere to development best practices, and foster discipline within the data engineering team.
• Build data pipelines, perform data cleansing, normalization, and transfer, and optimize ETL workflows using Azure Databricks and PySpark.
• Conduct testing phases, including unit, system integration, regression, and user acceptance testing.
• Translate business requirements into detailed technical solutions, and document and communicate data processes clearly.
• Implement data security measures, including encryption, access controls, and compliance with data regulations, ensuring data integrity and confidentiality.
• Troubleshoot data-related issues and assist in resolution processes.
• Build agent-driven workflows for data loading and transformation, leveraging MCP framework and agent orchestration.

• Minimum of 5 years of experience as a data developer using Python, PySpark, Spark SQL, and SQL Server, with a strong foundation in ETL concepts.
• Bachelor’s degree in Information Science, Computer Science, Mathematics, Statistics, or a related quantitative discipline.
• Experience with Azure Cloud Platform, Databricks, and Azure storage solutions.
• Effective written and verbal communication skills, including technical documentation.
• Strong analytical and organizational skills., * Experience working with unstructured, semi-structured data, and data lake environments; familiarity with data streaming and event-driven data pipelines.
• Hands-on experience with real-time or near real-time data processing.
• Knowledge of JSON, Parquet, and other file formats.
• Experience with NoSQL databases such as Hbase, MongoDB, Cosmos DB.
• Cloud experience on Azure or AWS platforms.
• Knowledge of Python with Spark, Spark Streaming, Azure SQL Server, Cosmos DB/MongoDB, Azure Event Hubs, Azure Data Lake Storage, and Azure Search.
• Reliable, self-motivated team player capable of managing multiple projects in a fast-paced environment.

Pay range is not a guarantee of compensation or salary, as final offer amount may vary based on factors including but not limited to experience and geographic location. Talascend also offers a variety of benefits including: health and disability insurance, 401(k), EAP, paid time off, and company-paid holidays. The specific programs and options available to an employee may vary depending on date of hire, plan requirements, schedule type, and client work site mandates.