Data Engineer

Our Client is looking for an experienced, motivated, and self-driven candidate who can contribute towards various aspects of taking the Live Analytics offering to the next level and build new data products for our clients. This involves design, development, and production support of the systems as well as working with product teams and engineering teams in shaping the future of the application., * Write new ETLs to support both real time and new features data and reporting products.

• Work with product teams to gather and understand requirements, discuss approaches, design solutions based on the requirements and create design documents for implementation keeping in mind costs, capacity, shelf-life and choosing the right tools and technologies.
• Design and develop peripheral systems and processes - monitoring and alerting services, ETL processes, automated offline workflows, automated build and deployment pipelines, systems for gathering application performance metrics, ensuring efficiency in cost.
• Ensure best practices in system development ranging from test coverage, use of enterprise standards, adherence to target architecture, high-availability and scalability, low latency, and fault-tolerance, thus ensuring delivery of high-quality systems.
• Create detailed documentation on systems as well as processes ranging from comprehensive overview of systems, runbooks for developing, monitoring and troubleshooting system issues and recommendations on future improvements.
• Managing project delivery from a day-to-day standpoint - participating in daily agile stand-ups, escalating blockers to stakeholders, ensuring timely completion of tasks & overall project & spring goals. Communicating updates to stakeholders and working with them on timelines.

• Bachelor’’s degree in Computer Science or related technical field. (NOT REQUIRED TO HAVE A DEGREE)
• Experience building Data Lakes, Data Warehouse using big data and cloud stack.
• 5+ years of experience on software development using Agile, dev ops that include designing, developing, and implementing large-scale data engineering solutions.
• Expert Level Proficiency in Spark streaming and spark concepts.
• Expert level proficiency in Python Programming language and PySpark with at least 4 years of experience.
• Experience working with Kafka streams and creating ingestion ETL’’s.
• Expert level proficiency in ANSI SQL /Spark SQL with at least 5 year’’s experience.
• Experience working with diverse data sources and data formats ( json, parquet, csv, psv, delta)
• Experience working with cloud services such as AWS along with cloud-based data sources(S3)
• Experience building ETL with batch and real time stream processing.
• Experience using version control system such as git and CI/CD

Nice to know / good to have

• Experience with building workflows and ETL’’s in the databricks.
• Experience ingesting data produced by real time systems like Kafka
• Guiding and mentoring skills - guiding and mentoring junior members - helping them understand technologies, the project, best practices and in general setting them up for success.
• Experience building and deploying ML models using ML Flow.

YOU (BEHAVIOURAL SKILLS/COMPETENCIES)

• Strong verbal communication, ability to clearly communicate ideas, work status, advocate for (or against) a particular approach or design.
• Documentation skills - ability to make the path easier for new team members / new hires / interns by organizing knowledge in Confluence, wiki, or similar.
• Ability to manage self and own time, deliver work on time, evaluate and estimate work.
• Ability to work with various teams, including tech and product, incorporate their inputs into the work being done, and manage expectations on issues & enhancements.

Our benefits package includes: (EXCLUDE on perm placements)

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• …and much more!

About INSPYR Solutions Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients’’ business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.