Sr. Data Engineer

Capstone IT is helping our client grow! We are looking for Sr. Data Engineers to design, develop, test, and maintain the scripts and jobs that are required to extract, transform, clean, and move data and metadata so that it can be loaded into a data warehouse, data mart, or operational data store. Python and Pyspark experience are required! They will mentor less experienced developers and serves as a project technical data lead for data projects. Trains, coaches and mentors team members. Duties and responsibilities include:

DATA ANALYSIS

• Maps source system data to data warehouse models or other file transformations (source to target mappings).

• Works with business systems analysts and business team members to understand source systems data and data quality issues/performs data quality analysis.

• Defines and captures metadata and rules associated with ETL processes.

DATA DESIGN

• Creates basic to larger physical designs of ETL processes. May be responsible for the higher-level logical design. For smaller projects, the physical design may be the only design. For larger projects, this may be based on a higher-level logical design.

• Employs cost effective techniques in design of physical ETL processes to meet immediate and future business needs.

• Conducts ETL process design / sizing estimates for data warehousing projects or other file transformations

• Designs data solutions to satisfy business information needs and receive design acceptance.

ETL DEVELOPMENT AND TESTING

• Develops complex ETL processes.

• Creates ETL processes that supports business functionality, performs well, and is easily supported.

• Creates scripts when necessary.

• Follows ETL standards, goals, and objectives.

• Works with teammates to define and resolve complex data warehouse or file transformation project issues.

• Adapts extraction, transformation, and load (ETL) processes to accommodate changes in source systems and new business user requirements

• Completes work according to the iteration plan or release plan.

• Participates in ETL Code Reviews, receives acceptance of code by peers at review. Responsible for code reviews for other developers.

• Tests (unit level, integration level, system level, and user acceptance level) complex ETL processes.

• Works with testing team to assist them with their testing processes.

• Assists with testing upgrades; identifies errors or inconsistencies with vendors’ software applied to Company’s data environment.

ETL CHANGE/CONFIGURATION MANAGEMENT AND PRODUCTION SUPPORT

• Follows Change Management/Configuration Management processes for the deployment of code into test and production environments.

• Leads teammates to define and resolve complex data warehouse or file transformation production issues.

• Troubleshoots data problems. Identifies, coordinates and helps resolve the root cause of day-to-day data production issues.

• Supports client areas in their analytical functions; corrects errors/resolves problems associated with data warehouse processes and other file transformations.

• Provides rotational on-call support, responding quickly to production issues.

• Minimum of 6 years IT experience to include at least 5 years’ experience required in data warehousing, relational database management systems, multi-dimensional database management systems. Strong knowledge of Business Intelligence architecture and tools such as Hyperion, Business Objects, MicroStrategy, Cognos, SAS, or WebFocus is desirable.
• Strong knowledge of at least one ETL tool such as IBM InfoSphere DataStage (or comparable ETL tool) is required.
• Strong knowledge of Data Warehousing data population techniques for target structures such as Star Schemas, Snowflake Schemas, highly normalized data models, and file structures.
• Strong knowledge of source to target mappings.

• Strong relational database knowledge; possesses a broad understanding of the current and prospective data architecture, and database performance tuning.

• Experience using Python/Pyspark to build data pipelines required.

• Experience with programming languages such as UNIX scripting, Java, or COBOL is essential. Strong ETL staging environment development skills and strong SQL programming skills in a DB2 environment.
• Strong verbal and written communication skills. Has ability to establish and maintain effective working relationships with external and internal personnel. Must be customer focused and excel in coordination of problem resolution.
• Flexibility to work in a changing, fast-paced environment and work with a sense of urgency and accountability.
• Experience with Agile methodologies is desirable.
• Rotational On-Call support is expected.
• Ability to effectively provide work direction to team members and delegate tasks; ability to train, mentor and provide constructive feedback to team members to meet or exceed performance in their job responsibilities.

• Demonstrated project management skills are desirable.

• Highly desirable: Insurance Information Warehouse (IIW) and DataStage.