Data Engineer

Information Technology Division's (ITD) vision is to become a dynamic and vibrant technology delivery organization of the New York City government that leads business transformations and technology innovations to enable citizen-centric smart urban planning, while promoting a culture of collaboration, talent development, and equity. ITD is responsible for developing technology solutions to support agency needs and citywide requests, including City Charter mandates, grant funding requirements, and citywide geospatial services, and maintaining systems infrastructure and endpoint assets, to optimize operational efficiency, promote innovation, and collaborate with other city agencies to enhance service accessibility to citizens. The division provides necessary computing infrastructure and productivity-enhancing technology tools in a hybrid mode (on premise and in the cloud), to support DCP's interdisciplinary staff across five boroughs, including planners, urban designers, project managers, and executive management. The division is comprised of experienced and talented software engineers, geographic specialists, service engineers, and technology leaders who specialize in managing system infrastructure, cyber security, geospatial data engineering, application services, and portfolio management.

Within ITD, the Geographic Data and Engineering (GDE) unit consists of application engineering, data engineering, geographic information systems, and geographic research teams, responsible for creating and implementing the agency's data strategy and data governance policy, updating and maintaining core citywide datasets in support of 911 dispatch and critical city operations, improving the creation, use, and availability of geospatial datasets within the agency, and providing data products and map-based visualization systems for the agency, other agencies, and the public., The Data Engineering team is seeking a Data Engineer. The team's mission is to create and publish high quality public datasets, build transparent and automated data pipelines using open-source technologies, develop a data ecosystem by offering comprehensive documentation and analytics resources, and bring people together across teams and agencies to share data and learn from each other. The team is continually evolving, and this new hire will have the ability and support to improve existing infrastructure and processes and drive forward new projects and initiatives. The ideal candidate will be excited about advancing the adoption of new technologies in DCP and City government and passionate about data production, quality, and accessibility.

Under limited supervision, the Data Engineer's primary areas of responsibility include, but are not limited to: o Designing, deploying, and scaling of current infrastructure to ensure that Data Engineering maintains and improves existing processes, and expands its abilities and offerings. o Understanding the technologies used by the team and recommending new technologies. Ideating, designing, and implementing improvements to products and processes to ensure that the team is using the best tools for the job. o Building Data Engineering data products, which involves: o operating data ingestion pipelines and managing data storage, o running scripts that transform input data by standardizing, geocoding, merging, aggregating, and performing basic spatial manipulations, and o reviewing outputs by writing custom scripts and/or using established QAQC tools to identify errors, inconsistencies, and edge cases in datasets. o Designing new data products from concept to completion, working on specifications with product owners, creating technical designs with team members, helping to plan sprints, contributing to documentation, and delivering a finished product. o Collaborating with other data engineers in sprint planning, design, code review, and pair programming. o Learning and wrangling complex legacy data systems and bringing the infrastructure into a modern development stack. o Sharing your accomplishments and lessons learned with NYC Planning, other city agencies, and the world via talks, presentations, blogs, or other mediums.

Do you have experience in Version control systems?, Do you have a Bachelor's degree?, 1. A baccalaureate degree in computer science, engineering or a related field from an accredited college and four years of satisfactory full-time experience related to IT automation engineering, monitoring engineering, management of infrastructure; or

2. Eight years of satisfactory full-time experience related to IT automation engineering, monitoring engineering, management of infrastructure;

3. Education and/or experience which is equivalent to "1" or "2" above.

Preferred Skills

• Four or more years of experience in data engineering, analytics, or related technical field - Two or more years of experience in data engineering, analytics, or related technical field. - Experience with SQL (PostgreSQL/DuckDB/MySQL/SQL-Server). - Experience with Python or similar scripting language. - Working knowledge of: -Data Warehousing/Data Lakes -ETL/ELT frameworks (dbt or other) -Working in cloud environments (AWS/GCP/Azure) and infrastructure as code principles -Containerization technologies -CI/CD -Git or other version control systems and collaborative development practices - Working knowledge of spatial data tools like PostGIS, GeoPandas, etc. - Understanding of data governance and metadata management principles. - Strong analytical and problem-solving mindset with attention to detail. - Strong communication skills and ability to work in a collaborative development environment. Nice to have: - Experience with orchestration tools (Airflow/Prefect/Dagster). - Experience with big data platforms/technologies (BigQuery/Snowflake/Redshift/Apache Spark).

Public Service Loan Forgiveness

4.04.0 out of 5 stars Manhattan, NY $100,000 - $112,000 a year - Full-time

The Department of City Planning (DCP) plans for the future of New York City, working to create thriving and dynamic neighborhoods with access to housing and jobs, resilient infrastructure, and a vibrant public realm. The Department engages communities to develop inclusive plans, expands housing access and economic opportunity, and plans for long-term sustainability. DCP supports the City Planning Commission in its annual review of hundreds of land use applications. The Department also advises on citywide strategic and capital planning, analyzes data, conducts policy analysis, and creates open data and civic engagement tools. The Department of City Planning offers staff meaningful opportunities to help shape the future of New York City and address some of its most pressing challenges. The Department values public service, collaboration, and creativity, and supports professional development. Learn more about DCP and view current job openings at www.nyc.gov/planning.