Data Engineer
Pegasus Knowledge Solutions
New York, United States of America
yesterday
Role details
Contract type
Temporary to permanent Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
SeniorJob location
New York, United States of America
Tech stack
Artificial Intelligence
JIRA
Big Data
Cloud Database
Information Engineering
Hadoop
Python
Salesforce
Data Logging
Spark
Data Layers
PySpark
Apache Flink
Kafka
Presto
Job description
- Clean, process & structure huge volumes of unstructured support data (Salesforce, Sprinklr, Bliss, JIRA).
- Build AI memory systems using vector databases (Pinecone, Milvus, Weaviate, pgvector).
- Create a central metrics hub for analytics & AI insights.
- Own high-reliability pipelines with monitoring & zero-downtime standards.
- Work closely with product/ops and push for better data logging at the source.
Tech Stack
- Python, PySpark
- Spark, Kafka, Flink, Hadoop, Hudi, Presto, Pinot
- Vector DBs (Pinecone/MilvWeaviate/pgvector)
- Cloud data warehouses
- AI tools (Claude, Codex, etc.) for automation & parsing
Requirements
- 5+ years in data engineering with big data systems
- Strong Python + PySpark
- Experience building AI-ready data layers
- Proactive, independent, business-focused
- Comfortable working in ambiguity and driving clarity
- Honest communicator who flags risks early