OpenSearch Engineer / AWS & Data Pipelines
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+11 more
Job description
We are building a high-scale payment orchestration platform that processes millions of financial transactions every month across multiple providers, regions, and products., OpenSearch
- Own and operate AWS OpenSearch Service in production.
- Design indices, mappings, analyzers, templates, aliases, shards, and retention policies.
- Optimize search queries, aggregations, filtering, pagination, and bulk indexing.
- Investigate slow queries, indexing bottlenecks, hot shards, memory pressure, and rejected requests.
- Plan safe upgrades, migrations, reindexing, and capacity scaling.
PostgreSQL-to-OpenSearch pipelines
- Build and maintain reliable pipelines that synchronize PostgreSQL data with OpenSearch.
- Support inserts, updates, deletes, backfills, and full reindexing.
- Use AWS-native services such as AWS DMS, OpenSearch Ingestion, Lambda, SQS, S3, ECS, or EKS where appropriate.
- Implement retries, idempotency, deduplication, replay, dead-letter handling, and failure recovery.
- Monitor ingestion lag, throughput, failed records, and data freshness.
- Build reconciliation processes to detect and repair missing, stale, or inconsistent documents.
- Ensure ingestion does not negatively affect the transactional PostgreSQL workload.
Reliability and operations
- Monitor cluster health, search latency, indexing latency, storage, CPU, JVM pressure, and shard distribution.
- Build CloudWatch dashboards, alerts, and operational runbooks.
- Design snapshot, restore, backup, and disaster-recovery procedures.
- Participate in production incident response and root-cause analysis.
- Implement secure access using IAM, role-based access control, encryption, and least-privilege principles.
Collaboration
- Work closely with backend, database, infrastructure, and product teams.
- Help design efficient OpenSearch document models for payments, customers, subscriptions, and transactions.
- Define standards for searchable fields, schema changes, and index versioning.
- Document architecture, data flows, and recovery procedures.
Requirements
- Strong production experience with OpenSearch or Elasticsearch.
- Hands-on experience with AWS OpenSearch Service.
- Experience building data pipelines between PostgreSQL and OpenSearch.
- Strong understanding of:
- Index and shard design
- Mappings and analyzers
- OpenSearch Query DSL
- Aggregations, sorting, filtering, and pagination
- Bulk indexing and cluster performance
- Data consistency, retries, idempotency, and replay
- Experience with large datasets and high indexing or query volumes.
- Experience performing backfills and reindexing safely.
- Strong troubleshooting and production-operations skills., * Experience with AWS DMS or OpenSearch Ingestion.
- Experience with Lambda, SQS, S3, ECS, EKS, or Terraform.
- Experience building custom ingestion services in Go, Java, or Python.
- Experience in payments, billing, fintech, or subscription systems.
- Experience with multi-tenant or high-volume search platforms.
- Familiarity with ClickHouse, Tinybird, or similar analytical systems.
- AWS certifications.
Benefits & conditions
Pulled from the full job description
- Unlimited paid time off, Discover the perks of being part of our vibrant team! We offer:
- Remote Work Environment: Embrace the freedom to work from anywhere, anytime, promoting a healthy work-life balance.
- Unlimited PTO: Enjoy unlimited paid time off to recharge and prioritize your well-being, without counting days.
- Paid National Holidays: Celebrate and relax on national holidays with paid time off to unwind and recharge.
- Company-provided MacBook: Experience seamless productivity with top-notch Apple MacBooks provided to all employees who need them.
- Flexible Independent Contractor Agreement: Unlock the benefits of flexibility, autonomy, and entrepreneurial opportunities. Benefit from tax advantages, networking opportunities, reduced employment obligations, and the freedom to work from anywhere. Read more about it here: https://docs.google.com/document/d/1nkrN76JlZkbKj9WSOhlT1_mni_CZeDkHdwfIjPXVwvk/preview?tab=t.0#heading=h.ndsdl4wapxtt
Be part of our fast-growing team and seize this excellent opportunity for personal and professional growth!
About the company
Ruby Labs is a leading tech company that creates and operates innovative consumer products. We offer a diverse range of opportunities across the health, education, and entertainment industries. Our innovative teams are driving the future of consumer-led products, and we’re always looking for passionate individuals to join us. Learn more about our story at: https://rubylabs.com/about-us, At Ruby Labs, we move fast, aim high, and expect the same from our team. We’re not here to play small-we’re here to build, grow, and win. That means we look for people who are ambitious, driven, and ready to give their best every single day.
This is a place for individuals who thrive under pressure, embrace challenges, and see opportunity in every obstacle. If you’re hungry to achieve, motivated by impact, and want to grow at the speed of your own ambition, Ruby Labs offers the platform to make it happen.
Here, effort is matched with reward. We recognize those who go all in and deliver results, and we create space for people who want more-more responsibility, more growth, and more success.
LI-Remote
You must create an Indeed account before continuing to the company website to apply
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on fr.indeed.comGood distractions
Talks and stories from around this role — technically off-topic, practically not.
Moments
Explore playlistsVideos
See allRelated articles
See all
Data Engineer Salary UK
Top Big Data Technologies That You Need to Know
Find a Developer Job: 12 Best Job Sites For Developers
Software Engineer Salary London