Software Development Engineer, AGI - Web information retrieval

Amazon AGI is pioneering the next frontier of artificial general intelligence, leveraging over 25 years of AI and machine learning expertise along with Amazon's world-class infrastructure and commitment to responsible AI. Join our diverse team of scientists, engineers, and researchers to push the boundaries of what's possible - from reinventing commerce and enterprise productivity to advancing universal agents and robotics.

We are looking for a Software Development Engineer within AGI to develop a multi-modal, multi-lingual web-scale information retrieval system to make the world’s information accessible to AI models and customers everywhere. We are pushing the boundaries of what's possible by combining the power of large language models with retrieval systems, enabling capabilities in question-answering, knowledge synthesis, and information grounding.

You will have an opportunity to directly impact the customer experience, design, architecture, and implementation of products that will be used every day by people you know. We're looking for someone passionate about innovating on behalf of customers, who demonstrates strong product ownership and is willing to think in new ways to solve difficult problems.

You will have track record of success in delivering new products, solving problems, and learning new technologies quickly. A commitment to teamwork, proactive approach to solving problems, and strong verbal and written communication skills are essential. Creating reliable, scalable, and high-performance products requires technical expertise, understanding of computer science fundamentals, and practical experience building efficient large-scale distributed systems. This person is comfortable delivering quality solutions in a fast-growing environment where priorities may change rapidly.

Key job responsibilities
* Build high-throughput, cost-effective data pipelines to support feature extraction and indexing for our web-scale Information Retrieval (IR) system
* Develop and optimize the core algorithms in Rust and ranking models that power the search engine's ability to retrieve and rank relevant results for user queries.
* Design and implement efficient data structures and indexing techniques to store and retrieve massive amounts of web data and content using Rust programming language.
* Optimize the performance, scalability, and reliability of the search engine's core components, including query parsing, retrieval, ranking, and result rendering.
* Collaborate with machine learning teams to integrate and deploy advanced machine learning models for query understanding, ranking, and personalization.
* Develop and maintain the control plane systems that manage and orchestrate the IR system infrastructure, including the distributed compute clusters, storage systems, and networking components.
* Design and implement real-time updates and freshness mechanisms to ensure the I engine reflects the latest web content and user behavior.
* Develop efficient, state-of-the-art streaming algorithms for processing large datasets (e.g. deduplication, topic clustering)
* Participate in the design and implementation of real-time updates and freshness mechanisms to ensure the search engine reflects the latest web content and user behavior.