Data Engineer, AI Platform

Team Introduction
The Intelligent Creation - AI Platform team is a team focusing on building advanced end-to-end AI production pipelines, including deep learning model training, optimization, deployment and applications. We provide AI capabilities to empower content creation and consumption on TikTok and serve billions of users.

Responsibilities
- Own the management of massive GenAI datasets (PB) scale, including storage for video/image data, data processing, and data validation.
- Build and evolve GenAI data-pipeline infrastructure with a focus on extreme processing speed and end-to-end throughput.
- Partner closely with ML researchers/engineers to accelerate training-data acquisition, improve data quality, support evaluation of model outputs, and enable a closed-loop data lifecycle.
- Lower the barrier to data acquisition and maximize the utility and value of data across use cases.

Minimum Qualifications
- 3+ years of experience building data services; strong proficiency in Python.
- Experience with high-concurrency and asynchronous programming is a plus.
- Hands-on with Hive, MySQL, MongoDB, and Elasticsearch; solid understanding of internals; capable of data abstraction and data modeling.
- Practical experience with large-scale data processing frameworks such as Hadoop, Spark, Flink, and Ray.
- Excellent communication and collaboration skills; detail-oriented; strong problem-solving and analytical abilities.

Preferred Qualifications
- 1–2 years of experience with the Ray framework, including proficient orchestration of GPU/CPU resources; deep understanding of Ray’s architecture and usage patterns; strong background in high-concurrency and async processing to boost overall throughput.
- Experience sourcing and curating training datasets for GenAI.