Senior Data Engineer

About the role:

We are looking for a talented Data Engineer to be part of our Data team. You will work alongside a growing multidisciplinary team of talented research scientists and machine learning engineers to improve and scale the efficiency within our models. You will contribute in building the data infrastructure that manages our data and powers the training of all Stability AI models. The position is based in Japan and is a hybrid work model with at least one day (recommended) in the office in Tokyo per week.

Responsibilities:

• Clean, normalize, and preprocess data in a scalable, parallelizable way to prepare it for ingestion into our machine learning model training pipelines while ensuring data quality
• Design, implement, and maintain scalable data infrastructure for generative AI
• Develop tools to search and serve the data at scale
• Collaborate with cross-functional teams to understand and meet their data requirements
• Develop and manage data processing pipelines to support machine learning teams
• Maintain and improve data quality and integrity across various databases and data stores
• Manage and organize large-scale unstructured data, including image, text, audio, video and 3D

Qualifications:

• Proven background within large-scale distributed workloads
• Experience with large-scale data loading for machine learning training runs
• Experience with cloud storage and file systems. AWS (S3) is strongly preferred, but open to other cloud platforms
• Experience with Python
• Experience with large-scale data processing and software development for unstructured data.
• Expertise in database, data lake, and data warehouse technologies (Redshift, BigQuery, Snowflake).
• Experience working with machine learning projects and ideally some deep learning / computer vision knowledge
• Good teamwork and communication skills based on experience working with a distributed international team with timezone and cultural differences
• Excellent communication skills to effectively collaborate with users, solve issues, and provide guidance
• Attention to detail and the ability to document processes and solutions effectively

Equal Employment Opportunity:

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.