AWS Data Engineer

Location: Remote US 

About the role: 

We are looking for a talented AWS Data Platform Engineer with a focus on scaling efficient distribution workloads. You will work alongside a growing multidisciplinary team of talented Research Scientists and Machine Learning Engineers to improve and scale the efficiency within our computing capacity. Stability AI operates a very large HPC cluster for training foundational AI models across several modalities. 

In this role, you will contribute to groundbreaking projects that redefine visual storytelling through advanced generative modeling techniques, optimizing and managing large-scale distributed workloads to drive project efficiency and success.

Responsibilities:

• Clean, normalize, and preprocess data in a scalable, parallelizable way to prepare it for ingestion into our machine learning model training pipelines while ensuring of data quality
• Building and maintaining highly scalable distributed workloads 
• Build data pipelines to ingest and process data (e.g. images and text) for feeding into ML models
• AWS Resource Management
• Keep up-to-date with papers / methods regarding how to improve data quality and/or curate data for Image, Video, LLMs etc.

Qualifications:

• Proven background within large scale distributed workloads 
• Experience with large scale data loading for machine learning training runs
• Knowledge of AWS cloud computing platforms, and experience with managing associated resources. 
• Experience with Python + Pytorch, Deep learning, Computer Vision
• Experience with multiprocessing and multithreading python workloads
• Experience with parallel dataframe manipulation using Pyspark / Ray
• Proficiency in HPC cluster management tools and technologies
• Excellent communication skills to effectively collaborate with users, solve issues, and provide guidance
• Attention to detail and the ability to document processes and solutions effectively

Compensation

The salary range for this role is between $130,000 and $190,000. Individual pay within the range is based on factors like job-related skills and experience. Total compensation also includes stock options and benefits.

Equal Employment Opportunity:

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.