Distributed Machine Learning

1. Demos & tutorials

• Distributed Training a Linear Regression Model for Housing Prediction Using Spark

• Distributed Training a Linear Regression Model with Ray for Housing Prediction

• Decentralized Training a Simple Neural Network for Housing Price Prediction with Horovod

2. Reference Systems

• Amazon

• Microsoft

• SAP

• ZhongAn Insurance (ML Group)

• ZhongAn Insurance (AML Group)

3. Ecosystems

• OpenMPI

• ClipperAI

• Apache Flink

• Google Vizier

• Horovod

• Apache Kafka (distributed event streaming platform)

• Spring Boot: Microservice

• Django Python Web framework

• Berkeley Ray

• Distributed PyTorch & Tensorflow

4. Paper Reading and Presentation

Papers:

• More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

• LightLDA: Big Topic Models on Modest Compute Clusters

• SaberLDA: Sparsity-Aware Learning of Topic Models on GPUs

• Deep learning with COTS HPC systems

• STRADS: A Distributed Framework for Scheduled Model Parallel Machine Learning

• Device Placement Optimization with Reinforcement Learning

• Large Scale Distributed Deep Networks

• A Survey on Distributed Machine Learning

Presentations:

• Large Scale Distributed Deep Networks

• Deep learning with COTS HPC system

• LightLDA: Big Topic Model on Modest Compute Clusters

• SaberLDA: Sparsity-Aware Learning of Topic Models on GPUs

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

• A Survey on Distributed Machine Learning: Part 1 (ML & optimization algorithms)

• A Survey on Distributed Machine Learning: Part 2 (Distributed ML Frameworks)

• STRADS: a Distributed Framework For Scheduled Model Parallel Machine Learning

• Device Placement Optimization with Reinforcement learning