Latest Innovations in TensorFlow Serving

Posted by Posted by Chris Olston, Research Scientist, and Noah Fiedel, Software Engineer, TensorFlow ServingSince initially open-sourcing TensorFlow Serving in February 2016, we’ve made some major enhancements. Let’s take a look back at where we started, review our progress, and share where we are headed next.Before TensorFlow Serving, users of TensorFlow inside Google had to create their own serving system from scratch. Although serving might appear easy at first, one-off serving solutions quickly grow in complexity. Machine Learning (ML) serving systems need to support model versioning (for model updates with a rollback option) and multiple models (for experimentation via A/B testing), while ensuring that concurrent models achieve high throughput on hardware accelerators (GPUs and TPUs) with low latency. So we set out to create a single, general TensorFlow Serving software stack.We decided to make it open-sourceable from the get-go, and…
Original Post: Latest Innovations in TensorFlow Serving