My MLOps Journey so far
MLOps, or Machine Learning Operations, is the practice of adopting to Machine Learning (ML) what DevOps has done for software development (see e.g. https://en.wikipedia.org/wiki/MLOps). An implemented MLOps system offers a plethora of very nice tools and artefacts. Examples of MLOps tools are:
- Easy retraining of ML models.
- Monitoring of ML model performance to guard against performance drop due to e.g. data drift.
Examples of MLOps artefacts are:
- Versioned ML models.
- Defined evaluation figures and metrics from each training run.
- Versioned code and environment associated with each training run.
About two years ago, I started working on a project that would combine sensor and image data to make predictions of a certain quantity at different lead times. At the outset, we did not have data available so the initial exercise was to locate the appropriate external hard disks containing the data. At this point, the journey necessary to move an operational ML model felt like having to hand walk across this:
After many struggles and eureka moments, we are almost at the end of the journey and I want to share my solutions to encountered challenges, experience, and outstanding challenges encountered on my journey to an MLOps system using the Azure MLStudio, Azure DevOps, and other Azure services. I have developed on a Windows 10 computer and used the Azure Machine Learning Python SDK.
I have organized my MLOps exerience in the following topics:
- Near real time ingestion of tabular data.
- Near real time ingestion and processing of image data.
- Use deep learning model as preprocessing step.
- Access to data from ML model training.
- Leverage private packages when training and deploying models.
- Allow consumer of deployed ML model to retrieve predictions without requiring consumer to have access to input data.
- Deploy new ML model only if better than currently deployed model.
- How to use secrets without relying on .config files.
I will add links to the above items as I find time to write up my learnings. I hope this can help others avoid the situation of finding answerless questions exactly matching your situation, very well depicted by XKCD creator Randall Munroe:
