Achieving production-level governance with machine-learning projects currently presents unique challenges. A new space of tools and practices is emerging under the name MLOps. The space is analogous to DevOps but tailored to the practices and workflows of machine learning.
Why MLOps is Needed
Machine learning models make predictions for new data based on the data they have been trained on. Managing this data in a way that can be safely used in live environments is challenging, and one of the key reasons why 80 per cent of data science projects never make it to production – an estimate from Gartner.
It is essential that the data is clean, correct, and safe to use without any privacy or bias issues. Real-world data can also continuously change, so inputs and predictions have to be monitored for any shifts that may be problematic for the model. These are complex challenges that are distinct from those found in traditional DevOps.
MLOps Not Just DevOps
DevOps practices are centred on the “build and release” process and continuous integration. Traditional development builds are packages of executable artifacts compiled from source code. Non-code supporting data in these builds tends to be limited to relatively small static config files. In essence, traditional DevOps is geared to building programs consisting of sets of explicitly defined rules that give specific outputs in response to specific inputs.
In contrast, machine-learning models make predictions by indirectly capturing patterns from data, not by formulating all the rules. A characteristic machine-learning problem involves making new predictions based on known data, such as predicting the price of a house using known house prices and details such as the number of bedrooms, square footage, and location. Machine-learning builds run a pipeline that extracts patterns from data and creates a weighted machine-learning model artifact. This makes these builds far more complex and the whole data science workflow more experimental. As a result, a key part of the MLOps challenge is supporting multi-step machine learning model builds that involve large data volumes and varying parameters.
To run projects safely in live environments, we need to be able to monitor for problem situations and see how to fix things when they go wrong. There are pretty standard DevOps practices for how to record code builds in order to go back to old versions. But MLOps does not yet have standardisation on how to record and go back to the data that was used to train a version of a model.
There are also special MLOps challenges to face in the live environment. There are largely agreed DevOps approaches for monitoring for error codes or an increase in latency. But it’s a different challenge to monitor for bad predictions. You may not have any direct way of knowing whether a prediction is good, and may have to instead monitor indirect signals such as customer behaviour (conversions, rate of customers leaving the site, any feedback submitted). It can also be hard to know in advance how well your training data represents your live data. For example, it might match well at a general level but there could be specific kinds of exceptions. This risk can be mitigated with careful monitoring and cautious management of the rollout of new versions.
The MLOps Tool Scene
The effort involved in solving MLOps challenges can be reduced by leveraging a platform and applying it to the particular case. Many organisations face a choice of whether to use an off-the-shelf machine-learning platform or try to put an in-house platform together themselves by assembling open-source components.
Some machine-learning platforms are part of a cloud provider’s offering, such as AWS SageMaker or AzureML. This may or may not appeal, depending on the cloud strategy of the organisation. Other platforms are not cloud-specific and instead offer self-install or a custom hosted solution (eg, Databricks MLflow).
Instead of choosing a platform, organisations can instead choose to assemble their own. This may be a preferred route when requirements are too niche to fit a current platform, such as needing integrations to other in-house systems or if data has to be stored in a particular location or format. Choosing to assemble an in-house platform requires learning to navigate the ML tool landscape. This landscape is complex with different tools specialising in different niches and in some cases there are competing tools approaching similar problems in different ways (see the Linux Foundation’s LF AI project for a visualization or categorised lists from the Institute for Ethical AI).
The Linux Foundation’s diagram of MLOps tools … Click for full detail
For organisations using Kubernetes, the kubeflow project presents an interesting option as it aims to curate a set of open-source tools and make them work well together on kubernetes. The project is led by Google, and top contributors (as listed by IBM) include IBM, Cisco, Caicloud, Amazon, and Microsoft, as well as ML tooling provider Seldon, Chinese tech giant NetEase, Japanese tech conglomerate NTT, and hardware giant Intel.
Governance
Challenges around reproducibility and monitoring of machine learning systems are governance problems. They need to be addressed in order to be confident that a production system can be maintained and that any challenges from auditors or customers can be answered. For many projects these are not the only challenges as customers might reasonably expect to be able to ask why a prediction concerning them was made. In some cases this may also be a legal requirement as the European Union’s General Data Protection Regulation states that a “data subject” has a right to “meaningful information about the logic involved” in any automated decision that relates to them.
Explainability is a data science problem in itself. Modelling techniques can be divided into “black-box” and “white-box”, depending on whether the method can naturally be inspected to provide insight into the reasons for particular predictions. With black-box models, such as proprietary neural networks, the options for interpreting results are more restricted and more difficult to use than the options for interpreting a white-box linear model. In highly regulated industries, it can be impossible for AI projects to move forward without supporting explainability. For example, medical diagnosis systems may need to be highly interpretable so that they can be investigated when things go wrong or so that the model can aid a human doctor. This can mean that projects are restricted to working with models that admit of acceptable interpretability. Making black-box models more interpretable is a fast-growth area, with new techniques rapidly becoming available.
The MLOps scene is evolving as machine-learning becomes more widely adopted, and we learn more about what counts as best practice for different use cases. Different organisations have different machine learning use cases and therefore differing needs. As the field evolves we’ll likely see greater standardisation, and even the more challenging use cases will become better supported. ®
Original post: https://www.theregister.co.uk/AMP/2020/03/07/devops_machine_learning_mlops/
I am the owner of JustCBD Store brand (justcbdstore.com) and I’m presently seeking to expand my wholesale side of business. It would be great if someone at targetdomain is able to provide some guidance ! I thought that the most suitable way to do this would be to talk to vape stores and cbd retail stores. I was really hoping if someone could recommend a qualified web site where I can get Vape Shop B2B Business Data List I am currently taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most suitable selection and would appreciate any assistance on this. Or would it be easier for me to scrape my own leads? Suggestions?
I’m the co-founder of JustCBD Store label (justcbdstore.com) and am seeking to expand my wholesale side of company. I really hope that anybody at targetdomain can help me ! I thought that the most suitable way to accomplish this would be to connect to vape shops and cbd stores. I was hoping if someone could suggest a reliable web site where I can buy CBD Shops Business Contact List I am currently considering creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not exactly sure which one would be the very best choice and would appreciate any guidance on this. Or would it be much simpler for me to scrape my own leads? Suggestions?
That is a very good tip especially to those fresh to the blogosphere. Simple but very precise info… Thank you for sharing this one. A must read post!
Great article. I will be experiencing many of these issues as well..
Excellent article. I absolutely appreciate this website. Continue the good work!
I couldn’t resist commenting. Well written!
Everyone loves it when folks come together and share opinions. Great website, keep it up!
I couldn’t refrain from commenting. Exceptionally well written!
Spot on with this write-up, I truly believe that this amazing site needs a great deal more attention. I’ll probably be returning to see more, thanks for the information!
There’s definately a lot to know about this subject. I really like all of the points you have made.
Great web site you have got here.. It’s hard to find quality writing like yours nowadays. I truly appreciate individuals like you! Take care!!
Saved as a favorite, I like your site!
I have to thank you for the efforts you have put in writing this website. I really hope to check out the same high-grade blog posts by you later on as well. In truth, your creative writing abilities has inspired me to get my own blog now 😉
An interesting discussion is worth comment. I do believe that you need to publish more on this topic, it may not be a taboo matter but usually people don’t discuss such subjects. To the next! Many thanks!!
Everything is very open with a clear clarification of the challenges. It was definitely informative. Your website is useful. Many thanks for sharing!
An outstanding share! I’ve just forwarded this onto a co-worker who had been conducting a little homework on this. And he in fact ordered me breakfast because I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to talk about this matter here on your web site.
Very good article! We will be linking to this particularly great content on our site. Keep up the great writing.
Good article! We will be linking to this particularly great post on our website. Keep up the good writing.
Excellent article. I definitely appreciate this website. Stick with it!
Hi there! This blog post could not be written much better! Going through this post reminds me of my previous roommate! He continually kept talking about this. I most certainly will forward this article to him. Pretty sure he’s going to have a great read. Thank you for sharing!
I’m impressed, I have to admit. Seldom do I come across a blog that’s both equally educative and engaging, and without a doubt, you have hit the nail on the head. The problem is something which too few people are speaking intelligently about. I’m very happy that I stumbled across this in my hunt for something regarding this.
I wanted to thank you for this good read!! I definitely loved every bit of it. I have got you saved as a favorite to check out new things you post…
When I initially commented I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on every time a comment is added I recieve four emails with the same comment. Is there a means you can remove me from that service? Cheers!
A fascinating discussion is worth comment. I do think that you should write more about this issue, it may not be a taboo matter but generally folks don’t discuss such topics. To the next! Best wishes!!
You’re so interesting! I do not believe I have read through anything like that before. So nice to find someone with a few genuine thoughts on this topic. Seriously.. many thanks for starting this up. This web site is something that is required on the web, someone with some originality!
Wonderful article! We are linking to this great content on our site. Keep up the good writing.
After I initially commented I seem to have clicked the -Notify me when new comments are added- checkbox and now every time a comment is added I get 4 emails with the exact same comment. There has to be a way you can remove me from that service? Thank you!
Way cool! Some extremely valid points! I appreciate you penning this write-up plus the rest of the website is also really good.
I just want to mention I’m new to blogs and certainly savored you’re blog site. Most likely I’m likely to bookmark your blog post . You absolutely have superb articles and reviews. Many thanks for sharing your blog site.
Hi, There’s no doubt that your blog might be having internet browser compatibility issues. Whenever I look at your web site in Safari, it looks fine but when opening in I.E., it’s got some overlapping issues. I simply wanted to give you a quick heads up! Aside from that, excellent website!
bookmarked!!, I love your website!
It’s nearly impossible to find experienced people on this topic, however, you sound like you know what you’re talking about! Thanks
I wanted to thank you for this very good read!! I absolutely enjoyed every bit of it. I have got you saved as a favorite to look at new stuff you post…
Pretty! This has been an extremely wonderful post. Thanks for providing this info.
Your style is so unique compared to other people I have read stuff from. Thank you for posting when you have the opportunity, Guess I’ll just bookmark this page.
I love it when people come together and share ideas. Great site, stick with it!
I’d like to thank you for the efforts you have put in penning this blog. I am hoping to view the same high-grade blog posts from you later on as well. In fact, your creative writing abilities has encouraged me to get my very own blog now 😉
Aw, this was an extremely good post. Spending some time and actual effort to produce a really good article… but what can I say… I put things off a whole lot and never seem to get nearly anything done.
Way cool! Some extremely valid points! I appreciate you writing this article plus the rest of the website is really good.
I seriously love your website.. Pleasant colors & theme. Did you build this site yourself? Please reply back as I’m looking to create my own personal blog and want to find out where you got this from or exactly what the theme is called. Appreciate it!
Oh my goodness! Impressive article dude! Thank you, However I am having difficulties with your RSS. I don’t know the reason why I can’t subscribe to it. Is there anyone else having similar RSS problems? Anyone who knows the answer will you kindly respond? Thanx!!
Hey there! I simply want to offer you a huge thumbs up for the great information you have got here on this post. I am coming back to your website for more soon.
You need to be a part of a contest for one of the finest sites on the internet. I most certainly will highly recommend this website!
This web site really has all of the info I needed about this subject and didn’t know who to ask.
Very good information. Lucky me I found your website by accident (stumbleupon). I’ve saved it for later!
A motivating discussion is worth comment. I believe that you need to write more on this subject, it may not be a taboo matter but usually people don’t discuss such issues. To the next! Many thanks!!
Hi, I do think this is a great blog. I stumbledupon it 😉 I may revisit once again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to help other people.
Everything is very open with a really clear description of the challenges. It was really informative. Your site is very useful. Thanks for sharing!
I’d like to thank you for the efforts you have put in writing this site. I really hope to check out the same high-grade blog posts from you later on as well. In truth, your creative writing abilities has inspired me to get my very own blog now 😉
I was very happy to find this site. I need to to thank you for ones time due to this fantastic read!! I definitely liked every part of it and i also have you saved as a favorite to check out new stuff in your web site.
You are so interesting! I do not think I’ve read anything like that before. So nice to find someone with a few original thoughts on this topic. Really.. many thanks for starting this up. This site is one thing that is needed on the internet, someone with a little originality!
This web site definitely has all the information I wanted about this subject and didn’t know who to ask.
Good info. Lucky me I found your site by chance (stumbleupon). I have book marked it for later!
Hello there! This post could not be written any better! Looking through this article reminds me of my previous roommate! He always kept talking about this. I am going to send this post to him. Fairly certain he’s going to have a good read. Thank you for sharing!
I truly love your site.. Excellent colors & theme. Did you develop this web site yourself? Please reply back as I’m wanting to create my very own site and would love to find out where you got this from or just what the theme is named. Appreciate it!
Oh my goodness! Impressive article dude! Thanks, However I am going through troubles with your RSS. I don’t understand why I cannot subscribe to it. Is there anyone else getting identical RSS problems? Anyone that knows the solution can you kindly respond? Thanx!!
It’s very easy to find out any matter on web as compared to textbooks, as I found this post at this website.|
Hi there! This blog post could not be written much better! Reading through this article reminds me of my previous roommate! He always kept preaching about this. I am going to send this information to him. Pretty sure he will have a good read. Many thanks for sharing!
I used to be able to find good info from your blog articles.
Nice post. I learn something new and challenging on websites I stumbleupon everyday. It will always be interesting to read through articles from other authors and practice something from their websites.
Hello my loved one! I want to say that this post is amazing, nice written and come with approximately all significant infos. I’d like to see extra posts like this .|
I noticed one of your pages have a 404 error.
Hey there! Someone in my Facebook group shared this site with us so I came to look it over. I’m definitely enjoying the information. I’m book-marking and will be tweeting this to my followers! Exceptional blog and brilliant design.
Wow, superb blog layout! How lengthy have you been blogging for? you make running a blog glance easy. The full look of your site is excellent, let alone the content material!
Hello.This post was extremely fascinating, especially since I was investigating for thoughts on this issue last Tuesday.
Hi, Neat post. There’s a problem with your web site in internet explorer, would check this… IE still is the market leader and a good portion of people will miss your excellent writing because of this problem.
It’s in reality a great and useful piece of info. I am happy that you simply shared this helpful information with us. Please keep us up to date like this. Thanks for sharing.
Howdy just wanted to give you a brief heads up and let you know a few of the pictures aren’t loading correctly. I’m not sure why but I think its a linking issue. I’ve tried it in two different internet browsers and both show the same outcome.