Building an MLOps strategy from the ground up

Crunch 2022

Isabel Zimmerman, RStudio, PBC

September 20, 2022

from the ground up

Tip

go to isabel.quarto.pub/crunch2022/ to follow along!

• what is MLOps anyway (and how can I start)? 🐶
• what do I need to keep in mind for tooling? 🔨

hi im isabel

started my career as a software engineer/data scientist, working on operationalizing models, mostly in kubernetes environments. if you know anything about k8s, they can be a little frustrating. so, giving my brain a break, i started training my little dog named toast to do tricks.

when first taught my dog to sit, stood in front of him…

now, as a data scientist, i realized i had not trained my dog very well. he only knew sit when i was standing in front of him.

go for a walk, side

back home in my local environment, side

toast taught me a hard lesson. even if youre training for the right outcome, sitting, being out of my house brought a new set of challenges, and he behaved differently

if you develop models…

you can operationalize them

but, this is not a dog training conference–so, i’ll tell you this.

if you develop models…

you should operationalize them

well, some of them

in fact, i’ll go so far as to say this

• 85 % of models don’t make it to production, that is, they don’t get deployed, and some people try to say that this is a reason why we need MLOps tools. you might update a model tens of times before it is truly ready for production.

• might be a good thing (in a cost saving sense as well as model quality sense).

information -> 🐶 -> actions

i gave my dog information, telling him to sit, or some kind of input. at home there was a great response, reproducible responses! but out in the wilderness of my suburban neighborhood, when I gave my dog instructions, it didn’t always look the way I expected it to. working together with my dog to create actions–greeting the neighbor by sitting in a polite manner–looked different and felt different in different environments

information -> model -> actions

models work the same way. when you’re training and tuning a model, data scientists often live in a cozy, familiar environment. but when exposed to the messiness of real data, and the uncertainty of world in general, they may act differently.

business value of models often comes from operationalizing a model–how that model works in the wild..

but, maybe we’re getting ahead of ourselves here

What is MLOps?

MLOps, or machine learning operations is where DevOps (development operations) and machine learning combine.

MLOps is…

a set of practices to deploy and maintain machine learning models in production reliably and efficiently

. . . and these practices can be HARD.

• i said i started with kubernetes + models. after a while where i STRUGGLED with the tools at hand, i felt that, as a data scientist, the tools were not built for me. they were built for a software engineer that had pretty deep knowledge of infrastructure–and i needed to productionalize models, but i didn’t want to have to be a cloud architect as well a data scientist.

• so i changed career paths to build new tools designed specifically for data scientists to deploy models. my current role is to develop an open source package called

• vetiver, which is an mlops framework to help data scientists in R and Python

a basic workflow

import pandas as pd
raw = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-03-02/youtube.csv')
df = raw[["like_count", "funny", "show_product_quickly", "patriotic", \
    "celebrity", "danger", "animals"]].dropna()

import pandas as pd
raw = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-03-02/youtube.csv')
df = raw[["like_count", "funny", "show_product_quickly", "patriotic", \
    "celebrity", "danger", "animals"]].dropna()
print(df)

     like_count  funny  show_product_quickly  patriotic  celebrity  danger  \
0        1233.0  False                 False      False      False   False   
1         485.0   True                  True      False       True    True   
2         129.0   True                 False      False      False    True   
3           2.0  False                  True      False      False   False   
4          20.0   True                  True      False      False    True   
..          ...    ...                   ...        ...        ...     ...   
241        10.0   True                 False       True       True   False   
243       572.0  False                  True       True      False   False   
244        14.0   True                 False      False       True    True   
245        12.0   True                 False      False      False    True   
246       334.0  False                 False      False       True   False   

     animals  
0      False  
1      False  
2       True  
3      False  
4       True  
..       ...  
241     True  
243     True  
244    False  
245    False  
246    False  

[225 rows x 7 columns]

import pandas as pd
raw = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-03-02/youtube.csv')
df = raw[["like_count", "funny", "show_product_quickly", "patriotic", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, pipeline, ensemble
X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

import pandas as pd
raw = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-03-02/youtube.csv')
df = raw[["like_count", "funny", "show_product_quickly", "patriotic", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, pipeline, ensemble
X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

oe = preprocessing.OrdinalEncoder().fit(X_train)
rf = ensemble.RandomForestRegressor().fit(oe.transform(X_train), y_train)

import pandas as pd
raw = pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-03-02/youtube.csv')
df = raw[["like_count", "funny", "show_product_quickly", "patriotic", \
    "celebrity", "danger", "animals"]].dropna()

from sklearn import model_selection, preprocessing, pipeline, ensemble
X_train, X_test, y_train, y_test = model_selection.train_test_split(
    df.drop(columns = ['like_count']),
    df['like_count'],
    test_size=0.2
)

oe = preprocessing.OrdinalEncoder().fit(X_train)
rf = ensemble.RandomForestRegressor().fit(oe.transform(X_train), y_train)

rf_pipe = pipeline.Pipeline([('ordinal_encoder',oe), ('random_forest', rf)])

vetiver focuses on 3 practices

MLOps is…

MLOps is… versioning

model

model_final

model_final_v2

model_final_v2_ACTUALLY

lacks context not scalable

versioning a model usually includes storing the model object, a hash for a robust and scalable version, and some metadata

MLOps is… versioning

managing change in models

across time (model_final, model_final_2). this is also important if you have different implementations– for example, staging and production.

really, anytime you have multiple models saved, some sort of versioning will help organize your models, making them easier to organize for yourself, share with your team members, and manage later on.

import pins

model_board = pins.board_temp(
    allow_pickle_read = True)

from vetiver import VetiverModel, vetiver_pin_write

v = VetiverModel(rf_pipe, "ads", ptype_data = X_train)

deployable model object

using the model object– however youre getting data can stay the same! however you are modeling can stay the same!

data prototype - a common failure point is deployed data looks different

deploying the WHOLE PIPELINE - had to fit the preprocessor too, otherwise you will be overestimating your models performance

• Python
• R

import pins

model_board = pins.board_temp(
    allow_pickle_read = True)

from vetiver import VetiverModel, vetiver_pin_write

v = VetiverModel(rf_pipe, "ads", ptype_data = X_train)

vetiver_pin_write(model_board, v)

library(vetiver)
library(pins)

model_board <- board_temp()

v <- vetiver_model(model, "ads")

model_board %>% 
  vetiver_pin_write(v)

writing data to the pin

import pins

model_board = pins.board_temp(
    allow_pickle_read = True)

from vetiver import VetiverModel, vetiver_pin_write

v = VetiverModel(rf_pipe, "ads", ptype_data = X_train)

vetiver_pin_write(model_board, v)

model_board.pin_meta("ads")

Meta(title='ads: a pinned Pipeline object', description="Scikit-learn <class 'sklearn.pipeline.Pipeline'> model", created='20221003T221304Z', pin_hash='612d4b523ca8c0ef', file='ads.joblib', file_size=432866, type='joblib', api_version=1, version=Version(created=datetime.datetime(2022, 10, 3, 22, 13, 4), hash='612d4'), name='ads', user={'ptype': '{"funny": true, "show_product_quickly": true, "patriotic": false, "celebrity": false, "danger": false, "animals": false}', 'required_pkgs': ['vetiver', 'scikit-learn']})

MLOps is… versioning

where are these boards hosted?

saw temp board, but its gone!

import pins

model_board = pins.board_temp(
    allow_pickle_read = True)

pins is a project my team built, also in R

import pins

model_board = pins.board_local(
    allow_pickle_read = True)

import pins

model_board = pins.board_s3(
    allow_pickle_read = True)

amazon web services

import pins

model_board = pins.board_gcs(
    allow_pickle_read = True)

google cloud storage

import pins

model_board = pins.board_azure(
    allow_pickle_read = True)

microsoft azure blob storage

you still need to download them or do some sort of operation to actually make a prediction, which is a pretty clunky workflow

which leads us to… however, model registries are storing the model, not running the model.

MLOps is… deploying

MLOps is… deploying

putting a model in production

deploying means many different things. in general, we say that deploying a model is putting a model in production

uh, what is production?

production loooks different for different groups–if you’ve put a model inside an application that is running somewhere else, you’ve deployed a model!

MLOps is… deploying

putting a model in production somewhere that is not on your local laptop

MLOps is… deploying

putting a model in production somewhere that is not on your local laptop

the IDEA of data science workflows are directly opposite of software engineering best practices.

where software engineers are interested in robust applications and strong testing and absolute certainty that something will behave as expected, data science requires agility and experimental behavior

if we’re thinking of making an app, we don’t want to copy and paste machine learning code over into an app’s source code, or send emails with pickled models in them…

MLOps is… deploying

putting a model in production somewhere that is not on your local laptop

✅ using REST APIs

REST APIs (Representational state transfer application programming interface) are standardized ways for machines to communicate to each other.

&&&

REST APIs are useful since they are endpoints that can be hosted in different computational environments, independent of the application

this makes the software engineering part of me very very happy, since this can live stably, decoupled from

they are testable,

from vetiver import VetiverModel, VetiverAPI, vetiver_pin_write
v = VetiverModel(rf_pipe, "ads", ptype_data = X_train) 

VetiverAPI(v).run()

flexible tool– creating a FastAPI endpoint and adding in things to make it model aware

start up

self documenting

health endpoint, make sure everything is online

but, this is still local! where can we move it

vetiver.deploy_rsconnect(
    connect_server = connect_server, 
    board = model_board, 
    pin_name = "ads", 
    version = "59869")

things break pretty quickly when your development cycle acts or even feels differently than your production environment–this is a nod to the reproducibility of APIs, we wanted the APIs to look the same, even if they’re hosted in different locations

vetiver.write_app(board=board, pin_name="ads")

will create a file named app.py with all the information needed to start up an API

just a python script. it will run out of the box

for some cloud providers, this might be enough

vetiver.write_app(board=board, pin_name="ads")
vetiver.write_docker(app_file="app.py")

maybe you’re a docker fan?

docker is pretty well received by cloud providers, bring your own container philosophy

currently working on explicit workflows for different cloud providers

also working on specialized docker containers (AWS Lambda)

MLOps is… monitoring

model is deployed

a data scientist’s work is not done!

continuous monitoring– continue to track how the model

MLOps is… monitoring

MLOps is… monitoring

it is SO IMPORTANT TO TRACK your model’s performance metrics start decaying.

software engineering–when things went wrong, ERROR

models fail silently! and they can still run with no error, even if your accuracy is zero percent–

if you are not monitoring your model in some way, you are oblivious to decay.

import vetiver
from sklearn import metrics
from datetime import timedelta

metric_set = [metrics.mean_absolute_error, metrics.mean_squared_error]

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    metric_set, 
    "like_count", 
    "preds"
    )

• have new data (saving from infra)
• contains things like date the prediction was made, the predicted value, and the actual value

give the function this data, but also a few other things: - amount of time to aggregate over - set of metrics to track

import vetiver
from sklearn import metrics
from datetime import timedelta

metric_set = [metrics.mean_absolute_error, metrics.mean_squared_error]

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    metric_set, 
    "like_count", 
    "preds"
    )

m = vetiver.plot_metrics(metrics)

import vetiver
from sklearn import metrics
from datetime import timedelta

metric_set = [metrics.mean_absolute_error, metrics.mean_squared_error]

metrics = vetiver.compute_metrics(
    new_data, 
    "date", 
    timedelta(weeks = 1), 
    metric_set, 
    "like_count", 
    "preds"
    )

m = vetiver.plot_metrics(metrics)
m.update_yaxes(matches=None)
m.show()

flexible tool– will create plot, you can customize (ie, rescale the y-axis)

MLOps is… monitoring

when things go wrong:

• retrain, retrain, retrain
• try a new model type
• remember to version!

if performance is declining, - retraining model? - new model type altogether?

if you are using model versioning, it becomes easier to roll back to the latest version

MLOps is… thinking about making good models

not only do we want our models to be GOOD in a statistical and business sense, but also GOOD in an ethical sense

there’s no surefire way to avoid bias, but being thoughtful is a good place to start

this is a space my team is actively investing time into to make sure we can integrate and support any effort to create a tool that helps data scientist make sure the models they deploy have given a holistic view to their model

MLOps is… thinking about making good models

vetiver.vetiver_pin_write(board, v)

Model Cards provide a framework for transparent, responsible reporting. 
 Use the vetiver `.qmd` Quarto template as a place to start, 
 with vetiver.model_card()

created by people who left google

looks at a model quantitatively and also qualitativvely

MLOps is… thinking about making good models

vetiver.model_card()

partially automated (pieces that we can), uses Quarto, a platform for technical documentation and communication (actually, fun fact, my slides are also made in quarto)

my model doesn’t have any ethical challenges, it’s predicting youtube likes

• Also, consider the possibility of gathering feedback from those impacted by the machine learning system, especially those with marginalized identities.

• However, we strongly advise that instead of deleting any such section because you have incomplete or imprecise information, you note your own process and considerations.

MLOps is… thinking about making good models

From Mitchell et al. (2019):

Therefore the usefulness and accuracy of a model card relies on the integrity of the creator(s) of the card itself.

The process of documenting the extent and limits of a machine learning system is part of transparent, responsible reporting.

ultimately the extent of its value depends on you

actually, my dad has a quote that he always tells me–“if you haven’t written it down, you haven’t thought it out”

from the ground up

• what is MLOps anyway (and how can I start)? 🐶 ✅
• what do I need to keep in mind for tooling? 🔨

from this, hopefully you are fairly oriented to what MLOps is, and what practices are helping you deploy models!

Tooling tips…

the landscape is vast! choosing the right tool is important, and can be difficult. there’s no one size fits all method for MLOps.

i make data science tools, but i also have spent a lot of time looking at tools, and here’s a few of my takeaways

Tooling tips…

you (and your team!) are unique!

before you dive into tools, its really important to recognize that teams are at different points of their MLOps journey. not every team needs a huuuuge pipeline to move data to some multicloud environment. maybe a big win for your team is to even start versioning your models, even if its just locally.

Tooling tips…

• composable
  • in different environments
  • with other tools

one thing i think about a lot is how does the software i build work in different scenarios, because each team has unique needs. not only do i want this software to

• work in different clouds.

• first class support for teams products

• make sure users can easily make an app.py or Dockerfile to bring to a public or private cloud

• work well with other tools

• pandas and scikit learn uses the same dataframe data structures to model the data

• the data science landscape in python would look very different if these two tools did not have pieces that worked as a natural extension of each other.

• in the same way, you can use vetiver alone to do the tasks of versioning, deploying, and monitoring, but also add in other softwares if you would like to do more.

• composable with itself

this is also making a straightforward point of entry to get started with the tool, but the ability to extend these modular functions to be quite complex.

Tooling tips…

• composable
• reproducible
  • open source 💖

i can point to about 1000 different ways reproduciblility is important in machine learning, especially MLOps.

however, i really want to highlight the value of open source tools here.

we all saw that landscape a few slides ago, there’s a lot of options out there! MLOps is a happening field right now

from a business perspective: open source is foreeeverrrr

• free!
• outlasts a company’s lifetime
• outlasts your company’s subscription

from a human perspective: open source is for everyonee

• free!
• equitable

if you have an internet connection, you can build the skills, be part of an amazing technical community

Tooling tips…

• composable
• reproducible
• ergonomic

your NEED is that you are ready to implement some MLOps practices

your tool help you do that by some magic of engineering

but, we are humans, not robots! we need a tool that feels natural to help us complete the task

Tooling tips…

• composable
• reproducible
• ergonomic

your tool needs to “fit the person.” people need some sort of tool that is ergonomic, feels GOOD to use.

vetiver is a tool made especially with data scientists in mind.

• natural extension of their workflow, using the model objects that they are creating in python or R

not all tools are built for data scientists, and that is a GOOD THING! find tools that were built with your persona in mind

Tooling tips…

• composable
• reproducible
• ergonomic

(and able to do the MLOps tasks we want)

MLOps definitely covers a lot of ground. there’s tools that focus on experiment tracking or maybe just pipelines, or other pieces

from the ground up

• what is MLOps anyway (and how can I start)? 🐶 ✅
• what do I need to keep in mind for tooling? 🔨 ✅

MLOps is…

a set of practices to deploy and maintain machine learning models in production reliably and efficiently

• versioning

• deploying

• monitoring

vetiver can help with this for your R and Python models!

Learn more

• Documentation at https://vetiver.rstudio.com/

• Recent screencast on deploying a model with Docker

• End-to-end demos from RStudio Solutions Engineering in R and Python

• These slides! Visit isabel.quarto.pub/crunch2022