The problem with metrics is a big problem for AI

Update: This post was expanded into a paper, The Problem with Metrics is a Fundamental Problem for AI, by Rachel Thomas and David Uminsky, which was accepted to the Ethics of Data Science Conference 2020. The paper version includes more grounding in previous academic work and a framework towards mitigating these harms.

Goodhart’s Law states that “When a measure becomes a target, it ceases to be a good measure.” At their heart, what most current AI approaches do is to optimize metrics. The practice of optimizing metrics is not new nor unique to AI, yet AI can be particularly efficient (even too efficient!) at doing so.

This is important to understand, because any risks of optimizing metrics are heightened by AI. While metrics can be useful in their proper place, there are harms when they are unthinkingly applied. Some of the scariest instances of algorithms run amok (such as Google’s algorithm contributing to radicalizing people into white supremacyteachers being fired by an algorithm, or essay grading software that rewards sophisticated garbage) all result from over-emphasizing metrics. We have to understand this dynamic in order to understand the urgent risks we are facing due to misuse of AI.

Headlines from HBR, Washington Post, and Vice on some of the outcomes of over-optimizing metrics: rewarding gibberish essays, promoting propaganda, massive fraud at Wells Fargo, and firing good teachers
Headlines from HBR, Washington Post, and Vice on some of the outcomes of over-optimizing metrics: rewarding gibberish essays, promoting propaganda, massive fraud at Wells Fargo, and firing good teachers

The following principles will be illustrated through a series of case studies:

We can’t measure the things that matter most

Metrics are typically just a proxy for what we really care about. The paper Does Machine Learning Automate Moral Hazard and Error? covers an interesting example: the researchers investigate which factors in someone’s electronic medical record are most predictive of a future stroke. However, the researchers found that several of the most predictive factors (such as accidental injury, a benign breast lump, or colonoscopy) don’t make sense as risk factors for stroke. So, just what is going on? It turned out that the model was just identifying people who utilize health care a lot. They didn’t actually have data of who had a stroke (a physiological event in which regions of the brain are denied new oxygen); they had data about who had access to medical care, chose to go to a doctor, were given the needed tests, and had this billing code added to their chart. But a number of factors influence this process: who has health insurance or can afford their co-pay, who can take time off of work or find childcare, gender and racial biases that impact who gets accurate diagnoses, cultural factors, and more. As a result, the model was largely picking out people who utilized healthcare versus who did not.

This an example of the common phenomenon of having to use proxies: You want to know what content users like, so you measure what they click on. You want to know which teachers are most effective, so you measure their students test scores. You want to know about crime, so you measure arrests. These things are not the same. Many things we do care about can not be measured. Metrics can be helpful, but we can’t forget that they are just proxies.

As another example, Google used hours spent watching YouTube as a proxy for how happy users were with the content, writing on the Google blog that “If viewers are watching more YouTube, it signals to us that they’re happier with the content they’ve found.” Guillaume Chaslot, an AI engineer who formerly worked at Google/YouTube, shares how this had the side effect of incentivizing conspiracy theories, since convincing users that the rest of the media is lying kept them watching more YouTube.

Metrics can, and will, be gamed

It is almost inevitable that metrics will be gamed, particularly when they are given too much power. One week this spring, Chaslot collected 84,695 videos from YouTube and analyzed the number of views and the number of channels from which they were recommended. This is what he found (also covered in the Washington Post):

Chart showing Russia Today's video on the Mueller Report as being an outlier in how many YouTube channels recommended it. <a href='https://twitter.com/gchaslot/status/1121603851675553793?s=20'>Source</a>
Chart showing Russia Today’s video on the Mueller Report as being an outlier in how many YouTube channels recommended it. Source

The state-owned media outlet Russia Today was an extreme outlier in how much YouTube’s algorithm had selected it to be recommended by a wide-variety of other YouTube channels. Such algorithmic selections, which begin autoplaying as soon as your current video is done, account for 70% of the time that users spend on YouTube. This chart strongly suggests that Russia Today has in some way gamed YouTube’s algorithm. (More evidence about issues with YouTube’s recommendation system is detailed here.) Platforms are rife with attempts to game their algorithms, to show up higher in search results or recommended content, through fake clicks, fake reviews, fake followers, and more.

Automatic essay grading software focuses primarily on metrics like sentence length, vocabulary, spelling, and subject-verb agreement, but is unable to evaluate aspects of writing that are hard to quantify, such as creativity. As a result, gibberish essays randomly generated by computer programs to contain lots of sophisticated words score well. Essays from students in mainland China, which do well on essay length and sophisticated word choice, received higher scores from the algorithms than from expert human graders, suggesting that these students may be using chunks of pre-memorized text.

As USA education policy began over-emphasizing student test scores as the primary way to evaluate teachers, there have been widespread scandals of teachers and principals cheating by altering students scores, in Georgia, Indiana, Massachusetts, Nevada, Virginia, Texas, and elsewhere. One consequence of this is that teachers who don’t cheat may be penalized or even fired (when it appears student test scores have dropped to more average levels under their instruction). When metrics are given undue importance, attempts to game those metrics become common.

Metrics tend to overemphasize short-term concerns

It is much easier to measure short-term quantities: click through rates, month-over-month churn, quarterly earnings. Many long-term trends have a complex mix of factors and are tougher to quantify. What is the long-term impact on user trust of having your brand associated with promoting pedophiliawhite supremacy, and flat-earth theories? What is the long-term impact on hiring to be the subject of years worth of privacy scandalspolitical manipulation, and facilitating genocide?

Simply measuring what users click on is a short-term concern, and does not take into account factors like the potential long-term impact of a long-form investigative article which may have taken months to research and which could help shape a reader’s understanding of a complex issue and even lead to significant societal changes.

A recent Harvard Business Review article looked at Wells Fargo as a case study of how letting metrics replace strategy can harm a business. After identifying cross-selling as a measure of long-term customer relationships, Wells Fargo went overboard emphasizing the cross-selling metric: intense pressure on employees combined with an unethical sales culture led to 3.5 million fraudulent deposit and credit card accounts being opened without customers’ consent. The metric of cross-selling is a much more short-term concern compared to the loftier goal of nurturing long-term customer relationships. Overemphasizing metrics removes our focus from long-term concerns such as our values, trust and reputation, and our impact on society and the environment, and myopically focuses on the short-term.

Many metrics gather data of what we do in highly addictive environments

It matters which metrics we gather and in what environment we do so. Metrics such as what users click on, how much time they spend on sites, and “engagement” are heavily relied on by tech companies as proxies for user preference, and are used to drive important business decisions. Unfortunately, these metrics are gathered in environments engineered to be highly addictive, laden with dark patterns, and where financial and design decisions have already greatly circumscribed the range of options.

Our online environment is a buffet of junk food
Our online environment is a buffet of junk food

Zeynep Tufekci, a professor at UNC and regular contributor to the New York Times, compares recommendation algorithms (such as YouTube choosing which videos to auto-play for you and Facebook deciding what to put at the top of your newsfeed) to a cafeteria shoving junk food into children’s faces. “This is a bit like an autopilot cafeteria in a school that has figured out children have sweet teeth, and also like fatty and salty foods. So you make a line offering such food, automatically loading the next plate as soon as the bag of chips or candy in front of the young person has been consumed.” As those selections get normalized, the output becomes ever more extreme: “So the food gets higher and higher in sugar, fat and salt – natural human cravings – while the videos recommended and auto-played by YouTube get more and more bizarre or hateful.” Too many of our online environments are like this, with metrics capturing that we love sugar, fat, and salt, not taking into account that we are in the digital equivalent of a food desert and that companies haven’t been required to put nutrition labels on what they are offering. Such metrics are not indicative of what we would prefer in a healthier or more empowering environment.

When Metrics are Useful

All this is not to say that we should throw metrics out altogether. Data can be valuable in helping us understand the world, test hypotheses, and move beyond gut instincts or hunches. Metrics can be useful when they are in their proper context and place. One way to keep metrics in their place is to consider a slate of many metrics for a fuller picture (and resist the temptation to try to boil these down to a single score). For instance, knowing the rates at which tech companies hire people from under-indexed groups is a very limited data point. For evaluating diversity and inclusion at tech companies, we need to know comparative promotion rates, cap table ownership, retention rates (many tech companies are revolving doors driving people from under-indexed groups away with their toxic cultures), number of harassment victims silenced by NDAs, rates of under-leveling, and more. Even then, all this data should still be combined with listening to first-person experiences of those working at these companies.

Columbia professor and New York Times Chief Data Scientist Chris Wiggins wrote that quantitative measures should always be combined with qualitative information, “Since we can not know in advance every phenomenon users will experience, we can not know in advance what metrics will quantify these phenomena. To that end, data scientists and machine learning engineers must partner with or learn the skills of user experience research, giving users a voice.”

Another key to keeping metrics in their proper place is to keep domain experts and those who will be most impacted closely involved in their development and use. Surely most teachers could have foreseen that evaluating teachers primarily on the standardized test scores of their students would lead to a host of negative consequences.

I am not opposed to metrics; I am alarmed about the harms caused when metrics are overemphasized, a phenomenon that we see frequently with AI, and which is having a negative, real-world impact. AI running unchecked to optimize metrics has led to Google/YouTube’s heavy promotion of white supremacist material, essay grading software that rewards garbage, and more. By keeping the risks of metrics in mind, we can try to prevent these harms.

 

 

Original post: https://www.fast.ai/2019/09/24/metrics/

107 comentários em “The problem with metrics is a big problem for AI

  1. Oh my goodness! Impressive article dude! Thanks, However I am experiencing difficulties with your RSS.
    I don’t know why I am unable to subscribe to it.
    Is there anyone else having similar RSS problems? Anyone that knows the answer can you kindly respond?
    Thanx!!

  2. That is very attention-grabbing, You are an excessively professional blogger.

    I have joined your feed and stay up for in search of extra
    of your magnificent post. Also, I’ve shared your web site in my social
    networks

  3. I love your blog.. very nice colors & theme. Did you make this website yourself or did you hire someone
    to do it for you? Plz reply as I’m looking
    to construct my own blog and would like to know where u got this from.
    cheers

  4. This design is spectacular! You certainly know how to keep a reader entertained.
    Between your wit and your videos, I was almost moved to start my own blog (well,
    almost…HaHa!) Wonderful job. I really enjoyed
    what you had to say, and more than that, how you presented it.
    Too cool!

  5. Pretty section of content. I just stumbled upon your blog and
    in accession capital to assert that I acquire actually enjoyed account your blog posts.
    Any way I’ll be subscribing to your feeds and even I achievement you access
    consistently fast.

  6. Hello, i believe that i noticed you visited
    my website so i came to go back the want?.I’m attempting to in finding issues to enhance my
    website!I suppose its good enough to use a few of your concepts!!

  7. Good day! I know this is kinda off topic however I’d figured I’d ask.

    Would you be interested in exchanging links or maybe guest writing a blog
    article or vice-versa? My blog addresses a lot of the same topics as
    yours and I feel we could greatly benefit from each other.

    If you happen to be interested feel free to send me an email.
    I look forward to hearing from you! Superb blog by the way!

  8. I am the owner of JustCBD brand (justcbdstore.com) and I’m presently looking to grow my wholesale side of business. I am hoping someone at targetdomain give me some advice ! I thought that the best way to do this would be to connect to vape shops and cbd retailers. I was really hoping if anybody could suggest a qualified website where I can get Vape Shop B2B Database I am currently looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the best option and would appreciate any advice on this. Or would it be simpler for me to scrape my own leads? Suggestions?

  9. I am the manager of JustCBD label (justcbdstore.com) and I am currently trying to broaden my wholesale side of business. I am hoping someone at targetdomain share some guidance . I considered that the most suitable way to do this would be to reach out to vape shops and cbd stores. I was hoping if anybody at all could suggest a trustworthy website where I can buy CBD Shops B2B Companies I am presently reviewing creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most ideal solution and would appreciate any support on this. Or would it be easier for me to scrape my own leads? Ideas?

  10. I’m impressed, I have to admit. Seldom do I encounter a blog that’s equally educative and engaging, and let me tell you, you’ve hit the nail on the head. The issue is something too few men and women are speaking intelligently about. Now i’m very happy I stumbled across this during my hunt for something regarding this.

  11. Having read this I thought it was rather informative. I appreciate you spending some time and energy to put this article together. I once again find myself personally spending a significant amount of time both reading and leaving comments. But so what, it was still worth it!

  12. A motivating discussion is definitely worth comment. I do think that you should write more on this topic, it may not be a taboo matter but typically people do not discuss these issues. To the next! Cheers!!

  13. I think this is one of the most important information for me.
    And i am glad reading your article. But should remark on few general
    things, The website style is ideal, the articles is really
    excellent : D. Good job, cheers

  14. You are so awesome! I don’t suppose I have read a single thing like this before. So great to discover someone with original thoughts on this topic. Really.. thank you for starting this up. This web site is one thing that is required on the web, someone with a little originality!

  15. Hello! I just would like to offer you a huge thumbs up for your excellent information you’ve got right here on this post. I will be coming back to your web site for more soon.

  16. I was suggested this blog by my cousin. I am not
    sure whether this publish is written by means of
    him as nobody else recognize such specified approximately
    my problem. You are amazing! Thanks!

  17. Everything is very open with a precise description of the challenges. It was truly informative. Your site is very helpful. Thank you for sharing!

  18. Having read this I thought it was very informative. I appreciate you finding the time and energy to put this information together. I once again find myself personally spending a lot of time both reading and commenting. But so what, it was still worthwhile!

  19. I have been surfing online more than three hours today, yet I never found any interesting article like yours.
    It’s pretty worth enough for me. Personally, if all website owners and bloggers
    made good content as you did, the web will be a lot more useful than ever
    before.

  20. This is the perfect site for anyone who hopes to find out about this topic. You understand so much its almost hard to argue with you (not that I actually will need to…HaHa). You definitely put a new spin on a topic that has been discussed for ages. Excellent stuff, just wonderful!

  21. May I just say what a comfort to find a person that really knows what they are discussing over the internet. You definitely know how to bring an issue to light and make it important. More and more people should read this and understand this side of the story. It’s surprising you are not more popular because you most certainly have the gift.

  22. Good post. I learn something totally new and challenging on sites I stumbleupon everyday. It will always be exciting to read through content from other authors and practice a little something from their websites.

  23. Good post. I learn something new and challenging on blogs I stumbleupon on a daily basis. It’s always useful to read through articles from other writers and practice a little something from other websites.

  24. Good post. I learn something totally new and challenging on blogs I stumbleupon everyday. It will always be exciting to read through content from other authors and practice a little something from their web sites.

  25. Next time I read a blog, I hope that it does not fail me just as much as this one. I mean, I know it was my choice to read, nonetheless I genuinely thought you’d have something interesting to say. All I hear is a bunch of whining about something you could fix if you were not too busy seeking attention.

  26. The very next time I read a blog, I hope that it doesn’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read through, nonetheless I genuinely believed you’d have something useful to talk about. All I hear is a bunch of crying about something that you could fix if you weren’t too busy seeking attention.

  27. I love your blog.. very nice colors & theme. Did you design this
    website yourself or did you hire someone to do it for you?
    Plz answer back as I’m looking to design my own blog and would like to know where
    u got this from. appreciate it

  28. Nice post. I learn something new and challenging on websites I stumbleupon every day.
    It will always be useful to read through content from other authors and practice something from their websites.

  29. This is the right website for anyone who wants to understand this topic. You know so much its almost hard to argue with you (not that I actually will need to…HaHa). You certainly put a fresh spin on a topic that’s been discussed for a long time. Great stuff, just excellent!

  30. An interesting discussion is worth comment. I do think that you ought to publish more about this subject matter, it might not be a taboo subject but typically folks don’t discuss such subjects. To the next! Best wishes!!

  31. You really make it seem so easy with your presentation but I find this topic to be actually something which I think I
    would never understand. It seems too complex and very broad for me.
    I am looking forward for your next post, I’ll try to get the hang
    of it!

  32. No matter if some one searches for his required thing, therefore
    he/she needs to be available that in detail, therefore that thing is maintained over here.

  33. I blog quite often and I really thank you for your content. This great article has truly peaked my interest. I’m going to take a note of your blog and keep checking for new information about once per week. I opted in for your Feed as well.

  34. Oh my goodness! Amazing article dude! Thank you, However I am going through problems with your RSS. I don’t know the reason why I cannot join it. Is there anybody getting identical RSS problems? Anyone who knows the solution can you kindly respond? Thanks!!

  35. Hello, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot
    of spam comments? If so how do you protect against it, any plugin or anything you can suggest?
    I get so much lately it’s driving me crazy so any help is very much appreciated.

  36. Hi! I realize this is sort of off-topic but I had to ask.
    Does operating a well-established blog such as yours require a massive amount work?
    I am completely new to writing a blog but I do write in my journal
    on a daily basis. I’d like to start a blog so I can easily share my personal experience and
    feelings online. Please let me know if you have any suggestions or tips for new aspiring bloggers.
    Appreciate it! adreamoftrains best hosting

  37. Today, I went to the beachfront with my children. I found a sea shell and gave it to my 4 year old daughter
    and said “You can hear the ocean if you put this to your ear.” She placed the shell to her
    ear and screamed. There was a hermit crab inside and it pinched her
    ear. She never wants to go back! LoL I know this
    is completely off topic but I had to tell someone!

  38. Aw, this was an extremely nice post. Taking a few minutes and actual effort to generate a great article… but what can I say… I procrastinate a whole lot and never seem to get nearly anything done.

  39. Aw, this was a very nice post. Taking the time and actual effort to produce a very good article… but what can I say… I procrastinate a whole lot and never manage to get nearly anything done.

  40. Your style is very unique in comparison to other people I have read stuff from.
    I appreciate you for posting when you’ve got the opportunity,
    Guess I will just book mark this site.

  41. You are so cool! I don’t believe I’ve truly read a single thing like this before. So nice to find another person with some original thoughts on this subject. Really.. thank you for starting this up. This website is one thing that’s needed on the internet, someone with a bit of originality!

  42. A fascinating discussion is worth comment. I do believe that you ought to write more on this subject, it may not be a taboo matter but typically folks don’t discuss these issues. To the next! Best wishes!!

  43. May I just say what a comfort to uncover someone that truly knows what they are talking about online. You definitely realize how to bring an issue to light and make it important. A lot more people have to read this and understand this side of the story. I can’t believe you are not more popular given that you definitely have the gift.

  44. Can I just say what a relief to discover somebody that actually understands what they’re talking about on the net. You actually understand how to bring a problem to light and make it important. A lot more people ought to read this and understand this side of the story. It’s surprising you aren’t more popular because you surely have the gift.

  45. Hi there! This post could not be written any better! Looking through this post reminds me of my previous roommate! He always kept talking about this. I will forward this article to him. Fairly certain he’s going to have a good read. Thank you for sharing!

  46. Aw, this was an extremely nice post. Finding the time and actual effort to create a superb article… but what can I say… I put things off a whole lot and don’t seem to get anything done.

  47. The next time I read a blog, Hopefully it does not disappoint me just as much as this one. I mean, Yes, it was my choice to read through, however I genuinely believed you would have something helpful to say. All I hear is a bunch of complaining about something you can fix if you weren’t too busy looking for attention.

  48. When I originally left a comment I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I recieve 4 emails with the exact same comment. There has to be an easy method you are able to remove me from that service? Thank you!

  49. Hi, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam feedback? If so how do you reduce it, any plugin or anything you can recommend? I get so much lately it’s driving me insane so any support is very much appreciated.|

  50. An interesting discussion is worth comment. I do believe that you should publish more about this subject, it might not be a taboo matter but usually people don’t discuss such subjects. To the next! Many thanks!!

  51. I absolutely love your site.. Great colors & theme. Did you make this website yourself? Please reply back as I’m planning to create my very own blog and would love to find out where you got this from or exactly what the theme is named. Appreciate it!

  52. May I just say what a comfort to discover someone that actually knows what they are discussing on the net. You definitely realize how to bring an issue to light and make it important. A lot more people have to check this out and understand this side of your story. I was surprised that you aren’t more popular given that you certainly have the gift.

  53. Hi there! This article could not be written any better! Reading through this post reminds me of my previous roommate! He continually kept preaching about this. I most certainly will forward this post to him. Pretty sure he will have a very good read. I appreciate you for sharing!

  54. After looking into a few of the blog posts on your site, I truly like your way of blogging. I bookmarked it to my bookmark webpage list and will be checking back soon. Please check out my web site as well and let me know your opinion.

  55. Achieving your fitness goal doesn’t have to require a certified personal trainer or an expensive gym memberships, especially when you have the budget and the space to consider practically every workout machine on the market.

  56. Great weblog right here! Also your web site rather a lot up very fast!
    What web host are you the use of? Can I am getting your associate link
    in your host? I want my website loaded up as quickly as
    yours lol 3gqLYTc cheap flights

  57. Thank you for some other wonderful article. Where else may just
    anyone get that kind of information in such a
    perfect method of writing? I’ve a presentation next
    week, and I’m on the search for such information. cheap flights 32hvAj4

  58. I’m amazed, I must say. Rarely do I come across a blog that’s equally educative and entertaining, and let me tell you, you have hit the nail on the head. The issue is something which too few folks are speaking intelligently about. I’m very happy I stumbled across this during my search for something regarding this.

  59. You’re so awesome! I don’t suppose I’ve truly read through anything like that before. So wonderful to discover somebody with some unique thoughts on this subject matter. Really.. thanks for starting this up. This site is one thing that’s needed on the web, someone with some originality!

  60. The next time I read a blog, I hope that it won’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read, nonetheless I actually believed you would have something useful to say. All I hear is a bunch of whining about something that you can fix if you were not too busy looking for attention.

  61. I blog quite often and I genuinely appreciate your content. Your article has truly peaked my interest. I will bookmark your website and keep checking for new details about once per week. I subscribed to your RSS feed too.

  62. Hi, I do believe this is an excellent blog. I stumbledupon it 😉 I’m going to return yet again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to guide other people.

  63. An outstanding share! I have just forwarded this onto a co-worker who was conducting a little research on this. And he actually ordered me breakfast because I stumbled upon it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to talk about this matter here on your website.

Leave a Reply

Your email address will not be published. Required fields are marked *