Deep Learning’s Climate Change Problem

Earlier this month, OpenAI announced it had built the biggest AI model in history. This astonishingly large model, known as GPT-3, is an impressive technical achievement. Yet it highlights a troubling and harmful trend in the field of artificial intelligence—one that has not gotten enough mainstream attention.

Modern AI models consume a massive amount of energy, and these energy requirements are growing at a breathtaking rate. In the deep learning era, the computational resources needed to produce a best-in-class AI model has on average doubled every 3.4 months; this translates to a 300,000x increase between 2012 and 2018. GPT-3 is just the latest embodiment of this exponential trajectory.

The bottom line: AI has a meaningful carbon footprint today, and if industry trends continue it will soon become much worse. Unless we are willing to reassess and reform today’s AI research agenda, the field of artificial intelligence could become an antagonist in the fight against climate change in the years ahead.

Bigger Is Not Always Better

In today’s deep learning-centric research paradigm, advances in artificial intelligence are primarily achieved through sheer scale: bigger datasets, larger models, more compute.

GPT-3 illustrates this phenomenon well. The model consists of a whopping 175 billion parameters. To put this figure in perspective, its predecessor model GPT-2—which was considered state-of-the-art when it was released last year—had only 1.5 billion parameters. While last year’s GPT-2 took a few dozen petaflop-days to train—already a massive amount of computational input—GPT-3 required several thousand.

The problem with relying on ever-larger models to drive progress in AI is that building and deploying these models entails a tremendous amount of energy expenditure and thus carbon emissions.

In a widely discussed 2019 study, a group of researchers led by Emma Strubell estimated that training a single deep learning model can generate up to 626,155 pounds of CO2 emissions—roughly equal to the total lifetime carbon footprint of five cars. As a point of comparison, the average American generates 36,156 pounds of CO2 emissions in a year.

To be sure, this estimate is for a particularly energy-intensive model. Training an average-sized machine learning model today generates far less than 626,155 pounds of carbon output.

At the same time, it is worth keeping in mind that when this analysis was conducted, GPT-2 was the largest model available for study and was treated by the researchers as an upper bound on model size. Just a year later, GPT-2 looks tiny—one hundred times smaller, in fact—compared to its successor.

Why exactly do machine learning models consume so much energy?

The first reason is that the datasets used to train these models continue to balloon in size. In 2018, the BERT model achieved best-in-class NLP performance after it was trained on a dataset of 3 billion words. XLNet outperformed BERT based on a training set of 32 billion words. Shortly thereafter, GPT-2 was trained on a dataset of 40 billion words. Dwarfing all these previous efforts, a weighted dataset of roughly 500 billion words was used to train GPT-3.

Neural networks carry out a lengthy set of mathematical operations (both forward propagation and back propagation) for each piece of data they are fed during training, updating their parameters in complex ways. Larger datasets therefore translate to soaring compute and energy requirements.

Another factor driving AI’s massive energy draw is the extensive experimentation and tuning required to develop a model. Machine learning today remains largely an exercise in trial and error. Practitioners will often build hundreds of versions of a given model during training, experimenting with different neural architectures and hyperparameters before identifying an optimal design.

The 2019 paper mentioned above includes a telling case study. The researchers picked an average-sized model—much smaller than headline-grabbing behemoths like GPT-3—and examined not just the energy required to train the final version, but the total number of trial runs that went into producing that final version.

Over the course of six months, 4,789 different versions of the model were trained, requiring 9,998 total days’ worth of GPU time (more than 27 years). Taking all these runs into account, the researchers estimated that building this model generated over 78,000 pounds of CO2 emissions in total—more than the average American adult will produce in two years.

To this point, the discussion has only addressed training machine learning models. But training is just the beginning of a model’s lifecycle. After a model is trained, it is then put to use in the real world.

Deploying AI models to take action in real-world settings—a process known as inference—consumes even more energy than training does. Indeed, Nvidia estimates that 80% to 90% of the cost of a neural network is in inference rather than training.

As an example, consider the AI underlying an autonomous vehicle. The neural networks first must be trained upfront to learn to drive. After training is completed and the autonomous vehicle is deployed, the model then performs inference on a continuous basis in order to navigate its environment—nonstop, day after day, for as long as the vehicle is in use.

Needless to say, the more parameters the model has, the steeper the energy requirements are for this ongoing inference.

Energy Use and Carbon Emissions

An assumption at the heart of this topic is the relationship between AI’s energy use and carbon emissions. What is the best way to think about this relationship?

According to the EPA, one kilowatt-hour of energy consumption generates 0.954 pounds of CO2 emissions on average in the United States. This average reflects the varying carbon footprints and relative proportions of different electricity sources across the U.S. energy grid (e.g., renewables, nuclear, natural gas, coal).

Strubell’s analysis, mentioned above, applies this U.S.-wide average in order to calculate the carbon emissions of various AI models based on their energy needs. It is a reasonable assumption. The power source mix for Amazon Web Services, for instance, roughly mirrors that of the U.S. as a whole, and most AI models are trained in the cloud.

Of course, if an AI model were trained using electricity generated primarily from renewables, then its carbon footprint would be correspondingly lower. For instance, Google Cloud Platform’s power mix is more heavily weighted toward renewables than is AWS’ (56% v. 17%, according to the Strubell paper).

Or, to give another example, a model that is trained on hardware located in the Pacific Northwest would generate less carbon output than the national average because of the abundant clean hydropower in that region. And it is worth mentioning that every cloud provider touts its investments in carbon offsets.

But taken in the aggregate, applying the overall U.S. power mix as Strubell does should yield a roughly accurate approximation of AI models’ carbon footprints.

Diminishing Returns

The problem with chasing progress in AI via ever-larger models is underscored by the relationship between model size and model performance. The data here is clear: increases in model size eventually exhibit sharply diminishing returns to performance.

An illustrative example will help make this point. ResNet was a well-known computer vision model released in 2015. An improved iteration of the model, known as ResNeXt, came out in 2017. ResNeXt required 35% more computational resources to train than did ResNet (measured by total floating point operations)—and achieved a 0.5% improvement in accuracy.

2019 paper from the Allen Institute for AI provides detailed data documenting the diminishing returns to model size across different tasks, models, and AI subfields. The most recent mega-model released, GPT-3, displays clear signs of diminishing returns compared to GPT-2.

If the AI community continues down its current path, greater and greater amounts of energy will need to be expended to build larger and larger models—in order to achieve ever-smaller incremental improvements in performance. Any cost/benefit analysis becomes increasingly disproportionate.

Given these diminishing returns, what motivates the continued development of ever-larger models? One major reason is the AI community’s current fixation on achieving “state-of-the-art” results on performance benchmarks. Building a model that sets a new accuracy record on a well-known benchmark—even if the improvement is only a fraction of a percent—can earn researchers recognition and acclaim.

As UCLA professor Guy Van den Broeck put it: “I think the best analogy is with some oil-rich country being able to build a very tall skyscraper. Sure, a lot of money and engineering effort goes into building these things. And you do get the ‘state of the art’ in building tall buildings. But…there is no scientific advancement per se.”

The “bigger is better” ethos that currently dominates the AI research agenda threatens to inflict major environmental damage in the years ahead. Thoughtful, bold change is needed to set the field of artificial intelligence on a more sustainable and productive trajectory.

Looking Forward

To start, there are near-term “quick wins” that every AI practitioner should consider to mitigate the carbon impact of their research.

An important first step is to increase transparency and measurement on this issue. When AI researchers publish results for new models, they should include—alongside performance and accuracy metrics—data on how much energy was expended in model development.

In a thoughtful analysis, the team from the Allen Institute for AI proposed floating point operations as the most universal and useful energy efficiency metric for researchers to track. Another group created a Maching Learning Emissions Calculator that practitioners can use to estimate the carbon footprints of the models they build (based on factors including hardware, cloud provider, and geographical region).

Along these lines, it should become best practice for researchers to plot energy costs against performance gains when training models. Explicitly quantifying this tradeoff will prompt researchers to make more informed, balanced decisions about resource allocation in light of diminishing returns.

Hopefully, as sustainable AI practices spread, the community will begin to take efficiency metrics like these into account when evaluating AI research, just as it does today with traditional performance metrics like accuracy: for conference paper submissions, speaking opportunities, academic roles and so forth.

There is other low-hanging fruit that can help reduce AI’s carbon footprint in the near term: using more efficient hyperparameter search methods, reducing the number of unnecessary experiments during training, employing more energy-efficient hardware.

But by themselves, these remedial actions are not enough to solve the problem. A more fundamental long-term shift is needed in the field of artificial intelligence.

We need to take a step back and acknowledge that simply building ever-larger neural networks is not the right path to generalized intelligence. From first principles, we need to push ourselves to discover more elegant, efficient ways to model intelligence in machines. Our ongoing battle with climate change, and thus the future of our planet, depend on it.

To quote AI legend Geoff Hinton, the godfather of deep learning: “The future depends on some graduate student who is deeply suspicious of everything I have said….My view is throw it all away and start again.”

The AI community must begin to work toward new paradigms in artificial intelligence that do not require exponentially growing datasets nor outrageously vast energy expenditures. Emerging research areas like few-shot learning are promising avenues.

The human brain—that original source of intelligence—provides important inspiration here. Our brains are incredibly efficient relative to today’s deep learning methods. They weigh a few pounds and require about 20 watts of energy, barely enough to power a dim lightbulb. Yet they represent the most powerful form of intelligence in the known universe.

“Human brains can do amazing things with little power consumption,” as AI researcher Siva Reddy put it. “The question is how can we build such machines.”

 

Original post: https://www.forbes.com/sites/robtoews/2020/06/17/deep-learnings-climate-change-problem/

114 comentários em “Deep Learning’s Climate Change Problem

  1. Good day! Would you mind if I share your blog with my myspace group?
    There’s a lot of people that I think would really appreciate your content.
    Please let me know. Cheers

  2. Having read this I thought it was rather enlightening. I appreciate
    you finding the time and energy to put this information together.
    I once again find myself spending a significant amount of time both
    reading and commenting. But so what, it was still worthwhile!

  3. Hey there! Do you know if they make any plugins to safeguard against hackers?
    I’m kinda paranoid about losing everything I’ve worked hard on. Any recommendations?

  4. I’m the proprietor of JustCBD Store brand (justcbdstore.com) and am looking to expand my wholesale side of business. It would be great if someone at targetdomain give me some advice ! I thought that the most effective way to accomplish this would be to reach out to vape shops and cbd stores. I was hoping if someone could recommend a reliable website where I can buy Vape Shop Business Leads I am already looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not sure which one would be the most suitable solution and would appreciate any support on this. Or would it be simpler for me to scrape my own leads? Ideas?

  5. I am the manager of JustCBD label (justcbdstore.com) and I’m presently trying to grow my wholesale side of business. I am hoping someone at targetdomain give me some advice 🙂 I considered that the most effective way to accomplish this would be to connect to vape companies and cbd retail stores. I was really hoping if anybody could recommend a trusted web-site where I can purchase Vape Shop B2B Sales Leads I am presently taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the best choice and would appreciate any guidance on this. Or would it be simpler for me to scrape my own leads? Suggestions?

  6. Right here is the perfect webpage for anyone who really wants to find out about this topic. You know so much its almost tough to argue with you (not that I really will need to…HaHa). You certainly put a brand new spin on a topic which has been written about for a long time. Excellent stuff, just wonderful!

  7. The very next time I read a blog, Hopefully it does not disappoint me as much as this particular one. I mean, I know it was my choice to read through, however I truly thought you would probably have something useful to talk about. All I hear is a bunch of complaining about something you could possibly fix if you weren’t too busy seeking attention.

  8. I was excited to find this website. I want to to thank you for ones time for this particularly fantastic read!! I definitely appreciated every part of it and i also have you book marked to see new things on your site.

  9. You are so awesome! I do not suppose I have read something like that before. So good to find someone with a few genuine thoughts on this subject matter. Seriously.. thank you for starting this up. This web site is something that is required on the web, someone with a bit of originality!

  10. An impressive share! I have just forwarded this onto a friend who had been doing a little research on this. And he in fact bought me lunch because I discovered it for him… lol. So let me reword this…. Thank YOU for the meal!! But yeah, thanx for spending some time to discuss this matter here on your blog.

  11. Aw, this was an extremely nice post. Taking the time and actual effort to create a very good article… but what can I say… I put things off a lot and never seem to get anything done.

  12. Hello there! I simply want to offer you a big thumbs up for your excellent information you’ve got here on this post. I am coming back to your blog for more soon.

  13. Having read this I thought it was extremely enlightening. I appreciate you spending some time and energy to put this information together. I once again find myself personally spending way too much time both reading and posting comments. But so what, it was still worth it!

  14. I’m impressed, I have to admit. Rarely do I come across a blog that’s both equally educative and amusing, and without a doubt, you’ve hit the nail on the head. The problem is an issue that not enough people are speaking intelligently about. I’m very happy I came across this in my hunt for something concerning this.

  15. Good post. I learn something new and challenging on blogs I stumbleupon on a daily basis. It will always be interesting to read through articles from other writers and practice a little something from other websites.

  16. Right here is the perfect blog for anyone who hopes to find out about this topic. You know a whole lot its almost tough to argue with you (not that I really will need to…HaHa). You definitely put a brand new spin on a subject which has been discussed for a long time. Great stuff, just great!

  17. An outstanding share! I’ve just forwarded this onto a co-worker who had been doing a little research on this. And he actually ordered me lunch simply because I discovered it for him… lol. So allow me to reword this…. Thanks for the meal!! But yeah, thanx for spending some time to discuss this topic here on your website.

  18. You made some good points there. I looked on the web to learn more about the issue and found most people will go along with your views on this website.

  19. Good post. I learn something new and challenging on websites I stumbleupon everyday. It will always be helpful to read through articles from other authors and practice a little something from other sites.

  20. Hola! I’ve been reading your web site for a long time now and finally got the bravery to go ahead
    and give you a shout out from Dallas Texas! Just wanted to mention keep up the fantastic job!

  21. Hey there! I know this is kind of off topic but I was
    wondering if you knew where I could locate a captcha plugin for my comment form?
    I’m using the same blog platform as yours and I’m having trouble finding one?
    Thanks a lot!

  22. Aw, this was an incredibly nice post. Taking a few minutes and actual effort to create a great article… but what can I say… I hesitate a whole lot and never seem to get anything done.

  23. Aw, this was an exceptionally good post. Taking a few minutes and actual effort to create a really good article… but what can I say… I put things off a lot and don’t seem to get nearly anything done.

  24. Howdy! I could have sworn I’ve been to this website before but after browsing through a few of the articles I realized it’s new to me. Anyhow, I’m definitely happy I came across it and I’ll be book-marking it and checking back frequently!

  25. Nice post. I learn something totally new and challenging on websites I stumbleupon every day. It will always be helpful to read articles from other writers and practice a little something from other websites.

  26. I’m amazed, I have to admit. Seldom do I encounter a blog that’s both equally educative and entertaining, and without a doubt, you have hit the nail on the head. The problem is something not enough folks are speaking intelligently about. I’m very happy that I found this during my search for something regarding this.

  27. Hello there! This article could not be written much better! Going through this post reminds me of my previous roommate! He continually kept talking about this. I most certainly will forward this article to him. Pretty sure he’s going to have a very good read. Thank you for sharing!

  28. Howdy, I do believe your website might be having browser compatibility problems. When I look at your blog in Safari, it looks fine however, if opening in Internet Explorer, it’s got some overlapping issues. I just wanted to give you a quick heads up! Aside from that, great blog!

  29. Great blog you have got here.. It’s hard to find good quality writing like yours these days. I truly appreciate people like you! Take care!!

  30. Hello there, I do believe your web site could be having web browser compatibility issues. Whenever I look at your blog in Safari, it looks fine however when opening in Internet Explorer, it has some overlapping issues. I just wanted to give you a quick heads up! Other than that, great blog!

  31. I blog often and I genuinely appreciate your information. The article has truly peaked my interest. I’m going to take a note of your site and keep checking for new information about once per week. I subscribed to your Feed as well.

  32. Right here is the right blog for anybody who wants to find out about this topic. You understand a whole lot its almost hard to argue with you (not that I actually would want to…HaHa). You certainly put a fresh spin on a topic which has been discussed for years. Excellent stuff, just wonderful!

  33. I’m amazed, I have to admit. Rarely do I encounter a blog that’s both educative and interesting, and let me tell you, you’ve hit the nail on the head. The issue is an issue that not enough people are speaking intelligently about. I’m very happy that I came across this in my hunt for something regarding this.

  34. You’re so awesome! I don’t believe I’ve read anything like that before. So wonderful to discover somebody with a few genuine thoughts on this subject. Really.. thank you for starting this up. This site is one thing that’s needed on the internet, someone with a bit of originality!

  35. I blog often and I seriously thank you for your content. This article has really peaked my interest. I will bookmark your blog and keep checking for new information about once per week. I subscribed to your RSS feed too.

  36. Right here is the right website for everyone who wishes to understand this topic. You realize a whole lot its almost hard to argue with you (not that I really will need to…HaHa). You certainly put a brand new spin on a topic which has been discussed for ages. Excellent stuff, just great!

  37. Having read this I thought it was really informative. I appreciate you taking the time and effort to put this informative article together. I once again find myself spending way too much time both reading and commenting. But so what, it was still worth it!

  38. Hello, i think that i saw you visited my blog so i came to ìreturn the favorî.I am attempting to find things to enhance my site!I suppose its ok to use a few of your ideas!!|

  39. Next time I read a blog, Hopefully it won’t disappoint me just as much as this particular one. I mean, I know it was my choice to read, however I actually thought you would have something interesting to talk about. All I hear is a bunch of whining about something you can fix if you were not too busy seeking attention.

  40. This is really interesting, You are a very skilled blogger.
    I’ve joined your rss feed and look forward to seeking more of your wonderful post.
    Also, I have shared your site in my social networks!

  41. Achieving your fitness goals doesn’t need a certified personal trainer or an expensive gym memberships, it’s not hard to exercise at home. It is easy to go down a training and fitness rabbit hole, however, when you are looking for the best home exercise equipment to outfit your personal home gym.

  42. An outstanding share! I have just forwarded this onto a coworker who had been conducting a little research on this. And he in fact ordered me breakfast due to the fact that I found it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to talk about this topic here on your website.

  43. A motivating discussion is worth comment. I do think that you ought to write more about this subject matter, it may not be a taboo matter but usually people do not talk about these subjects. To the next! All the best!!

  44. Having read this I believed it was really informative. I appreciate you taking the time and effort to put this short article together. I once again find myself spending a significant amount of time both reading and posting comments. But so what, it was still worthwhile!

  45. I’m amazed, I must say. Seldom do I come across a blog that’s both educative and interesting, and without a doubt, you have hit the nail on the head. The issue is something too few people are speaking intelligently about. I’m very happy that I found this during my search for something regarding this.

  46. I’m very pleased to discover this site. I need to to thank
    you for ones time due to this fantastic read!!
    I definitely savored every part of it and i also have you bookmarked to look at new stuff on your site.

  47. An intriguing discussion is definitely worth comment. I do believe that you need to write more on this topic, it might not be a taboo matter but usually people do not talk about such issues. To the next! Cheers!!

  48. Hi, There’s no doubt that your blog could possibly be having internet browser compatibility problems. When I look at your web site in Safari, it looks fine however, if opening in IE, it’s got some overlapping issues. I simply wanted to provide you with a quick heads up! Besides that, excellent site!

  49. Having read this I believed it was rather informative. I appreciate you taking the time and energy to put this content together. I once again find myself personally spending a lot of time both reading and posting comments. But so what, it was still worth it!

  50. An outstanding share! I have just forwarded this onto a co-worker who had been conducting a little homework on this. And he in fact bought me lunch because I found it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanks for spending some time to talk about this subject here on your web site.

  51. Oh my goodness! Impressive article dude! Thank you, However I am encountering troubles with your RSS. I don’t know the reason why I cannot join it. Is there anybody else having similar RSS problems? Anyone that knows the answer will you kindly respond? Thanks!!

  52. I’d like to thank you for the efforts you’ve put in penning this site. I am hoping to view the same high-grade content by you later on as well. In fact, your creative writing abilities has encouraged me to get my own blog now 😉

  53. I blog frequently and I genuinely appreciate your content. Your article has really peaked my interest. I am going to book mark your site and keep checking for new information about once per week. I subscribed to your Feed as well.

  54. After looking over a few of the blog articles on your blog, I really appreciate your way of writing a blog. I added it to my bookmark site list and will be checking back in the near future. Please check out my web site as well and let me know your opinion.

  55. Right here is the right blog for anybody who wishes to find out about this topic. You know so much its almost hard to argue with you (not that I personally would want to…HaHa). You definitely put a brand new spin on a subject that’s been discussed for ages. Great stuff, just wonderful!

  56. I’ve been exploring for a bit for any high-quality articles or weblog posts in this kind of house . Exploring in Yahoo I finally stumbled upon this site. Studying this info So i am satisfied to convey that I have an incredibly just right uncanny feeling I found out exactly what I needed. I most for sure will make certain to don?t disregard this site and provides it a glance regularly.|

  57. I blog quite often and I really appreciate your content. Your article has really peaked my interest. I will take a note of your blog and keep checking for new details about once per week. I subscribed to your Feed too.

  58. Can I simply say what a relief to find an individual who really knows what they are talking about online. You actually understand how to bring a problem to light and make it important. More people need to check this out and understand this side of your story. It’s surprising you aren’t more popular since you most certainly possess the gift.

  59. This is the perfect blog for anybody who hopes to understand this topic. You know a whole lot its almost tough to argue with you (not that I personally would want to…HaHa). You certainly put a brand new spin on a subject that has been written about for ages. Wonderful stuff, just excellent!

  60. I really love your blog.. Great colors & theme. Did you create this web site yourself? Please reply back as I’m trying to create my own personal blog and would love to know where you got this from or what the theme is named. Thanks!

  61. Nice post. I learn something new and challenging on sites I stumbleupon on a daily basis. It’s always interesting to read articles from other authors and use something from their web sites.

  62. A motivating discussion is definitely worth comment. I do think that you need to publish more on this topic, it might not be a taboo subject but typically folks don’t discuss these issues. To the next! Many thanks!!

Leave a Reply

Your email address will not be published. Required fields are marked *