Update: This post was expanded into a paper, The Problem with Metrics is a Fundamental Problem for AI, by Rachel Thomas and David Uminsky, which was accepted to the Ethics of Data Science Conference 2020. The paper version includes more grounding in previous academic work and a framework towards mitigating these harms.
Goodhart’s Law states that “When a measure becomes a target, it ceases to be a good measure.” At their heart, what most current AI approaches do is to optimize metrics. The practice of optimizing metrics is not new nor unique to AI, yet AI can be particularly efficient (even too efficient!) at doing so.
This is important to understand, because any risks of optimizing metrics are heightened by AI. While metrics can be useful in their proper place, there are harms when they are unthinkingly applied. Some of the scariest instances of algorithms run amok (such as Google’s algorithm contributing to radicalizing people into white supremacy, teachers being fired by an algorithm, or essay grading software that rewards sophisticated garbage) all result from over-emphasizing metrics. We have to understand this dynamic in order to understand the urgent risks we are facing due to misuse of AI.
The following principles will be illustrated through a series of case studies:
- Any metric is just a proxy for what you really care about
- Metrics can, and will, be gamed
- Metrics tend to overemphasize short-term concerns
- Many online metrics are gathered in highly addictive environments
- Metrics are most likely to be useful when they are treated as one piece of a bigger picture
We can’t measure the things that matter most
Metrics are typically just a proxy for what we really care about. The paper Does Machine Learning Automate Moral Hazard and Error? covers an interesting example: the researchers investigate which factors in someone’s electronic medical record are most predictive of a future stroke. However, the researchers found that several of the most predictive factors (such as accidental injury, a benign breast lump, or colonoscopy) don’t make sense as risk factors for stroke. So, just what is going on? It turned out that the model was just identifying people who utilize health care a lot. They didn’t actually have data of who had a stroke (a physiological event in which regions of the brain are denied new oxygen); they had data about who had access to medical care, chose to go to a doctor, were given the needed tests, and had this billing code added to their chart. But a number of factors influence this process: who has health insurance or can afford their co-pay, who can take time off of work or find childcare, gender and racial biases that impact who gets accurate diagnoses, cultural factors, and more. As a result, the model was largely picking out people who utilized healthcare versus who did not.
This an example of the common phenomenon of having to use proxies: You want to know what content users like, so you measure what they click on. You want to know which teachers are most effective, so you measure their students test scores. You want to know about crime, so you measure arrests. These things are not the same. Many things we do care about can not be measured. Metrics can be helpful, but we can’t forget that they are just proxies.
As another example, Google used hours spent watching YouTube as a proxy for how happy users were with the content, writing on the Google blog that “If viewers are watching more YouTube, it signals to us that they’re happier with the content they’ve found.” Guillaume Chaslot, an AI engineer who formerly worked at Google/YouTube, shares how this had the side effect of incentivizing conspiracy theories, since convincing users that the rest of the media is lying kept them watching more YouTube.
Metrics can, and will, be gamed
It is almost inevitable that metrics will be gamed, particularly when they are given too much power. One week this spring, Chaslot collected 84,695 videos from YouTube and analyzed the number of views and the number of channels from which they were recommended. This is what he found (also covered in the Washington Post):
The state-owned media outlet Russia Today was an extreme outlier in how much YouTube’s algorithm had selected it to be recommended by a wide-variety of other YouTube channels. Such algorithmic selections, which begin autoplaying as soon as your current video is done, account for 70% of the time that users spend on YouTube. This chart strongly suggests that Russia Today has in some way gamed YouTube’s algorithm. (More evidence about issues with YouTube’s recommendation system is detailed here.) Platforms are rife with attempts to game their algorithms, to show up higher in search results or recommended content, through fake clicks, fake reviews, fake followers, and more.
Automatic essay grading software focuses primarily on metrics like sentence length, vocabulary, spelling, and subject-verb agreement, but is unable to evaluate aspects of writing that are hard to quantify, such as creativity. As a result, gibberish essays randomly generated by computer programs to contain lots of sophisticated words score well. Essays from students in mainland China, which do well on essay length and sophisticated word choice, received higher scores from the algorithms than from expert human graders, suggesting that these students may be using chunks of pre-memorized text.
As USA education policy began over-emphasizing student test scores as the primary way to evaluate teachers, there have been widespread scandals of teachers and principals cheating by altering students scores, in Georgia, Indiana, Massachusetts, Nevada, Virginia, Texas, and elsewhere. One consequence of this is that teachers who don’t cheat may be penalized or even fired (when it appears student test scores have dropped to more average levels under their instruction). When metrics are given undue importance, attempts to game those metrics become common.
Metrics tend to overemphasize short-term concerns
It is much easier to measure short-term quantities: click through rates, month-over-month churn, quarterly earnings. Many long-term trends have a complex mix of factors and are tougher to quantify. What is the long-term impact on user trust of having your brand associated with promoting pedophilia, white supremacy, and flat-earth theories? What is the long-term impact on hiring to be the subject of years worth of privacy scandals, political manipulation, and facilitating genocide?
Simply measuring what users click on is a short-term concern, and does not take into account factors like the potential long-term impact of a long-form investigative article which may have taken months to research and which could help shape a reader’s understanding of a complex issue and even lead to significant societal changes.
A recent Harvard Business Review article looked at Wells Fargo as a case study of how letting metrics replace strategy can harm a business. After identifying cross-selling as a measure of long-term customer relationships, Wells Fargo went overboard emphasizing the cross-selling metric: intense pressure on employees combined with an unethical sales culture led to 3.5 million fraudulent deposit and credit card accounts being opened without customers’ consent. The metric of cross-selling is a much more short-term concern compared to the loftier goal of nurturing long-term customer relationships. Overemphasizing metrics removes our focus from long-term concerns such as our values, trust and reputation, and our impact on society and the environment, and myopically focuses on the short-term.
Many metrics gather data of what we do in highly addictive environments
It matters which metrics we gather and in what environment we do so. Metrics such as what users click on, how much time they spend on sites, and “engagement” are heavily relied on by tech companies as proxies for user preference, and are used to drive important business decisions. Unfortunately, these metrics are gathered in environments engineered to be highly addictive, laden with dark patterns, and where financial and design decisions have already greatly circumscribed the range of options.
Zeynep Tufekci, a professor at UNC and regular contributor to the New York Times, compares recommendation algorithms (such as YouTube choosing which videos to auto-play for you and Facebook deciding what to put at the top of your newsfeed) to a cafeteria shoving junk food into children’s faces. “This is a bit like an autopilot cafeteria in a school that has figured out children have sweet teeth, and also like fatty and salty foods. So you make a line offering such food, automatically loading the next plate as soon as the bag of chips or candy in front of the young person has been consumed.” As those selections get normalized, the output becomes ever more extreme: “So the food gets higher and higher in sugar, fat and salt – natural human cravings – while the videos recommended and auto-played by YouTube get more and more bizarre or hateful.” Too many of our online environments are like this, with metrics capturing that we love sugar, fat, and salt, not taking into account that we are in the digital equivalent of a food desert and that companies haven’t been required to put nutrition labels on what they are offering. Such metrics are not indicative of what we would prefer in a healthier or more empowering environment.
When Metrics are Useful
All this is not to say that we should throw metrics out altogether. Data can be valuable in helping us understand the world, test hypotheses, and move beyond gut instincts or hunches. Metrics can be useful when they are in their proper context and place. One way to keep metrics in their place is to consider a slate of many metrics for a fuller picture (and resist the temptation to try to boil these down to a single score). For instance, knowing the rates at which tech companies hire people from under-indexed groups is a very limited data point. For evaluating diversity and inclusion at tech companies, we need to know comparative promotion rates, cap table ownership, retention rates (many tech companies are revolving doors driving people from under-indexed groups away with their toxic cultures), number of harassment victims silenced by NDAs, rates of under-leveling, and more. Even then, all this data should still be combined with listening to first-person experiences of those working at these companies.
Columbia professor and New York Times Chief Data Scientist Chris Wiggins wrote that quantitative measures should always be combined with qualitative information, “Since we can not know in advance every phenomenon users will experience, we can not know in advance what metrics will quantify these phenomena. To that end, data scientists and machine learning engineers must partner with or learn the skills of user experience research, giving users a voice.”
Another key to keeping metrics in their proper place is to keep domain experts and those who will be most impacted closely involved in their development and use. Surely most teachers could have foreseen that evaluating teachers primarily on the standardized test scores of their students would lead to a host of negative consequences.
I am not opposed to metrics; I am alarmed about the harms caused when metrics are overemphasized, a phenomenon that we see frequently with AI, and which is having a negative, real-world impact. AI running unchecked to optimize metrics has led to Google/YouTube’s heavy promotion of white supremacist material, essay grading software that rewards garbage, and more. By keeping the risks of metrics in mind, we can try to prevent these harms.
Original post: https://www.fast.ai/2019/09/24/metrics/
Oh my goodness! Impressive article dude! Thanks, However I am experiencing difficulties with your RSS.
I don’t know why I am unable to subscribe to it.
Is there anyone else having similar RSS problems? Anyone that knows the answer can you kindly respond?
Thanx!!
Hmm is anyone else experiencing problems with the
images on this blog loading? I’m trying to find out if its a problem on my
end or if it’s the blog. Any suggestions would be greatly appreciated.
That is very attention-grabbing, You are an excessively professional blogger.
I have joined your feed and stay up for in search of extra
of your magnificent post. Also, I’ve shared your web site in my social
networks
I love your blog.. very nice colors & theme. Did you make this website yourself or did you hire someone
to do it for you? Plz reply as I’m looking
to construct my own blog and would like to know where u got this from.
cheers
This design is spectacular! You certainly know how to keep a reader entertained.
Between your wit and your videos, I was almost moved to start my own blog (well,
almost…HaHa!) Wonderful job. I really enjoyed
what you had to say, and more than that, how you presented it.
Too cool!
Good day! Do you use Twitter? I’d like to follow you if that would be ok.
I’m undoubtedly enjoying your blog and look forward to new posts.
Pretty section of content. I just stumbled upon your blog and
in accession capital to assert that I acquire actually enjoyed account your blog posts.
Any way I’ll be subscribing to your feeds and even I achievement you access
consistently fast.
Hello, i believe that i noticed you visited
my website so i came to go back the want?.I’m attempting to in finding issues to enhance my
website!I suppose its good enough to use a few of your concepts!!
Good day! I know this is kinda off topic however I’d figured I’d ask.
Would you be interested in exchanging links or maybe guest writing a blog
article or vice-versa? My blog addresses a lot of the same topics as
yours and I feel we could greatly benefit from each other.
If you happen to be interested feel free to send me an email.
I look forward to hearing from you! Superb blog by the way!
Currently it seems like BlogEngine is the best blogging platform
available right now. (from what I’ve read) Is that what you’re
using on your blog?
I am the owner of JustCBD brand (justcbdstore.com) and I’m presently looking to grow my wholesale side of business. I am hoping someone at targetdomain give me some advice ! I thought that the best way to do this would be to connect to vape shops and cbd retailers. I was really hoping if anybody could suggest a qualified website where I can get Vape Shop B2B Database I am currently looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the best option and would appreciate any advice on this. Or would it be simpler for me to scrape my own leads? Suggestions?
I am the manager of JustCBD label (justcbdstore.com) and I am currently trying to broaden my wholesale side of business. I am hoping someone at targetdomain share some guidance . I considered that the most suitable way to do this would be to reach out to vape shops and cbd stores. I was hoping if anybody at all could suggest a trustworthy website where I can buy CBD Shops B2B Companies I am presently reviewing creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most ideal solution and would appreciate any support on this. Or would it be easier for me to scrape my own leads? Ideas?
I’m impressed, I have to admit. Seldom do I encounter a blog that’s equally educative and engaging, and let me tell you, you’ve hit the nail on the head. The issue is something too few men and women are speaking intelligently about. Now i’m very happy I stumbled across this during my hunt for something regarding this.
Everything is very open with a very clear clarification of the issues.
It was really informative. Your site is extremely helpful.
Many thanks for sharing!
Having read this I thought it was rather informative. I appreciate you spending some time and energy to put this article together. I once again find myself personally spending a significant amount of time both reading and leaving comments. But so what, it was still worth it!
Thanks for sharing, this is a fantastic blog post.Thanks Again. Really Great.
This blog was… how do you say it? Relevant!! Finally I’ve found something which helped me. Many thanks!
This website was… how do I say it? Relevant!! Finally I’ve found something that helped me. Thanks!
Great article.
I quite like reading through a post that can make people think. Also, thank you for allowing me to comment!
It’s hard to find educated people for this topic, but you sound like you know what you’re talking about! Thanks
A motivating discussion is definitely worth comment. I do think that you should write more on this topic, it may not be a taboo matter but typically people do not discuss these issues. To the next! Cheers!!
That is a good tip especially to those new to the blogosphere. Brief but very precise information… Appreciate your sharing this one. A must read article!
I think this is one of the most important information for me.
And i am glad reading your article. But should remark on few general
things, The website style is ideal, the articles is really
excellent : D. Good job, cheers
Greetings! Very useful advice in this particular post! It is the little changes that make the biggest changes. Thanks a lot for sharing!
You are so awesome! I don’t suppose I have read a single thing like this before. So great to discover someone with original thoughts on this topic. Really.. thank you for starting this up. This web site is one thing that is required on the web, someone with a little originality!
Hello! I just would like to offer you a huge thumbs up for your excellent information you’ve got right here on this post. I will be coming back to your web site for more soon.
I was suggested this blog by my cousin. I am not
sure whether this publish is written by means of
him as nobody else recognize such specified approximately
my problem. You are amazing! Thanks!
Everything is very open with a precise description of the challenges. It was truly informative. Your site is very helpful. Thank you for sharing!
This is a topic that’s close to my heart… Best wishes! Exactly where are your contact details though?
Having read this I thought it was very informative. I appreciate you finding the time and energy to put this information together. I once again find myself personally spending a lot of time both reading and commenting. But so what, it was still worthwhile!
I have been surfing online more than three hours today, yet I never found any interesting article like yours.
It’s pretty worth enough for me. Personally, if all website owners and bloggers
made good content as you did, the web will be a lot more useful than ever
before.
This is the perfect site for anyone who hopes to find out about this topic. You understand so much its almost hard to argue with you (not that I actually will need to…HaHa). You definitely put a new spin on a topic that has been discussed for ages. Excellent stuff, just wonderful!
May I just say what a comfort to find a person that really knows what they are discussing over the internet. You definitely know how to bring an issue to light and make it important. More and more people should read this and understand this side of the story. It’s surprising you are not more popular because you most certainly have the gift.
Good post. I learn something totally new and challenging on sites I stumbleupon everyday. It will always be exciting to read through content from other authors and practice a little something from their websites.
Good post. I learn something new and challenging on blogs I stumbleupon on a daily basis. It’s always useful to read through articles from other writers and practice a little something from other websites.
Excellent post! We will be linking to this great article on our site. Keep up the good writing.
Pretty! This was an extremely wonderful post. Thank you for providing these details.
Good post. I learn something totally new and challenging on blogs I stumbleupon everyday. It will always be exciting to read through content from other authors and practice a little something from their web sites.
Next time I read a blog, I hope that it does not fail me just as much as this one. I mean, I know it was my choice to read, nonetheless I genuinely thought you’d have something interesting to say. All I hear is a bunch of whining about something you could fix if you were not too busy seeking attention.
The very next time I read a blog, I hope that it doesn’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read through, nonetheless I genuinely believed you’d have something useful to talk about. All I hear is a bunch of crying about something that you could fix if you weren’t too busy seeking attention.
I love your blog.. very nice colors & theme. Did you design this
website yourself or did you hire someone to do it for you?
Plz answer back as I’m looking to design my own blog and would like to know where
u got this from. appreciate it
I like it when people get together and share ideas. Great website, keep it up!
Greetings! Very helpful advice within this article! It is the little changes which will make the most important changes. Thanks for sharing!
Nice post. I learn something new and challenging on websites I stumbleupon every day.
It will always be useful to read through content from other authors and practice something from their websites.
This is the right website for anyone who wants to understand this topic. You know so much its almost hard to argue with you (not that I actually will need to…HaHa). You certainly put a fresh spin on a topic that’s been discussed for a long time. Great stuff, just excellent!
An interesting discussion is worth comment. I do think that you ought to publish more about this subject matter, it might not be a taboo subject but typically folks don’t discuss such subjects. To the next! Best wishes!!
You really make it seem so easy with your presentation but I find this topic to be actually something which I think I
would never understand. It seems too complex and very broad for me.
I am looking forward for your next post, I’ll try to get the hang
of it!
No matter if some one searches for his required thing, therefore
he/she needs to be available that in detail, therefore that thing is maintained over here.
I blog quite often and I really thank you for your content. This great article has truly peaked my interest. I’m going to take a note of your blog and keep checking for new information about once per week. I opted in for your Feed as well.
Oh my goodness! Amazing article dude! Thank you, However I am going through problems with your RSS. I don’t know the reason why I cannot join it. Is there anybody getting identical RSS problems? Anyone who knows the solution can you kindly respond? Thanks!!
WOW just what I was searching for. Came here by searching for web host
Hello, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot
of spam comments? If so how do you protect against it, any plugin or anything you can suggest?
I get so much lately it’s driving me crazy so any help is very much appreciated.
Hi! I realize this is sort of off-topic but I had to ask.
Does operating a well-established blog such as yours require a massive amount work?
I am completely new to writing a blog but I do write in my journal
on a daily basis. I’d like to start a blog so I can easily share my personal experience and
feelings online. Please let me know if you have any suggestions or tips for new aspiring bloggers.
Appreciate it! adreamoftrains best hosting
Great web site you’ve got here.. It’s hard to find high-quality writing like yours nowadays. I seriously appreciate people like you! Take care!!
Today, I went to the beachfront with my children. I found a sea shell and gave it to my 4 year old daughter
and said “You can hear the ocean if you put this to your ear.” She placed the shell to her
ear and screamed. There was a hermit crab inside and it pinched her
ear. She never wants to go back! LoL I know this
is completely off topic but I had to tell someone!
Spot on with this write-up, I really believe this site needs far more attention. I’ll probably be back again to read more, thanks for the info!
Saved as a favorite, I really like your web site!
Aw, this was an extremely nice post. Taking a few minutes and actual effort to generate a great article… but what can I say… I procrastinate a whole lot and never seem to get nearly anything done.
There’s certainly a lot to learn about this topic. I really like all the points you have made.
Aw, this was a very nice post. Taking the time and actual effort to produce a very good article… but what can I say… I procrastinate a whole lot and never manage to get nearly anything done.
Your style is very unique in comparison to other people I have read stuff from.
I appreciate you for posting when you’ve got the opportunity,
Guess I will just book mark this site.
It’s really a nice and useful piece of info. I’m satisfied
that you just shared this helpful info with us.
Please keep us informed like this. Thank you for sharing.
Very good post. I will be facing some of these issues as well..
Pretty! This was an extremely wonderful article. Thank you for supplying this info.
You are so cool! I don’t believe I’ve truly read a single thing like this before. So nice to find another person with some original thoughts on this subject. Really.. thank you for starting this up. This website is one thing that’s needed on the internet, someone with a bit of originality!
A fascinating discussion is worth comment. I do believe that you ought to write more on this subject, it may not be a taboo matter but typically folks don’t discuss these issues. To the next! Best wishes!!
May I just say what a comfort to uncover someone that truly knows what they are talking about online. You definitely realize how to bring an issue to light and make it important. A lot more people have to read this and understand this side of the story. I can’t believe you are not more popular given that you definitely have the gift.
Can I just say what a relief to discover somebody that actually understands what they’re talking about on the net. You actually understand how to bring a problem to light and make it important. A lot more people ought to read this and understand this side of the story. It’s surprising you aren’t more popular because you surely have the gift.
Hi there! This post could not be written any better! Looking through this post reminds me of my previous roommate! He always kept talking about this. I will forward this article to him. Fairly certain he’s going to have a good read. Thank you for sharing!
bookmarked!!, I like your website!
This site was… how do I say it? Relevant!! Finally I have found something that helped me. Appreciate it!
Aw, this was an extremely nice post. Finding the time and actual effort to create a superb article… but what can I say… I put things off a whole lot and don’t seem to get anything done.
The next time I read a blog, Hopefully it does not disappoint me just as much as this one. I mean, Yes, it was my choice to read through, however I genuinely believed you would have something helpful to say. All I hear is a bunch of complaining about something you can fix if you weren’t too busy looking for attention.
Great blog you’ve got here.. It’s difficult to find high-quality writing like yours these days. I seriously appreciate individuals like you! Take care!!
You have made some good points there. I checked on the internet for more info about the issue and found most individuals will go along with your views on this web site.
When I originally left a comment I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on each time a comment is added I recieve 4 emails with the exact same comment. There has to be an easy method you are able to remove me from that service? Thank you!
This is a topic which is close to my heart… Cheers! Exactly where are your contact details though?
Your style is so unique in comparison to other people I have read stuff from. I appreciate you for posting when you’ve got the opportunity, Guess I’ll just bookmark this page.
Your style is unique in comparison to other people I have read stuff from. I appreciate you for posting when you’ve got the opportunity, Guess I will just book mark this page.
Hi, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam feedback? If so how do you reduce it, any plugin or anything you can recommend? I get so much lately it’s driving me insane so any support is very much appreciated.|
An interesting discussion is worth comment. I do believe that you should publish more about this subject, it might not be a taboo matter but usually people don’t discuss such subjects. To the next! Many thanks!!
I absolutely love your site.. Great colors & theme. Did you make this website yourself? Please reply back as I’m planning to create my very own blog and would love to find out where you got this from or exactly what the theme is named. Appreciate it!
May I just say what a comfort to discover someone that actually knows what they are discussing on the net. You definitely realize how to bring an issue to light and make it important. A lot more people have to check this out and understand this side of your story. I was surprised that you aren’t more popular given that you certainly have the gift.
Hi there! This article could not be written any better! Reading through this post reminds me of my previous roommate! He continually kept preaching about this. I most certainly will forward this post to him. Pretty sure he will have a very good read. I appreciate you for sharing!
After looking into a few of the blog posts on your site, I truly like your way of blogging. I bookmarked it to my bookmark webpage list and will be checking back soon. Please check out my web site as well and let me know your opinion.
Way cool! Some extremely valid points! I appreciate you writing this article and the rest of the website is also very good.
Achieving your fitness goal doesn’t have to require a certified personal trainer or an expensive gym memberships, especially when you have the budget and the space to consider practically every workout machine on the market.
It’s very trouble-free to find out any matter on web as compared to textbooks,
as I found this piece of writing at this web page.
Great weblog right here! Also your web site rather a lot up very fast!
What web host are you the use of? Can I am getting your associate link
in your host? I want my website loaded up as quickly as
yours lol 3gqLYTc cheap flights
Thank you for some other wonderful article. Where else may just
anyone get that kind of information in such a
perfect method of writing? I’ve a presentation next
week, and I’m on the search for such information. cheap flights 32hvAj4
Saved as a favorite, I really like your site!
It’s nearly impossible to find experienced people about this topic, however, you seem like you know what you’re talking about! Thanks
Good post! We will be linking to this great post on our site. Keep up the good writing.
Great article. I am facing some of these issues as well..
This website truly has all of the information and facts I wanted concerning this subject and didn’t know who to ask.
This is a topic that’s close to my heart… Cheers! Exactly where are your contact details though?
I’m amazed, I must say. Rarely do I come across a blog that’s equally educative and entertaining, and let me tell you, you have hit the nail on the head. The issue is something which too few folks are speaking intelligently about. I’m very happy I stumbled across this during my search for something regarding this.
Saved as a favorite, I like your website!
You’re so awesome! I don’t suppose I’ve truly read through anything like that before. So wonderful to discover somebody with some unique thoughts on this subject matter. Really.. thanks for starting this up. This site is one thing that’s needed on the web, someone with some originality!
I like it when folks get together and share views. Great blog, stick with it!
The next time I read a blog, I hope that it won’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read, nonetheless I actually believed you would have something useful to say. All I hear is a bunch of whining about something that you can fix if you were not too busy looking for attention.
Good blog you’ve got here.. It’s difficult to find high-quality writing like yours these days. I truly appreciate people like you! Take care!!
I blog quite often and I genuinely appreciate your content. Your article has truly peaked my interest. I will bookmark your website and keep checking for new details about once per week. I subscribed to your RSS feed too.
I like it when folks get together and share thoughts. Great website, continue the good work!
Hi, I do believe this is an excellent blog. I stumbledupon it 😉 I’m going to return yet again since I bookmarked it. Money and freedom is the greatest way to change, may you be rich and continue to guide other people.
An outstanding share! I have just forwarded this onto a co-worker who was conducting a little research on this. And he actually ordered me breakfast because I stumbled upon it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to talk about this matter here on your website.
Hi, I think your website could be having internet browser compatibility problems. When I take a look at your blog in Safari, it looks fine however, when opening in IE, it has some overlapping issues. I merely wanted to give you a quick heads up! Aside from that, fantastic blog!
I need to to thank you for this good read!! I certainly enjoyed every little bit of it. I have got you saved as a favorite to look at new stuff you post…
Greetings, There’s no doubt that your blog could possibly be having web browser compatibility issues. Whenever I look at your website in Safari, it looks fine however, if opening in IE, it has some overlapping issues. I just wanted to give you a quick heads up! Apart from that, wonderful site!
I seriously love your site.. Very nice colors & theme. Did you create this web site yourself? Please reply back as I’m attempting to create my own website and would love to know where you got this from or just what the theme is called. Thank you!
Hi there! This article could not be written much better! Reading through this post reminds me of my previous roommate! He continually kept preaching about this. I most certainly will forward this information to him. Fairly certain he’ll have a great read. Many thanks for sharing!
There is certainly a lot to find out about this subject. I like all of the points you made.
I blog often and I seriously appreciate your information. This great article has truly peaked my interest. I am going to bookmark your blog and keep checking for new information about once a week. I opted in for your Feed as well.
I just want to say I am beginner to blogs and truly loved this web blog. More than likely I’m going to bookmark your blog . You actually come with very good articles. Bless you for sharing with us your webpage.
Thanks a lot for the article post.
Thanks for an concept, you sparked at thought from a angle I hadn’t given thoguht to yet. Now lets see if I can do one thing with it.