In this special guest feature, Wilson Pang, CTO of Appen, offers a few quality controls that organizations can implement to allow for the most accurate and consistent data annotation process possible. Wilson joined Appen in November 2018 and is responsible for the company’s products and technology. Wilson has over seventeen years’ experience in software engineering and data science. Prior to joining Appen, Wilson was Chief Data Officer of CTrip in China, the second largest online travel agency company in the world where he led data engineers, analysts, data product managers and scientists to improve user experience and increase operational efficiency that grew the business. Before that, he was senior director of engineering in eBay in California and provided leadership to various domains including data service and solutions, search science, marketing technology and billing systems. Wilson obtained his Masters and Bachelor’s degrees of Electric Engineering from Zhejiang University in China.
Using poor-quality data to train your machine learning system is like preparing for a physics test by studying geometry. You’ll learn something, but your efforts probably won’t help you answer your test questions correctly. For example, if you train a computer vision system for autonomous vehicles with images of sidewalks mislabeled as streets, the results could be disastrous. In order to develop accurate algorithms, you will need high-quality training data. To generate high-quality data, you’ll need skilled annotators to carefully label the information you plan to use with your algorithm.
When we talk about quality training data, we’re talking about both the accuracy and consistency of those labels. Accuracy is how close a label is to the truth. Consistency is the degree to which multiple annotations on various training items agree with one another.
Here are a few quality controls that organizations can implement to allow for the most accurate and consistent data annotation process possible.
Standard Quality-Assurance Methods Offer a Baseline
Typically, organizations that are creating high-quality training data sets use three standard methods for ensuring accuracy and consistency: gold sets, consensus, and auditing.
- Gold sets, or benchmarks, measure accuracy by comparing annotations (or annotators) to a “gold set” or vetted example. This helps to measure how well a set of annotations from a group or individual matches the benchmark.
- Consensus, or overlap, measures consistency and agreement amongst a group, and does so by dividing the sum of agreeing data annotations by the total number of annotations. This is the most common method of quality control for projects with relatively objective rating scales. The goal is to arrive at a consensus decision for each item. Any disagreement amongst the overlapped judgments is typically arbitrated by an auditor.
- Auditing measures both accuracy and consistency by having an expert review the labels, either by spot-checking or reviewing them all. This method is important for projects where arriving at a consensus judgment may not be feasible—tasks such as transcription, where auditors review and rework the content until it reaches the highest levels or accuracy.
Detailed Controls Can Provide In-Depth Quality Assessment
These baseline quality measurements are a solid foundation for monitoring the quality of data annotations. However, no two AI projects are alike, and organizations should establish quality assessments more tailored to a specific initiative. Organizational leaders responsible for AI initiatives can create in-depth quality controls by considering the following processes:
- Multiple quality measurement metrics: Leverage multiple methods of quality measurement discussed earlier.
- Weekly data deep-dives: Implement a project management team to investigate data weekly and set stretch productivity and quality goals. For example, if you require data that is accurate 92% of the time, set a stretch goal of 95% and try to ensure your annotation process exceeds your initial goal.
- Management testing and auditing: To build up your project managers’ quality-assurance skill set, ask them to complete perform annotation work and quality audits so that they have first-hand experience of the annotation process. This gives the management team a 360-degree view of the project and a full understanding of the annotators’ process.
High-Quality Training Data Helps Mitigate Bias
High-quality training data ensures more accurate algorithms, and it can also help mitigate the potential bias in many AI projects. Bias can manifest as uneven voice or facial recognition performance for different genders, accents, or ethnicities. Fighting bias during your data annotation process is another way to infuse your training data set with quality.
To avoid bias at the project level, organizations should actively build diversity into the data teams defining goals, roadmaps, metrics, and algorithms. Hiring a diverse team of data talent is easier said than done, but if the makeup of your team isn’t representative of the population the algorithm will impact, then the end product risks only working for, or appealing to, a subset of people, or discriminating against certain subsets of the population.
It’s important to also account for bias at the data level. Here are considerations to help you mitigate bias in training data:
- When internal team members label the data, they will always add some bias because they have expectations about what their system should conclude. If you decide to use an internal team, consult an outside source to help foster an objective annotation environment.
- Find or create a representative training dataset. Quantity always helps, especially if you’re using data from internal systems. Try to find the most comprehensive data, and experiment with different datasets, metrics, and segmentation to ensure you’ve covered the bases.
- If you’re engineering or annotating data, take care to design the instructions and tasks for your annotators in ways that don’t bias them from the outset. It’s important that annotators have enough instruction to correctly perform the task, but not know what the data will be used for, which can bias behavior.
- Check for implicit bias in the data as part of your quality-assurance process.
- Once your product is live, monitor performance using the data it generates to determine whether it’s delivering equitable opportunities and outcomes for all users.
High-Quality Data Is an AI Keystone
High-quality training data is necessary for a successful AI initiative. And, while the quality-assurance processes are numerous, they are a vital component of your AI initiative. Quality training data not only begets algorithms that work in the real world, it also helps mitigate some of the bias inherent in manual data annotations. Before you begin to launch your AI initiative, develop data quality assurance practices to realize the best return on your investment.
Original post: https://insidebigdata.com/2019/11/17/how-to-ensure-data-quality-for-ai/
This article is truly a pleasant one it helps new web users, who are wishing in favor of blogging.
First of all I would like to say wonderful blog!
I had a quick question that I’d like to ask if you do not mind.
I was interested to know how you center yourself and clear your
thoughts prior to writing. I’ve had trouble clearing my thoughts in getting my ideas out.
I truly do enjoy writing but it just seems like the first 10 to 15 minutes are wasted just trying to figure
out how to begin. Any suggestions or hints? Many thanks!
Nice post. I learn something totally new and challenging
on blogs I stumbleupon on a daily basis. It will always be helpful
to read through articles from other writers and use something
from their web sites.
Spot on with this write-up, I actually feel this web site needs
much more attention. I’ll probably be back again to read more,
thanks for the advice!
Good day! I know this is somewhat off topic but I was wondering which blog platform are you using for this website?
I’m getting sick and tired of WordPress because I’ve had problems with hackers and I’m
looking at alternatives for another platform.
I would be fantastic if you could point me in the direction of a good
platform.
I have read so many content regarding the blogger
lovers except this paragraph is genuinely a fastidious post,
keep it up.
My relatives all the time say that I am wasting my time
here at net, but I know I am getting familiarity all the time by reading such fastidious content.
You ought to be a part of a contest for one of the highest quality websites online.
I will highly recommend this web site!
Highly energetic article, I loved that a lot. Will there be a part 2?
obviously like your website however you have to test the
spelling on quite a few of your posts. Several of them are rife with spelling problems and I find it very troublesome to inform the truth then again I’ll definitely come again again.
I am the co-founder of JustCBD company (justcbdstore.com) and I’m presently aiming to expand my wholesale side of company. I really hope that someone at targetdomain can help me ! I thought that the best way to do this would be to reach out to vape shops and cbd retail stores. I was hoping if someone could suggest a dependable website where I can get Vape Shop Business Sales Leads I am already taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most ideal choice and would appreciate any assistance on this. Or would it be easier for me to scrape my own leads? Ideas?
Currently it sounds like Movable Type is the best blogging platform
available right now. (from what I’ve read) Is that what you’re using on your blog?
I am the proprietor of JustCBD Store label (justcbdstore.com) and I am currently planning to expand my wholesale side of business. I am hoping someone at targetdomain can help me 🙂 I considered that the best way to do this would be to talk to vape companies and cbd retail stores. I was really hoping if anyone could recommend a reputable website where I can buy CBD Shops Business Contact List I am currently considering creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the most ideal option and would appreciate any guidance on this. Or would it be easier for me to scrape my own leads? Suggestions?
There is definately a great deal to find out about this issue. I like all the points you’ve made.
Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
You clearly know what youre talking about, why throw away
your intelligence on just posting videos to your site when you
could be giving us something enlightening to read?
Way cool! Some extremely valid points! I appreciate you writing this post plus the rest of the site is really good.
Having read this I thought it was really informative. I appreciate you finding the time and effort to put this short article together. I once again find myself personally spending a significant amount of time both reading and commenting. But so what, it was still worthwhile!
I must thank you for the efforts you’ve put in penning this site. I am hoping to see the same high-grade content from you in the future as well. In truth, your creative writing abilities has encouraged me to get my very own blog now 😉
I’m gone to inform my little brother, that he should also visit
this blog on regular basis to get updated from newest gossip.
Good post. I learn something new and challenging on blogs I stumbleupon everyday. It will always be useful to read through articles from other authors and practice something from other sites.
I would like to thank you for the efforts you’ve put in writing this blog. I am hoping to check out the same high-grade blog posts from you in the future as well. In truth, your creative writing abilities has motivated me to get my own website now 😉
Saved as a favorite, I really like your blog!
Your style is really unique in comparison to other people I’ve read stuff from. I appreciate you for posting when you have the opportunity, Guess I will just book mark this blog.
Good day I am so glad I found your site, I really found
you by error, while I was searching on Bing for something else,
Anyhow I am here now and would just like to say cheers for a
remarkable post and a all round interesting blog (I also love the
theme/design), I don’t have time to read through it
all at the moment but I have bookmarked it and also included your RSS feeds, so when I have time
I will be back to read much more, Please do keep up the great jo.
There’s definately a lot to know about this issue. I like all the points you’ve made.
I really love your website.. Great colors & theme. Did you build this website yourself? Please reply back as I’m planning to create my own site and want to learn where you got this from or just what the theme is named. Many thanks!
Hello! I could have sworn I’ve been to this blog before but after looking at many of the posts I realized it’s new to me. Regardless, I’m certainly delighted I found it and I’ll be bookmarking it and checking back regularly!
Excellent post. I will be going through some of these issues as well..
An impressive share! I’ve just forwarded this onto a colleague who was conducting a little research on this. And he in fact bought me lunch because I discovered it for him… lol. So allow me to reword this…. Thanks for the meal!! But yeah, thanks for spending time to discuss this topic here on your internet site.
Hi! I could have sworn I’ve visited this site before but after looking at some of the articles I realized it’s new to me. Anyhow, I’m definitely delighted I found it and I’ll be book-marking it and checking back regularly!
An interesting discussion is worth comment. I think that you should publish more about this topic, it might not be a taboo subject but generally people do not speak about such subjects. To the next! Best wishes!!
You’ve made some decent points there. I looked on the web to find out more about the issue and found most people will go along with your views on this website.
I’m impressed, I must say. Seldom do I come across a blog that’s both equally educative and interesting, and let me tell you, you’ve hit the nail on the head. The problem is something not enough folks are speaking intelligently about. I am very happy that I found this in my search for something concerning this.
This blog was… how do I say it? Relevant!! Finally I’ve found something which helped me. Thank you!
I really love your blog.. Excellent colors & theme. Did you make this amazing site yourself? Please reply back as I’m trying to create my own personal blog and would like to learn where you got this from or just what the theme is named. Kudos!
It’s nearly impossible to find educated people for this topic, however, you sound like you know what you’re talking about! Thanks
Hi there! I simply want to offer you a huge thumbs up for your great info you have got right here on this post. I am returning to your blog for more soon.
Everything is very open with a precise explanation of the challenges. It was definitely informative. Your website is very useful. Thanks for sharing!
Simply want to say your article is as amazing.
The clearness in your publish is simply cool and that i
can assume you’re knowledgeable in this subject.
Well with your permission allow me to take hold of your RSS feed to stay up to
date with coming near near post. Thank you one million and please carry on the
gratifying work.
Hi! I know this is kinda off topic but I was wondering which blog platform are you using for this website?
I’m getting tired of WordPress because I’ve had issues with hackers and I’m
looking at alternatives for another platform.
I would be awesome if you could point me in the direction of
a good platform.
Oh my goodness! Amazing article dude! Thank you so much, However I am going through difficulties with your RSS. I don’t know the reason why I can’t join it. Is there anyone else having similar RSS problems? Anyone who knows the answer will you kindly respond? Thanks!!
I’m impressed, I must say. Seldom do I come across a blog that’s both educative and amusing, and let me tell you, you have hit the nail on the head. The issue is something which too few folks are speaking intelligently about. Now i’m very happy that I found this during my hunt for something regarding this.
Way cool! Some extremely valid points! I appreciate you penning this article plus the rest of the website is also very good.
Can I just say what a relief to discover a person that really understands what they’re discussing on the internet. You certainly know how to bring a problem to light and make it important. A lot more people need to read this and understand this side of your story. I was surprised that you’re not more popular since you most certainly possess the gift.
I for all time emailed this webpage post page to all my associates,
for the reason that if like to read it next my contacts will too.
Nice post. I was checking continuously this blog and I am
impressed! Very useful information specially the last part 🙂 I care
for such information much. I was looking for this certain info for a very long time.
Thank you and good luck.
Everything is very open with a precise description of the challenges. It was definitely informative. Your site is very helpful. Thanks for sharing!
Good post. I’m experiencing a few of these issues as well..
Hello this is kind of of off topic but I was
wanting to know if blogs use WYSIWYG editors or if you have to manually code with HTML.
I’m starting a blog soon but have no coding know-how so I
wanted to get advice from someone with experience.
Any help would be enormously appreciated!
That is a really good tip particularly to those new to the blogosphere. Simple but very accurate info… Thank you for sharing this one. A must read post!
This blog was… how do I say it? Relevant!! Finally I’ve found something which helped me. Many thanks!
I want to to thank you for this great read!! I absolutely enjoyed every bit of it. I have got you saved as a favorite to check out new things you post…
I love reading through an article that will make people think. Also, many thanks for allowing for me to comment!
After I originally left a comment I appear to have clicked on the -Notify me when new comments are added- checkbox and from now on every time a comment is added I get four emails with the exact same comment. Perhaps there is a means you are able to remove me from that service? Appreciate it!
You made some really good points there. I checked on the internet for additional information about the issue and found most individuals will go along with your views on this website.
This is a topic that is close to my heart… Many thanks! Exactly where are your contact details though?
This excellent website truly has all of the info I wanted concerning this subject and didn’t know who to ask.
I blog often and I truly thank you for your information. Your article has really peaked my interest. I will bookmark your blog and keep checking for new information about once per week. I subscribed to your Feed too.
I’m impressed, I must say. Seldom do I encounter a blog that’s both educative and amusing, and let me tell you, you have hit the nail on the head. The problem is something not enough people are speaking intelligently about. I’m very happy that I came across this during my hunt for something concerning this.
Good information. Lucky me I found your website by accident (stumbleupon). I’ve bookmarked it for later!
Very good post! We are linking to this great post on our website. Keep up the great writing.
An outstanding share! I have just forwarded this onto a coworker who was doing a little homework on this. And he in fact ordered me lunch because I stumbled upon it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending some time to talk about this subject here on your internet site.
I must thank you for the efforts you’ve put in penning this blog. I am hoping to see the same high-grade blog posts by you in the future as well. In fact, your creative writing abilities has encouraged me to get my own blog now 😉
When I initially commented I seem to have clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I recieve 4 emails with the same comment. There has to be an easy method you can remove me from that service? Thanks!
Spot on with this write-up, I really believe that this website needs far more attention. I’ll probably be returning to see more, thanks for the info!
You ought to take part in a contest for one of the greatest websites on the internet. I most certainly will recommend this website!
This is the right blog for anyone who would like to find out about this topic. You realize so much its almost hard to argue with you (not that I actually will need to…HaHa). You definitely put a fresh spin on a subject that has been written about for decades. Wonderful stuff, just wonderful!
Hi, I think your web site might be having web browser compatibility issues. Whenever I look at your blog in Safari, it looks fine but when opening in IE, it’s got some overlapping issues. I merely wanted to give you a quick heads up! Aside from that, great website!
I wanted to thank you for this great read!! I absolutely enjoyed every bit of it. I’ve got you saved as a favorite to look at new stuff you post…
Good article! We will be linking to this great article on our website. Keep up the good writing.
This is a great tip especially to those fresh to the blogosphere. Brief but very accurate info… Many thanks for sharing this one. A must read post!
This site certainly has all of the information and facts I wanted concerning this subject and didn’t know who to ask.
Hi there! I simply would like to give you a big thumbs up for the great info you have got right here on this post. I will be coming back to your website for more soon.
Hey just wanted to give you a quick heads up. The words in your article seem to be running off the screen in Safari. I’m not sure if this is a formatting issue or something to do with internet browser compatibility but I thought I’d post to let you know. The design and style look great though! Hope you get the issue fixed soon. Many thanks|
You ought to be a part of a contest for one of the greatest websites on the net. I’m going to recommend this blog!
Aw, this was a very nice post. Finding the time and actual effort to make a good article… but what can I say… I procrastinate a whole lot and don’t manage to get anything done.
Excellent article. I’m dealing with a few of these issues as well..
Pretty! This has been an incredibly wonderful article. Thanks for supplying this info.
I used to be able to find good advice from your content.
This is a topic that is near to my heart… Thank you! Where are your contact details though?
This blog was… how do you say it? Relevant!! Finally I’ve found something that helped me. Cheers!
Aw, this was a very nice post. Taking the time and actual effort to generate a top notch article… but what can I say… I put things off a lot and never manage to get nearly anything done.
You have made some really good points there. I checked on the internet for more info about the issue and found most people will go along with your views on this site.
Thanks for the marvelous posting! I truly enjoyed reading
it, you could be a great author. I will be sure to bookmark your blog and will come
back later on. I want to encourage you continue your
great writing, have a nice holiday weekend!
Achieving your fitness goals does not need a certified personal trainer or an expensive gym memberships, especially if you have the budget and the space to consider practically every workout machine on the market.
An intriguing discussion is worth comment. I do think that you should write more on this topic, it might not be a taboo subject but usually people do not discuss such issues. To the next! All the best!!
Hey! I just wanted to ask if you ever have any issues
with hackers? My last blog (wordpress) was
hacked and I ended up losing many months of hard work due
to no data backup. Do you have any methods to prevent
hackers? yynxznuh cheap flights
I enjoy reading through a post that can make people think. Also, thank you for permitting me to comment!
Good info. Lucky me I found your website by accident (stumbleupon). I have saved it for later!
I must thank you for the efforts you have put in penning this site. I am hoping to check out the same high-grade content by you in the future as well. In truth, your creative writing abilities has encouraged me to get my own site now 😉
Excellent post! We will be linking to this great article on our site. Keep up the good writing.
There’s certainly a lot to know about this issue. I like all the points you made.
I couldn’t resist commenting. Exceptionally well written!|
You’re so interesting! I don’t believe I’ve read through anything like this before. So nice to discover someone with genuine thoughts on this subject. Seriously.. many thanks for starting this up. This web site is one thing that’s needed on the internet, someone with some originality!
This website really has all of the information and facts I needed concerning this subject and didn’t know who to ask.
Greetings! Very helpful advice in this particular post! It’s the little changes that produce the most significant changes. Thanks for sharing!
I’m impressed, I have to admit. Rarely do I encounter a blog that’s both equally educative and amusing, and let me tell you, you’ve hit the nail on the head. The issue is something that too few people are speaking intelligently about. I am very happy that I stumbled across this in my search for something concerning this.
I used to be able to find good information from your content.
Spot on with this write-up, I honestly believe this site needs much more attention. I’ll probably be returning to read through more, thanks for the information!
I’m very happy to find this page. I want to to thank you for your time due to this fantastic read!! I definitely really liked every little bit of it and I have you book-marked to look at new information in your web site.
I love reading through an article that will make men and women think. Also, many thanks for allowing for me to comment!
Fact of the future: There are a lot of people that would pay a minimum of $10,000,000 USD for each founder token of appreciation known as 777 immortality smart contract.
You made some decent points there. I looked on the web for more info about the issue and found most people will go along with your views on this web site.
I like reading through a post that will make people think. Also, many thanks for allowing for me to comment!
Oh my goodness! Incredible article dude! Thanks, However I am having difficulties with your RSS. I don’t know why I cannot subscribe to it. Is there anybody having identical RSS problems? Anybody who knows the answer can you kindly respond? Thanx!!
Can I just say what a comfort to uncover somebody who genuinely understands what they are discussing on the web. You actually know how to bring a problem to light and make it important. More and more people ought to read this and understand this side of your story. I was surprised that you’re not more popular given that you definitely have the gift.
After looking over a number of the blog posts on your blog, I truly like your way of writing a blog. I saved it to my bookmark webpage list and will be checking back in the near future. Please check out my web site as well and let me know what you think.
Right here is the perfect web site for everyone who wishes to find out about this topic. You understand so much its almost tough to argue with you (not that I personally will need to…HaHa). You certainly put a fresh spin on a topic that’s been written about for ages. Great stuff, just excellent!
Very good article! We are linking to this great article on our site. Keep up the great writing.
Way cool! Some very valid points! I appreciate you penning this write-up and also the rest of the site is also really good.
I must thank you for the efforts you have put in writing this website. I am hoping to view the same high-grade content by you later on as well. In truth, your creative writing abilities has encouraged me to get my very own blog now 😉
Greetings! Very useful advice in this particular article! It is the little changes which will make the most important changes. Thanks for sharing!
Your style is very unique in comparison to other folks I have read stuff from. Thank you for posting when you’ve got the opportunity, Guess I’ll just bookmark this blog.
Great post! We will be linking to this particularly great content on our website. Keep up the good writing.
There’s certainly a lot to learn about this topic. I love all of the points you’ve made.
Thanks for the article. Want more.