Minimum amount of reviews
Does anyone else here think there should be a minimum amount of reviews for a tea before it can appear on the front page of the “best teas?”
It seems to dilute the effectiveness of it when almost all the pages are full of teas with only a couple of ratings.
To me, a 79 rating on a tea that has 30+ reviews means a lot more than a tea that has a rating of 89 with only 4 reviews. It doesn’t mean the tea is bad, it just doesn’t have the sample, size to give me confidence.
Sounds good to me, but where do you draw the line? I guess ideally, it would be great if we could choose the parameters ourselves—set it to show the top teas with at least 10, 20, 30… whatever ratings.
I think the bigger issue here is still with tea companies gaming the system. I still think there is something fishy going on with the East Pacific Tea Co. They seem to have a lot more ratings than they have tasting notes, and if you rate a tea without creating a tasting note, you can apparently do so anonymously.
I have looked into the East Pacific stuff and there are only 2 tasters notes that appear from over 6 months ago with 100 ratings that were a little questionable…the others were legit…people we see most days here on steepster or more consistently I should say. Most of the ratings have been that way but never showed on that page or highly rated overall up until a week ago…that is what I think is strange! I agree! I did mention to Jason about the 2 tasters – but like I said they only logged like 1 or 2 teas each or so…AND they haven’t been around for 6 months. I really don’t think it was this actual tea company with this specific one. I do agree there are obvious companies that do do this tho, sigh.
I share your frustration, I do!
I think that if there was a 3 or 5 minimum rule for posting reviews that would CERTAINLY help this situation! I also think that if someone is going to post a 100 (for example) they should tell us why at least once…maybe not every time (maybe the first time) – but I think that the extremely high and extremely low ratings should have some content somehow…but now? Not sure how we could police it, correctly, ya know?
I will say I am not all that happy about the number one tea on steepster being Ginger Snap from them and not having any actual reviews – by that I mean tasting notes! It’s the #1 tea but no notes!? Ug. That DOES bother me. If there were notes I wouldn’t be as frustrated, I guess…so the more I think about it the more I agree with some sort of min review type thing for that front page!
I think gaming is universally disliked here. But even with honest reviews there is a problem.
Take Yelp for example. It would literally be impossible to use if they allowed the lesser reviewed places to be listed as #1. The popular places would just get drowned out by the unknown places.
I guess you can always click on “Most Popular” too, to see which teas have the MOST ratings (starting with mostly store brands).
Unfortunately, the most popular teas on the first page or two are the ones on the grocery store shelves. Nothing wrong with that, but you can’t discover the great smaller companies that way either.
I’m not sure how well that would work out. If there was a required amount of reviews to get onto the “top teas” list. Wouldn’t people still “game” the system by creating more accounts and still rating the tea?
I think its a good idea, however I don’t believe it would be enough to balance the system.
That is certainly one approach we could take. I think even though that isn’t technically how the formula is calculated, in a sense it does that, even though that minimum level has the possibility of fluctuating based on a variety of factors.
Part of the issue we believe needs to be tweaked is how all of the other teas on Steepster weigh into the rating. Factors in the formula include how many teas there are overall and how many overall ratings each tea has. But with over 16k teas, a majority of those have 0 or just 1 tasting note/rating, so they are lowering the barrier for the number of tasting notes/ratings a tea needs to be considered “popular”. If the average number of tasting notes/ratings for each tea were higher, these teas with fewer tasting notes wouldn’t rate so highly. I hope that makes sense…
It’s just part of the equation we’ll have to consider when we adjust the rating system formula.
I just figured out one way you guys could do it and it would (in theory) be fairly simple.
Right now, there are 4 categories for looking at the teas list…. “Highest Rated”, “Most Popular”, “Newly Added”, and “By Name” (alphabetically).
Why not simply add a 5th category that is called something like “Customer’s Favorites” or something — this category will be just like “Highest Rated” except it will only include teas that have 10, 15, or 20 or more reviews from separate users (and better yet, throw out the ratings that companies have given their own teas). That way we can get a look at the ones that many people have tried and see how those stack up.
Just so you know, users can only leave 1 rating per tea. When you see a users leave multiple tasting notes on a tea, you’ll notice that they all have the same rating because their rating is only counted once (and when they change the rating it just changes that single existing rating).
Also, we don’t have the ability to distinguish company accounts from general user accounts. That’s another things we’d love to add but haven’t been able to yet. So in order to automatically pull those out, we’d have to create an additional user class.
Either way, the idea of creating a separate ratings sort is really interesting and could work well! We’ll have to look into that.
I think you should look at a similar algorithm as BoardGameGeek if you can. I believe they use a lot of dummy votes to keep things towards the average, and looks at the users votes to determine how much weight that user should carry. Also, someone who votes very high on a few teas, and very low on others would have a penalty, as a normal person has more of a bell curve…
Woohoo – thank you William Adam McDuff.
I love BGG! Their ratings system works very well. You don’t see any random games in the top 50…
Jason, if you guys ever wanted to talk to the admins of BGG about how they do their board game ratings, I’m sure we could help you get their contact info…
Thanks for the suggestion. Adding some sort of credibility system has always been something we’ve wanted to add since it could not only help support ratings but also act as a way to determine tea swap credibility. Both of your suggestions would definitely be something we’d want to include in that system, so I think you’re right on track!
And thanks for the offer to connect us with the BGG people. We’ll certainly hit you up for that, if/when we get to adding such a system.
Login or sign up to leave a comment.