Poe Probes > Reviewing Samples, Part 9: Reader Responses

This is the ninth in a series of articles about the practice of toy reviewers being given free samples for review, and whether that represents a problem for readers looking for honest assessments to make purchase decisions. You can find the other articles in the series here.

In addition to interviewing toy reviewers, I also asked readers what they thought of the practice of reviewing samples. Here’s a sampling of some of their responses. (I tried to get as many quotes as I could; some readers wrote thoughtful but very long pieces that simply couldn’t be broken down into bite-size chunks. If there’s enough interest I’d consider posting them as editorials.)

In reading over all the responses (and I got over thirty of them), I noticed some common themes. On the whole, readers said they did think that reviewers who received samples softened their reviews. However, for the last question, most said this was not a concern for them, because either they were aware of the bias and altered their assessment of the review accordingly; or  because they only cared about the pictures/videos of the toys anyway.

1.) As a consumer, does whether or not a reviewer got a toy for free affect how seriously you take the review, or how much weight you give to its opinions?

No. What matters is whether the review itself is well-written, balanced, provides a solid description of the toy itself, and is “apolitical”. Meaning no blatant cheerleading for the manufacturer, as that clearly demonstrates a bias. This also applies to sites or reviewers who shall remain unnamed who choose to suppress commentary from their readers that they deem overly critical of a particular company. –Dark Angel

As a rule, no.  I understand that it’s a way for retailers to provide reviewers with their product so that the reviewer doesn’t have to shell out their own money.  If that were the case, some reviewers on a much tighter income might not able to purchase said to, thus limiting that exposure for the retailer.  Toy reviewing seems to mostly be a labor of love anyway, I can’t imagine many of these people are paying off several mortgages this way, so it’s definitely a pick and choose process when it comes to buying.   –JG

It can, but usually doesn’t. The good thing about toy collecting after a long time is that you learn what you see is what you get. As long as there are good in hand pictures you can pretty much see most of what you need. The review and opinions an help confirm or deny your observations. Many times I read reviews simply because I want to hear what other people think even if I already own the toy. In reality though, it depends on the reviewer. If its someone you are familiar with its easier to gauge how well their opinions would potentially align with your own.  –T16SkyHopp

Absolutely, yes.  The issue is not necessarily whether the person has the intention of being impartial or not, but whether they can really help it.  I’ve gotten free figures in the past (throw-ins for large purchases, gifts from friends, leftovers friends didn’t want, etc.) and its just far easier to be satisfied with a toy you didn’t pay for than it is to like one you plopped down $20-50 for.  Having money invested gives you that added motivation to be bothered by faults/flaws, and to inherently expect higher quality.  In other words, when you pay for a toy, you look at it with more of a critical eye, and when viewing it, you’re unconsciously/subconsciously rationalizing the purchase in your head…..was it worth it or not, and if not, why.  Free toys, I think, just don’t provoke that same level of scrutiny. –AA

I won’t name any specific reviewers, but recently I watched two different video reviews for Mattel’s Red Lion and Lance figures which were overwhelmingly positive. This was before having the toys in my hands, but even then I could easily see problems they (the reviewers) were overlooking in their reviews and were basically just gushing over the toys. I can appreciate the nostalgia associated with these, but once I had them in hand they were far more disappointing than I could imagine.  Maybe my opinion can be blamed on having to pay for the toys, but it seems the “value” of these toys was lost on the reviewers. Some reviewers just make it seem like every toy they review is the “Toy of the Year” and it comes off as more a commercial for the toy than an actual review. –3B

If  a reviewer gets a free sample of a product I certainly do take that into account when analyzing the review, but in no way does it cause me to discount the review.  For me, the key factor is disclosure: I appreciate it when the reviewer makes it known that the product was received free or that the reviewer is in some way affiliated with the company involved.  That helps me to see that the reviewer is being honest and laying all their cards out on the table.  In regards to how seriously I take the review, well it depends on the reviewer.  If they have a past record of being critical, balanced, and solid in their reviewing and opinions towards companies, I’m going to have no issue with their review.  –JR

2.) If you do tend to view free sample reviews more skeptically, in what way are you skeptical and why?

Generally speaking, I don’t trust toy fans to maintain their integrity in the face of a future filled with free toys. It can be a case of a toy being free subconsciously negating anything one might find wrong with it. I think it might not come as naturally to be critical of something you did not have to spend any of your own money to acquire. But more often than not I’m absolutely of the opinion that toy reviewers will intentionally gloss over (or neglect to mention at all) faults with a toy if they think positive reviews will equate to more free toys. Bottom line: No toy nerd is going to screw themselves out of free toys. –AL

Even the best reviewers, and especially the best reviewers will never say “This is not worth your money” with a free sample.  They can’t, since they didn’t spend anything.  I do like knowing that Pixel Dan’s got a MOTUC sub, so if he’s reviewing something with problems, he’s still paying for another one soon enough.  He’s gotten more critical lately it seems, but he’s not a jerk about it and I imagine he’s not in a postition where he’s afraid the freebies will stop coming.  I imagine his reviews are also more critical since people will call him out on being too soft and because his views reflect the fandom and the issues they’ve had with the products. –FakeEyes22

Primarily my skepticism stems from these reviewers understanding that the bulk of their audience is based on access to early/expensive/obscure review samples. Again, while I believe in the integrity of everyone I follow, I’m sure none of them would jeopardize their advantage by being overly critical on a repeated basis. – -Slick McFavorite

I am very skeptical with free sample reviews.  I generally find these reviews are heavily favourable, so it’s the glass is half full situtation. So I don’t put much weight on the written review, but I will use the photos and pics as reference to influence my decision to purchase.  Free sample reviews are likely to be published before the toys are available at retail (b&m and online), so the photos of the finished product and packaging are invaluable in helping me decide whether to buy or not. –BamBam

3.) Do you think there’s any difference between free sample reviewing in the toy industry and, say, IGN reviewing a videogame, CNET reviewing a new smartphone, or Rolling Stone reviewing a new CD?

From my understanding, those venues are receiving office copies, and I would suppose stay at the office (although I could be wrong).  If that is true, I’d say it’s a big difference in a more one-on-one relationship, over a professional one. –DG

There is a difference.  Though the reviewers working for IGN, CNET and Rolling Stone may have pursued careers in that field because they have a passion for games/tech/music, their reputations as reviewers depend upon being objective and critical.  The corporate entities that employ them may massage the relationships they share with the publishers/providers offering material for review, but the reviewers themselves likely need not worry themselves that if they honestly critique the subject doing so could result in that group taking their ball and going home.  The allegiance of these reviewers is focused clearly on their audience.  Their readers/viewers look to them to uphold a certain standard.

We, as adult collectors, are still fighting for our hobby to be taken seriously in general.  Music, film, literature and tech are over that hump.  Telling someone you review music or movies or books or gadgets for a living is seen as legit, even cool, by the world at large.  Video games and comics are coming along, even if they are not quite there yet.  Toys, however, are still seen largely as the dominion of children, despite our best efforts to prove otherwise.  For toy criticism to be taken seriously, it needs to be objective, it needs to have cultural relevancy and substance, it needs to elevate the product by encouraging those that design and produce it to bring a better product to market.  A better product and intelligent discourse concerning that product brings legitimacy to the hobby overall. –AB

In concept no. In reality yes. IGN reviewers will receive every game by every manufacturer, same with Rolling Stone and new cd’s. The toy industry is too new at being reviewed though. Hasbro doesn’t send out one of every Star Wars figure to get reviewed. Bandai doesn’t send out any free Thundercats for review. The only major company sending out free review samples of all their products is Mattel. That in essence creates an unfair playing field where a reviewer is going into any review of a non-Mattel product already miffed at having to pay for it when they are looking at a box of free Matty stuff on their desk.  –KJG

Fundamentally there is not much of a difference, but the nature of collectibles is different. None of the items above are going to be hard to get, video games, phones, and CDs, are always there for purchase. Many times you may only get one or two chances to buy a figure, then the price starts to rise and the options to buy start to fall. To me this means that the objectivity of a toy review is much more important. […]

Toy sites also work on some kind of bizarro logic. If IGN or CNET was to indorse more than a few  stinkers, [readers] would fall away. Toy sites however, make excuses, produce sites trying save lines, write good review for substandard products and in general take sides. This lack of objectivity does [not] seem to result in the lose of fans, and may even gain the site fans. –Tribsaint

4.) Ultimately, do you think there’s any problem with reviewing free samples of toys? 

No, I don’t think there is any issue with it, with a couple of caveats. First, I do think the reviewer does need to say that it is a free review sample. Second, I think it is important the reviewer does a mix of both paid and free reviews. If all I saw was free reviews on a site, I don’t think I’d feel as comfortable with the reviewer. I also would think that company would have a duty to spread samples to different reviewers to keep a single review site becoming their “go-to” reviewer. –Engineernerd

No, as long as they are honest with their readers about the source of the items so the reader can include that knowledge in their assessment of the review. But at the end of the day, I will always place far more trust in a reviewer who bought the toy off the peg or a website like I did than a reviewer who didn’t have to pay for the item. There are lots of toys I’d like more if I hadn’t had to buy them, and I think that pretty well sums up my opinion. –Valo487

I have seen reviewers who are basically unofficial sales reps for companies give stellar reviews to toys which were inspected to be good and have no issues as the company knew they were for [the reviews].  Unlike the fans who received the figures which had multiple QC issues.  Also this reviewer gave overly-critical negative reviews to toys which have no QC issues, just because they are competition from a rival company.  Also some reviews skip issues about durability which result, in my own experience, in the toys breaking upon opening or posing. –Mark

No, because I mostly rely on photos when making my purchasing decisions anyway. But I do respect reviewers who are willing to disclose any potential conflicts or things that may be affect their judgment one way or the other.  –Izdawiz

None whatsoever. As long as the product gets reviewed. –Axel

Not at all, as long as the reviews are realistic and consider the difficulties that collectors have when purchasing the items. The price has to be taken into account as well, and if the reviewer fails to mention how much the toy costs and how difficult it is to purchase it, then the review is lacking, whether the reviewer got it as a free sample or not. There are many internet forums where collectors talk about the products they buy, so, if a reviewer writes a completely crazy review, people will notice. –MM