(completely random photos from my studio: this precious little ceramic pot was hand-made by my daughter’s dear friend , artist Olivia Shea)
In my last post about pdf pattern REVIEWS, I mentioned that tester reviews are in a special category, since the testers represent the designer. This time the subject is pdf pattern TESTING. This post is mostly questions, (LOTS of questions) because I’m not a tester, I’m a consumer. Like you. Trying to spend my money wisely. (But if you are also a beta pattern tester, I’d love to hear your insights. Really truly.) Whenever you buy anything, you want to be assured that it was tested before it went into production and was released for sale. Nobody wants to be a “guinea pig”. There are 2 kinds of product testing: Alpha and Beta.
Alpha testing is performed by engineers and asks “Can you make this work?”
Beta-testing is performed by consumers and asks “Does this work for you?”
Alpha testing is done during product development, either “in house”, or sent out to professional testing services. The purpose of Alpha testing is to work out the kinks in a product through back and forth communication between the testers and the product manager, in a technical manner. Alpha testers have a background in not only how to use the product, but also in how it is made, so that they can suggest fixes, involving the product itself, the directions to use the product, and possibly access to obtaining the product (packaging, downloads).
Beta testing is done after product development and alpha is complete. It is performed by “average consumers”, who try the product and report back their findings in a non-technical manner. The purpose of beta testing is to see if you and I can use the product without problems. Beta testing does not require a background in knowing how the product was made, and the beta tester is not expected to suggest fixes, only to report problems. In fact, it would be counter-productive to have beta testers with too much technical knowledge, because they might be tempted to overlook issues as “no big deal, I can work around that”, leading the manufacturer to believe that everything’s fine when it isn’t.
Today I’ll discuss beta testing of pdf patterns; next time I’ll go into alpha testing. I know way more about alpha testing because I’ve worked in the garment business forever, and alpha tested thousands of patterns.
Social media has changed the way that consumers are aware of beta testing. In high school, I earned spending money by working at a beta-testing lab, trying out new cosmetics, sampling soft-drinks, or watching potential tv ads. I don’t remember ever telling anybody what products I tested that day, except maybe at the dinner table. These days beta testing is very vocal, very visible, blasted all over the twitter-sphere, tumbler, FB, and blog-land. Beta testers love to report about the latest cosmetic, videogame, or kitchen gadget they were “honored to test”: it has morphed into “product reviewing”. Has it become a game of attracting “likes” online, fulfilling a desire for attention? Does this cloud objectivity? More than ever, the customer, if she absorbs this promotional activity, needs to consider the source, and take everything with a grain of salt.
How does beta-testing plus social-media affect pdf pattern sales? Have you ever been enticed to purchase a pdf pattern by seeing the preview “sneak peek” photos, and then reading the gushing tester reviews at launch? I have. Many times. And most of the time, those patterns are wonderful. I LOVE sewing patterns the same way my friend Elisa loves recipes, and for the same reason: they help you take raw ingredients and turn them into masterpieces. But….. have you ever been disappointed when you download the pdf? If not, count yourself lucky. I have bought so many duds. Sometimes there are errors in drafting and grading that immediately jump out at me straight from the printer. (That’s my own personal issue because I’m in the apparel production industry: as a Technical Designer it’s in my DNA to spot pattern and sample problems…and fix them..)
Other times there are problems that don’t become evident until sewing. These come from patterns that haven’t been measured, walked, or trued (to be explained in another post on alpha testing). Sometimes the problems are in the instructions. I don’t pay all that much attention to the instructions, but I know many other people do because I read FB posts talking about missing steps, the number of pieces to cut is wrong, layouts have pieces that should have been cut on fold but weren’t indicated that way, references to waist and chest are mixed up, elastic measurements don’t make sense, etc …so I know there are instruction errors that were never caught. And it always makes me wonder: Why weren’t these things caught by the testers, the same ones who raved about the pattern on social media?
Beta testers aren’t expected to recognize or fix drafting or grading errors, that’s not their job. Testers are home-sewers, that’s the whole PURPOSE of beta testing, to see if the average sewer (with no pattern engineering background) will be able to put together the project smoothly. In a FB pattern review group, we were discussing how to review patterns, and a member posted this:
”I’ve tested in the past and wonder if I’m ever qualified as I don’t know proper pattern drafting technique. I know I’ve been chosen based on past photos I’ve done as designers also want “pretty pictures” but there’s no way I’d ever be able to critique on drafting and what’s supposed to line up where .”
She is actually a perfectly qualified to be beta-tester. (I would be the worst beta-tester: I’ve never met a pattern I didn’t want to tweak.) She represents the consumer who does not have technical background. What the beta testers should be reporting are the instruction typos, any sewing problems, and fit issues. The problems could indeed stem from drafting errors, but she is only expected to report the OUTCOME of the errors, not the CAUSE. This is where I’m perplexed with patterns that have serious flaws. Didn’t anybody have a problem setting in that sleeve with the armscye an inch bigger than the sleeve-cap? Wasn’t there an issue with that tiny bodice side-seam, only an inch high, that had to encompass 16 layers of fabric (front/back X left/right X bodice/skirt X body/seam-allowance)? Didn’t that non-graded strap, same length in 12 months and girls 12 bother anybody? Okay, I can understand why the grading issues (that bother the heck out of me) go by unnoticed by the beta tester if she is making one piece, in one size. I make multiples to sell; I NEED perfectly proportioned grading. Still, some of the outcomes in the tester photos are obvious: straps than can barely be tied in a bow on size girls 12, and are dragging below the hemline on the bubs. Nobody noticed this?
Now, this shouldn’t bother me personally: I can fix any pattern. But what about the many sewers I’ve heard from who blame themselves if their project doesn’t turn out well? This breaks my heart because when a sewing project goes wrong, 9 times out of 10 it’s because of the pattern, not the stitcher. So I wonder: How could so many testers sew up a pattern that still has errors, yet those errors never got fixed? There are only 2 possibilities as I see it (please let me know if there’s some other reason that I’m missing):
Beta testers didn’t report the problems they encountered, or
Beta testers did report them, but nothing was fixed
Do most beta testers report most problems?
Probably most of the time. Many beta testers in pattern review FB groups do say they always fill in the questionnaires that are required for beta testing, they do bring up problems, they do communicate with the designer. However I’ve also read anecdotal evidence that this doesn’t always happen. What would stop a tester from fulfilling that part of the agreement?
If they don’t always report errors, why not? Let’s start by asking why do beta testers volunteer anyway?
At face value, beta testing is a swap : free product in exchange for testing it. But with sewing, there’s a lot more expected of the tester than there is for, let’s say, nail polish. It can be MORE expensive to participate in the test, than the product you’re getting in exchange is worth . Even if your fabric was from the thrift shop, your model works for free, and you are using the camera in the phone you already use, your time has enormous value. So there must be more benefit to it, right? In a FB pattern review group, a member put out an honest inquiry last week, asking ”Can someone please tell me some of the reasons testing is so incredibly appealing to so many? What are the benefits to you as a tester? I mean, what is the payoff? “ The replies were very informative.:
Love the process, liked having input, created better pdfs: 7
Liked having a deadline , pushed me to sew, forced me to finish: 7
Likes showing off photos, especially if chosen to be published with the pdf: 5
Fun, social, made friendships: 4
Free pattern (although some admitted this was financially illogical): 4
Last week I discovered that some testers do actually get paid, so I put out an inquiry to find out how common this is, and the replies were:
Received payment upon test completion OR received credit to spend on fabric at a sponsor of the designer’s blog (average amount: $20 to $30): 4
Had to pay deposit to test but received refund: 4
Testers were entered in a raffle or drawing for cash, gift card, or fabric: 3
Earned cash from affiliated link: 4
Received discount on all future patterns OR from entire pattern shop: 3
Received final version of tested pattern plus an additional pattern: 14
Received only the final version of the tested pattern: 14
So it’s more of a social benefit, than a financial one, but it’s not a financial sacrifice either: many justify the time by saying “I would have sewn it anyway” and “I need pictures of my kids anyway”, and many appreciated the nudge to get these things done. There is a big social/emotional element going on here, which makes sense, due to the relationships developed between indie designers and their home-sewing testers. Blog posts abound on “how to be chosen for testing” and “how to recruit and manage your testers.” As with beta-testing in many product categories, there is sometimes a popularity contest at stake, a longing for recognition and acceptance, with some testers who get called on again and again, and others who never get “picked”. It’s a point of pride to have your photo chosen for publicity on the designer’s website or pdf file. But is this an objective rational way to finalize product approval? Is the beta testing process overly influenced by emotion? Is it possible that there are goals at odds with each other, which compromise “best practices”? (https://en.wikipedia.org/wiki/Best_practice)
When you read reviews at release, do you wonder if pdf beta testers are being overly complimentary? This makes sense: we learn in kindergarten that it is wrong to hurt anybody’s feelings . Do beta testers know how important their feedback is to us, as consumers? Not “Do I like it?” (emotional.) but rather “Does it work?” (rational).
Have you noticed that emotions run high at launch time? Who doesn’t want to see the much anticipated and hyped next design? It’s so fun to look at the gorgeous photos, and read about the new patterns. I totally get caught up in it, buying way more pdfs than I can use. But what happens when all is not dandy, and as a customer you purchase a pattern that turns out to have mistakes, and then you ask a question about it online? You risk angry push-back by (some) testers, who insist that everything is perfect with the pattern, nothing is wrong, just follow the instructions, you’re probably not a very good sewer. Even if you show them obvious pattern errors, charts that are wrong, incorrect grading, they still defend the pattern. Can they not see the problems in the pattern? Or did they notice the problem, but not want to advise the designer? I applaud their loyalty, but how about thinking of the consumers who are paying for the pattern? Is it possible that beta-testers are worried about:
Fear of push-back from fellow beta-testers ? (the same ones who attack paying customers if they ask a question)
Fear of hurting the designers feelings? (as if this weren’t a business, but rather a social group)
Fear of not being invited to test again?
So the other question is, do some pdf designers NOT WANT feedback? The designer doesn’t need feedback on her design; just as in the garment industry where the designer doesn’t (usually) want to hear the fashion opinion of the fit model. In her blog Elisalex de Castro Peake explained :
“What we aren’t really looking for feedback on is the design. Mostly because that’s a very subjective issue, and one that remains our call (and of course, too late to change once it’s in the testing stage! “
But what about reporting back on the quality of the pattern? Is it also possible that beta testers feel reporting THAT isn’t wanted? Maybe beta testers have reported errors before, and discovered that nothing was changed or corrected (ie: they’ve learned to NOT report). Maybe they tested for a designer who told them, upon hearing that an outfit sewed up way too big, to just safety-pin or clip the back tighter and take the photo anyway. Maybe they’ve been to this rodeo before, and seen that it’s not so much about testing but rather product promotion. In the Cashmerette blog, Jenny Rushmore wrote:
“Theoretically testing and reviewing are separate, but in fact they’re not in practice. In the majority of cases (it seems to me, anyway), testing leads to a finished object, which leads to a review post pretty soon after a pattern comes out. “
The comments that Jenny received were eye-opening:
“i think this is likely even more the case where bloggers aren’t actually testing, but just given a final pattern for free in exchange for being part of a blog hop promo for a new pattern or bundle. “
“The cynical part of me thinks it also has to do with some of those bloggers’ desires to continue to be asked to pattern test/blog tour. “
“It seems some things which are called testing are really receiving an advanced copy of the production pattern just before release. Yes, this does require sewing up a version, but is this testing or just reviewing? “
“how much of testing is truly testing and how much is really just the marketing rollout? Why would a dozen testers of relatively similar build and sewing ability be required to catch typos and missed notches? When they’re all popular bloggers it makes sense. “
“It makes perfect sense to me that bloggers are given promo patterns ahead of a launch, I don’t mind that, but why call them testers? … I don’t think anyone wants to call themselves a promoter but in some cases tester is just a euphemism for that “
So SOMETIMES feedback is requested, but sometimes it’s avoided, possibly because it’s not always wanted. Like a wife asking her husband “Do I look fat in this dress?” Better to say nothing than to appear “mean”. Better not tell the truth, even if she would look better in the other outfit. Don’t rock the boat. Leave well enough alone. Don’t stir the pot.
Let me ask you a question: wouldn’t you tell a friend if she had spinach in her teeth?
Shouldn’t testers feel free to report? Shouldn’t designers listen?
Shouldn’tBeta testers be looking out for their sisters who are going to be using that same pattern?
But ultimately doesn’t the responsibility lies with the designer: who she selects for beta-testing, how she processes feedback, what she does to make sure the pattern meets her standards?
And then we come to the other question which was:
If beta testers DO report problems, why aren’t errors always corrected? I can think of 2 reasons:
Is there not enough time?
Beta testing is typically done one week prior to launch. Is that even enough time to gather all of the responses and correct errors? If the pattern engineering is done in-house, it could be. If drafting and grading is sent out, or done overseas, that could add to the schedule. But the schedule is self-imposed, after all. This is not testing for a mass-manufacturing business who must work within the fashion calendar. Indie businesses can release any product at any time they desire. Why not allow more time for sufficient testing? What if big problems arise, causing the need for another round of samples? Does the designer care enough about getting out a quality product, to give the testing process the time it requires? Or, is this not really beta ”testing” but more for “product review” and publicity?
2. Is there too much focus on short-term gain, to get those photos done and rush that release? The problem with that is, just as social media can boost sales, social media also spreads information about problems in products. Like wildfire. So if designers don’t want to hear the problems to their face, the issues could surface behind their backs.
The bottom line is, as a consumer, do you trust that the patterns you purchase have really been tested?
It’s your money, your investment in not only the pattern, but more importantly your time. When shopping for patterns on the launch day (discount!), the only information you have is the listing, the reputation of the designer, and the beta-tester photos & reviews. Do you trust that the testers were objective and non-biased, able to catch errors and willing to report them back, and interested in you receiving a quality product? Are you confident that the designer cared enough to make corrections, and really wanted to hear about what a consumer, like you, would experience with her product?