Should they, though? Designers have a finite amount of time and energy they can spend. If option A has 65% approval, and option B has 75%, why spend time fiddling with A in the hopes of raising to 80? They have however many other projects to work on and hard deadlines. Sure, with unlimited resources, they can try and polish up a diamond on the rough idea. But since they don’t have unlimited resources, they have to work with the most realistic options for making most of the people happy most of the time.
That's a somewhat different issue. Given the reality of limited resources, it's certainly possible to decide that option B is 'good enough', but you want to make an informed decision, rather than "option A is lower rated than option B, and I have no idea why", and that means your poll should be designed to answer that question... even if down the line you choose not to use the answer.
why ask someone a question you have no intention to act on. Whatever they decide for this phb is basically staying until there is a new phb. And it dilutes your results, I want to now how much you like this thing I made, not how much you like what it could be.
with current questions you know how many hate something, how many could take or leave it, and how many like it. If you throw in needs work, suddenly you are taking feedback on something that doesnt exist and may never exist, and its taking some of the feedback aaway from the other thing. Someone may hate the current implementation, but think the idea is OK, or love the implementation but think it needs work. Or not really care but think it needs work.
now you ve taken 3 people and you no longer have a clue how much they like or dislike what you have presented. To do the question justice, you'd need more questions. And another factor in surveys, is asking more questions sometimes influences responses. Asking what would you improve makes people respond differently than, do you like this.
and more questions alsomequals more time and factors to process. Its a not simple bonus to ask more questions and why bother if you aren't going to act on that info..
I'm not saying they magically know what players mean, I'm saying they weren't interested in players opinions on what to refine and what to drop.
Clearly they were or they wouldn't have run the playtests or the surveys at all as there's no point if they don't want to gather feedback from it.
But once again you seem to be missing the point I've been making; they gave us a five option scale that served no truly useful purpose. It could have been a simple binary option (satisfied or dissatisfied) and obtained exactly the same results, because if you're just boiling the responses down into a "percent satisfied" score then they didn't need the other three options in the first place.
Also once again, that kind of satisfied vs. dissatisfied response is only so useful, because it's pretty useless as a metric upon which to make decisions unless it's overwhelmingly one or the other, the moment you get into the middle ranges you don't have any information on which to base a decision. The purpose of any survey should be to get back useful information, but the point I tried to make in the first place is that the survey is not well structured for doing so, because they limited the usefulness of the answers from the outset.
I really do no understand why my criticism of the surveys not being very good has elicited multiple people rushing to their defence; they're not great surveys, this should not be a controversial opinion to hold. I'd like to see the style of survey changed for future playtests, because I do not think they're fit for purpose as they're either unnecessarily complicated if all they want is a yes/no score, or they're both unnecessarily complicated and unnecessarily limiting if they want more information to act upon, as a three option response can be both more useful, and simpler.
Not sure why you'd want the surveys to be both complicated and bad, but you do you.
no, they were interested in how much players liked or disliked what they presented, not what players think needed more work, those are two very different questions as you and others pointed out. If a match maker asks you which of these people is attractive enough to marry, it doesnt mean they are interested in knowing which ones might be attractive enough if they lost weight, or changed their hair.
And that fact doesnt mean asking the question was pointless.
The point of degrees of satisfaction, is to create data on how satisfied or hated something is, which they did make use of when they really want to. They claimed that even though uniform progression tested well, above their metrics for satisfaction, it was not "very satisfied" enough to keep against the lower number of people who emphatically disliked it, and between the lines, the extra effort it would require to implement. So they do take and make use of degree of satisfaction data when they are looking for granularity.
And I'm not saying the surveys were great, I'm saying the surveys were 'ok' because the information they were interested in was limited. Or another way to put it, its not the surveys you have a problem with, you want the designers to incorporate different feedback into their process. But flawed or not the creator has to decide what type of feedback they are willing to take action on. There is no strong need to take feedback on things you are not going to change. Its not wrong for you as a tester/user to wish they cared or took feedback you give, but making something is only as heavily driven by feedback as the creator chooses, its their prerogative. And there are pros and cons to being more or less driven by feedback in design.
When I first saw the surveys I thought they were really bad, but now that I see the full process, I realized they simply were not that interested in things I thought they were, and they didnt really have time, interest, or resources to take or act on the type of feedback I wanted them too. Look at the results, could they have actually spent more time refining druid templates for this phb based on player feedback? not really, it takes them 2 months per cycle, and this thing needed to be finished by a certain date. I personally think brawler should have had more passes, but I can't really claim they could actually do it. They barely got to the planned dates. In fact they seem to have delayed it, as first it seemed they were going to attempt to get 3 books in before the end of the year. Ideally, I would set up a better process and content pipeline so they could test and act on more feedback faster, but thats not what they have right now.
does that mean the UAs weren't improved by feedback? imo they were.
why ask someone a question you have no intention to act on.
If that were the case, they wouldn't have the poll at all. There are plenty of things they have decided to produce revised versions of, and the decision whether to revise, try something entirely new, revert, or keep the new version is not one they'd make before producing the survey. This certainly does not mean that everything will be acted on, it just means they should design their survey to be useful.
no, they were interested in how much players liked or disliked what they presented, not what players think needed more work, those are two very different questions as you and others pointed out.
Except they're not; because whether people like or dislike something only tells part of the story, it tells you nothing about why someone disliked something, it doesn't tell you that a feature might be a sound idea but poorly executed, it tells you nothing about what the player base may or may not support if implemented differently.
It's cutting out a massive chunk of useful information on which better, more efficient decisions can be made.
But I feel like you're still not listening to what I and others are saying at all; there's no use repeating that they only care about pass/fail, because that's exactly the problem that we're complaining about. The entire point is that they should be gathering more useful data via simpler questions, rather than less useful data using more complex, more ambiguous ones.
And that fact doesnt mean asking the question was pointless.
I never said asking the question is pointless, I said giving us five options is. If all they care about is pass/fail then they only need to ask for a pass/fail answer to get the same results, because it doesn't matter if respondent 693 was "satisfied" rather than "very satisfied" by a change, because with thousands responding the volume of responses gives you the exact same pass/fail percentage in the end.
Once again, the complaint is that they either:
Could be giving a much simpler two options to get the same data.
Could be asking a simpler three option question and gathering more information.
Either way you cut it the survey is poorly constructed, because the options are overly complex and ambiguous, and wasting time as a result.
The point of degrees of satisfaction, is to create data on how satisfied or hated something is, which they did make use of when they really want to. They claimed that even though uniform progression tested well, above their metrics for satisfaction, it was not "very satisfied" enough to keep against the lower number of people who emphatically disliked it, and between the lines, the extra effort it would require to implement. So they do take and make use of degree of satisfaction data when they are looking for granularity.
And the point of complaints about that methodology is that it tells you nothing about why people dislike something; again, there have been multiple ideas presented in the playtest that weren't bad ideas in themselves (templated Wild Shape, half-caster Warlock) but they were extremely poorly implemented, so it was inevitable they got overwhelmingly dissatisfied results, because we were given no means by which to communicate that we liked the idea but hated the execution.
We never really got to playtest either of those proposals because they were doomed from the start by poor execution, so it was inevitable they'd get a negative pass/fail score. But that never meant they were bad ideas; many of these discussing in the forums recognised that there were many, many ways to make these things viable, some not even all that difficult, but the pass/fail nature meant the features were never going to get an update, because there was no possibility of that happening.
So once again, they eliminated useful feedback they could have used to come back with better versions of those ideas, rather than axing them and starting again, which is potentially wasting developer time when they could have just been refining what they already put forward. Everyone should want want them to use better surveys, because even if you don't use all of the extra data, having the data is better than not, and the developers wasting last time for better results is better for everybody.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
why ask someone a question you have no intention to act on.
If that were the case, they wouldn't have the poll at all. There are plenty of things they have decided to produce revised versions of, and the decision whether to revise, try something entirely new, revert, or keep the new version is not one they'd make before producing the survey. This certainly does not mean that everything will be acted on, it just means they should design their survey to be useful.
They have the poll to figure out if the things they are doing are broadly hated or liked, not because they plan for the testers to decide how they plan to handle design process. There is a difference between a chef taking a survey on how many people like the dishes they have created, and asking people how to cook.
I'm not saying no one has any good advice on how to cook, but most chefs aren't asking for that type of feedback, many actively don't want it. And thats their prerogative as the chef. It doesnt mean finding out how the various meals were received is pointless at all.
I want to know if you ve liked what I made, and I as the designer will decide what to do with that information based on myriad factors you aren't aware of is completely valid. At the end of the day this is their product and they are the ones who will sink or swim with it.
if you are trying to get rating scale, introducing an option that isnt about how satisfied people are messes up the information. You created a third category which says nothing about how much responents like what you produced. lets say 70% of people choose an option templates needs more work. You now have no idea how much 70% of the people who answered a question about how much they liked the templates you created , liked or hated your idea. You got a vague response of needs work. They could like or hate it and say it needs work. Does it need a little or a lot of work. And thats even worse if you already know you have 3 options. Now you can't compare the satisfaction rating of plan B to plan A, and you can't compare it to satisfaction levels of the live product.
Even if they wanted that information, it wouldn't be the appropriate place to seek it. And, its not always better to try to get more information from a single survey. You think its an objectively better survey if it has those questions, but thats questionable in survey design.
They have the poll to figure out if the things they are doing are broadly hated or liked, not because they plan for the testers to decide how they plan to handle design process. There is a difference between a chef taking a survey on how many people like the dishes they have created, and asking people how to cook.
Your own example doesn't pan out (pun intended); if a chef asks if people like a meal and they say they hate it, then a good chef wants to know why, otherwise they're ditching a recipe forever. If a meal was just too salty, or the vegetables weren't soft enough or whatever then these are fixable problems.
But again your argument is essentially "they only care if you liked it" but that's exactly the problem we're complaining about. Because only caring a feature scores low or high, but not why, means they're making decisions with only half of the information. What is the point of repeatedly telling us that the problem is exactly what we've said it is? Why are you so determined to shout down legitimate complaints about the process? What the hell do you get out of that?
And again you're accusing us of wanting us to tell them how to fix it, but that's not what we're asking for; a three option response with a clear choice of "needs improving" isn't telling them what to do, it's just expressing a desire to see something improved rather than immediately axed, it's still on them to actually listen, or to focus on improvements rather than going back to the drawing board. But a feature with an overwhelming response of "needs improving" is a much more useful result than a middling figure on the pass/fail scale.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
They have the poll to figure out if the things they are doing are broadly hated or liked, not because they plan for the testers to decide how they plan to handle design process. There is a difference between a chef taking a survey on how many people like the dishes they have created, and asking people how to cook.
Your own example doesn't pan out (pun intended); if a chef asks if people like a meal and they say they hate it, then a good chef wants to know why, otherwise they're ditching a recipe forever. If a meal was just too salty, or the vegetables weren't soft enough or whatever then these are fixable problems.
But a good chef doesn't give the tasters only 2 options to describe why they don't like it, she lets them describe why they don't like it in a freeform way - like a write in box which was included in the survey.
Your own example doesn't pan out (pun intended); if a chef asks if people like a meal and they say they hate it, then a good chef wants to know why, otherwise they're ditching a recipe forever. If a meal was just too salty, or the vegetables weren't soft enough or whatever then these are fixable problems.
But a good chef doesn't give the tasters only 2 options to describe why they don't like it, she lets them describe why they don't like it in a freeform way - like a write in box which was included in the survey.
Which is of precisely no value if you already boiled the decision down to pass/fail and ditched the menu before reading the comments. Because pass/fail can't give you a sense of how many people disliked the meal but would like an improved version. Because some people might like the idea of a restaurant serving spaghetti, they'd just prefer it not to include literal hot garbage as a garnish.
Turns out a poor metaphor doesn't change the problem when you're not really listening to what people's complaints are. 😉
For the millionth time; the problem is that the survey gives five options that serve no purpose, as they could offer either two (pass/fail) and get the exact same results, or three (good/improve/redesign) and get more information upon which to base a decision, without adding reading they won't do.
But this feels like it's going nowhere; the argument is that it was a bad survey, and the defences are… that it's okay because it's a bad survey? Why even reply to say that? It's saying nothing. Bad survey is bad is hardly rocket science.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
Getting away from this whole chef analogy. The argument is not that it’s a bad survey, it’s that they may not have been looking for the information you think they were looking for. Unless you know what they are doing internally with the information, you can’t say it was poorly designed. They might be getting exactly what they wanted from it. Or it could be they initially thought it was important to have more granular data, and they moved away from it, but re-designing the survey was a pain. It could be they weight the answers. It could be a million things. And, again, they’ve said repeatedly that they read every comment. The fact the some posters think that’s impractical doesn’t mean it is or that they don’t do it. It actually seems pretty reasonable to me. You figure each designer is working on one thing, a subclass, spells, feats, etc., and they only read the comments about the thing they’re working on, it really cuts down the volume.
As far as them needing to know why people didn’t like it, it really doesn’t matter that much. People don’t like this subclass, they’ve got 10 more ideas on the shelf, maybe one of those will hit. And as the ardling showed us, trying to adjust a thing to make it more popular can be a fool’s errand. They released it, some people liked it, some wanted it to be something different, some wanted it different, but in a different way. They make a change that makes some people happier, others are still unhappy, and some of the people who liked it initially fall away off the idea. They spend a couple months of staff time on it, and eventually they realize no amount of tinkering is going to get it across the finish line. Or at least, the amount of tinkering it would take can’t happen in the amount of time they have. The lesson is, it doesn’t matter why a given player doesn’t like it, because someone else liked it the original way, and a third person wanted the opposite kind of change.
The argument is not that it’s a bad survey, it’s that they may not have been looking for the information you think they were looking for.
The problem is that we do know what they were at least claiming to be looking for, because they put out videos talking about it.
Yeah. They want to know, subjectively, if people like it, and possibly how much they like it. They’re not interested in John Q Public saying they would like it if X. They don’t care if some Internet forum dweller (like me, hi) thinks the white room dpr calculation is lower then some platonic ideal. Just if it’s fun. This isn’t new. There have been dozens of things they’ve pitched in UAs for the last 10 years that ended up being cut. The entire process is, and has been, a popularity contest, not an attempt to solicit advice from the community.
The argument is not that it’s a bad survey, it’s that they may not have been looking for the information you think they were looking for.
Why do people keeping trying to make this non-argument? If they were only looking for pass/fail results, then it is a bad survey for multiple reasons:
If all they care about it pass/fail, then they don't need five options in the questions, because the middle three options serve no purpose. It's making answering the questions more difficult for no reason.
Only asking for pass/fail doesn't give good results for making a decision; the decision the developers are supposed to be trying to make is whether to keep a feature, refine it, or go back to the drawing board, but boiling it down into a pass/fail score can't do that, because at best you're guessing at what people meant. With the five option format the middle three are poorly defined, so can't be used to infer anything useful.
No matter how you cut it the survey is a) presenting options as overly complex and b) gathering less data than it should be. To make good decisions you need better data, and either way they're not doing that. Now they could gather that extra data and still not use it properly, or ignore it, those always remain a possibility, but not gathering it at all makes it a certainty.
Because either the survey is more of a pain in the ass to fill out than it needs to be to deliver a simple pass/fail score at the end, or they could be gathering results that are a whole lot more useful for their stated purpose (gauging support for proposed changes). Either way the surveys need improving – why on earth would you want them to remain worse than they could be?
They want to know, subjectively, if people like it, and possibly how much they like it.
Then they're doing it wrong the way with they've framed the survey, because it's impossible to give useful in between answers with the questions they are asking. If at the end the answer is pass/fail, there is no other possibility with the way they're conducting it, so it's not actually possible to gauge how much people really liked something, or why they didn't.
They’re not interested in John Q Public saying they would like it if X.
Literally nobody is proposing that they be told what to implement, and you are now one of several people parroting this ridiculous, and frankly insulting, straw-man argument. It's extremely bad faith for people to keep dismissing critics of the survey as wanting to be in charge.
What I and others are asking for is a way to indicate "pass/improve/fail" rather than merely "pass/fail". THAT'S IT.
So they would actually have clearer, more useful data to work with, where it's actually possible to give a proper "middle" answer for features that are interesting but not good in their present form. It's still up to them if they act on it, or how they improve a feature if they choose to do that.
We have repeatedly given the same examples of things like the proposed Wild Shape templates and Warlock half-casting as ideas that people had no means of which to say "it's an okay idea, but it needs more work"; features we know they immediately axed because they didn't get a high enough pass/fail score. Because we know they can't be reading all the text replies, no matter what they say (they'd need to employ dozens of staff at least for that sole purpose just to stand a chance of getting through it, and I doubt anyone believes they've done that).
I'm getting real tired of trying to explain what the argument is to people who don't seem to want to know, yet keep replying anyway.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
Your own example doesn't pan out (pun intended); if a chef asks if people like a meal and they say they hate it, then a good chef wants to know why, otherwise they're ditching a recipe forever. If a meal was just too salty, or the vegetables weren't soft enough or whatever then these are fixable problems.
But a good chef doesn't give the tasters only 2 options to describe why they don't like it, she lets them describe why they don't like it in a freeform way - like a write in box which was included in the survey.
Which is of precisely no value if you already boiled the decision down to pass/fail and ditched the menu before reading the comments. Because pass/fail can't give you a sense of how many people disliked the meal but would like an improved version. Because some people might like the idea of a restaurant serving spaghetti, they'd just prefer it not to include literal hot garbage as a garnish.
Turns out a poor metaphor doesn't change the problem when you're not really listening to what people's complaints are. 😉
For the millionth time; the problem is that the survey gives five options that serve no purpose, as they could offer either two (pass/fail) and get the exact same results, or three (good/improve/redesign) and get more information upon which to base a decision, without adding reading they won't do.
But this feels like it's going nowhere; the argument is that it was a bad survey, and the defences are… that it's okay because it's a bad survey? Why even reply to say that? It's saying nothing. Bad survey is bad is hardly rocket science.
See the problem with your argument is that is based on the premise that they did not use the write-in information at all at any point, which they have explicitly stated is not true. They do read & consider the write in comments and have repeated brought up specific issues people had with specific classes or features. Some examples:
- mentioned that people want Wild Shape to scale so they can continue to WS into the same animal across all levels rather than being forced to swap to a completely different animal / theme at higher levels b/c there are not higher CR versions of them.
- mentioned that people want warlocks to be able to cast more spells.
- mentioned that people don't like the exhaustion cost to the Beserker Barbarian's 3rd level feature.
- mentioned that people want monks to be able to use their features more often
So no, the survey results were not boiled down into purely pass/fail, they also considered the write-in comments for what people wanted changed for those features that scored either middling or poorly.
A thing to bear in mind about write-in comments is that it isn't really necessary to read all of them. Pick a random sample of 100 and you should get 95% of the issues that at least 3% of the respondents care about, and issues that are rare enough that they don't appear in your sample are likely also rare enough to not be worth doing anything about.
See the problem with your argument is that is based on the premise that they did not use the write-in information at all at any point
No it isn't, though I do have legitimate doubts about their capacity to read all written feedback, as I do not believe for a moment that they've hired enough staff just to do it considering they've had lay offs in the past.
which they have explicitly stated is not true. They do read & consider the write in comments
All we know is that they've read some comments; to have considered them all would require them to have a lot of staff dedicated purely to doing that, or to be feeding into some kind of algorithm, though if it's the latter the results of such are dubious.
The most charitable option is that they're grabbing a random sampling of comments from negative or positive responses where they're interested in more detail, because it's the only practical way to do something like that at scale. But that still means an enormous amount of information has to be ignored, and crucially, key information is lost.
And yet again, you're ignoring what is being asked for. A three option response where one of the responses indicates "room for improvement" is what I am saying is needed, because it allows them to easily and immediately see features that players could support in an improved form. It gives access to a crucial piece of information for a proper quick decision, without any reading required, and makes answers both simpler and more explicit.
If your argument is "but they don't need that because they've got five options, three more than they need, and can just read thousands of responses and manually count up the key arguments to maybe glean what you could have got instantly if it were simply an option" then your argument is still that you want the survey less useful, and less easy to use?
But at this point we've had what… 3-4 pages of people defending the survey without actually reading what the criticisms of it are, or bothering to read what the proposals for improving the process are, so why should I expect that to change now? I'm unsubscribing now as I've already said at least a dozen times exactly what my argument/proposal actually is if anyone would like to go back and read it for a change.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
I'm pretty sure there's science behind why surveys use 5-point scales. But it's not my field so I won't try to go into it.
I'm also pretty sure they don't care about "I like it but it needs fixing" answers on the scale, because they are not soliciting design advice. They are not very interested in the difference between "I don't like this idea" and "I like this idea but not this implementation," because all that matters in the end is the ultimate product (both are just "I don't like this to some degree" and the marketers are asking, not the designers).
And if there is some magic community-sourced idea that actually comes up, they figure they'll catch it in the short-answer comments (because if it's a worthwhile idea, it'll show up way more than once).
I'm also pretty sure they don't care about "I like it but it needs fixing" answers on the scale, because they are not soliciting design advice. They are not very interested in the difference between "I don't like this idea" and "I like this idea but not this implementation," because all that matters in the end is the ultimate product
This is contradicted by both what they've said and what they've done. They aren't looking for design advice, but having an idea of why people like or don't like something is definitely important to making a successful product.
I'm also pretty sure they don't care about "I like it but it needs fixing" answers on the scale, because they are not soliciting design advice. They are not very interested in the difference between "I don't like this idea" and "I like this idea but not this implementation," because all that matters in the end is the ultimate product
This is contradicted by both what they've said and what they've done. They aren't looking for design advice, but having an idea of why people like or don't like something is definitely important to making a successful product.
The "why," to the extent that it's useful feedback for them, is put in the short-answer fields. Can't fit all the dimensions of possible reasons in a voting scale.
See the problem with your argument is that is based on the premise that they did not use the write-in information at all at any point
No it isn't, though I do have legitimate doubts about their capacity to read all written feedback, as I do not believe for a moment that they've hired enough staff just to do it considering they've had lay offs in the past.
which they have explicitly stated is not true. They do read & consider the write in comments
All we know is that they've read some comments; to have considered them all would require them to have a lot of staff dedicated purely to doing that, or to be feeding into some kind of algorithm, though if it's the latter the results of such are dubious.
The most charitable option is that they're grabbing a random sampling of comments from negative or positive responses where they're interested in more detail, because it's the only practical way to do something like that at scale. But that still means an enormous amount of information has to be ignored, and crucially, key information is lost.
And yet again, you're ignoring what is being asked for. A three option response where one of the responses indicates "room for improvement" is what I am saying is needed, because it allows them to easily and immediately see features that players could support in an improved form. It gives access to a crucial piece of information for a proper quick decision, without any reading required, and makes answers both simpler and more explicit.
If your argument is "but they don't need that because they've got five options, three more than they need, and can just read thousands of responses and manually count up the key arguments to maybe glean what you could have got instantly if it were simply an option" then your argument is still that you want the survey less useful, and less easy to use?
But at this point we've had what… 3-4 pages of people defending the survey without actually reading what the criticisms of it are, or bothering to read what the proposals for improving the process are, so why should I expect that to change now? I'm unsubscribing now as I've already said at least a dozen times exactly what my argument/proposal actually is if anyone would like to go back and read it for a change.
whether you believe them or not is up to you, but they have claimed after the rumor they weren't reading that they in fact read every comment, and they didnt have a ton of people doing it. Regardless based on crawfords talks, and the way the UA evolved, they are definitely aware, through some means or another what peoples thought on many issues are. Not everyone agrees with how they resolved or didnt resolve those concerns, but there is a bunch of evidence they considered many things, not just the satisfaction scale.
I'm pretty sure there's science behind why surveys use 5-point scales. But it's not my field so I won't try to go into it.
For those interested, Qualtrics is my favorite website for constructing surveys and understanding many aspects of surveys.
One reason for not using yes/no questions is called Acquiescence bias. People filling out surveys tend to choose a yes or a positive response as a default answer. See also Acquiescence bias - Wikipedia. Surveys use a rating scale, some of which are called Likert Scales, to provide several options that can reduce and remove bias.
When customers share their responses to different product versions, insights about their preferences and other factors, such as whether they perceive the difference, can be gleaned. Remember the two versions of Warlock in UA Playtest?
Question design for surveys is essential. Those text boxes are open-ended questions and very useful. For example, do responders who give extreme answers also provide a text box answer? Search patterns and AI extract significant insights. Each time the tools improve, further re-analysis is possible.
Keep answering surveys; ratings and text box content make a difference.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
why ask someone a question you have no intention to act on. Whatever they decide for this phb is basically staying until there is a new phb. And it dilutes your results, I want to now how much you like this thing I made, not how much you like what it could be.
with current questions you know how many hate something, how many could take or leave it, and how many like it. If you throw in needs work, suddenly you are taking feedback on something that doesnt exist and may never exist, and its taking some of the feedback aaway from the other thing. Someone may hate the current implementation, but think the idea is OK, or love the implementation but think it needs work. Or not really care but think it needs work.
now you ve taken 3 people and you no longer have a clue how much they like or dislike what you have presented. To do the question justice, you'd need more questions. And another factor in surveys, is asking more questions sometimes influences responses. Asking what would you improve makes people respond differently than, do you like this.
and more questions alsomequals more time and factors to process. Its a not simple bonus to ask more questions and why bother if you aren't going to act on that info..
no, they were interested in how much players liked or disliked what they presented, not what players think needed more work, those are two very different questions as you and others pointed out. If a match maker asks you which of these people is attractive enough to marry, it doesnt mean they are interested in knowing which ones might be attractive enough if they lost weight, or changed their hair.
And that fact doesnt mean asking the question was pointless.
The point of degrees of satisfaction, is to create data on how satisfied or hated something is, which they did make use of when they really want to. They claimed that even though uniform progression tested well, above their metrics for satisfaction, it was not "very satisfied" enough to keep against the lower number of people who emphatically disliked it, and between the lines, the extra effort it would require to implement. So they do take and make use of degree of satisfaction data when they are looking for granularity.
And I'm not saying the surveys were great, I'm saying the surveys were 'ok' because the information they were interested in was limited. Or another way to put it, its not the surveys you have a problem with, you want the designers to incorporate different feedback into their process. But flawed or not the creator has to decide what type of feedback they are willing to take action on. There is no strong need to take feedback on things you are not going to change. Its not wrong for you as a tester/user to wish they cared or took feedback you give, but making something is only as heavily driven by feedback as the creator chooses, its their prerogative. And there are pros and cons to being more or less driven by feedback in design.
When I first saw the surveys I thought they were really bad, but now that I see the full process, I realized they simply were not that interested in things I thought they were, and they didnt really have time, interest, or resources to take or act on the type of feedback I wanted them too. Look at the results, could they have actually spent more time refining druid templates for this phb based on player feedback? not really, it takes them 2 months per cycle, and this thing needed to be finished by a certain date. I personally think brawler should have had more passes, but I can't really claim they could actually do it. They barely got to the planned dates. In fact they seem to have delayed it, as first it seemed they were going to attempt to get 3 books in before the end of the year. Ideally, I would set up a better process and content pipeline so they could test and act on more feedback faster, but thats not what they have right now.
does that mean the UAs weren't improved by feedback? imo they were.
If that were the case, they wouldn't have the poll at all. There are plenty of things they have decided to produce revised versions of, and the decision whether to revise, try something entirely new, revert, or keep the new version is not one they'd make before producing the survey. This certainly does not mean that everything will be acted on, it just means they should design their survey to be useful.
Except they're not; because whether people like or dislike something only tells part of the story, it tells you nothing about why someone disliked something, it doesn't tell you that a feature might be a sound idea but poorly executed, it tells you nothing about what the player base may or may not support if implemented differently.
It's cutting out a massive chunk of useful information on which better, more efficient decisions can be made.
But I feel like you're still not listening to what I and others are saying at all; there's no use repeating that they only care about pass/fail, because that's exactly the problem that we're complaining about. The entire point is that they should be gathering more useful data via simpler questions, rather than less useful data using more complex, more ambiguous ones.
I never said asking the question is pointless, I said giving us five options is. If all they care about is pass/fail then they only need to ask for a pass/fail answer to get the same results, because it doesn't matter if respondent 693 was "satisfied" rather than "very satisfied" by a change, because with thousands responding the volume of responses gives you the exact same pass/fail percentage in the end.
Once again, the complaint is that they either:
Either way you cut it the survey is poorly constructed, because the options are overly complex and ambiguous, and wasting time as a result.
And the point of complaints about that methodology is that it tells you nothing about why people dislike something; again, there have been multiple ideas presented in the playtest that weren't bad ideas in themselves (templated Wild Shape, half-caster Warlock) but they were extremely poorly implemented, so it was inevitable they got overwhelmingly dissatisfied results, because we were given no means by which to communicate that we liked the idea but hated the execution.
We never really got to playtest either of those proposals because they were doomed from the start by poor execution, so it was inevitable they'd get a negative pass/fail score. But that never meant they were bad ideas; many of these discussing in the forums recognised that there were many, many ways to make these things viable, some not even all that difficult, but the pass/fail nature meant the features were never going to get an update, because there was no possibility of that happening.
So once again, they eliminated useful feedback they could have used to come back with better versions of those ideas, rather than axing them and starting again, which is potentially wasting developer time when they could have just been refining what they already put forward. Everyone should want want them to use better surveys, because even if you don't use all of the extra data, having the data is better than not, and the developers wasting last time for better results is better for everybody.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
They have the poll to figure out if the things they are doing are broadly hated or liked, not because they plan for the testers to decide how they plan to handle design process. There is a difference between a chef taking a survey on how many people like the dishes they have created, and asking people how to cook.
I'm not saying no one has any good advice on how to cook, but most chefs aren't asking for that type of feedback, many actively don't want it. And thats their prerogative as the chef. It doesnt mean finding out how the various meals were received is pointless at all.
I want to know if you ve liked what I made, and I as the designer will decide what to do with that information based on myriad factors you aren't aware of is completely valid. At the end of the day this is their product and they are the ones who will sink or swim with it.
if you are trying to get rating scale, introducing an option that isnt about how satisfied people are messes up the information. You created a third category which says nothing about how much responents like what you produced. lets say 70% of people choose an option templates needs more work. You now have no idea how much 70% of the people who answered a question about how much they liked the templates you created , liked or hated your idea. You got a vague response of needs work. They could like or hate it and say it needs work. Does it need a little or a lot of work. And thats even worse if you already know you have 3 options. Now you can't compare the satisfaction rating of plan B to plan A, and you can't compare it to satisfaction levels of the live product.
Even if they wanted that information, it wouldn't be the appropriate place to seek it. And, its not always better to try to get more information from a single survey. You think its an objectively better survey if it has those questions, but thats questionable in survey design.
Your own example doesn't pan out (pun intended); if a chef asks if people like a meal and they say they hate it, then a good chef wants to know why, otherwise they're ditching a recipe forever. If a meal was just too salty, or the vegetables weren't soft enough or whatever then these are fixable problems.
But again your argument is essentially "they only care if you liked it" but that's exactly the problem we're complaining about. Because only caring a feature scores low or high, but not why, means they're making decisions with only half of the information. What is the point of repeatedly telling us that the problem is exactly what we've said it is? Why are you so determined to shout down legitimate complaints about the process? What the hell do you get out of that?
And again you're accusing us of wanting us to tell them how to fix it, but that's not what we're asking for; a three option response with a clear choice of "needs improving" isn't telling them what to do, it's just expressing a desire to see something improved rather than immediately axed, it's still on them to actually listen, or to focus on improvements rather than going back to the drawing board. But a feature with an overwhelming response of "needs improving" is a much more useful result than a middling figure on the pass/fail scale.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
But a good chef doesn't give the tasters only 2 options to describe why they don't like it, she lets them describe why they don't like it in a freeform way - like a write in box which was included in the survey.
Which is of precisely no value if you already boiled the decision down to pass/fail and ditched the menu before reading the comments. Because pass/fail can't give you a sense of how many people disliked the meal but would like an improved version. Because some people might like the idea of a restaurant serving spaghetti, they'd just prefer it not to include literal hot garbage as a garnish.
Turns out a poor metaphor doesn't change the problem when you're not really listening to what people's complaints are. 😉
For the millionth time; the problem is that the survey gives five options that serve no purpose, as they could offer either two (pass/fail) and get the exact same results, or three (good/improve/redesign) and get more information upon which to base a decision, without adding reading they won't do.
But this feels like it's going nowhere; the argument is that it was a bad survey, and the defences are… that it's okay because it's a bad survey? Why even reply to say that? It's saying nothing. Bad survey is bad is hardly rocket science.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
Getting away from this whole chef analogy.
The argument is not that it’s a bad survey, it’s that they may not have been looking for the information you think they were looking for. Unless you know what they are doing internally with the information, you can’t say it was poorly designed. They might be getting exactly what they wanted from it. Or it could be they initially thought it was important to have more granular data, and they moved away from it, but re-designing the survey was a pain. It could be they weight the answers. It could be a million things. And, again, they’ve said repeatedly that they read every comment. The fact the some posters think that’s impractical doesn’t mean it is or that they don’t do it. It actually seems pretty reasonable to me. You figure each designer is working on one thing, a subclass, spells, feats, etc., and they only read the comments about the thing they’re working on, it really cuts down the volume.
As far as them needing to know why people didn’t like it, it really doesn’t matter that much. People don’t like this subclass, they’ve got 10 more ideas on the shelf, maybe one of those will hit.
And as the ardling showed us, trying to adjust a thing to make it more popular can be a fool’s errand. They released it, some people liked it, some wanted it to be something different, some wanted it different, but in a different way. They make a change that makes some people happier, others are still unhappy, and some of the people who liked it initially fall away off the idea. They spend a couple months of staff time on it, and eventually they realize no amount of tinkering is going to get it across the finish line. Or at least, the amount of tinkering it would take can’t happen in the amount of time they have.
The lesson is, it doesn’t matter why a given player doesn’t like it, because someone else liked it the original way, and a third person wanted the opposite kind of change.
The problem is that we do know what they were at least claiming to be looking for, because they put out videos talking about it.
Yeah. They want to know, subjectively, if people like it, and possibly how much they like it. They’re not interested in John Q Public saying they would like it if X. They don’t care if some Internet forum dweller (like me, hi) thinks the white room dpr calculation is lower then some platonic ideal. Just if it’s fun. This isn’t new. There have been dozens of things they’ve pitched in UAs for the last 10 years that ended up being cut. The entire process is, and has been, a popularity contest, not an attempt to solicit advice from the community.
Why do people keeping trying to make this non-argument? If they were only looking for pass/fail results, then it is a bad survey for multiple reasons:
No matter how you cut it the survey is a) presenting options as overly complex and b) gathering less data than it should be. To make good decisions you need better data, and either way they're not doing that. Now they could gather that extra data and still not use it properly, or ignore it, those always remain a possibility, but not gathering it at all makes it a certainty.
Because either the survey is more of a pain in the ass to fill out than it needs to be to deliver a simple pass/fail score at the end, or they could be gathering results that are a whole lot more useful for their stated purpose (gauging support for proposed changes). Either way the surveys need improving – why on earth would you want them to remain worse than they could be?
Then they're doing it wrong the way with they've framed the survey, because it's impossible to give useful in between answers with the questions they are asking. If at the end the answer is pass/fail, there is no other possibility with the way they're conducting it, so it's not actually possible to gauge how much people really liked something, or why they didn't.
Literally nobody is proposing that they be told what to implement, and you are now one of several people parroting this ridiculous, and frankly insulting, straw-man argument. It's extremely bad faith for people to keep dismissing critics of the survey as wanting to be in charge.
What I and others are asking for is a way to indicate "pass/improve/fail" rather than merely "pass/fail". THAT'S IT.
So they would actually have clearer, more useful data to work with, where it's actually possible to give a proper "middle" answer for features that are interesting but not good in their present form. It's still up to them if they act on it, or how they improve a feature if they choose to do that.
We have repeatedly given the same examples of things like the proposed Wild Shape templates and Warlock half-casting as ideas that people had no means of which to say "it's an okay idea, but it needs more work"; features we know they immediately axed because they didn't get a high enough pass/fail score. Because we know they can't be reading all the text replies, no matter what they say (they'd need to employ dozens of staff at least for that sole purpose just to stand a chance of getting through it, and I doubt anyone believes they've done that).
I'm getting real tired of trying to explain what the argument is to people who don't seem to want to know, yet keep replying anyway.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
See the problem with your argument is that is based on the premise that they did not use the write-in information at all at any point, which they have explicitly stated is not true. They do read & consider the write in comments and have repeated brought up specific issues people had with specific classes or features. Some examples:
- mentioned that people want Wild Shape to scale so they can continue to WS into the same animal across all levels rather than being forced to swap to a completely different animal / theme at higher levels b/c there are not higher CR versions of them.
- mentioned that people want warlocks to be able to cast more spells.
- mentioned that people don't like the exhaustion cost to the Beserker Barbarian's 3rd level feature.
- mentioned that people want monks to be able to use their features more often
So no, the survey results were not boiled down into purely pass/fail, they also considered the write-in comments for what people wanted changed for those features that scored either middling or poorly.
A thing to bear in mind about write-in comments is that it isn't really necessary to read all of them. Pick a random sample of 100 and you should get 95% of the issues that at least 3% of the respondents care about, and issues that are rare enough that they don't appear in your sample are likely also rare enough to not be worth doing anything about.
No it isn't, though I do have legitimate doubts about their capacity to read all written feedback, as I do not believe for a moment that they've hired enough staff just to do it considering they've had lay offs in the past.
All we know is that they've read some comments; to have considered them all would require them to have a lot of staff dedicated purely to doing that, or to be feeding into some kind of algorithm, though if it's the latter the results of such are dubious.
The most charitable option is that they're grabbing a random sampling of comments from negative or positive responses where they're interested in more detail, because it's the only practical way to do something like that at scale. But that still means an enormous amount of information has to be ignored, and crucially, key information is lost.
And yet again, you're ignoring what is being asked for. A three option response where one of the responses indicates "room for improvement" is what I am saying is needed, because it allows them to easily and immediately see features that players could support in an improved form. It gives access to a crucial piece of information for a proper quick decision, without any reading required, and makes answers both simpler and more explicit.
If your argument is "but they don't need that because they've got five options, three more than they need, and can just read thousands of responses and manually count up the key arguments to maybe glean what you could have got instantly if it were simply an option" then your argument is still that you want the survey less useful, and less easy to use?
But at this point we've had what… 3-4 pages of people defending the survey without actually reading what the criticisms of it are, or bothering to read what the proposals for improving the process are, so why should I expect that to change now? I'm unsubscribing now as I've already said at least a dozen times exactly what my argument/proposal actually is if anyone would like to go back and read it for a change.
Former D&D Beyond Customer of six years: With the axing of piecemeal purchasing, lack of meaningful development, and toxic moderation the site isn't worth paying for anymore. I remain a free user only until my groups are done migrating from DDB, and if necessary D&D, after which I'm done. There are better systems owned by better companies out there.
I have unsubscribed from all topics and will not reply to messages. My homebrew is now 100% unsupported.
I'm pretty sure there's science behind why surveys use 5-point scales. But it's not my field so I won't try to go into it.
I'm also pretty sure they don't care about "I like it but it needs fixing" answers on the scale, because they are not soliciting design advice. They are not very interested in the difference between "I don't like this idea" and "I like this idea but not this implementation," because all that matters in the end is the ultimate product (both are just "I don't like this to some degree" and the marketers are asking, not the designers).
And if there is some magic community-sourced idea that actually comes up, they figure they'll catch it in the short-answer comments (because if it's a worthwhile idea, it'll show up way more than once).
This is contradicted by both what they've said and what they've done. They aren't looking for design advice, but having an idea of why people like or don't like something is definitely important to making a successful product.
The "why," to the extent that it's useful feedback for them, is put in the short-answer fields. Can't fit all the dimensions of possible reasons in a voting scale.
whether you believe them or not is up to you, but they have claimed after the rumor they weren't reading that they in fact read every comment, and they didnt have a ton of people doing it. Regardless based on crawfords talks, and the way the UA evolved, they are definitely aware, through some means or another what peoples thought on many issues are. Not everyone agrees with how they resolved or didnt resolve those concerns, but there is a bunch of evidence they considered many things, not just the satisfaction scale.
For those interested, Qualtrics is my favorite website for constructing surveys and understanding many aspects of surveys.
One reason for not using yes/no questions is called Acquiescence bias. People filling out surveys tend to choose a yes or a positive response as a default answer. See also Acquiescence bias - Wikipedia. Surveys use a rating scale, some of which are called Likert Scales, to provide several options that can reduce and remove bias.
When customers share their responses to different product versions, insights about their preferences and other factors, such as whether they perceive the difference, can be gleaned. Remember the two versions of Warlock in UA Playtest?
Question design for surveys is essential. Those text boxes are open-ended questions and very useful. For example, do responders who give extreme answers also provide a text box answer? Search patterns and AI extract significant insights. Each time the tools improve, further re-analysis is possible.
Keep answering surveys; ratings and text box content make a difference.