It should be noted that there is not a single poll in existence which does not have sampling problems. It is not the job of the surveyor to design a perfect poll—it is their job to recognise the biases, determine which data is useful, and find ways to mitigate any biases which exist.
The first point has been expounded upon already by folks on this thread - most notably limited visibility of the poll and the length turning off some users. Wizards, who has been doing data collection on their game for decades and who utilises professional statisticians, certainly recognises these biases.
Acknowledging the biases exist, there are still a number of ways this poll’s data is, in and of itself, useful.
For starters, Wizards does want to know more about Beyond users. They are trying to build up Beyond as the major D&D hub, which means they’ll want to know more about Beyond customers. Having a survey that disproportionately catches Beyond users (and includes questions verifying if they are Beyond users), results in a great deal of information about one group of customers you are specifically trying to reach, weeding out a large number of responses that might not be as relevant to those interests.
Second, longer surveys weed out the indifferent respondents—those who feel strongly about an issue (either positively or negatively) are more likely to sit through a longer survey to make sure their voices can be heard. That reduces the amount of data you have to sift through and ensures you can collect the most amount of data from each respondent. After all, often it is better to have more data from the extremes, from which a middle ground might be extrapolated, than less data from the middle, that doesn’t get you all the information you need due to the shorter length.
Next, the survey collects a lot of data about player habits and preferences, which can be used to compare data within a survey. It might be hard to generalise “players who play in DM like X,” and “players who are only players like Y” to all DMs and all player-only players due to sampling issues. But within the sample, the comparison can be useful—“these two groups collected with the same methodology are similar in these ways, and different in these ways.” That provides a fair bit of utility. This is another reason why a longer survey with more variables to sort by might be more valuable than a wider-reaching survey that does not provide as many variables.
The data is also useful in terms of monitoring trends. Wizards does these kinds of surveys fairly regularly - watching how the data changes within similarly-situated survey groups is extremely helpful. Those kinds of trends are some of the more easy to extrapolate generalised information from - if you are seeing patterns in the group you are consistently surveying, that likely means those patterns exist beyond the surveyed group.
It should also be noted that Wizards relies on these surveys, but does not solely rely on these surveys. Book sales are a big part of their data collection - you can get a fairly generalised picture of what folks like based on what books individuals are buying. Beyond expands on that by giving Wizards a way to see what subclasses, species, etc. are being purchased individually. Additionally, they may have other data collection methods, such as information received from LGSes and other sources.
So, yes, the methodology has flaws. That doesn’t mean the survey is without utility and it certainly doesn’t mean the survey is the only data point relied upon.
It should be noted that there is not a single poll in existence which does not have sampling problems. It is not the job of the surveyor to design a perfect poll—it is their job to recognise the biases, determine which data is useful, and find ways to mitigate any biases which exist.
The first point has been expounded upon already by folks on this thread - most notably limited visibility of the poll and the length turning off some users. Wizards, who has been doing data collection on their game for decades and who utilises professional statisticians, certainly recognises these biases.
Acknowledging the biases exist, there are still a number of ways this poll’s data is, in and of itself, useful.
For starters, Wizards does want to know more about Beyond users. They are trying to build up Beyond as the major D&D hub, which means they’ll want to know more about Beyond customers. Having a survey that disproportionately catches Beyond users (and includes questions verifying if they are Beyond users), results in a great deal of information about one group of customers you are specifically trying to reach, weeding out a large number of responses that might not be as relevant to those interests.
Second, longer surveys weed out the indifferent respondents—those who feel strongly about an issue (either positively or negatively) are more likely to sit through a longer survey to make sure their voices can be heard. That reduces the amount of data you have to sift through and ensures you can collect the most amount of data from each respondent. After all, often it is better to have more data from the extremes, from which a middle ground might be extrapolated, than less data from the middle, that doesn’t get you all the information you need due to the shorter length.
Next, the survey collects a lot of data about player habits and preferences, which can be used to compare data within a survey. It might be hard to generalise “players who play in DM like X,” and “players who are only players like Y” to all DMs and all player-only players due to sampling issues. But within the sample, the comparison can be useful—“these two groups collected with the same methodology are similar in these ways, and different in these ways.” That provides a fair bit of utility. This is another reason why a longer survey with more variables to sort by might be more valuable than a wider-reaching survey that does not provide as many variables.
The data is also useful in terms of monitoring trends. Wizards does these kinds of surveys fairly regularly - watching how the data changes within similarly-situated survey groups is extremely helpful. Those kinds of trends are some of the more easy to extrapolate generalised information from - if you are seeing patterns in the group you are consistently surveying, that likely means those patterns exist beyond the surveyed group.
It should also be noted that Wizards relies on these surveys, but does not solely rely on these surveys. Book sales are a big part of their data collection - you can get a fairly generalised picture of what folks like based on what books individuals are buying. Beyond expands on that by giving Wizards a way to see what subclasses, species, etc. are being purchased individually. Additionally, they may have other data collection methods, such as information received from LGSes and other sources.
So, yes, the methodology has flaws. That doesn’t mean the survey is without utility and it certainly doesn’t mean the survey is the only data point relied upon.
If you need to be reminded many were those who spent the month of January and many are those still saying Wizards don't really understand their own customers.
You don't think a major source of that concern beyond the OGL and other concerns is their plan to make Beyond "the major D&D hub" like this doesn't send a message to those who don't even use Beyond that they and how they play the game are now an afterthought for those who make the game?
Before the advent of the internet there was no such "hub." There needn't be.
I am very, very skeptical of anything that tries to "spotify" a traditional pastime. I ain't alone in that skepticism.
There is every possibility that players who use Beyond have disparate views to those who do not. Both are players of D&D. But they are very different types of consumers.
It's like someone who buys books exclusively from an online bookstore or who will only read e-books and someone who only buys physical books from brick-and-mortal stores. You can't expect the results from surveys targeting only the first of these to reflect the views of the general reading population.
And you can't pretend it's only due to indifference that many will not take longer surveys. Some simply don't have the patience to have to repeat themselves as some even in this thread have already said. Some face the usual time constraints work or life force upon us. I've lost count of how many times over the years I've started to complete some survey and not then reached the end by the end of my break or before I've had tend to a matter of some urgency. The survey in question may very well be weeding out the indifferent. But it is also weeding out those who have more important things to do but whose engagement with the hobby is no less important than those who don't.
As I already mentioned, I didn't even know such a survey was taking place until I saw the original post. That is someone who is at the table on average four to seven times a month who wasn't even conscious of the survey's existence. If you'd like another reason for why many are saying Wizards don't really understand their own customers.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
Will we have access to the ANOVA when this survey is done?
It's been a long time, if ever, that's WotC has actually published any data gathered from these surveys, which are put out pretty regularly, I'd say at least twice a year if not quarterly.
Seems a lot of the contempt for the survey being based in D&D Beyond seems to be based in some presumption that WotC or their market research consultant is exclusively using DDB as a means to gather information on the D&D market. As if the OGL debacle recovery wasn't largely communicated through channels outside D&D Beyond, engaging with a community outside DDB of whom existence is well aware of. I mean, how many DDB regulars are attending the D&D Influencers Content Creator's Junket Summit next month? DDB is one tool on which WotC among many.
Anyway, I got a sense they were focusing on two things this time around. 1.) They're still trying to figure out the best balance between digital and physical content and determine how they're being used, might be the first survey that's asked about the physical bundle experiment. 2.) There seems to be a false dichotomy in the questions between playing as written and playing "homebrew". I think it's good that they're asking questions about homebrew in a way where I don't think they're trying to "compete" with homebrew options but hopefully find more ways they can offer support for home brewing, or at least that's my optimistic take on why they may be pushing inquiry into players "as written" to "homebrew" ratios. I don't know off the top of my head what would be a better way of asking but it was one of those things where the survey read like they really didn't know how to account for homebrew or didn't recognize that "as written" to "homebrew" is more a spectrum or kaleidoscope than static binary.
It's all weird because really they have so much data in DDB usage itself I don't see why that's not relied on, or maybe that's another tool. I mean early days DDB used to pull "hey this is sorta neat" user usage stats out in the dev summaries. Now those were made more for entertainment purposes, but I'm pretty sure someone could actually produce findings or data from DDB users based on their actual usage that might help drive some decisions regarding the game.
Rollback Post to RevisionRollBack
Jander Sunstar is the thinking person's Drizzt, fight me.
Will we have access to the ANOVA when this survey is done?
It's been a long time, if ever, that's WotC has actually published any data gathered from these surveys, which are put out pretty regularly, I'd say at least twice a year if not quarterly.
Seems a lot of the contempt for the survey being based in D&D Beyond seems to be based in some presumption that WotC or their market research consultant is exclusively using DDB as a means to gather information on the D&D market. As if the OGL debacle recovery wasn't largely communicated through channels outside D&D Beyond, engaging with a community outside DDB of whom existence is well aware of. I mean, how many DDB regulars are attending the D&D Influencers Content Creator's Junket Summit next month? DDB is one tool on which WotC among many.
Anyway, I got a sense they were focusing on two things this time around. 1.) They're still trying to figure out the best balance between digital and physical content and determine how they're being used, might be the first survey that's asked about the physical bundle experiment. 2.) There seems to be a false dichotomy in the questions between playing as written and playing "homebrew". I think it's good that they're asking questions about homebrew in a way where I don't think they're trying to "compete" with homebrew options but hopefully find more ways they can offer support for home brewing, or at least that's my optimistic take on why they may be pushing inquiry into players "as written" to "homebrew" ratios. I don't know off the top of my head what would be a better way of asking but it was one of those things where the survey read like they really didn't know how to account for homebrew or didn't recognize that "as written" to "homebrew" is more a spectrum or kaleidoscope than static binary.
It's all weird because really they have so much data in DDB usage itself I don't see why that's not relied on, or maybe that's another tool. I mean early days DDB used to pull "hey this is sorta neat" user usage stats out in the dev summaries. Now those were made more for entertainment purposes, but I'm pretty sure someone could actually produce findings or data from DDB users based on their actual usage that might help drive some decisions regarding the game.
Cast your mind back to January and how commentary on these forums could have given anyone the false impression "most players" supported measures to replace the existing open game licence.
How did that work out for you all?
It would be corporate suicide for Wizards to put all their eggs in one basket and just hope the general opinion here is going to be that of most players.
Beyond might be transformed into the official website for D&D but it is where a particular type of consumer engages with the hobby and as we've learned is not the most reliable measure of what most players think or feel.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
I'll note, for those who skimmed the survey itself, one of the 'what tools do you use to play D&D' questions had an answer option of 'I've never heard of D&D Beyond.' That said, if they were intending this to be a genuine answer option, I also looked to see if WotC had linked the survey on their twitter or website, and didn't see it linked in either place. Possibly it was linked on some other social media? But it seems odd to have that poll option and not at least have the survey linked on the website.
Will we have access to the ANOVA when this survey is done?
It's been a long time, if ever, that's WotC has actually published any data gathered from these surveys, which are put out pretty regularly, I'd say at least twice a year if not quarterly.
Seems a lot of the contempt for the survey being based in D&D Beyond seems to be based in some presumption that WotC or their market research consultant is exclusively using DDB as a means to gather information on the D&D market. As if the OGL debacle recovery wasn't largely communicated through channels outside D&D Beyond, engaging with a community outside DDB of whom existence is well aware of. I mean, how many DDB regulars are attending the D&D Influencers Content Creator's Junket Summit next month? DDB is one tool for WotC among many.
Anyway, I got a sense they were focusing on two things this time around. 1.) They're still trying to figure out the best balance between digital and physical content and determine how they're being used, might be the first survey that's asked about the physical bundle experiment. 2.) There seems to be a false dichotomy in the questions between playing as written and playing "homebrew". I think it's good that they're asking questions about homebrew in a way where I don't think they're trying to "compete" with homebrew options but hopefully find more ways they can offer support for home brewing, or at least that's my optimistic take on why they may be pushing inquiry into players "as written" to "homebrew" ratios. I don't know off the top of my head what would be a better way of asking but it was one of those things where the survey read like they really didn't know how to account for homebrew or didn't recognize that "as written" to "homebrew" is more a spectrum or kaleidoscope than static binary.
It's all weird because really they have so much data in DDB usage itself I don't see why that's not relied on, or maybe that's another tool. I mean early days DDB used to pull "hey this is sorta neat" user usage stats out in the dev summaries. Now those were made more for entertainment purposes, but I'm pretty sure someone could actually produce findings or data from DDB users based on their actual usage that might help drive some decisions regarding the game.
Cast your mind back to January and how commentary on these forums could have given anyone the false impression "most players" supported measures to replace the existing open game licence.
How did that work out for you all?
It would be corporate suicide for Wizards to put all their eggs in one basket and just hope the general opinion here is going to be that of most players.
Beyond might be transformed into the official website for D&D but it is where a particular type of consumer engages with the hobby and as we've learned is not the most reliable measure of what most players think or feel.
This is a weirdly antagonistic take, and confusing since you seem to think you're writing a rebuttal. I don't know what "you all" you think I represent; but if you cast your mind (I tend to just read back in this instance and in other instances I remember, but if that's how you operate, cast away) to what you're actually responding to, it's pretty clear I don't believe a survey posted by WotC on DDB is some sort of ultimate source of insight into how D&D is played. I only question why formally poll DDB because there's enough DDB data re: use and sales and that probably tells them something Beyond, so to speak, what DDB users will selectively report. But it's pretty clear WotC uses many tools and vectors to get insight into the game. Hope that clears you up.
Rollback Post to RevisionRollBack
Jander Sunstar is the thinking person's Drizzt, fight me.
Will we have access to the ANOVA when this survey is done?
It's been a long time, if ever, that's WotC has actually published any data gathered from these surveys, which are put out pretty regularly, I'd say at least twice a year if not quarterly.
Seems a lot of the contempt for the survey being based in D&D Beyond seems to be based in some presumption that WotC or their market research consultant is exclusively using DDB as a means to gather information on the D&D market. As if the OGL debacle recovery wasn't largely communicated through channels outside D&D Beyond, engaging with a community outside DDB of whom existence is well aware of. I mean, how many DDB regulars are attending the D&D Influencers Content Creator's Junket Summit next month? DDB is one tool for WotC among many.
Anyway, I got a sense they were focusing on two things this time around. 1.) They're still trying to figure out the best balance between digital and physical content and determine how they're being used, might be the first survey that's asked about the physical bundle experiment. 2.) There seems to be a false dichotomy in the questions between playing as written and playing "homebrew". I think it's good that they're asking questions about homebrew in a way where I don't think they're trying to "compete" with homebrew options but hopefully find more ways they can offer support for home brewing, or at least that's my optimistic take on why they may be pushing inquiry into players "as written" to "homebrew" ratios. I don't know off the top of my head what would be a better way of asking but it was one of those things where the survey read like they really didn't know how to account for homebrew or didn't recognize that "as written" to "homebrew" is more a spectrum or kaleidoscope than static binary.
It's all weird because really they have so much data in DDB usage itself I don't see why that's not relied on, or maybe that's another tool. I mean early days DDB used to pull "hey this is sorta neat" user usage stats out in the dev summaries. Now those were made more for entertainment purposes, but I'm pretty sure someone could actually produce findings or data from DDB users based on their actual usage that might help drive some decisions regarding the game.
Cast your mind back to January and how commentary on these forums could have given anyone the false impression "most players" supported measures to replace the existing open game licence.
How did that work out for you all?
It would be corporate suicide for Wizards to put all their eggs in one basket and just hope the general opinion here is going to be that of most players.
Beyond might be transformed into the official website for D&D but it is where a particular type of consumer engages with the hobby and as we've learned is not the most reliable measure of what most players think or feel.
This is a weirdly antagonistic take, and confusing since you seem to think you're writing a rebuttal. I don't know what "you all" you think I represent; but if you cast your mind (I tend to just read back in this instance and in other instances I remember, but if that's how you operate, cast away) to what you're actually responding to, it's pretty clear I don't believe a survey posted by WotC on DDB is some sort of ultimate source of insight into how D&D is played. I only question why formally poll DDB because there's enough DDB data re: use and sales and that probably tells them something Beyond, so to speak, what DDB users will selectively report. But it's pretty clear WotC uses many tools and vectors to get insight into the game. Hope that clears you up.
You suggested Wizards could rely on data pulled from Beyond to make decisions regarding and impacting the development of the game.
I have pointed out how players who use Beyond and those who don't are different types of consumers.
Have pointed out how general opinion here does not appear to reflect that of a broader cross-section of the hobby's playerbase.
And pointed out how it would be folly for the company to think otherwise.
If I bring up the OGL mess it's as a sore reminder of just how wrong Wizards were to think they knew what most players want.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
You're fighting your own misunderstanding. Your map of misreading is simple. Yes, I wondered if the survey hosted on this site is worth it since DDB itself has much more and much more raw data than a survey respondent will selectively support. I _also_ wrote at the very start the notion that this survey or DDB is the only tool through which WotC gathers players, a notion that seems to be behind so much of the silly antagonism such as yours, is a false premise. That's the second time I've explained this to you. If you continue to misrepresent what I've written here, I'll just assume you either are simply trolling or have some sort of disorder where your misunderstandings are held in your faulty world view as strong opinion or something.
This is what you are doing. Because you keep responding to me as if I'm just talking about the survey and not more broadly about Beyond.
How does raw data taken from Beyond provide any indication of what the broader community want?
Can you answer that question instead of just repeating that it does as if it isn't the notion this site does so that I've been contesting?
How does the data that can be harvested here on a site that for the most part is only used by those of us who play online reflect the views of those of us who only play at tables?
It doesn't.
Remember what happened when Wizards tried to cater to video game players by producing the failure that was 4e?
Redesigning the game to appeal only to those who consume and play the game one way would be corporate suicide.
If that's what you want then don't act surprised when the official version of game goes the way of the dinosaur.
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
How does raw data taken from Beyond provide any indication of what the broader community want?
The data from Beyond is much like the data from any survey - incomplete, but incomplete doesn’t mean the data is useless. There are things you can derive from Beyond’s data - most notably what users of Beyond like and trends which can be extrapolated to a general population (as mentioned earlier, trends within data taken from the same sample can be extrapolated because you have similar sampling between data collection times).
Anyone who took a high school statistic or science class knows that every single source of data collection is flawed. As a general rule, the best way to get better data is by diversifying the sources of your data. As was already said on this thread, you do not want to just use Beyond data; you do not want to just use corporeal book sale data; you do not want to just use survey data—you want to take a bunch of different points of data and combine them together to get a more clear picture of the whole.
heres the few things i can tell from those who say the survey is too long to take... - you don't have patience... you should, those surveys are there to help shape the very fundation of the game, this takes time, its not something that gets fixed on a small 5 minute break... all surveys, even on th ephone with people, always take between 10 and 30 minutes. When they ask your opinion on it, you should be ready to take that survey and reserve 30 minutes for it.
- you feel like those questions repeat serves no purpose and think they are detrimental to the survey or the answers... i will ask you this much... what good is a survey written by someone who just want the most abusive class on the planet and be godlike and unkillable in a game ? as an exemple to this... the new druid wildshape is already making the big round table on the net as the nerf that kills the druid and rendersit the worse possible class ever made. the only reason to that is that they removed its ability to have infinite hit points by just wildshaping often. the people who blame this nerf are the ones we don't want feedback from, because all they want is not a blanced game at all, all they want is to be godlike and will take survey only to unbalance the game more. so again... are those questions and weighting really unnecessary ? those weighted question becomes necessary when there are thousands and thousands of people answering. because we want ot categorise them for ease of reading content.
- you start skipping because you dont want to repeat yourself... a perfectly good point, but then the question becomes... why would we think your survey is important if you dont take it seriously ? this is the real concern of any surveys... the last thing we want is to have surveys about people just venting for sake of venting. we want "FEEDBACK" not "VENTING" now that doesn't mean you skipping things will be ignored, many parts of what you actually did will be used in certain categories. so basically any of the feedback you gave will be used, reguardless of you filling it in entirely or not. but the question remains, if people can't finish the survey because they get bored... why would the company take that person seriously if that person isn't serious about it ?
- you also seem to think those surveys are studied on a personnal basis, as in your specific opinion is of utmost importance. it is not utmost important, it is "as important" as all the others... meaning your personnal opnion is mixed in with those people that thinks like you. AKA categories. if enough people wants the druid wildshape to be more balanced, then of course reguardless of what you thinkm, that druid change will happen because majority, not personnal opnion, but majority of opinion wanted it to be ! so in that reguard, it not their fault you think its boring to take the survey, it is yours for thinking your personnal and individual opinion was important when its not what they are after. they are after many opinions of the same thing. they searching for patterns of what people desires.
knowing all of this, then people who are venting are not doing the survey justice. we don't want them... we want all feedback because we wanna know how many people want those changes. and in the end, the most important lesson is... we cannot please everyone, thats why we want to please the majority. important to note that i use "WE" as in the company and those doing surveys. i am not really an expert on such things, but even i know that creating a company you need to create a survey to se if there is a demand to begin with. thi si the final point one has to consider when answering surveys...
they are searching demands, not request ! request is you wanting something for yourself, as an individual... demand is the sum of all the request in an entire district ! they are seeking demands, not request !
i love their survey cause they are straight to the point and each questions explains itself as to what they wanna know. but never take a survey on a personnal level... thats never a good thing to do. those surveys do not want your personnal individuality.
Rollback Post to RevisionRollBack
DM of two gaming groups. Likes to create stuff. Check out my homebrew --> Monsters --> Magical Items --> Races --> Subclasses If you like --> Upvote, If you wanna comment --> Comment
Play by Post Games --> One Shot Adventure - House of Artwood (DM) (Completed)
How does raw data taken from Beyond provide any indication of what the broader community want?
The data from Beyond is much like the data from any survey - incomplete, but incomplete doesn’t mean the data is useless. There are things you can derive from Beyond’s data - most notably what users of Beyond like and trends which can be extrapolated to a general population (as mentioned earlier, trends within data taken from the same sample can be extrapolated because you have similar sampling between data collection times).
Anyone who took a high school statistic or science class knows that every single source of data collection is flawed. As a general rule, the best way to get better data is by diversifying the sources of your data. As was already said on this thread, you do not want to just use Beyond data; you do not want to just use corporeal book sale data; you do not want to just use survey data—you want to take a bunch of different points of data and combine them together to get a more clear picture of the whole.
They need to be doing this. But I've grave doubts they are.
Beyond book sales how they are harvesting data from those who don't use Beyond? Knowing too that some who do also buy physical books? How many players who don't use Beyond are not even conscious of their being such surveys? How else can they make themselves heard if indeed that's the case?
Seems to me one type of consumer is being forgotten. Those who exclusively play in person and who only went online during lockdowns if indeed that was necessary. I think it would be unwise for them to think their numbers are of little consequence and what Beyond users want as far as development decisions go is sufficient in that regard.
Someone else mentioned how they now see Beyond as the "hub" of the game. Many players are concerned this means they will ultimately move the game more and more away from the table until whatever the latest edition of the game will then be has become something no longer played at tables. Because it is here where they can make more money from groups of players who might otherwise just buy one or two sets of core rules between them. It's a cop-out to tell people they can just play an older edition if they want to do that if the only reason the game would have taken that path was because one type of consumer got to dominate the conversation.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
How does raw data taken from Beyond provide any indication of what the broader community want?
The data from Beyond is much like the data from any survey - incomplete, but incomplete doesn’t mean the data is useless. There are things you can derive from Beyond’s data - most notably what users of Beyond like and trends which can be extrapolated to a general population (as mentioned earlier, trends within data taken from the same sample can be extrapolated because you have similar sampling between data collection times).
Anyone who took a high school statistic or science class knows that every single source of data collection is flawed. As a general rule, the best way to get better data is by diversifying the sources of your data. As was already said on this thread, you do not want to just use Beyond data; you do not want to just use corporeal book sale data; you do not want to just use survey data—you want to take a bunch of different points of data and combine them together to get a more clear picture of the whole.
They need to be doing this. But I've grave doubts they are.
Beyond book sales how they are harvesting data from those who don't use Beyond? Knowing too that some who do also buy physical books? How many players who don't use Beyond are not even conscious of their being such surveys? How else can they make themselves heard if indeed that's the case?
Seems to me one type of consumer is being forgotten. Those who exclusively play in person and who only went online during lockdowns if indeed that was necessary. I think it would be unwise for them to think their numbers are of little consequence and what Beyond users want as far as development decisions go is sufficient in that regard.
Someone else mentioned how they now see Beyond as the "hub" of the game. Many players are concerned this means they will ultimately move the game more and more away from the table until whatever the latest edition of the game will then be has become something no longer played at tables. Because it is here where they can make more money from groups of players who might otherwise just buy one or two sets of core rules between them. It's a cop-out to tell people they can just play an older edition if they want to do that if the only reason the game would have taken that path was because one type of consumer got to dominate the conversation.
You seem to have two concerns - one actually related to this thread and one that is not. I’ll respond to your point about data, but will not derail this thread discussing your conspiracy theories about Beyond (conspiracy theories that Wizards has pretty clearly disavowed).
Wizards is very good at data collection. While their implementation of that data at a game design and management level might be lacking, they still clearly understand how to collect and manage data.
Take, for example, your fear that Beyond purchased and paper purchases might have overlap—they control for that specifically with surveys like this, which specifically ask if the respondent bought the books digitally, in paper, or both. That gives them some insight into how large the “both” might be as a percentage of sales, better controlling for issues in data obtained elsewhere.
Now, are there methods perfect? No. But, as you have ignored a couple of times, there simply isn’t a way to collect perfect data. That’s why they collect from diverse sources.
And, even past the data, they have talked numerous times about their interfacing with local game stores as an alternative method of gaining information about players who might not otherwise interface online. It is not direct information from those players, but they’re a group who isn’t easy to poll and something is better than nothing.
So, yes, you are correct that there are flaws. But, if you—someone who clearly doesn’t understand data science and collection—can see the flaws, you can bet that the professional statisticians on Wizards’ staff also see them and are working to address them.
I just wAnt to point out that data is only ever flawed in relation to the expectations for the analysis and use for that date -- the data itself is only flawed if it contains information that is not desired as it doesn't meet the needs or purpose of data collection aspect as a whole -- methodology merely points to an approach, and has implications for how that analysis can be used, and so forth.
A company that rakes in 5.86 billion dollars in a year and that is built on having a broad foundational data set -- especially one that has significant assets in the form of IP -- does not need to go out and collect data from people who are not using DDB in order to develop a strong and broad data structure regarding those who play D&D.
Thinking otherwise is something worthy of derision and mockery, even if it stems from a deep and abiding level of ignorance. The reason is simple -- proof positive rests in the simple continued existence of D&D and the generally successful product assortment found.
Some of the quoted bits I am seeing -- and I only see them through quotes -- are filled with poor grasp of much beyond the simple fact that they did a survey, it was done here, and no body knows what they plan to do with that information.
wild fantasies and imaginings about those poor people who don't use DDB or equally lurid horror stories of those poor people who do are equally and foundationally bad, because they are "whataboutisms" and pearl clutching and sorry but it makes as much sense as me pearl clutching because my 78 year old neighbor was not walking her dog as fast today as she was yesterday when I see them out my office window.
in part because there are no pearls here.
I get that some folks are just angry at WotC. That nothing they do will ever be good enough, that everything they do will be suspect and a problem and it will somehow ruin a game older than most of the people who play it, just like it ruined monopoly in the 30's, 50's, 70's, and 90's.
Which is an intentionally written statement meant to provoke a sense of the absurd.
I am having a tad bit of success in avoiding them thanks to the tools that DDB provides to ignore them, but nevertheless, it is important to note that DDB is merely a tool. It is going to be the home of D&D online. It is going to perpetuate some things that many different people find upsetting for many different reasons. And while the survey may or may not have something to do with that (because any kind of talking about what they might do with it or how they will do it or even the value of the data they collected is utterly and wholly being stated without any kind of real knowledge beyond the most uselessly vague stuff), it happened, and it was not a badly done survey.
What they plan to do with it, why they did it,and all the rest isn't something anyone can address because this is not a place for those people who are behind the surveys to interact openly.
it is a place for those who don't have anything to do with them to interact with each other.
Rollback Post to RevisionRollBack
Only a DM since 1980 (3000+ Sessions) / PhD, MS, MA / Mixed, Bi, Trans, Woman / No longer welcome in the US, apparently
Wyrlde: Adventures in the Seven Cities .-=] Lore Book | Patreon | Wyrlde YT [=-. An original Setting for 5e, a whole solar system of adventure. Ongoing updates, exclusies, more. Not Talking About It / Dubbed The Oracle in the Cult of Mythology Nerds
You seem to have two concerns - one actually related to this thread and one that is not. I’ll respond to your point about data, but will not derail this thread discussing your conspiracy theories about Beyond (conspiracy theories that Wizards has pretty clearly disavowed).
Wizards is very good at data collection. While their implementation of that data at a game design and management level might be lacking, they still clearly understand how to collect and manage data.
Take, for example, your fear that Beyond purchased and paper purchases might have overlap—they control for that specifically with surveys like this, which specifically ask if the respondent bought the books digitally, in paper, or both. That gives them some insight into how large the “both” might be as a percentage of sales, better controlling for issues in data obtained elsewhere.
Now, are there methods perfect? No. But, as you have ignored a couple of times, there simply isn’t a way to collect perfect data. That’s why they collect from diverse sources.
And, even past the data, they have talked numerous times about their interfacing with local game stores as an alternative method of gaining information about players who might not otherwise interface online. It is not direct information from those players, but they’re a group who isn’t easy to poll and something is better than nothing.
So, yes, you are correct that there are flaws. But, if you—someone who clearly doesn’t understand data science and collection—can see the flaws, you can bet that the professional statisticians on Wizards’ staff also see them and are working to address them.
Only time will tell as far as where they intend to take the game in the future. Brinks may have clearly disavowed the idea the game might become exclusively digital in a series of interviews. Tomorrow is another day and all it would take is for all those "professional statisticians" to determine that seems to be way to go if the company is to make much more money for itself and for its investors.
None of us can predict what is going to happen. Not me. Not you. You can call them conspiracy theories all you like. I really don't care. You're not going to scare me out of having my doubts the company actually cares about the game's legacy as much as it does making a buck and doing so from newer and newer generations of players who practically live on the internet.
They control for that overlap with a question on a survey disproportionately completed by those who use Beyond. OK. That was hardly the point I was making. It is mostly players who use either both or just digital who use Beyond while most if not indeed all of those who only buy physical books might not even visit this site and are then likely not even participating.
Despite responses from that demographic who only buy physical books and who only play in person being crucial to the data's having any real value in telling them what they want to know if what they want to know is where to take the game if they intend to keep the game at tables and not make these conspiracies theories a reality?
I obviously don't understand data science and collection if that is supposed to make any sense.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
They control for that overlap with a question on a survey disproportionately completed by those who use Beyond. OK. That was hardly the point I was making. It is mostly players who use either both or just digital who use Beyond while most if not indeed all of those who only buy physical books might not even visit this site and are then likely not even participating.
Despite responses from that demographic who only buy physical books and who only play in person being crucial to the data's having any real value in telling them what they want to know if what they want to know is where to take the game if they intend to keep the game at tables and not make these conspiracies theories a reality?
I obviously don't understand data science and collection if that is supposed to make any sense.
Let’s try rephrasing the above in a way you might be able to understand.
Mr. Survey wants to learn about the whole population of the game. Unfortunately, Mr. Survey cannot really poll the offline portion of the game - he doesn’t really have a way of getting to them because they tend to be offline.
Mr. Survey does have the ability to poll online people - they’re far easier to get data from. He also knows that an online poll might pick up some offline players, and that a lot of offline players use the online tools. He also has the ability to see how many total physical books and total digital books are sold.
His online poll suggests 25% of online poll respondents bought both a physical and digital book. He has 400 sales of that digital book. Extrapolating that data, it can be estimated that 100 people bought both versions. He knows he has 1,000 physical books. Subtract out the 100 who are estimated to have bought the digital as well, that lets you estimate 900 people bought physical only.
Is that going to be 100% accurate? Of course not, but nothing would be. Will it be enough to get a decent, if not perfect, picture? Sure, especially if used in conjunction with other sources of data.
At its core, that is Mr. Survey’s job—it isn’t about just collecting the data and getting the information about the data you have, it is about using the data you have to derive information about the data you do not have.
They control for that overlap with a question on a survey disproportionately completed by those who use Beyond. OK. That was hardly the point I was making. It is mostly players who use either both or just digital who use Beyond while most if not indeed all of those who only buy physical books might not even visit this site and are then likely not even participating.
Despite responses from that demographic who only buy physical books and who only play in person being crucial to the data's having any real value in telling them what they want to know if what they want to know is where to take the game if they intend to keep the game at tables and not make these conspiracies theories a reality?
I obviously don't understand data science and collection if that is supposed to make any sense.
Let’s try rephrasing the above in a way you might be able to understand.
Mr. Survey wants to learn about the whole population of the game. Unfortunately, Mr. Survey cannot really poll the offline portion of the game - he doesn’t really have a way of getting to them because they tend to be offline.
Mr. Survey does have the ability to poll online people - they’re far easier to get data from. He also knows that an online poll might pick up some offline players, and that a lot of offline players use the online tools. He also has the ability to see how many total physical books and total digital books are sold.
His online poll suggests 25% of online poll respondents bought both a physical and digital book. He has 400 sales of that digital book. Extrapolating that data, it can be estimated that 100 people bought both versions. He knows he has 1,000 physical books. Subtract out the 100 who are estimated to have bought the digital as well, that lets you estimate 900 people bought physical only.
Is that going to be 100% accurate? Of course not, but nothing would be. Will it be enough to get a decent, if not perfect, picture? Sure, especially if used in conjunction with other sources of data.
At its core, that is Mr. Survey’s job—it isn’t about just collecting the data and getting the information about the data you have, it is about using the data you have to derive information about the data you do not have.
This is twice now you've acted like this is all about collecting the numbers for that overlap between those who buy digital books and those who buy physical books and those who only do the latter so you can tell me what I already know. That was not the point of my questions concerning those who only do the latter.
I'm not talking about who is buying what. I'm talking about these being different types of consumers wanting very different things out of the game.
You are saying it's nothing more than a "conspiracy theory" the idea that the game might become more and more digitized until whatever the then latest edition is is no longer intended to be played around a table ...
But those of us who will only ever play around tables and only ever play with pen and paper are the least likely to be participating in a survey hosted on a website used mostly by those who play over the internet the results of which are said to be able to help the company make decisions that will impact the future direction of the game.
So it's we don't really need to listen to those who only play the game as we promise it will always be played?
What utter poppycock.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
Though a brand new account whose first post was on a thread known to be frequented by alternate accounts of trolls and who seems chronically incapable of understanding what others are saying is not exactly inspiring confidence, I'll bite and give you the benefit of the doubt.
Let's try to explain this in as simple of terms as possible.
1. There are other sources being used in addition to sales. These have been mentioned before on this thread.
2. Sales are a great way of obtaining data. Seeing how well Book X sells, particularly as compared to Book Y, allows you to see what players like or do not like.
3. The group you want to find out more information about is notoriously hard to collect data on--the very thing that makes them a distinct demographic (lack of online presence) is the very thing that makes them very difficult to get data on.
4. Sales data is often the best way of collecting numerical data on those folks--since you cannot survey them easily you have to make do with the data you can get.
5. Anyone who understands how data collection works--which you admit you do not--understands that this is simply how the world works. You use what information you can to find the information you cannot.
Though a brand new account whose first post was on a thread known to be frequented by alternate accounts of trolls and who seems chronically incapable of understanding what others are saying is not exactly inspiring confidence, I'll bite and give you the benefit of the doubt.
Let's try to explain this in as simple of terms as possible.
1. There are other sources being used in addition to sales. These have been mentioned before on this thread.
2. Sales are a great way of obtaining data. Seeing how well Book X sells, particularly as compared to Book Y, allows you to see what players like or do not like.
3. The group you want to find out more information about is notoriously hard to collect data on--the very thing that makes them a distinct demographic (lack of online presence) is the very thing that makes them very difficult to get data on.
4. Sales data is often the best way of collecting numerical data on those folks--since you cannot survey them easily you have to make do with the data you can get.
5. Anyone who understands how data collection works--which you admit you do not--understands that this is simply how the world works. You use what information you can to find the information you cannot.
So that's a yes: Wizards don't really need to listen to those who only play the game as Wizards promise it will always be played.
It's "not exactly inspiring confidence" to be told by people who play online there's no need to worry.
One doesn't have to have a sound understanding of how data collection works to understand it only takes a drop of common sense to see why people are worried if it's newer and newer generations of players who might strictly or mostly play online who will then get to transform the game.
We are talking about different types of consumers. Their differences are not limited to how they play D&D any more than the only difference between those who choose to support a local bookstore and those who always buy books from Amazon is how they buy books.
As if their attitudes, beliefs and values play no part in informing how they go about consuming things and no correlation exists between these and what might then inform their other decisions and what they then think might be best for the game.
When those deciding the direction the game will take—and I'm not just talking about whether or not it will remain at tables—are just one type of consumer you wind up with a situation that would be like a publishing house only listening to what one type of consumer among those who buy their books has to say to determine what it will publish in the future.
They have openly said the game is "under-monetized" and we all know this is because some groups get by with one or two set of rules.
Just how do you reckon they intend to "fix that problem"?
Make it a necessity to have an account here at the "hub" and to subscribe to view books?
These are all "what ifs" ...
Only time will tell.
I love D&D and don't want to be telling you I told you so.
Rollback Post to RevisionRollBack
INSPIRATIONS:Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
Another way of explaining this is this survey is not The Ring of Power, it is not the One Survey to Rule All Other Understandings of the future of D&D. It is but a member of fellowship of research tools used to together to get some sense of the state of D&D use.
This survey, as discussed, comes out in iterations at least 2-4 times a year. It is not the sole driver of WotC's understanding of the interests and practices of the entirety of the D&D player base. Anyone who thinks otherwise isn't coming from an evidence based perspective but one of misdirected passions and rancor. The survey just isn't the high stakes McGuffin whose casting into Mt.Doom will save the World's Greatest Role Playing Game. There's no meta to be played here to apex the development of the game to one particular mode of affinity.
Rollback Post to RevisionRollBack
Jander Sunstar is the thinking person's Drizzt, fight me.
To post a comment, please login or register a new account.
It should be noted that there is not a single poll in existence which does not have sampling problems. It is not the job of the surveyor to design a perfect poll—it is their job to recognise the biases, determine which data is useful, and find ways to mitigate any biases which exist.
The first point has been expounded upon already by folks on this thread - most notably limited visibility of the poll and the length turning off some users. Wizards, who has been doing data collection on their game for decades and who utilises professional statisticians, certainly recognises these biases.
Acknowledging the biases exist, there are still a number of ways this poll’s data is, in and of itself, useful.
For starters, Wizards does want to know more about Beyond users. They are trying to build up Beyond as the major D&D hub, which means they’ll want to know more about Beyond customers. Having a survey that disproportionately catches Beyond users (and includes questions verifying if they are Beyond users), results in a great deal of information about one group of customers you are specifically trying to reach, weeding out a large number of responses that might not be as relevant to those interests.
Second, longer surveys weed out the indifferent respondents—those who feel strongly about an issue (either positively or negatively) are more likely to sit through a longer survey to make sure their voices can be heard. That reduces the amount of data you have to sift through and ensures you can collect the most amount of data from each respondent. After all, often it is better to have more data from the extremes, from which a middle ground might be extrapolated, than less data from the middle, that doesn’t get you all the information you need due to the shorter length.
Next, the survey collects a lot of data about player habits and preferences, which can be used to compare data within a survey. It might be hard to generalise “players who play in DM like X,” and “players who are only players like Y” to all DMs and all player-only players due to sampling issues. But within the sample, the comparison can be useful—“these two groups collected with the same methodology are similar in these ways, and different in these ways.” That provides a fair bit of utility. This is another reason why a longer survey with more variables to sort by might be more valuable than a wider-reaching survey that does not provide as many variables.
The data is also useful in terms of monitoring trends. Wizards does these kinds of surveys fairly regularly - watching how the data changes within similarly-situated survey groups is extremely helpful. Those kinds of trends are some of the more easy to extrapolate generalised information from - if you are seeing patterns in the group you are consistently surveying, that likely means those patterns exist beyond the surveyed group.
It should also be noted that Wizards relies on these surveys, but does not solely rely on these surveys. Book sales are a big part of their data collection - you can get a fairly generalised picture of what folks like based on what books individuals are buying. Beyond expands on that by giving Wizards a way to see what subclasses, species, etc. are being purchased individually. Additionally, they may have other data collection methods, such as information received from LGSes and other sources.
So, yes, the methodology has flaws. That doesn’t mean the survey is without utility and it certainly doesn’t mean the survey is the only data point relied upon.
Will we have access to the ANOVA when this survey is done?
Blank
If you need to be reminded many were those who spent the month of January and many are those still saying Wizards don't really understand their own customers.
You don't think a major source of that concern beyond the OGL and other concerns is their plan to make Beyond "the major D&D hub" like this doesn't send a message to those who don't even use Beyond that they and how they play the game are now an afterthought for those who make the game?
Before the advent of the internet there was no such "hub." There needn't be.
I am very, very skeptical of anything that tries to "spotify" a traditional pastime. I ain't alone in that skepticism.
And you can't pretend it's only due to indifference that many will not take longer surveys. Some simply don't have the patience to have to repeat themselves as some even in this thread have already said. Some face the usual time constraints work or life force upon us. I've lost count of how many times over the years I've started to complete some survey and not then reached the end by the end of my break or before I've had tend to a matter of some urgency. The survey in question may very well be weeding out the indifferent. But it is also weeding out those who have more important things to do but whose engagement with the hobby is no less important than those who don't.
As I already mentioned, I didn't even know such a survey was taking place until I saw the original post. That is someone who is at the table on average four to seven times a month who wasn't even conscious of the survey's existence. If you'd like another reason for why many are saying Wizards don't really understand their own customers.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
It's been a long time, if ever, that's WotC has actually published any data gathered from these surveys, which are put out pretty regularly, I'd say at least twice a year if not quarterly.
Seems a lot of the contempt for the survey being based in D&D Beyond seems to be based in some presumption that WotC or their market research consultant is exclusively using DDB as a means to gather information on the D&D market. As if the OGL debacle recovery wasn't largely communicated through channels outside D&D Beyond, engaging with a community outside DDB of whom existence is well aware of. I mean, how many DDB regulars are attending the D&D
InfluencersContent Creator'sJunketSummit next month? DDB is one tool on which WotC among many.Anyway, I got a sense they were focusing on two things this time around. 1.) They're still trying to figure out the best balance between digital and physical content and determine how they're being used, might be the first survey that's asked about the physical bundle experiment. 2.) There seems to be a false dichotomy in the questions between playing as written and playing "homebrew". I think it's good that they're asking questions about homebrew in a way where I don't think they're trying to "compete" with homebrew options but hopefully find more ways they can offer support for home brewing, or at least that's my optimistic take on why they may be pushing inquiry into players "as written" to "homebrew" ratios. I don't know off the top of my head what would be a better way of asking but it was one of those things where the survey read like they really didn't know how to account for homebrew or didn't recognize that "as written" to "homebrew" is more a spectrum or kaleidoscope than static binary.
It's all weird because really they have so much data in DDB usage itself I don't see why that's not relied on, or maybe that's another tool. I mean early days DDB used to pull "hey this is sorta neat" user usage stats out in the dev summaries. Now those were made more for entertainment purposes, but I'm pretty sure someone could actually produce findings or data from DDB users based on their actual usage that might help drive some decisions regarding the game.
Jander Sunstar is the thinking person's Drizzt, fight me.
Cast your mind back to January and how commentary on these forums could have given anyone the false impression "most players" supported measures to replace the existing open game licence.
How did that work out for you all?
It would be corporate suicide for Wizards to put all their eggs in one basket and just hope the general opinion here is going to be that of most players.
Beyond might be transformed into the official website for D&D but it is where a particular type of consumer engages with the hobby and as we've learned is not the most reliable measure of what most players think or feel.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
I'll note, for those who skimmed the survey itself, one of the 'what tools do you use to play D&D' questions had an answer option of 'I've never heard of D&D Beyond.' That said, if they were intending this to be a genuine answer option, I also looked to see if WotC had linked the survey on their twitter or website, and didn't see it linked in either place. Possibly it was linked on some other social media? But it seems odd to have that poll option and not at least have the survey linked on the website.
Birgit | Shifter | Sorcerer | Dragonlords
Shayone | Hobgoblin | Sorcerer | Netherdeep
This is a weirdly antagonistic take, and confusing since you seem to think you're writing a rebuttal. I don't know what "you all" you think I represent; but if you cast your mind (I tend to just read back in this instance and in other instances I remember, but if that's how you operate, cast away) to what you're actually responding to, it's pretty clear I don't believe a survey posted by WotC on DDB is some sort of ultimate source of insight into how D&D is played. I only question why formally poll DDB because there's enough DDB data re: use and sales and that probably tells them something Beyond, so to speak, what DDB users will selectively report. But it's pretty clear WotC uses many tools and vectors to get insight into the game. Hope that clears you up.
Jander Sunstar is the thinking person's Drizzt, fight me.
You suggested Wizards could rely on data pulled from Beyond to make decisions regarding and impacting the development of the game.
I have pointed out how players who use Beyond and those who don't are different types of consumers.
Have pointed out how general opinion here does not appear to reflect that of a broader cross-section of the hobby's playerbase.
And pointed out how it would be folly for the company to think otherwise.
If I bring up the OGL mess it's as a sore reminder of just how wrong Wizards were to think they knew what most players want.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
This is what you are doing. Because you keep responding to me as if I'm just talking about the survey and not more broadly about Beyond.
How does raw data taken from Beyond provide any indication of what the broader community want?
Can you answer that question instead of just repeating that it does as if it isn't the notion this site does so that I've been contesting?
How does the data that can be harvested here on a site that for the most part is only used by those of us who play online reflect the views of those of us who only play at tables?
It doesn't.
Remember what happened when Wizards tried to cater to video game players by producing the failure that was 4e?
Redesigning the game to appeal only to those who consume and play the game one way would be corporate suicide.
If that's what you want then don't act surprised when the official version of game goes the way of the dinosaur.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
The data from Beyond is much like the data from any survey - incomplete, but incomplete doesn’t mean the data is useless. There are things you can derive from Beyond’s data - most notably what users of Beyond like and trends which can be extrapolated to a general population (as mentioned earlier, trends within data taken from the same sample can be extrapolated because you have similar sampling between data collection times).
Anyone who took a high school statistic or science class knows that every single source of data collection is flawed. As a general rule, the best way to get better data is by diversifying the sources of your data. As was already said on this thread, you do not want to just use Beyond data; you do not want to just use corporeal book sale data; you do not want to just use survey data—you want to take a bunch of different points of data and combine them together to get a more clear picture of the whole.
heres the few things i can tell from those who say the survey is too long to take...
- you don't have patience... you should, those surveys are there to help shape the very fundation of the game, this takes time, its not something that gets fixed on a small 5 minute break... all surveys, even on th ephone with people, always take between 10 and 30 minutes. When they ask your opinion on it, you should be ready to take that survey and reserve 30 minutes for it.
- you feel like those questions repeat serves no purpose and think they are detrimental to the survey or the answers... i will ask you this much... what good is a survey written by someone who just want the most abusive class on the planet and be godlike and unkillable in a game ? as an exemple to this... the new druid wildshape is already making the big round table on the net as the nerf that kills the druid and rendersit the worse possible class ever made. the only reason to that is that they removed its ability to have infinite hit points by just wildshaping often. the people who blame this nerf are the ones we don't want feedback from, because all they want is not a blanced game at all, all they want is to be godlike and will take survey only to unbalance the game more. so again... are those questions and weighting really unnecessary ? those weighted question becomes necessary when there are thousands and thousands of people answering. because we want ot categorise them for ease of reading content.
- you start skipping because you dont want to repeat yourself... a perfectly good point, but then the question becomes... why would we think your survey is important if you dont take it seriously ? this is the real concern of any surveys... the last thing we want is to have surveys about people just venting for sake of venting. we want "FEEDBACK" not "VENTING" now that doesn't mean you skipping things will be ignored, many parts of what you actually did will be used in certain categories. so basically any of the feedback you gave will be used, reguardless of you filling it in entirely or not. but the question remains, if people can't finish the survey because they get bored... why would the company take that person seriously if that person isn't serious about it ?
- you also seem to think those surveys are studied on a personnal basis, as in your specific opinion is of utmost importance. it is not utmost important, it is "as important" as all the others... meaning your personnal opnion is mixed in with those people that thinks like you. AKA categories. if enough people wants the druid wildshape to be more balanced, then of course reguardless of what you thinkm, that druid change will happen because majority, not personnal opnion, but majority of opinion wanted it to be ! so in that reguard, it not their fault you think its boring to take the survey, it is yours for thinking your personnal and individual opinion was important when its not what they are after. they are after many opinions of the same thing. they searching for patterns of what people desires.
knowing all of this, then people who are venting are not doing the survey justice. we don't want them... we want all feedback because we wanna know how many people want those changes. and in the end, the most important lesson is... we cannot please everyone, thats why we want to please the majority. important to note that i use "WE" as in the company and those doing surveys. i am not really an expert on such things, but even i know that creating a company you need to create a survey to se if there is a demand to begin with. thi si the final point one has to consider when answering surveys...
they are searching demands, not request !
request is you wanting something for yourself, as an individual...
demand is the sum of all the request in an entire district !
they are seeking demands, not request !
i love their survey cause they are straight to the point and each questions explains itself as to what they wanna know.
but never take a survey on a personnal level... thats never a good thing to do. those surveys do not want your personnal individuality.
DM of two gaming groups.
Likes to create stuff.
Check out my homebrew --> Monsters --> Magical Items --> Races --> Subclasses
If you like --> Upvote, If you wanna comment --> Comment
Play by Post Games
--> One Shot Adventure - House of Artwood (DM) (Completed)
They need to be doing this. But I've grave doubts they are.
Beyond book sales how they are harvesting data from those who don't use Beyond? Knowing too that some who do also buy physical books? How many players who don't use Beyond are not even conscious of their being such surveys? How else can they make themselves heard if indeed that's the case?
Seems to me one type of consumer is being forgotten. Those who exclusively play in person and who only went online during lockdowns if indeed that was necessary. I think it would be unwise for them to think their numbers are of little consequence and what Beyond users want as far as development decisions go is sufficient in that regard.
Someone else mentioned how they now see Beyond as the "hub" of the game. Many players are concerned this means they will ultimately move the game more and more away from the table until whatever the latest edition of the game will then be has become something no longer played at tables. Because it is here where they can make more money from groups of players who might otherwise just buy one or two sets of core rules between them. It's a cop-out to tell people they can just play an older edition if they want to do that if the only reason the game would have taken that path was because one type of consumer got to dominate the conversation.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
You seem to have two concerns - one actually related to this thread and one that is not. I’ll respond to your point about data, but will not derail this thread discussing your conspiracy theories about Beyond (conspiracy theories that Wizards has pretty clearly disavowed).
Wizards is very good at data collection. While their implementation of that data at a game design and management level might be lacking, they still clearly understand how to collect and manage data.
Take, for example, your fear that Beyond purchased and paper purchases might have overlap—they control for that specifically with surveys like this, which specifically ask if the respondent bought the books digitally, in paper, or both. That gives them some insight into how large the “both” might be as a percentage of sales, better controlling for issues in data obtained elsewhere.
Now, are there methods perfect? No. But, as you have ignored a couple of times, there simply isn’t a way to collect perfect data. That’s why they collect from diverse sources.
And, even past the data, they have talked numerous times about their interfacing with local game stores as an alternative method of gaining information about players who might not otherwise interface online. It is not direct information from those players, but they’re a group who isn’t easy to poll and something is better than nothing.
So, yes, you are correct that there are flaws. But, if you—someone who clearly doesn’t understand data science and collection—can see the flaws, you can bet that the professional statisticians on Wizards’ staff also see them and are working to address them.
I just wAnt to point out that data is only ever flawed in relation to the expectations for the analysis and use for that date -- the data itself is only flawed if it contains information that is not desired as it doesn't meet the needs or purpose of data collection aspect as a whole -- methodology merely points to an approach, and has implications for how that analysis can be used, and so forth.
A company that rakes in 5.86 billion dollars in a year and that is built on having a broad foundational data set -- especially one that has significant assets in the form of IP -- does not need to go out and collect data from people who are not using DDB in order to develop a strong and broad data structure regarding those who play D&D.
Thinking otherwise is something worthy of derision and mockery, even if it stems from a deep and abiding level of ignorance. The reason is simple -- proof positive rests in the simple continued existence of D&D and the generally successful product assortment found.
Some of the quoted bits I am seeing -- and I only see them through quotes -- are filled with poor grasp of much beyond the simple fact that they did a survey, it was done here, and no body knows what they plan to do with that information.
wild fantasies and imaginings about those poor people who don't use DDB or equally lurid horror stories of those poor people who do are equally and foundationally bad, because they are "whataboutisms" and pearl clutching and sorry but it makes as much sense as me pearl clutching because my 78 year old neighbor was not walking her dog as fast today as she was yesterday when I see them out my office window.
in part because there are no pearls here.
I get that some folks are just angry at WotC. That nothing they do will ever be good enough, that everything they do will be suspect and a problem and it will somehow ruin a game older than most of the people who play it, just like it ruined monopoly in the 30's, 50's, 70's, and 90's.
Which is an intentionally written statement meant to provoke a sense of the absurd.
I am having a tad bit of success in avoiding them thanks to the tools that DDB provides to ignore them, but nevertheless, it is important to note that DDB is merely a tool. It is going to be the home of D&D online. It is going to perpetuate some things that many different people find upsetting for many different reasons. And while the survey may or may not have something to do with that (because any kind of talking about what they might do with it or how they will do it or even the value of the data they collected is utterly and wholly being stated without any kind of real knowledge beyond the most uselessly vague stuff), it happened, and it was not a badly done survey.
What they plan to do with it, why they did it,and all the rest isn't something anyone can address because this is not a place for those people who are behind the surveys to interact openly.
it is a place for those who don't have anything to do with them to interact with each other.
Only a DM since 1980 (3000+ Sessions) / PhD, MS, MA / Mixed, Bi, Trans, Woman / No longer welcome in the US, apparently
Wyrlde: Adventures in the Seven Cities
.-=] Lore Book | Patreon | Wyrlde YT [=-.
An original Setting for 5e, a whole solar system of adventure. Ongoing updates, exclusies, more.
Not Talking About It / Dubbed The Oracle in the Cult of Mythology Nerds
Only time will tell as far as where they intend to take the game in the future. Brinks may have clearly disavowed the idea the game might become exclusively digital in a series of interviews. Tomorrow is another day and all it would take is for all those "professional statisticians" to determine that seems to be way to go if the company is to make much more money for itself and for its investors.
None of us can predict what is going to happen. Not me. Not you. You can call them conspiracy theories all you like. I really don't care. You're not going to scare me out of having my doubts the company actually cares about the game's legacy as much as it does making a buck and doing so from newer and newer generations of players who practically live on the internet.
They control for that overlap with a question on a survey disproportionately completed by those who use Beyond. OK. That was hardly the point I was making. It is mostly players who use either both or just digital who use Beyond while most if not indeed all of those who only buy physical books might not even visit this site and are then likely not even participating.
Despite responses from that demographic who only buy physical books and who only play in person being crucial to the data's having any real value in telling them what they want to know if what they want to know is where to take the game if they intend to keep the game at tables and not make these conspiracies theories a reality?
I obviously don't understand data science and collection if that is supposed to make any sense.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
Let’s try rephrasing the above in a way you might be able to understand.
Mr. Survey wants to learn about the whole population of the game. Unfortunately, Mr. Survey cannot really poll the offline portion of the game - he doesn’t really have a way of getting to them because they tend to be offline.
Mr. Survey does have the ability to poll online people - they’re far easier to get data from. He also knows that an online poll might pick up some offline players, and that a lot of offline players use the online tools. He also has the ability to see how many total physical books and total digital books are sold.
His online poll suggests 25% of online poll respondents bought both a physical and digital book. He has 400 sales of that digital book. Extrapolating that data, it can be estimated that 100 people bought both versions. He knows he has 1,000 physical books. Subtract out the 100 who are estimated to have bought the digital as well, that lets you estimate 900 people bought physical only.
Is that going to be 100% accurate? Of course not, but nothing would be. Will it be enough to get a decent, if not perfect, picture? Sure, especially if used in conjunction with other sources of data.
At its core, that is Mr. Survey’s job—it isn’t about just collecting the data and getting the information about the data you have, it is about using the data you have to derive information about the data you do not have.
This is twice now you've acted like this is all about collecting the numbers for that overlap between those who buy digital books and those who buy physical books and those who only do the latter so you can tell me what I already know. That was not the point of my questions concerning those who only do the latter.
I'm not talking about who is buying what. I'm talking about these being different types of consumers wanting very different things out of the game.
You are saying it's nothing more than a "conspiracy theory" the idea that the game might become more and more digitized until whatever the then latest edition is is no longer intended to be played around a table ...
But those of us who will only ever play around tables and only ever play with pen and paper are the least likely to be participating in a survey hosted on a website used mostly by those who play over the internet the results of which are said to be able to help the company make decisions that will impact the future direction of the game.
So it's we don't really need to listen to those who only play the game as we promise it will always be played?
What utter poppycock.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
Though a brand new account whose first post was on a thread known to be frequented by alternate accounts of trolls and who seems chronically incapable of understanding what others are saying is not exactly inspiring confidence, I'll bite and give you the benefit of the doubt.
Let's try to explain this in as simple of terms as possible.
1. There are other sources being used in addition to sales. These have been mentioned before on this thread.
2. Sales are a great way of obtaining data. Seeing how well Book X sells, particularly as compared to Book Y, allows you to see what players like or do not like.
3. The group you want to find out more information about is notoriously hard to collect data on--the very thing that makes them a distinct demographic (lack of online presence) is the very thing that makes them very difficult to get data on.
4. Sales data is often the best way of collecting numerical data on those folks--since you cannot survey them easily you have to make do with the data you can get.
5. Anyone who understands how data collection works--which you admit you do not--understands that this is simply how the world works. You use what information you can to find the information you cannot.
So that's a yes: Wizards don't really need to listen to those who only play the game as Wizards promise it will always be played.
It's "not exactly inspiring confidence" to be told by people who play online there's no need to worry.
One doesn't have to have a sound understanding of how data collection works to understand it only takes a drop of common sense to see why people are worried if it's newer and newer generations of players who might strictly or mostly play online who will then get to transform the game.
We are talking about different types of consumers. Their differences are not limited to how they play D&D any more than the only difference between those who choose to support a local bookstore and those who always buy books from Amazon is how they buy books.
As if their attitudes, beliefs and values play no part in informing how they go about consuming things and no correlation exists between these and what might then inform their other decisions and what they then think might be best for the game.
When those deciding the direction the game will take—and I'm not just talking about whether or not it will remain at tables—are just one type of consumer you wind up with a situation that would be like a publishing house only listening to what one type of consumer among those who buy their books has to say to determine what it will publish in the future.
They have openly said the game is "under-monetized" and we all know this is because some groups get by with one or two set of rules.
Just how do you reckon they intend to "fix that problem"?
Make it a necessity to have an account here at the "hub" and to subscribe to view books?
These are all "what ifs" ...
Only time will tell.
I love D&D and don't want to be telling you I told you so.
INSPIRATIONS: Clark Ashton Smith, Mervyn Peake, Jack Vance, Michael Moorcock, Fritz Leiber, M. John Harrison, Gene Wolfe, Steven Brust, Terry Pratchett, China Miéville.
SYSTEMS: ShadowDark, C&C, AD&D.
GEAR: pencils, graph paper, dice.
Another way of explaining this is this survey is not The Ring of Power, it is not the One Survey to Rule All Other Understandings of the future of D&D. It is but a member of fellowship of research tools used to together to get some sense of the state of D&D use.
This survey, as discussed, comes out in iterations at least 2-4 times a year. It is not the sole driver of WotC's understanding of the interests and practices of the entirety of the D&D player base. Anyone who thinks otherwise isn't coming from an evidence based perspective but one of misdirected passions and rancor. The survey just isn't the high stakes McGuffin whose casting into Mt.Doom will save the World's Greatest Role Playing Game. There's no meta to be played here to apex the development of the game to one particular mode of affinity.
Jander Sunstar is the thinking person's Drizzt, fight me.