LLMs don't follow rules, they can't do logic. GPT has read all the D&D rules but it can't understand them so it can't follow / enforce them, neither can any other LLM.
That's a popular sentiment. I believe it to be largely untrue, mostly because the RPG community don't like the idea. Not pointing at anyone in particular, it's just ... an observation.
GPT is pretty good at legal texts. I know this for a fact because I work with legal texts, and I use GPT to look up stuff and find interpretations. And it does this pretty well. Does it do it flawlessly? Oh hell no. But well enough to save me time? Heck yes.
I work with Danish unemployment regulations. It's the single largest law text in our fair nation, and is a factor 5:1 to the core rulebooks of DND.
There's a core difference: Looking up §56 is not the same as understanding how class mechanics synergize when multi classing (as an example). But I see no reason to expect that GPT should be able to handle the rules in general.
I could be wrong, though. I've had a few simple - but largely succesful - games with GPT as game master. But they were a fun exercise for my 5yo, not a full group of adults.
And on the flip side if I send GPT a decklist for a Magic: The Gathering deck and ask it which cards to trim and why, it just makes up what some of the cards do and how certain interactions work.
If I ask it to help fix a Powershell script, it will literally make up cmdlets.
Also I was asking questions about ITIL (an IT framework) not too long ago and it kept contradicting itself. Mind you, ITIL is just like that but it can't follow simple reasoning more complicated than one or two steps.
It really is just generative and not analytical. The greatest flaw as well is that it always comes across as confident and is unable to say, "I don't know." If it doesn't know, it will make something up. That's what generative AI does and that's why it can't be relied on.
Now, I've used it in the past to help name small, boring towns. Like the starter towns I want the players to forget about once they're swept up in adventure. Like, "What are some old names for settlements near water. Or hills. Or a forest." And it gives me all sorts of prefixes and suffixes to mix and match. Most of them are at least semi historically accurate, so it sounds like something a medieval town might have been named. I let it do the dull/uninspired work that doesn't matter, but it can't be relied upon for anything important.
Answering the original question. It doesn't sound like a resource like this exists yet. It certainly could be done--maybe that's the way of the future for niche-content. They'll get a group of thousands of DM's who will provide feedback such to the point that a model can make accurate predictions.
You can ask a thousand different DMs how to rule something and you might get a thousand different answers. This game is social and subjective by nature. The rules are open to interpretation. RAW vs DM Fiat is a spectrum and DMs swing wildly between the two styles.
And on the flip side if I send GPT a decklist for a Magic: The Gathering deck and ask it which cards to trim and why, it just makes up what some of the cards do and how certain interactions work.
If I ask it to help fix a Powershell script, it will literally make up cmdlets.
Also I was asking questions about ITIL (an IT framework) not too long ago and it kept contradicting itself. Mind you, ITIL is just like that but it can't follow simple reasoning more complicated than one or two steps.
It really is just generative and not analytical. The greatest flaw as well is that it always comes across as confident and is unable to say, "I don't know." If it doesn't know, it will make something up. That's what generative AI does and that's why it can't be relied on.
Now, I've used it in the past to help name small, boring towns. Like the starter towns I want the players to forget about once they're swept up in adventure. Like, "What are some old names for settlements near water. Or hills. Or a forest." And it gives me all sorts of prefixes and suffixes to mix and match. Most of them are at least semi historically accurate, so it sounds like something a medieval town might have been named. I let it do the dull/uninspired work that doesn't matter, but it can't be relied upon for anything important.
Don't get me wrong: GPT is dumb as a brick, and I acknowledge that. But it's like a hammer in that way, and a hammer still performs admirably in it's function.
I think ... if you drew a map of MTG synergies, you'd wind up with something mindbogglingly complex. I don't play, but I imagine it would fill stadiums with red string and unreadable hieroglyphs.
I understand it can code, and does so .... sort of acceptably well, but inefficiently?
I dunno, there's no brain behind any of it. It can't, it just does. But yes, it's greatest flaw is DEFINITELY it's lack of ... any real concept of it's shortcomings. I needed to do something with my CV once, and I asked it: There's this paid add-on that says it does CV's. Will it do what I need - and the reply I got was an exuberant HELL YES!
Well, guess what: Not only can it not do that, at all. It also couldn't do anything at all even remotely related to CV's. It was horrible. 'Can you make this as a PDF for me?' HELL YES, gimme a second. And then nothing. Forever.
Bloody thing =)
Rollback Post to RevisionRollBack
Blanket disclaimer: I only ever state opinion. But I can sound terribly dogmatic - so if you feel I'm trying to tell you what to think, I'm really not, I swear. I'm telling you what I think, that's all.
I understand it can code, and does so .... sort of acceptably well, but inefficiently?
To be absolutely clear, Chat-GPT can't code. It can't do math more complicated than simple addition, subtraction, and maybe multiplication. It doesn't understand logic. It can't produce usable code.
There are models built to produce or help fix code and to perform mathematics, but GPT isn't one of them. LLMs aren't built for that sort of thing. They're built to interact with human languages, tell you what it thinks you want to hear, and generate text based off of that. It's not very good at analysis and will usually give a lukewarm answer, unless you tell it you're leaning one way or another. In which case it will try to give you some sort of emotional encouragement and tell you you're right because that's what it thinks you want to hear. It can't think critically. When asked to cite sources, it will often cite unreliable sources like random blog posts or opinions on Reddit. Sometimes it will just say, "I don't have any sources for this, it's just knowledge I have."
What it produces is "AI slop," or content that's uninspired, unimaginative, and generally feels soulless or dull. It can produce images, but they tend to be muddy, plain, and not stylized. This is partly because an artist's flaws become their style. Art is supposed to be one's own unique and deeply personal interpretation of the world. Image-generating AI is a goulash made by training the model on all kinds of artists' styles. When you take all of the colors and slop them together, you just get brown. When you take everything in your pantry and throw it in a pot, you just get this weird, bland, cafeteria food taste.
This is what AI slop is. It's uninspired. It's "technically, this is edible." Will it get better in the future? I don't know. I can't predict the future. But if no one's creating new content and only using AI to do so, then AI models only have AI-produced slop to train on. You end up with a copy of a copy of a copy... of a pile of slop.
I just asked GPT to make me a JS code to display a laughing smiley as a background on my phone. It did that without problems at all.
Can I ask it to code a AAA game? No. Can I ask it bit by bit to create stuff that's useful in practical terms - without being great in any way? Absolutely
Rollback Post to RevisionRollBack
Blanket disclaimer: I only ever state opinion. But I can sound terribly dogmatic - so if you feel I'm trying to tell you what to think, I'm really not, I swear. I'm telling you what I think, that's all.
I understand it can code, and does so .... sort of acceptably well, but inefficiently?
To be absolutely clear, Chat-GPT can't code. It can't do math more complicated than simple addition, subtraction, and maybe multiplication. It doesn't understand logic. It can't produce usable code.
That isn't really true. Chat GPT is actually quite good with creating code. I've built entire sub-systems with chatGPT. Its not a universal answer to general questions, but the more you give it, the more intelgent it becomes.
To give an example, if I ask chatGPT to write a story, it will likely be generic and relatively boring, but if I write a story and ask it to help me edit it, make it sound more thematic with specific ques the result is quite extrodinary. The same is true with code. If I ask it to "write a script that does X", it probobly won't work that well, but if I create a script that doesn't quite do what I want and give it appropriate ques of what I want, it can create pure magic.
Its a sort of productivity tool that rewards creative, experienced and knowledgeable people, but if you are simply looking for complex answers with easy questions, its not going to come out well.
I use A.I. every day in my work (Solution Architect), I have 30 years of experience in software design and I can tell you that A.I. like ChatGPT has made me roughly 500% more productive. I think without experience, education and knowledge, it would be relatively useless.
I do agree that there are better A.I.'s then chatGPT, but so far as free tools go, it's quite strong and very effective. Its kind of a go to tool for quick assessments, analysis and design spec creation. Its a fantastic editor, great with research, awesome as a sounding board when working project and awesome at answering detailed questions that would be time consuming to google.
To be absolutely clear, Chat-GPT can't code. It can't do math more complicated than simple addition, subtraction, and maybe multiplication.
ChatGPT itself is just a neural net architecture; whether it can perform a given task depends on the dataset it's been trained on. If you're going to ask it to code, you'd better provide it a large amount of code as training material, and if you do, you can expect that its ability to perform coding tasks depends on how similar the task is to something it's been trained on.
AIs do seem to have a pretty significant problem with handling lack of knowledge, though; they are reluctant to give an "I don't know" or "I don't understand what you want" response, and instead just make stuff up. There's some question about whether this is a fundamental problem with current generative AI models or just a quirk that can be fixed. Of course, for purposes of running a game, a fast and plausible sounding response (even if wrong) may be better than spending an extended period looking for a correct answer.
And on the flip side if I send GPT a decklist for a Magic: The Gathering deck and ask it which cards to trim and why, it just makes up what some of the cards do and how certain interactions work.
If I ask it to help fix a Powershell script, it will literally make up cmdlets.
Also I was asking questions about ITIL (an IT framework) not too long ago and it kept contradicting itself. Mind you, ITIL is just like that but it can't follow simple reasoning more complicated than one or two steps.
It really is just generative and not analytical. The greatest flaw as well is that it always comes across as confident and is unable to say, "I don't know." If it doesn't know, it will make something up. That's what generative AI does and that's why it can't be relied on.
Now, I've used it in the past to help name small, boring towns. Like the starter towns I want the players to forget about once they're swept up in adventure. Like, "What are some old names for settlements near water. Or hills. Or a forest." And it gives me all sorts of prefixes and suffixes to mix and match. Most of them are at least semi historically accurate, so it sounds like something a medieval town might have been named. I let it do the dull/uninspired work that doesn't matter, but it can't be relied upon for anything important.
I have Darkvision, by the way.
You can ask a thousand different DMs how to rule something and you might get a thousand different answers. This game is social and subjective by nature. The rules are open to interpretation. RAW vs DM Fiat is a spectrum and DMs swing wildly between the two styles.
I have Darkvision, by the way.
Don't get me wrong: GPT is dumb as a brick, and I acknowledge that. But it's like a hammer in that way, and a hammer still performs admirably in it's function.
I think ... if you drew a map of MTG synergies, you'd wind up with something mindbogglingly complex. I don't play, but I imagine it would fill stadiums with red string and unreadable hieroglyphs.
I understand it can code, and does so .... sort of acceptably well, but inefficiently?
I dunno, there's no brain behind any of it. It can't, it just does. But yes, it's greatest flaw is DEFINITELY it's lack of ... any real concept of it's shortcomings. I needed to do something with my CV once, and I asked it: There's this paid add-on that says it does CV's. Will it do what I need - and the reply I got was an exuberant HELL YES!
Well, guess what: Not only can it not do that, at all. It also couldn't do anything at all even remotely related to CV's. It was horrible. 'Can you make this as a PDF for me?' HELL YES, gimme a second. And then nothing. Forever.
Bloody thing =)
Blanket disclaimer: I only ever state opinion. But I can sound terribly dogmatic - so if you feel I'm trying to tell you what to think, I'm really not, I swear. I'm telling you what I think, that's all.
To be absolutely clear, Chat-GPT can't code. It can't do math more complicated than simple addition, subtraction, and maybe multiplication. It doesn't understand logic. It can't produce usable code.
There are models built to produce or help fix code and to perform mathematics, but GPT isn't one of them. LLMs aren't built for that sort of thing. They're built to interact with human languages, tell you what it thinks you want to hear, and generate text based off of that. It's not very good at analysis and will usually give a lukewarm answer, unless you tell it you're leaning one way or another. In which case it will try to give you some sort of emotional encouragement and tell you you're right because that's what it thinks you want to hear. It can't think critically. When asked to cite sources, it will often cite unreliable sources like random blog posts or opinions on Reddit. Sometimes it will just say, "I don't have any sources for this, it's just knowledge I have."
What it produces is "AI slop," or content that's uninspired, unimaginative, and generally feels soulless or dull. It can produce images, but they tend to be muddy, plain, and not stylized. This is partly because an artist's flaws become their style. Art is supposed to be one's own unique and deeply personal interpretation of the world. Image-generating AI is a goulash made by training the model on all kinds of artists' styles. When you take all of the colors and slop them together, you just get brown. When you take everything in your pantry and throw it in a pot, you just get this weird, bland, cafeteria food taste.
This is what AI slop is. It's uninspired. It's "technically, this is edible." Will it get better in the future? I don't know. I can't predict the future. But if no one's creating new content and only using AI to do so, then AI models only have AI-produced slop to train on. You end up with a copy of a copy of a copy... of a pile of slop.
I have Darkvision, by the way.
I just asked GPT to make me a JS code to display a laughing smiley as a background on my phone. It did that without problems at all.
Can I ask it to code a AAA game? No. Can I ask it bit by bit to create stuff that's useful in practical terms - without being great in any way? Absolutely
Blanket disclaimer: I only ever state opinion. But I can sound terribly dogmatic - so if you feel I'm trying to tell you what to think, I'm really not, I swear. I'm telling you what I think, that's all.
That isn't really true. Chat GPT is actually quite good with creating code. I've built entire sub-systems with chatGPT. Its not a universal answer to general questions, but the more you give it, the more intelgent it becomes.
To give an example, if I ask chatGPT to write a story, it will likely be generic and relatively boring, but if I write a story and ask it to help me edit it, make it sound more thematic with specific ques the result is quite extrodinary. The same is true with code. If I ask it to "write a script that does X", it probobly won't work that well, but if I create a script that doesn't quite do what I want and give it appropriate ques of what I want, it can create pure magic.
Its a sort of productivity tool that rewards creative, experienced and knowledgeable people, but if you are simply looking for complex answers with easy questions, its not going to come out well.
I use A.I. every day in my work (Solution Architect), I have 30 years of experience in software design and I can tell you that A.I. like ChatGPT has made me roughly 500% more productive. I think without experience, education and knowledge, it would be relatively useless.
I do agree that there are better A.I.'s then chatGPT, but so far as free tools go, it's quite strong and very effective. Its kind of a go to tool for quick assessments, analysis and design spec creation. Its a fantastic editor, great with research, awesome as a sounding board when working project and awesome at answering detailed questions that would be time consuming to google.
ChatGPT itself is just a neural net architecture; whether it can perform a given task depends on the dataset it's been trained on. If you're going to ask it to code, you'd better provide it a large amount of code as training material, and if you do, you can expect that its ability to perform coding tasks depends on how similar the task is to something it's been trained on.
AIs do seem to have a pretty significant problem with handling lack of knowledge, though; they are reluctant to give an "I don't know" or "I don't understand what you want" response, and instead just make stuff up. There's some question about whether this is a fundamental problem with current generative AI models or just a quirk that can be fixed. Of course, for purposes of running a game, a fast and plausible sounding response (even if wrong) may be better than spending an extended period looking for a correct answer.