Help - Search - Members - Calendar
Full Version: Probability and Statistics
Dumpshock Forums > Discussion > Shadowrun
Pages: 1, 2
Jrayjoker
Hi all you play testers, developers, and all other assorted opinionated SOBs who hang out on these here forums. wink.gif

I have seen some of the code for probability of x number of dice having y number of successes against target number z bandied about in these forums. I find them somewhat fun and helpful, but my question is this:

Is anyone on the dev team actually applying Probability & Statistics to the new system and making sure it is reasonable, or are the play testers expected to do the number crunching if they feel like it?

I guess a way to paraphrase my question would be, how mathematically rigorous are the developers choosing to be at this time?

Anyone in the know care to comment at this time?

Also, does anyone have any idea how common or uncommon it is for a game developer to apply p&s to the game mechanic?
Nikoli
well, probability and statistical analysys should he high on the design end, but chances are they are just plugging in numbers and guestimating impact basedon what they see. Nothing wrong with this as it's faithfully served P&P rpg's since their inception.
It's difficult to imagine the true impact of what .05% increase in probability actually means ove rthe lifetime of a character. what they likely look at is, how likely does it seem over the course of a gaming session that this will happen, not over the next 1000000 sessions.
mfb
my hope would be that they're actually checking probabilities and stuff like that, and not just making up die rolls for their examples. unless you play a game on a near-daily basis, it's hard to do that and come up with realistic results, because you're simply not familiar enough with the mechanics to be able to reliably guestimate levels of success.
Jrayjoker
True, true. But you can calculate your probabilities based on a smaller samples and get valid (and perhaps sound) results IIRC, it just ups the chance for anomolies.

There are fields of mathematical study that look for trends in small populations, etc. that would apply quite well to a night's worth of dice rolls. I am in no way versed in those maths, but I know they exist.
Kagetenshi
Given how relatively easy it is to calculate a lot of the probabilities (and, more importantly, how easy it is to find or write small programs to calculate the probabilities of the more iffy sections), there's no excuse whatsoever for using "samples". The actual probabilities are easy to find, unless they're doing something really weird with the mechanics.

~J
Charon
QUOTE (Jrayjoker)
True, true. But you can calculate your probabilities based on a smaller samples and get valid (and perhaps sound) results IIRC, it just ups the chance for anomolies.

What? This ain't a survey. No need for extrapolation. We are talking dice rolls here, not "How many people intend to vote republicans in the next election".

The exact probabilities of success for any given dice roll can be easily calculated and considering that these are pretty basic maths, I'm certain that at least a few people on the design team are intimately familiar with the ins and out of the system.
blakkie
Working out the probabilities explicitly is important because gamers do it. Not just consciously such as by keen people here, but subconciously gamers learn over time tend to learn what is roughly more/less probable, and what is generally more "profitable". Knowing these the odds up front the developer can make a better guess at how the game will be eventually be played without actual playtesting. The playtesting is still required to proof the system.

Note: You have to be careful with playtesting. Learning the probabilities in a system emperically can take time, so it can takes some time for the playtesting to find the groove where the game will normally get played.
mfb
which is why you never guestimate during playtest or development. ever.
Jrayjoker
QUOTE (Charon)
QUOTE (Jrayjoker @ May 12 2005, 01:31 PM)
True, true. But you can calculate your probabilities based on a smaller samples and get valid (and perhaps sound) results IIRC, it just ups the chance for anomolies.

What? This ain't a survey. No need for extrapolation. We are talking dice rolls here, not "How many people intend to vote republicans in the next election".

The exact probabilities of success for any given dice roll can be easily calculated and considering that these are pretty basic maths, I'm certain that at least a few people on the design team are intimately familiar with the ins and out of the system.

So, are you suggesting that a rigorous approach isn't necessary? I am not sure that it is necessary to write a thesis on the mechanics, but with a system that has been highly dependant on making things harder or easier based on conditional modifiers I think it should be considered. If we like the feel of the game because we can play with the odds, then the odds need to be considered in the new system.
mfb
indeed. a rigorous approach is very necessary, especially when you're playing with a new system.
Charon
QUOTE (Jrayjoker @ May 12 2005, 02:12 PM)
QUOTE (Charon @ May 12 2005, 01:02 PM)
QUOTE (Jrayjoker @ May 12 2005, 01:31 PM)
True, true. But you can calculate your probabilities based on a smaller samples and get valid (and perhaps sound) results IIRC, it just ups the chance for anomolies.

What? This ain't a survey. No need for extrapolation. We are talking dice rolls here, not "How many people intend to vote republicans in the next election".

The exact probabilities of success for any given dice roll can be easily calculated and considering that these are pretty basic maths, I'm certain that at least a few people on the design team are intimately familiar with the ins and out of the system.

So, are you suggesting that a rigorous approach isn't necessary?

Err, no. The contrary.

When dealing with dice rolls, it's relying on samples and extrapolation that isn't rigorous. You can calculate the exact value with few basic formulas. I can give you the exact probabilities of X successes with Y dice in less than a minute with Excel or a calculator.

Samples is what you do when you can't build a mathematical model. Like determining who is gonna win the elections with a survey.
Kagetenshi
QUOTE (Jrayjoker)
So, are you suggesting that a rigorous approach isn't necessary? I am not sure that it is necessary to write a thesis on the mechanics, but with a system that has been highly dependant on making things harder or easier based on conditional modifiers I think it should be considered. If we like the feel of the game because we can play with the odds, then the odds need to be considered in the new system.

As others have already said, using a sample when the actual probability is available through simple calculation is the antithesis of a rigorous approach.

~J
blakkie
QUOTE (mfb @ May 12 2005, 01:10 PM)
which is why you never guestimate during playtest or development. ever.

You mean you -shouldn't-, ever. smile.gif

I remember about 3 months or so after Ultima Online went retail (fall '97). A new member was brought onto the development team, his handle was Evil John i believe. He mentioned that he was doing damage over time calculations to try balance out the combat system. These were calculations that had never been done before! eek.gif They had just winged it....and if you had played UO in the first few months you would have seen just how much it showed. It was really, really bad.
mfb
indeed. i hope the same thing doesn't happen with SR4.
Jrayjoker
QUOTE (Charon)
Err, no. The contrary.

When dealing with dice rolls, it's relying on samples and extrapolation that isn't rigorous. You can calculate the exact value with few basic formulas. I can give you the exact probabilities of X successes with Y dice in less than a minute with Excel or a calculator.

Samples is what you do when you can't build a mathematical model. Like determining who is gonna win the elections with a survey.

Thanks for the clarification.
Lady Anaka
Numbers have been run. Probabilities have been calculated. Contrary to popular belief, dartboards and random throws were not involved.

smile.gif
blakkie
QUOTE (Lady Anaka)
Numbers have been run. Probabilities have been calculated. Contrary to popular belief, dartboards and random throws were not involved.

smile.gif

Can you comment on the use or non-use of halucinatory fauna?
Lady Anaka
To my knowledge, no bits or pieces of animals were ingested, smoked, or otherwise rubbed onto anyone in an effort to aid in game design and or cause halucinations during this process.

Oh, and EW. Just saying.
Pistons
No animals were harmed in the creation of this game system. It is entirely possible that plant life may have been abused, but none of them are commenting and thus it is difficult to verify. nyahnyah.gif
Kagetenshi
For the purposes of that statement, are babies considered animals or plant life?

~J
Phantom Runner
QUOTE (Lady Anaka)
To my knowledge, no bits or pieces of animals were ingested, smoked, or otherwise rubbed onto anyone in an effort to aid in game design and or cause halucinations during this process.

Oh, and EW. Just saying.

But that does not rule out ritual enemas ...

eek.gif
Crimsondude 2.0
QUOTE (Pistons @ May 12 2005, 02:54 PM)
No animals were harmed in the creation of this game system. It is entirely possible that plant life may have been abused, but none of them are commenting and thus it is difficult to verify.  nyahnyah.gif

Well, duh. If that wasn't true, the devs wouldn't have been able to hit the freelancer crack pipe.

That is unless you've switched to freelancer crank.
Ellery
Does anyone have solid evidence that any game designers actually rigorously analyze and understand the probability and statistics of the game they're creating? Game designers often have a literary background rather than an analytic one. Publication schedules are tight, leaving little time for anything viewed as extra. Players don't typically have time to analyze rules statistically before they decide to purchase a book. And so on. All the short-term pressures are against performing any analysis at all (aside from what can be inferred from experience while playtesting), let alone a thorough one.

Of course, from a long-term perspective, a little extra work early on can avoid enormous, ongoing, systematic problems that arise from a system that just doesn't work probabilistically. (It can also help avoid lots of smaller glitches.)

I've played enough MMORPGs to be very skeptical that they really bother to analyze anything there--and this is with teams of programmers who are mostly from a CS background, and thus have at least been exposed to a lot of math, including in most places a required course that covers probability among other things (usually called "discrete math" or somesuch). The design teams there are much larger, and the people are more suited to perform the analysis, and I still see people altering powers so they become completely useless or absurdly overpowered to a level that would be caught by a few dozen hours of playtesting let alone a careful analysis.

So, yes, I think the analysis is very important to do. But unless I'm missing something, SR would be the exception rather than the rule if an analysis were done carefully here. I'd encourage the developers to perform the analysis themselves, or ask their playtesters to, or both! But until I hear official word that they are doing this, I don't see much justification for thinking that they are or will based on my perception of the industry standard.
Phantom Runner
QUOTE (Ellery)
Does anyone have solid evidence that any game designers actually rigorously analyze and understand the probability and statistics of the game they're creating?... SR would be the exception rather than the rule if an analysis were done carefully here.  I'd encourage the developers to perform the analysis themselves, or ask their playtesters to, or both!  But until I hear official word that they are doing this, I don't see much justification for thinking that they are or will based on my perception of the industry standard.

I know from personal experience both playtesting several games and from discussion with a few developers that some (only the ones I have personal experience with) are created with an eye toward the statistics, in so much as that statistics are taken into consideration. Some games companies do employ people with advanced mathematics degree(s).

Saddly I have also partaken in playtesting of games where there is almost no statistical support and we playtesters had to do all the number crunching ourselves (not very fun I assure you)...
Ellery
We can count the major contributors to SR on our fingers. Do we know anything about their affinity for statistics? (I don't know many of them.)
Solstice
No we don't. But we also know that as far as is common knowledge, literary degrees do not include even the most rudimentary of mathematical analysis. When I picture game designers I picture some people sitting around a coffee shop swaping quotes from The Secret Garden , not huddled around a spreadsheet mumbling about P not and P hat etc.
Crimsondude 2.0
I'd rather have 1 guy with a degree in Stats run a game line than a dozen people with doctorates in any liberal arts.
Ellery
I wouldn't. I'd give up quite a few literary arts PhD's to have at least one person with a degree in mathematics or statistics, though. Writing is important. Plot is important. Giving a feeling to the imaginary world is important. But having rules that mesh with that world is important too. It's not good if you have to default on your dice mechanics creation test.
The White Dwarf
Uh, it already got answered. The people involved said yea, theyre looking at the numbers to (hopefully) avoid problems. Which of course doesnt garuntee itll be perfect, but does garuntee that it wont be totally off the wall. While there may not be a statistician on the team, or a literary degree for that matter as Im frankly not sure what all they got degrees in, anyone including trained cybermonkys can run numbers thur a computer program so Im gonna have to at least assume it was done well enough to be playable. I mean if you put some of the statistical craziness from sr2 and advanced dnd and compared it to sr3, your brain would melt. Hopefully this will be the same way... or rather the other way, where your brain doesnt melt cause its better =)
Pistons
QUOTE (Ellery)
I'd encourage the developers to perform the analysis themselves, or ask their playtesters to, or both! But until I hear official word that they are doing this, I don't see much justification for thinking that they are or will based on my perception of the industry standard.

QUOTE
Lady Anaka:
Numbers have been run. Probabilities have been calculated. Contrary to popular belief, dartboards and random throws were not involved.

smile.gif


Lady Anaka = Michelle Lyons
(Information provided in case there were a still a few people who did not know.)

I can also attest that yes, numbers have been run and probabilities calculated.
Ellery
That's good to know, P. I hope a good job has been done. Calculating that TN 5 is a 1/3 chance is calculating a probability, but one shouldn't stop there.
Jrayjoker
Cool. thanks to all the people in the know who have commented.


Now for a new qestion:

Have the probabilities from SR3 tests been compared to the probabilities fro SR4 tests where applicable to help ensure that crossover and conversion is as seamless as possible?
Nikoli
I would, but without some VB time, it's a bitch to write an excel spreadsheet that will properly handle exploding dice.
Ellery
Check out the dice paper linked in this thread. It gives recurrance formulae that can be plugged into Excel without the need for VB.

Of course, this requires some mathematical knowledge; some may find VB easier than math.
Charon
QUOTE (Ellery)
Some may find VB easier than math.

I sure don't.

Thanks.
blakkie
QUOTE (Nikoli)
I would, but without some VB time, it's a bitch to write an excel spreadsheet that will properly handle exploding dice.

I've got a spread sheet with the formula in it, but it's mixed in with a bunch of other junk and the variable parts should be the constants and the constants should be the variables. I could send you it, but it would take some work to get it to the point where you could easily grasp what is going on with it.

It isn't that hard though to reproduce, assuming you can read the notation in that PDF Ellery linked. Personally it's been a long time for me since i'd done that stuff, i can only gronk it with lots of thinking time (building up from the scratch basics of mathematics) with help of knowledge of what the final product should roughly look like.
The White Dwarf
You can also just d/l a dice roller appl, set the roll you want, and have it run like 2 mil combinations. Sorta tend to see ballpark values after that and it takes all of 30 seconds.

Eitherway Im sure if theyve run the numbers for the new version they were mindful enough to set them at values they wanted, whether the old stats entered the mix or not. Knowing the current chance to hit a tn 10 on 3 dice doesnt make you any more or less capable of saying "a person with this skill should be able to do that half the time, so that roll has to have odds = 50%".
blakkie
QUOTE (The White Dwarf)
You can also just d/l a dice roller appl, set the roll you want, and have it run like 2 mil combinations. Sorta tend to see ballpark values after that and it takes all of 30 seconds.

http://forums.dumpshock.com/index.php?showtopic=8385

I prefer theoretically determined though, because once you have them set up you can break stuff out into charts and senarios quicker. Plus it's more accurate (although with a 1 million rolls we're pretty much close enough).
The White Dwarf
Oh I agree, math is obviously better. But for the non programming inclined anti-math ppl just setting up a dice set and hitting "do 2 mil trys" is close enough to work with conversations sake. Im sure the testers used actual math was just throwin that out there for ppl anti-math and pro mouse clicking heh.
Eyeless Blond
QUOTE (The White Dwarf @ May 14 2005, 06:11 AM)
Oh I agree, math is obviously better.  But for the non programming inclined anti-math ppl just setting up a dice set and hitting "do 2 mil trys" is close enough to work with conversations sake.  Im sure the testers used actual math was just throwin that out there for ppl anti-math and pro mouse clicking heh.

I don't think they did, actually, or we'd probably have things set at a static TN of 4 instead of 5. 4 gets you a wider range of useful target thresholds, which is what you really want for this kind of system. Granularity == good.
blakkie
QUOTE (Eyeless Blond @ May 14 2005, 08:35 AM)
QUOTE (The White Dwarf @ May 14 2005, 06:11 AM)
Oh I agree, math is obviously better.  But for the non programming inclined anti-math ppl just setting up a dice set and hitting "do 2 mil trys" is close enough to work with conversations sake.  Im sure the testers used actual math was just throwin that out there for ppl anti-math and pro mouse clicking heh.

I don't think they did, actually, or we'd probably have things set at a static TN of 4 instead of 5. 4 gets you a wider range of useful target thresholds, which is what you really want for this kind of system. Granularity == good.

They might have felt that it created too many successes/die for what they are trying to do. They wanted a bit more instability in the dice rolls. *shrug*

EDIT: To explain what i mean by "instability", imagine that you used TN 2, only 1's would be non-successes. Then the number of successes would always be close to the number of die you rolled. More predictable. More weighted towards the person rolling the most die.
Charon
QUOTE (Eyeless Blond @ May 14 2005, 09:35 AM)
QUOTE (The White Dwarf @ May 14 2005, 06:11 AM)
Oh I   Im sure the testers used actual math was just throwin that out there for ppl anti-math and pro mouse clicking heh.

I don't think they did, actually, or we'd probably have things set at a static TN of 4 instead of 5.

It's the contrary, IMO.

A fixed TN 4 would be the default option, IMO. If they went with a TN 5 instead, I would tend to think it indicates they ran the numbers and liked 5 better, otherwise we'd have a TN 4.
Ellery
Maybe they played nWoD and decided that TN 5 on d6 is similar to TN 7 on d10.
Charon
nWoD is TN 8 on D10.

Exalted and Trinity are 7.

---

What are you suggesting anyway? That the designers are clueless dolt that copy their competetion without understanding the reasons behind a game design decision or that SR4 should use TN 4 just for the sake of being different from WW?
Jrayjoker
I'm sure that is not what is being suggested. Sticking with d6 is just good marketing at this point. Let the odds follow the polyhedron. smile.gif
Ellery
Having not played nWoD, Exalted, or Trinity, I accept your information about what the target numbers are. I note, however, that 5 for d6 is even closer to 8 on d10.

QUOTE (Charon)
What are you suggesting anyway? That the designers are clueless dolt that copy their competetion without understanding the reasons behind a game design decision
I'm suggesting that the reasons do not primarily include the statistical properties of the mechanic in situations where it will be applied.

There are other more plausible reasons to pick 5. 4 is boring--that's like flipping a coin. Who wants to play a game where you flip coins? Why even have dice? Bah. 3 means success is common, and therefore boring. We want success to be exciting! Yeah, I got a success! Woohoo! That's not going to happen with TN3. Hey, I've played Trinity and nWoD. Those were some good times. Works for them, why not for us?

When we hear reports that, "Yes, the numbers were run," that's nice. But unless we're hearing stuff like, "The coefficient of variation on typical rolls with TN3 was too small, so we picked TN5," I'm going to have a hard time believing that statistics played an important role many decisions. Most of the population doesn't even know what a coefficient of variation is, and for good reason. Statistical analysis isn't something most people need for work or play. For example, people don't normally think, "Wow, I want to develop a game with elves and magic and high technology and dragons and huge megacorporations where the players take on the dirty illegal jobs that the corps don't want made public! I better go dig out my stats book, or order one from Amazon today!"
blakkie
QUOTE (Ellery @ May 14 2005, 02:33 PM)
Most of the population doesn't even know what a coefficient of variation is, and for good reason. Statistical analysis isn't something most people need for work or play.

That is also a good reason why you wouldn't try to explain it to the general public that way. They might try use the term "standard deviation", what CV is derived from, which is something that is a bit more common knowledge. Personally i went with the term "instability" as it seemed to conjure a more accessable visual.

Of course there is also a real possibility that those sorts of specifics might be beyond what they want to talk about right now.
DrJest
QUOTE
Lady Anaka = Michelle Lyons
(Information provided in case there were a still a few people who did not know.)


That still creeps me out smile.gif Nothing personal, Lady A - I used to work with a Michelle Lyons when I was still trapped in the corporate hell that is Norwich Union GI. The unavoidable consequence of that, of course, is that I have no idea what you look like and will probably never meet you, so now and for all eternity you will have her face in my mind's eye (relax, she's pretty smile.gif ).
Solstice
QUOTE (Ellery)
Most of the population doesn't even know what a coefficient of variation is, and for good reason. Statistical analysis isn't something most people need for work or play. For example, people don't normally think, "Wow, I want to develop a game with elves and magic and high technology and dragons and huge megacorporations where the players take on the dirty illegal jobs that the corps don't want made public! I better go dig out my stats book, or order one from Amazon today!"

Maybe not, but a good grasp of the theories is important in almost all aspects of life. I can't think of too many professional degrees where stats isn't helpful if not necessary.
Jon Szeto
QUOTE (Crimsondude 2.0)
I'd rather have 1 guy with a degree in Stats run a game line than a dozen people with doctorates in any liberal arts.

How about a BS in engineering physics, with an additional four years of work experience in quality engineering, statistical process control, and Six Sigma?
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Dumpshock Forums © 2001-2012