re: The Elusive Million Hand Sample Analysis

Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
re: The Elusive Million Hand Sample Analysis

Hi, I've heard it mentioned many times in the 'online poker is rigged' debate that there are million hand analyses out there to prove it one way or the other. It doesn't matter which side they're on, if they think it's rigged, they've seen these giant analyses that confirm it, and vice versa if they think it's not rigged.

I've searched but never found any at all, can anyone point me in the direction of a million+ hand analysis? :)
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
I feel this is a slippery slope- we can't claim that a large enough sample size will prove it while we extrapolate from our own smaller sample sizes. Then people who claim it is rigged could extrapolate from their own small samples 'proving' the opposite. The sample size is inherently critical ;)

The sample size is critical. But, if someone has a 5k hand sample where things are very wrong, it is at least a starting point. We expect some 5k hand samples to be several standard deviations from the mean, simply because over billions of hands there will be some 5k chunks in there that are far away from the mean. If there wasn't, we would know something was up because the data wasn't following a normal distribution. Sample size should always be mentioned.

What we tend to find, and I have read many threads on 2+2 and elsewhere on this, is that people claiming it is rigged don't actually produce any data that supports their claims. When you have people posting their analysis of their 37k hands played, they find that it doesn't look weird at all. If it was rigged, it is very weird that everyone who has used their actual records to look for it happens to be part of the group it is not rigged for or against.

If you have 100 different people talking about their 20k hand database statistics, you're looking at 2 million hands.

Link, please? I believe you and understand it's not entirely relevant, but I'm interested nonetheless :)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4346402/

And, I mistyped. This study was for nearly 77 million distinct hands, not 37 million. I have no idea where the 37 came from, my only defense is that I was tired. It is a very good study, and they make no mention of finding unusual distributions of hands/flops/etc. Unfortunately, they did not explicitly look for such things, so we can't be sure they didn't miss them.

I am going to look for more of these studies, but this post is growing large and I will post what I find below.

Note: There was one, years back, that was exactly like what everyone is talking about. I remember reading it. It was a website that analyzed a million+ hands and looked for anomalies. It didn't find anything. That site seems to be gone now, and I can't find any mirrors. So, I know it sounds like we're all making up this statement, but it was a big deal when the site came out and we're probably recalling the same thing.

I only have ~20k hands, I don't think it's rigged but I'd really like to find some of these 'million hand reviews' that are mentioned all the time. It seems any time someone mentions poker being rigged somebody mentions 'the elusive Million hand sample analysis' that everyone has read but nobody has a link to :p

Oh, and thanks for replying! You're one of a few people I seem to often interact with here :top:

Alright, 20k is not a huge sample, but it is enough that you can at least look at some of your results. For example, are you getting the right amount of pairs?

  1. Figure how many pairs you should have: #_of_hands / 17
  2. How much you expect that to vary: 0.23529411764 * sqrt( #_of_hands)
  3. Figure how many of each pair you should have: #_of_hands / 221
  4. How much you expect that to vary: 0.06711491843 * sqrt( #_of_hands)

From my tournament play (7,367 hands) -- as I only play hold'em when I play tournaments and I don't play very many. And, I have no tracking for BOL, so those tournaments don't count.
  1. Number of pairs: 7367 / 17 = 433.35
  2. Should vary by: 20.2
  3. Number of pairs I should have: 7367 / 221 = 33.33
  4. Should vary by 5.76

Actual Pairs: 406. That's 27.35 away from expected. Divide that by how much I expect it to vary, it is 1.35 standard deviations away. That's pretty normal. Numbers less than 3 are normal. Greater than 3 are rare and may indicate a problem or sample size issues.

Each Pair:
  • AA: 40
  • KK: 28
  • QQ: 34
  • JJ: 36
  • TT: 30
  • 99: 36
  • 88: 29
  • 77: 33
  • 66: 37
  • 55: 19
  • 44: 23
  • 33: 31
  • 22: 30

You can see that most of these are right on top of the expected value and easily +/- 5.76 away from the calculated value. We really only need to look at the highest and lowest counts, to see how they faired. (40-33.33)/5.76 = 1.16 which is perfectly normal. (33.33-19)/5.76 = 2.49 which is the biggest number we have seen. But, there is a 6.39% chance of it happening. It's not that unlikely.

Just looking at this we can determine that I have gotten fewer total pairs than I expected--although I got AA more times--but nothing was outside the normal distribution of probabilities. And this is a very small sample. It is at least a starting point. You could easily do this with your ~20k hands, and see how the numbers come out for you. Does it determine anything with 100% certainty? No. But, it gives you a starting point. If you have numbers that are way out of line, we can determine how likely it is that you have a sample that far out of line based on the sample size.
 
T

titiduru

Visionary
Silver Level
Joined
May 26, 2016
Total posts
586
Chips
0
I have never seen such a hand analysis before either, sorry.
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
I just remembered that I had a bunch of hold'em hands on Carbon in this database. I never play there anymore. They gave me a bunch of free money, and I would run it up on the Hold'em tables and then lose it on the PLO tables without any regard to bankroll management (I had like $50 on there and was playing $10 PLO) because it was free. I eventually busted that roll, but went to ACR, where I had more money and enough PLO action that I didn't have to play hold'em.

Total hold'em hands: 20,152
Expected pairs: 1,185.4 (+/- 33.4 per stdev)
Actual pairs: 1,223
Expected of each pair: 91.2 (+/- 9.53 per stdev)
Actual of each pair: range from 78-104

The difference in total pairs is 1.13 standard deviations from the mean. The largest standard deviation from the mean for any individual pair is 1.39. In short, this data is pretty darn normal.

How about, do I flop sets (or better) enough? If I see a flop while holding a pair, I should have a set or better (hit my hole cards) on the flop 11.7551% of the time.

I saw 872 flops while holding a pair. I should have had a set or better 102.5 times (+/- 9.51 per stdev). I actually had 110 sets, quads, or full houses (excluding boards like 2-2-2 from full houses). That's 7.5 away from expected and less than one standard deviation from what I expect. In other words, it was perfectly normal.

If I had flopped like 200 sets or better, we would have some serious investigation to do. That's way outside the expected. In that case, we would have a datapoint that suggested something might be up. We would then investigate how often someone would have a 20k sample of hands 10 standard deviations from the mean. How many 20k samples would we need before we expect one of them to be that far away? And, other people would start looking for this exact same issue in their database. If I am flopping way more sets than I should, are other people? If they are, then the site is probably rigged. If they're flopping way less, that is also a good clue that someone is wrong. If this is the only sample where we find this happening, we still have questions and reason to investigate, but it is not definitive. It's a strong datapoint though.

The simple fact is, we don't have people coming out with datasets that are far away from the mean. And, when people test their own data, it tends to be normal and no one is finding anything consistently out of line with expectation. If they did, others could easily check to see if their samples were out of line with that expectation. In this way, many small samples do work as well as a single large sample.
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
This website is pretty good: http://www.spadebidder.com/about/

But, it's based on a database with over a billion hands, not a million. So, sorry about that. His analysis involved hundreds of millions of flops and situations. His primary focus is on discussion of flop distributions and what we should expect (based on the hands people see flops with) and what we actually see.

You really should read all 7 parts, because it does explain some observations that he gets to later. For example, flops will have fewer Aces than you would expect if you just dealt out 3 random cards from the deck. This is because, you only see a flop when two or more players have decided to play their hands. The hands with Aces are more frequently chosen to be played, so there is a high degree of probability that one or more Aces is not in the deck. When you account for this, the numbers work out very nicely.

The math is a bit heavy, but not overwhelming. This guy has done a lot of work on this dataset, and published what he has found. I don't believe he has access to all the hole cards, so the type of analysis is a little different from what you would do with your own database, where you do have access to your own cards to prove you are being dealt the right distribution and are hitting hands with the right frequency.
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
Other fun facts from my hold'em sample. This time looking at suited cards dealt and flops, turns, and rivers.

Dealt suited cards (20,152 hand sample)
Expected: 4,741.6
Actual: 4,724 (0.29 stdev from expected)

Flops seen with suited cards: 1,386
Flopping a flush expected: 11.7
Flopping a flush actual: 10 (0.49 stdev from expected)
Flopping a flush draw expected: 151.7
Flopping a flush draw actual: 174 (1.92 stdev from expected)


The only one that is even curious is the flush draw. Over that sample size, the standard deviation is 11.6. So, we are 1.92 standard deviations from the expected value. It isn't even that far from expectation, especially with a 1.4k sample size. Obviously, there's nothing here that makes me think rigged.

I could go further, and see how many turns (137) I saw. And, how often I hit my flush on the turn (21). That's a tad low. I expect to make it 26.2 times. It still isn't horrible at 1.13 stdev away from the expected value. 14 of those flushes saw the river and I saw 107 rivers. I already had a flush on 14. So, 93 rivers drawing to a flush. I should hit my flush on the river 18.2 times. I hit my flush on 18 of them. That is pretty spot on (0.05 stdev from expected).

Nothing with flush draws makes me think there's weirdness going on.

And, we could do this for any class of hands and draws. The fact that the sample size is small is an issue when we discuss rare events, but we can see if things are happening roughly as often as expected.

Edit: my point of posting these hand reviews from a 20k sample is that you could be doing the same thing with your own sample. The fact that it is small is a limitation when it comes to "proof" but not one when it comes to looking for something out of place. If you review it and find data that shouldn't be there, it provides a place to start looking for other people. Although, I suspect that when you look, you will end up finding similar data to what I found. Like I said, those with databases who go and review them inevitably come back with results that suggest things are normal.
 
Last edited:
Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
Thanks for all the info and links, Vinnie! :top:

I was reviewing my PP frequency while playing a couple freerolls, and I've never had so many PP in my life. It was so strange, 14 PP out of 127 hands! I wonder what the odds are there/how many SD :hmmmm:

I understand it's an outlier, but it did kind of mess with my head when I was specifically focused on counting them :p

I have 20,087 hands of tournament play, here are my results:

POCKET PAIRS
Code:
Cards   #    Expected
Any PP 1187    1182
AA       93      91
KK       77      91
QQ       98      91
JJ       73      91
TT      106      91
99       82      91
88       83      91
77       98      91
66       98      91
55       94      91
44       94      91
33      106      91
22       85      91

FS      107     100 ( I saw 849 flops with PP; 849/8.5)
(Flopped Set)

SUITED
Code:
Cards           #       Expected
ANY           4686      4671
Saw Flop      1582       N/A
Flop Flush    14          13
Flop 4F       186        174

I couldn't figure out the SD thing, my math wasn't matching yours on your own numbers so I didn't want to waste my time making erroneous claims :dontknow:

I didn't expect to find anything fishy, but I agree that it's foolish to think it is rigged without doing at least the math above. I wish I understood SD better, just to see how far off I am with JJ/TT.

If anyone else wants to throw some figures in here, feel free! I've actually considered trying to get some of our reviews together so we could create a large sample size to check. All of my (tracked) play is on ACR, it would be even better if we could focus on a specific site.

Thanks again Vinnie, very helpful :adore:
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
z-score = how many standard deviations a data point is from the mean. I have prefaced the numbers with that, from your data. This is the same thing I was talking about before, I just wasn't using this term.


I have 20,087 hands of tournament play, here are my results:

POCKET PAIRS
Code:
Cards   #    Expected
Any PP 1187    1182  (z-score 0.16)
AA       93      91       (z-score 0.22)
KK       77      91       (z-score 1.46)
QQ       98      91      (z-score 0.75) 
JJ       73      91        (z-score 1.88)
TT      106      91      (z-score 1.59)
99       82      91       (z-score 0.93)
88       83      91       (z-score 0.83)
77       98      91       (z-score 0.75)       
66       98      91       (z-score 0.75)
55       94      91       (z-score 0.33)
44       94      91       (z-score 0.33)
33      106      91      (z-score 1.59)
22       85      91       (z-score 0.62)

FS      107     100 ( I saw 849 flops with PP; 849/8.5)
(Flopped Set)

I am not sure if you mean flopped set, or flopped set or better. I suspect you mean set or better, as that is the 7.5-1 number. I don't know if you filtered for that, or excluded quads and full houses. If you did filter for set or better (aka, hitting one of your cards) then your results of 107 are only 0.77 standard deviations from the mean. If you filtered for exactly a set and not quads or boats, you expect 91.5 flops and would be 1.72 standard deviations from the mean. I suspect you meant the first, but even the second is within normal expectations.

I didn't expect to find anything fishy, but I agree that it's foolish to think it is rigged without doing at least the math above. I wish I understood SD better, just to see how far off I am with JJ/TT.

Not very far off, less than 2 standard deviations.

SUITED
Code:
Cards           #       Expected
ANY           4686      [S]4671[/S] 4726   (z-score 0.67)
Saw Flop      1582       N/A
Flop Flush    14          13             (z-score 0.19)
Flop 4F       186        [S]174[/S] 173     (z-score 1.04)

There were a couple errors with the expected numbers, but they were minor and don't change much. In any case, your numbers here are perfectly in the expected range.

I couldn't figure out the SD thing, my math wasn't matching yours on your own numbers so I didn't want to waste my time making erroneous claims :dontknow:

I don't know what you were doing wrong with the calculations. Either way, I ran the numbers. There's nothing here that shocks me. Your data is very close to my data for a similar sample size (20k hands each). I would say it was "shockingly" close, but this is exactly the sort of thing that isn't shocking. Even though the hands were played on completely different sites, they represent a random distribution and show up as expected. :D
 
Last edited:
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
Here is my pair list. Note, I totally missed that I had a pair of 7s 106 times. So my range was 78-106. Either way, it works. My expected number of pairs was also 91.

  • AA: 85
  • KK: 94
  • QQ: 98
  • JJ: 104
  • TT: 78
  • 99: 88
  • 88: 99
  • 77: 106
  • 66: 91
  • 55: 93
  • 44: 104
  • 33: 95
  • 22: 88

If we were to combine these, I suspect we find things get even closer to expectation. I might do that.

0pTnkIS.png


As you can see, pretty much as you expect. Things conform to the data. I am not going to run all these numbers combined for everything. But, it wouldn't change anything.
 
Last edited:
Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
I am not sure if you mean flopped set, or flopped set or better

It was filtered for set only (not better), sorry about all the inaccuracies.. It seems I've sourced bad numbers for odds of set and even suited cards :eek:


Wow, thanks for putting so much time into this! Love that chart with the combined breakdown. You're awesome :adore:

If anyone else wants to contribute their figures, we could do our own independent analysis. :deal:
 
toots babos

toots babos

ex-tornament grinder
Bronze Level
Joined
Oct 19, 2014
Total posts
3,869
Awards
16
Chips
109
here's some more data for you both from my database


358,644 tournament hands

pairs amount
AA- 1572
KK -1637
QQ -1596
JJ -1574
TT - 1624
99 -1662
88 -1564
77 -1623
66 -1598
55 -1570
44 -1653
33 -1581
22 -1615
 
Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
Holy sample size, Batman! :eek2:

I can add it to the chart, though Vinnie would probably do a better job being able to calculate the z-score.

Thanks! :top:
 
toots babos

toots babos

ex-tornament grinder
Bronze Level
Joined
Oct 19, 2014
Total posts
3,869
Awards
16
Chips
109
it will be interesting to see how the figures line up :)
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
Your numbers weren't far off. I assume you used percentages found online. I was using the exact odds. For example, odds of holding suited cards are 4/17. Looks like you used 23.25% which is pretty close. The 174 instead of the 173 was just rounding. When I went and got the z-scores, I avoided all rounding until the very end. So, I didn't used 91 for expected number of hands, I used the 90.8914... number until the very end.

The odds of hitting your card, when you hold a pair, is easily calculated by [1-(48*47*46)/(50*49*48)] or basically, find the odds of "NOT" hitting your set and subtract from 1. This is where we get the 7.5-1 number. That's the number almost every single website uses, because most people just want to know the odds of improving. It isn't like you want to hit a set, but not hit quads or a full house.

For hitting exactly your set. You need to find the combos. First, there are 19,600 different 3 card flops from a 50 card deck. There are 2 ways to hit your card and (48*44)/2 or 1,056 ways to make a non-paired combination without your card. That's 2,112 flop combinations that give you a set, but not better. 2112/19600 is the odds of hitting exactly a set. That is 8.28-1. It's a number that is useless to know, because it doesn't have any practical application during play.

Some Examples of Exact Odds:
* Odds of set or better using your pair? 2304/19600
* Odds of flopping exactly a set? 2112/19600
* Odds of flopping quads? 48/19600
* Odds of flopping a full house hitting your card? 144/19600

* Odds of flopping a flush? 165/19600
* Odds of flopping a flush draw? 2145/19600
 
J

Jimmy Crane

Rising Star
Bronze Level
Joined
Aug 1, 2017
Total posts
8
Chips
0
Where can I fish this kind of data? Can anyone suggest hand analysis software?
 
J

Jimmy Crane

Rising Star
Bronze Level
Joined
Aug 1, 2017
Total posts
8
Chips
0
Nice, I'll check it out. Thanks!
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
I remember when I had a database this big. Except it was almost all cash games.

here's some more data for you both from my database

358,644 tournament hands

pairs amount (we expect ~1622.8)
AA- 1572 (z-score 1.26)
KK -1637 (z-score 0.35)
QQ -1596 (z-score 0.67)
JJ -1574 (z-score 1.21)
TT - 1624 (z-score 0.03)
99 -1662 (z-score 0.97)
88 -1564 (z-score 1.46)
77 -1623 (z-score 0.004)
66 -1598 (z-score 0.62)
55 -1570 (z-score 1.31)
44 -1653 (z-score 0.75)
33 -1581 (z-score 1.04)
22 -1615 (z-score 0.19)
Total pairs: 20,869 (expected: 21,096.7) (z-score 1.62)

As you may have expected, there's nothing crazy out of line here. And, when we put that with the others, it also doesn't raise any flags.

YWZ1AwN.png
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
Holy sample size, Batman! :eek2:

I can add it to the chart, though Vinnie would probably do a better job being able to calculate the z-score.

Thanks! :top:

I got lazy and have a spreadsheet that's doing all the heavy lifting. I just put in the exact odds, sample size, and actual results. It spits out the z-score and expected number. The math isn't super hard. It's all explained in The Mathematics of Poker by Chen and Ankenman in Chapters 1 and 2. I grant that I have taken some statistics courses and other higher maths, but this isn't too complicated. I also really enjoy this sort of thing.
 
Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
Thanks so much Vinnie! :top:

And what a fitting, well-constructed 750th post ;)

Congrats on becoming CC Elite!

:party:
 
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
lol, I didn't even notice. It's just a sign that I spend too much time on here. My wife came out, while I was posting that and getting the exact numbers, and she just said, "Oh, you're doing poker math." And, then she walked away. LOL, I might have missed out on getting lucky, and I don't mean in cards. But, I am off to bed, now, to try and see if there's still a chance. ;)
 
darthdimsky

darthdimsky

Legend
Silver Level
Joined
Nov 21, 2016
Total posts
1,085
Chips
0
I wonder if we can chalk up a million of our own. Here's my sample of tourney hands

2017 2016

Op PS PP PP Total

Tot#s 27482 84017 234383 345882

AA 120 341 1092 1553
KK 122 400 1034 1556
QQ 122 355 1066 1543
JJ 127 375 1016 1518
TT 108 388 1031 1527
99 133 373 1042 1548
88 130 364 1030 1524
77 126 398 1004 1528
66 129 395 1061 1585
55 122 360 1040 1522
44 110 363 1025 1498
33 144 400 1123 1667
22 124 409 1111 1644

Edit: I don't have a spreadsheet programme with me. Sorry about the alignment.

PS: Op = Operator; PS = poker stars; PP = partypoker
 
Last edited:
vinnie

vinnie

Legend
Silver Level
Joined
Apr 12, 2013
Total posts
1,208
Awards
1
US
Chips
50
I added those totals to the sheet, but haven't broken down things site by site. 744,765 hands and there's nothing out of line. The z-scores range from 0.035 to 1.743. I did the ones for PS with similar results as for the whole sample (min of 0.03 and max like 1.5). I suspect we'll find the same for the rest, if we run the numbers.

I can run the other two sites, but I doubt there will be anything. The expected value for the second site is 380 and we are close on all values. The third site is 1,061 and everything seems pretty close to that also. The expected is 1,565 for the total, and everything lines up on that front also.

Of course, maybe we're not the people the sites are rigged against.
 
darthdimsky

darthdimsky

Legend
Silver Level
Joined
Nov 21, 2016
Total posts
1,085
Chips
0
I added those totals to the sheet, but haven't broken down things site by site. 744,765 hands and there's nothing out of line. The z-scores range from 0.035 to 1.743. I did the ones for PS with similar results as for the whole sample (min of 0.03 and max like 1.5). I suspect we'll find the same for the rest, if we run the numbers.

I can run the other two sites, but I doubt there will be anything. The expected value for the second site is 380 and we are close on all values. The third site is 1,061 and everything seems pretty close to that also. The expected is 1,565 for the total, and everything lines up on that front also.

Of course, maybe we're not the people the sites are rigged against.

Hey Vinnie, I'm really sorry about the confusion. The 3 columns ideally are:

  1. 2017 - PS
  2. 2017 - PP
  3. 2016 - PP
I figured I'd separate it just in case this db grew. After all, folks could just claim that one operator is a crook at x period. It's just for future ref. But it makes more sense to add it to the totals you guys have arrived at.

Maybe we're the outliers?! ;)
 
Bob23bk

Bob23bk

Visionary
Silver Level
Joined
Mar 24, 2015
Total posts
653
Chips
0
We're over 700k hands, does anyone else have any hands they can contribute to help us reach 1 million? :)
 
Starting Hands - Poker Hand Nicknames Rankings - Poker Hands
Top