Exclusive | How to compare two or more distribution forms (attached links)

The distribution of the processing group and the control group, the picture comes from the author

Now the two sets of histograms can be compared!

However, an important issue still exists: the size of the group is arbitrary. In extreme cases, if we tie less data together, we will finally get each group to at most observation data. If we bind more data together, we may eventually get a group. In two cases, if we exaggerate, the figure will lose the amount of information. This is the classic deviation-mutant exchanges.

Nuclear density chart

One possible solution is to use nuclear density functions to use nuclear density estimates (KDE) to use continuous functions to approach the histogram.

Sns.kDeplot (x = 'Income', Data = DF, Hue = 'Group', Common_norm = False); PLT.TITLE ("Kernel Density Function");

The income distribution of the processing group and the control group, the picture comes from the author

It can be seen from the figure that the estimated nuclear density of the income of the processing group seems to have "fatter tails" (higher variance), but the average value of the group is more similar.

The problem of nuclear density estimation is a black box. It may cover the relevant characteristics of data.

Cumulative distribution map

A more transparent representation of the two distributions is to accumulate distribution functions. At each point (income) of the X -axis, we draw a percentage of equal or lower data points. The advantage of accumulating distribution functions is:

We don't need to make any arbitrary decisions (for example, the number of groups)

We don't need to do any approximation (for example: KDE), but we can represent all the data points

Sns.histPlot (x = 'Income', Data = DF, Hue = 'Group', Bins = Len (DF), Stat = "Density", Element = "STEP", Fill = FALSE, Cumulative = True, common_norm = false); PLT.TITLE ("Cumulative distribution function");

The cumulative distribution of the processing group and the control group, the picture comes from the author

How should we explain this picture?

The two lines are crossing near 0.5 (Y axis), which means that their median is similar

The orange line on the left is on the blue line, and the right side is the opposite, which means that the tail of the processing group is more fat (more extreme value)

Q-q chart

A related method is the Q-Q chart, where Q represents the division. Q-Q Figure draws out the number of two distributions. If the distribution is the same, you will get a straight line of 45 degrees.

There is no local Q-Q chart function in Python. Although the Statmodels package provides a QQplot function, it is quite troublesome. Therefore, we need to do it manually.

First, we need to use the Percentile function to calculate the quadrangers of the two groups.

Income = df ['Income']. Valuesincome_t = df.loc [df.group == 'timent', 'infome']. Valuesincome_c = df.loc [df.group == == 'Control', 'Income']. Valuesdf_pct = pd.dataframe () df_pct ['q_treatment'] = np.percentile (Income_t, Range (100)) df_pct [ 'q_control'] = NP.Percentile (Income_c, Range (100)) Now we can compare the two segments to each other, plus the 45 degrees line to indicate the benchmark benchmark Perfect fit.

PLT.FIGURE (FIGSIZE = (8, 8)) PLT.SCatter (x = 'q_control', y = 'q_treatment', dta = df_pct, label = 'actual file'); sns.lineplot (x = 'q_control', y = 'q_control', data = df_pct, color = 'r', label = 'line of perfect Fit'); PLT.XLABEL ('Quantile of Income, Control Group') PLT.YLabel ('Quantile of Income, Treatment Group') PLT.LEGEND () PLT.TITLE ("" QQ plot ");

Q-q picture, the picture comes from the author

The Q-Q diagram provides a very similar insight from the cumulative distribution map: the income of the processing group has the same median number (in the center of the center), but the wider tail (point below the left, above the right line).

Two group -inspection

So far, we have seen different methods between visual distribution. The main advantage of visualization is intuitive: we can observe differences through the naked eye and evaluate them intuitively.

However, we may want to evaluate the statistical significance of the difference between distribution more strictly, that is, answering this question "Is the difference observed whether the difference is systematic or due to sampling noise?"

We now analyze different tests to distinguish two distributions.

T test

The first and most common test is student T test. The T test is usually used for a relatively average. In this case, we want to test whether the average income distribution of the two groups is the same. The statistics of the test of two average comparison tests are:

T test statistics, the picture comes from the author

The sample average is in the formula, and S is the sample standard deviation. Under gentle conditions, the test statistics are the studs T distribution that is gradually distributed.

We use the ttest_ind function in scipy to perform the T test. This function returns test statistics and hidden P values.

From Scipy.stats Import TTEST_INDSTAT, P_value = TTEST_IND (Income_c, Income_t) Print (F "T-TESTIC = Stat: .4f, P-Value = P_value: .4f") T-Test: Statistics = -1.5549, P-Value = 0.1203

The P value of the inspection is 0.12, so we do not refuse the zero -fake setting of the average value of the processing group and the control group.

Note: The T test assumes that the variance of the two samples is the same, so it is estimated to be calculated on the combined sample. Welch ’s T test allows the two samples to be equal.

Differences for standardization mean (SMD)

Generally speaking, when we conduct random control tests or A /B tests, it is always a good way to test the average value difference in all variables in the entire processing group and control group.

However, because the nurse of the T -test statistics depends on the amount of samples, the T test is criticized because it makes it difficult for P value to compare across research. In fact, we may obtain significant results in experiments with small differences but large samples, and in an experiment with a large difference but a small sample volume, we may obtain unpredictable results.

A solution that has been proposed is the standardization mean difference (SMD). As the name suggests, this is not a suitable inspection statistics, but just a standardized difference. The formula is as follows: standardization average difference, picture from the author

Generally speaking, the value below 0.1 can be considered "small differences."

The best way is to collect the average value of all variables and the control group, and the distance between the two -either T test or SMD -to a table called a balance table. You can use the Create_table_one function in the causalml library to generate it. As the name of the function implies, when performing the A/B test, the balance table should be the first table you present.

From Causalml.Match Import Create_table_onedf ['Treatment'] = df ['Group'] == 'Treatment'create_table_one (df,' Treatment ', [Genger', 'AgEnter, 'Income'])

Balance table, the picture comes from the author

In the first two columns, we can see the average value of the processing group and the control group, and the standard error in parentheses. In the last column, the value of SMD indicates that the standardization difference between all variables is greater than 0.1, indicating that the two groups may be different.

Mann -Whitney U test

Another optional test is Mann -Whitney U test. Zero fakes are the same powder in the two groups, and the selection assumptions are the value of one group larger (or smaller).

Different from the inspection we have seen before, the Mann -Whitney U test does not pay attention to the abnormal value, but places attention on the center of distribution.

The inspection process is as follows.

1. Merge all data points and sort (upgrade or order)

2. Calculate U₁− = R算n₁ (n₁+ 1)/2, R₁ is the rank of the first group, and N₁ is the number of the first group of data.

3. Calculate the second group of U 计 in a similar method

4. Statistical inspection volume is stat = min (u₁, u₂)

Under the zero -fake setting between the two distributions (that is, the same medium number), the inspection statistics are gradually distributed nearly normal when the mean value and square difference are known.

The intuitive method of calculating R and U is: if the value of the first sample is greater than the value of the second sample, then R₁ = N₁ (N₁+ 1)/2, so U₁ will be zero (the minimum value can be obtained) Essence Otherwise, if the two samples are similar, U₁ and U₂ will be very close to N₁n₂/ 2 (the maximum value can be obtained).

We use the Mannwhitneyu function from Scipy to perform testing.

From Scipy.stats Import MannwhitNeyustat, P_value = Mannwhitneyu (Income_T, Income_c) Print (F "Mann-Whitney u Test: Statistic = STAT: .4f, P-Valuel p_value: .4f ") Mann -Whitney u test: statistic = 106371.5000, P-Value = 0.6012