Measure that takes samples that is minimized in expectation for a uniformly-distributed random variable?
$begingroup$
I am having trouble thinking of a function that operates on a set of samples, that is, single-valued random variables between zero and one, $x_i in (0,1), iin{1,2,...I}$, and provides a measure of "non uniformity" of the samples:
$$
F({x_i}) : [0,1]^I to R
$$
This measure would be uniquely minimized (in expectation) when the samples are from a uniform distribution.
My first reaction was to bin these samples by cutting $[0,1]$ into equal-sized peices. Then the standard deviation of the counts across the bins would be my measure, minimized when the counts in each bin are equal.
But the binning allows for "cheating", that is, the samples may not actually be uniformly distributed, but only appear that way due to the binning procedure; a different choice of bins would show the problem.
Any better ideas?
The motivation is to evaluate the quality of a CDF fitting procedure. The fitting procedure is optimal when the CDFs (different CDF for each sample) map all the "real" samples $y_i in R$ uniformly to the [0,1] interval.
sampling descriptive-statistics uniform minimum uniformity
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
$endgroup$
add a comment |
$begingroup$
I am having trouble thinking of a function that operates on a set of samples, that is, single-valued random variables between zero and one, $x_i in (0,1), iin{1,2,...I}$, and provides a measure of "non uniformity" of the samples:
$$
F({x_i}) : [0,1]^I to R
$$
This measure would be uniquely minimized (in expectation) when the samples are from a uniform distribution.
My first reaction was to bin these samples by cutting $[0,1]$ into equal-sized peices. Then the standard deviation of the counts across the bins would be my measure, minimized when the counts in each bin are equal.
But the binning allows for "cheating", that is, the samples may not actually be uniformly distributed, but only appear that way due to the binning procedure; a different choice of bins would show the problem.
Any better ideas?
The motivation is to evaluate the quality of a CDF fitting procedure. The fitting procedure is optimal when the CDFs (different CDF for each sample) map all the "real" samples $y_i in R$ uniformly to the [0,1] interval.
sampling descriptive-statistics uniform minimum uniformity
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
$endgroup$
add a comment |
$begingroup$
I am having trouble thinking of a function that operates on a set of samples, that is, single-valued random variables between zero and one, $x_i in (0,1), iin{1,2,...I}$, and provides a measure of "non uniformity" of the samples:
$$
F({x_i}) : [0,1]^I to R
$$
This measure would be uniquely minimized (in expectation) when the samples are from a uniform distribution.
My first reaction was to bin these samples by cutting $[0,1]$ into equal-sized peices. Then the standard deviation of the counts across the bins would be my measure, minimized when the counts in each bin are equal.
But the binning allows for "cheating", that is, the samples may not actually be uniformly distributed, but only appear that way due to the binning procedure; a different choice of bins would show the problem.
Any better ideas?
The motivation is to evaluate the quality of a CDF fitting procedure. The fitting procedure is optimal when the CDFs (different CDF for each sample) map all the "real" samples $y_i in R$ uniformly to the [0,1] interval.
sampling descriptive-statistics uniform minimum uniformity
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
$endgroup$
I am having trouble thinking of a function that operates on a set of samples, that is, single-valued random variables between zero and one, $x_i in (0,1), iin{1,2,...I}$, and provides a measure of "non uniformity" of the samples:
$$
F({x_i}) : [0,1]^I to R
$$
This measure would be uniquely minimized (in expectation) when the samples are from a uniform distribution.
My first reaction was to bin these samples by cutting $[0,1]$ into equal-sized peices. Then the standard deviation of the counts across the bins would be my measure, minimized when the counts in each bin are equal.
But the binning allows for "cheating", that is, the samples may not actually be uniformly distributed, but only appear that way due to the binning procedure; a different choice of bins would show the problem.
Any better ideas?
The motivation is to evaluate the quality of a CDF fitting procedure. The fitting procedure is optimal when the CDFs (different CDF for each sample) map all the "real" samples $y_i in R$ uniformly to the [0,1] interval.
sampling descriptive-statistics uniform minimum uniformity
sampling descriptive-statistics uniform minimum uniformity
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
edited Dec 9 '18 at 6:18
OrangeSherbet
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
asked Dec 8 '18 at 5:00
OrangeSherbetOrangeSherbet
1155
1155
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):
https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size
Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:
KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
etc.
answered Dec 8 '18 at 6:35
123learn123learn
761
$endgroup$
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
add a comment |
$begingroup$
Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.
answered Dec 8 '18 at 6:00
ChrisChris
512211
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f380927%2fmeasure-that-takes-samples-that-is-minimized-in-expectation-for-a-uniformly-dist%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):
https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size
Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:
KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
etc.
answered Dec 8 '18 at 6:35
123learn123learn
761
$endgroup$
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
add a comment |
$begingroup$
Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):
https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size
Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:
KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
etc.
answered Dec 8 '18 at 6:35
123learn123learn
761
$endgroup$
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
add a comment |
$begingroup$
Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):
https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size
Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:
KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
etc.
answered Dec 8 '18 at 6:35
123learn123learn
761
$endgroup$
Like @Chris said, you can use some kind of divergence or information theory ideas. KL divergence like he said is good, or more generally f-divergence, etc., but more simply just use entropy. You said minimized when uniform, so entropy would be maximized, so do negative entropy. You said maps to real numbers, maybe you want for easier comparison, [0,1] interval, so use (negative) normalized entropy (divide by logN):
https://math.stackexchange.com/questions/395121/how-entropy-scales-with-sample-size
Or since you said the end goal is to compare some empirical CDF's, then depending on your problem maybe there is a more direct way you can do this like:
KS test:
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
Anderson-Darling:
https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
Cramer von Mises
https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion
etc.
answered Dec 8 '18 at 6:35
123learn123learn
761
answered Dec 8 '18 at 6:35
123learn123learn
761
answered Dec 8 '18 at 6:35
123learn123learn
761
answered Dec 8 '18 at 6:35
123learn123learn
761
761
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
add a comment |
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
2
2
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
$begingroup$
The biggest revelation for me was realizing I could calculate the "empirical CDF" using the samples, and then use distance measures between the empirical CDF and the cdf for the uniform distribution (a straight line from [0,0] to [1,1]), all without binning anything.
$endgroup$
– OrangeSherbet
Dec 8 '18 at 10:52
add a comment |
$begingroup$
Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.
answered Dec 8 '18 at 6:00
ChrisChris
512211
$endgroup$
add a comment |
$begingroup$
Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.
answered Dec 8 '18 at 6:00
ChrisChris
512211
$endgroup$
add a comment |
$begingroup$
Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.
answered Dec 8 '18 at 6:00
ChrisChris
512211
$endgroup$
Take your samples and form a random variable $X$. To measure the non-uniformity of the samples measure the KL divergence between $X$ and $X'$ as $KL(X||X')$.
answered Dec 8 '18 at 6:00
ChrisChris
512211
answered Dec 8 '18 at 6:00
ChrisChris
512211
answered Dec 8 '18 at 6:00
ChrisChris
512211
answered Dec 8 '18 at 6:00
ChrisChris
512211
512211
add a comment |
add a comment |
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f380927%2fmeasure-that-takes-samples-that-is-minimized-in-expectation-for-a-uniformly-dist%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
