Forum: help

RE: Some questions regarding the usage of the anovaBF function [ Reply ]
By: Vincent Laflamme on 2013-07-18 09:26

Thank you very much for your swift response.

1a) I think I understand your point about how the Bayesian ANOVA is more about model selection than the testing of effects. I guess my question is more along the line of "how do you report the results of a Bayesian ANOVA?". Would you report that, among every possible models, the one with only the main effect A has the highest Bayes factor? Or would you start by saying that A+B is better than A+B+A:B and then that A is better than A+B?

1b) Let's say that the bayes factor comparing the A+B+A:B model with the A+B model (I'll denote this as BF(A+B+A:B / A+B) ) is 0.9 . How sure can we be that A:B doesn't contribute to the prediction enough to be included? Let's say, on the other hand, that BF(A+B+A:B/A+B) is rather high (5 or more). Then, would you say that I need to run a BANOVA for each level of one of the factors (i.e. check simple effects) since the main effects can't be interpreted in the presence of an interaction?

1c) Actually, the Bayes factor of the model that includes only a single main effect is the best among the bunch BF(A+B / A+B+A:B)=3.5, BF(A/A+B) = 6, BF(A/NULL) = 20. My incredulity stems from the fact that, in the classical ANOVA's results, the interaction effect came out relatively strongly (p<.001; partial eta squared = 0.12). I would have expected the best model to include the interaction effect and yet it clearly didn't. I just wanted to make sure that this discrepancy wasn't due to a programming error on my part. You can't be too prudent when you use a package for the first time, after all.

2) Ah, ok. It seems straightforward enough. Thanks for the tip.

At any rate, thank you very much for your time.

RE: Some questions regarding the usage of the anovaBF function [ Reply ]
By: Richard Morey on 2013-07-18 07:33

[forum:39733]

Hi Vincent,

I'll answer the questions in turn.

1. My viewpoint is that there is no analogous concept to the "significance" of individual effects with Bayes factor. It is best considered a model comparison technique, where several models are compared. It sounds like in your case, you got a one main effect and an interaction - thus, theBayes factor that includes both main effects and the interaction is the "best" model.

The BayesFactor package actually requires that for an interaction to be included, both main effects must also be included. This is fairly standard (see http://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf, for instance). It is unclear to me whether testing a main effect while an interaction including it is assumed makes much sense. See also the discussion here: https://github.com/richarddmorey/BayesFactor/pull/1. If you really want to do this, you could use the argument whichModels="top" (see the help). I do not recommend this, but I included this option for people who are really sure that this makes sense for their application (you could also try whichModels="all").

The main thing to remember is that we are comparing *models*, not effects. Effects exist in the context of the models that are compared, and we cannot really consider effects separately. This actually a benefit of Bayes factor - Bayes factor considers how well a *model* predicts data. Effects don't predict data by themselves, so we can't directly make inferences about them. All inferences must be based on the quality of prediction.

2. You do not need to make corrections for multiple comparisons, assuming the comparisons were selected a priori. If these are not post hoc comparisons, then all you need to do is create a new column of data that includes the constraint you want. For instance, if you have a column called "Condition" with three levels: "Lev1", "Lev2", and "Lev3", and you wanted to test the hypothesis that Lev1 and Lev2 were equal, you would create a new column - say "ConditionRecode" - in which all the "Lev2"s were replaced with "Lev1". Then you include ConditionRecode where you would have included Condition, and combine the Bayes factors objects (you should have two - one that used Condition, and one that used ConditionRecode) together with the c() function.

If the comparison is post hoc, however, this is more difficult. There are currently no methods to deal with post hoc Bayesian testing, and it is indeed an issue, because the evidence needs to be discounted somehow. But how exactly this is to be done is an open question.

Some questions regarding the usage of the anovaBF function [ Reply ]
By: Vincent Laflamme on 2013-07-18 00:55

[forum:39732]

First of all, I'd like to thank you and congratulate you for making such a comprehensive and easy to use package for Bayesian ANOVAs. I'm quite enthusiastic about it and I'll try to use it in my studies in psychophysics. I have a couple of questions regarding its usage though. If you could answer them, I would be most grateful.

1) Just how divergent from regular ANOVAs can the results of a Bayes factor analysis be? The reason I ask is that I ran both a run-of-the-mill and a Bayesian ANOVA on my data and it yielded some pretty different results. The data itself is pretty simple: 1 dependant variable and 2 within subject factors. The regular ANOVA found a statistically significant effect of the second within subject factor (p <.001, partial eta-squared = .12) and a statistically significant interaction effect (p<.001, partial eta-squared = .12).

Now, if I understand correctly, the way to assess the "statistical significance" of a factor in a Bayesian ANOVA (if such a term exists in that context), one has to compute the Bayes factor that relate the model without the effect to be tested with the full model. When I apply this procedure, I find a Bayes factor of 27 for the main effect and a Bayes factor of only .31 for the interaction. I find that fairly surprising considering how the partial eta squared of that interaction effect on par with that of the main effect in the classic ANOVA. Also, when I look at the means, it would seem that an interaction effect is plausible. Do you think such a discrepancy is possible or would you say that I probably entered something wrong? This is my call to the anovaBF function:
anovaBF(formula=par1 ~ newCondS*newCondC +Subject1 ,data=fitsG_conc,whichRandom='Subject1',whichModels="top",iterations=40000)

2) If I see that the Bayes factor of a main effect is significantly high, how would I proceed to test specific paired comparisons? What I did was to only select two levels from the factor of interest and then I re-ran the anovaBF. Is that correct? Would I have to take into account any correction factor for multiple comparisons?

Thank you very much.