Brief Online Socio-Cognitive Mindfulness Interventions Neither Improve Socio-Cognitive Mindfulness nor Cognitive Biases: A Two-Study Conceptual Replication and Reanalysis of a Randomized Controlled Trial

An a-priori power analysis was conducted using G*power (Faul et al., 2007), setting power to 80% and type 1 error rate to 5% (one-sided). The target effect size was derived as follows: Taking Table 3 of the original study, odds ratios for the success probabilities in the “mindful” and combined “low mindful” and “mindless” groups were computed for all cognitive-bias tasks, which resulted in the selection of the 11 tasks listed above. The odds ratios of these 11 tasks were then transformed into the metric of Cohen’s d (for the conversion formula, see Borenstein et al., 2021, p. 44), yielding a median d of 0.21. This resulted in a target sample size of at least n = 564 (n = 282 per group).

Participants were recruited from the German-speaking community via a publicly available link to the online survey on various social media platforms (Facebook, WhatsApp, LinkedIn, Instagram, and Telegram) and via personal contacts of 11 research assistants. Participation was voluntary, fully anonymous, and uncompensated. To partake in the study, participants had to be at least 18 years of age and give their informed consent (via button-click). Participants could withdraw at any time. Data were collected between April 7 and May 12, 2022. The original study drew its sample from the English-speaking community, using a similar online survey, but did not collect any sociodemographic information of its participants.

Participants were excluded from analysis if they (1) did not pass the manipulation check of the intervention, (2) had more than 25% missing in the cognitive-bias tasks, (3) had completed the survey unusually quickly (using a cutoff of > 2 for a built-in relative speed index in SoSci Survey; Leiner, 2019), or (4) required too much time for the cognitive-bias tasks (taking longer than the 75th percentile of the total sample plus two times the interquartile range, rounding all numbers up, which is slightly more lenient than the standard definition of an outlier in a boxplot).

An RCT with two between-subject conditions (intervention, control) and a single post-interventional outcome assessment was conducted online using SoSci Survey (Leiner, 2019). The BSCMI consisted only of the main component of the intervention for the “mindful” group in the original study. Participants were asked to notice three new things in their environment they had never noticed before and list them in a text box (“Look around where you are right now, and try to notice three new things that you have never noticed before. What are those three things?”). This task was investigated in tandem with a number of other tasks (that were also tested in Study 2; see below) in the original study, but was considered to be solely responsible for the reported effect by the authors of the original study themselves (p. 8). There was no such task in the control condition of the present study. Thus, Study 1 directly tested whether the “notice new things” intervention in isolation had the effect that was ascribed to it by the authors of the original study.

The built-in random sequence generator of the online survey tool was used to allocate participants to either the control or intervention group (simple randomization). This assignment was concealed to the participants. After randomization, participants in the control group were directed to the cognitive-bias tasks, receiving no intervention, whereas participants in the BSCMI condition first received the intervention, and then completed the cognitive-bias tasks.

The participants were blind to the goals of the study and were not aware that there is another condition. Specifically, participants were also not made aware that the cognitive-bias tasks tested cognitive biases, but were told that the respective items presented various scenarios of decision-making and should be answered “intuitively.”

We followed the CONSORT guidelines (Montgomery et al., 2018) in reporting Study 1 (and also Study 2; below). The filled-out CONSORT checklist can be found in Online Resource 2. Open data (of both studies), analysis code, and materials are provided at https://​osf.​io/​sr6x8/​.

For economic reasons (i.e., brevity), only half of the 22 cognitive-bias tasks of the original study were used as main outcomes in the present study. The selected items concerned cognitive biases that appeared to be highly malleable by the BSCMI according to Table 3 in the original study and did not require pictures for visualization or responses to more than one question per item for ease of presentation and comprehension. The items related to the conjunction fallacy, gambling fallacy, positional bias, emotional attachment, fairness consideration, availability heuristic, equiprobability illusion, representativeness heuristic, base-rate neglect, endowment effect, and disposition effect (for item contents and full references, see Online Resource 1). Note that Maymin and Langer (2021) provided no rationale for their selection of cognitive-bias tasks. Items were translated to German using the parallel-blind technique (Behling & Law, 2000). All but two items (Items 4 and 10) were in single-choice format. For analysis, items were scored 0 = false and 1 = correct, and, extending the original study, averaged to yield a total cognitive-bias score. This score can be interpreted as the average probability of solving each task correctly. Higher scores thus indicated a lower overall susceptibility to the assessed cognitive biases. Even though the reliability of the cognitive-bias score appeared to be relatively low (McDonald’s ω = 0.40, 95% confidence interval = [0.32, 0.49]), it was of comparable size as in the original study (ω = 0.58 [0.47, 0.70]). While this low reliability suggested heterogeneity in the cognitive-bias tasks, total scores could still be sensibly interpreted as the average probability of solving the tasks correctly. Potential differences in the individual cognitive-bias tasks were addressed by investigating each task also individually (see the “Data Analyses” section). Whether participants had actually written something down in the text box following the BSCMI (see the “Procedure” section) was used as a manipulation check for each individual.

Participants were also queried for gender, age, and education. The online survey comprised further sociodemographic items, questions on meditation practice, and psychological scales, which were not of interest in Study 1. However, information on meditation practice and scores of one psychological scale, a short form of the Five-Facet Mindfulness Questionnaire (Burzler et al., 2019; ω = 0.82), were used for exploratory joint analyses of the combined Study 1 and Study 2 data on the associations of other variables with cognitive biases in Study 2.

Baseline comparisons between the intervention and control groups were conducted with t tests and chi-squared tests (two-sided). Dropout after group allocation was investigated with logistic regression analysis, using the variables at baseline (combining diverse with female participants and primary with secondary education, because of small group sizes) as predictors. The mean cognitive-bias scores in the intervention and control groups were then compared with Welch’s t test and additionally checked with the nonparametric Mann–Whitney U test. These two analyses were preregistered and are reported as the main results of Study 1. Effect sizes were interpreted following Funder and Ozer (2019), converting their benchmarks in the metric of r into the metric of Cohen d for mean differences: very small effect = 0.05 (d ≈ 0.10), small = 0.10 (d ≈ 0.20), medium = 0.20 (d ≈ 0.40), large = 0.30 (d ≈ 0.60), and very large = 0.40 (d ≈ 0.90). Significance was set to p < 0.05.

Additional exploratory, not preregistered, analyses also investigated differences between groups in individual tasks, using odds ratios and chi-squared tests, and investigated the original study data with the standard parametric and nonparametric statistical tests listed above. Further, Welch’s equivalence test (e.g., Lakens, 2017) was conducted with the Study 1 data. This approach turns traditional null hypothesis significance testing (NHST) around and tested whether the difference in cognitive-bias scores in the intervention and control groups was significantly less than a minimally meaningful effect, which was set to solving at least one item more in a rational manner (for the lack of literature for cognitive biases, the criterion of “one item more” was adopted from memory research, see Otgaar et al., 2025). Since NHST cannot provide evidence of absence of an effect, equivalence testing was recommended as a suitable alternative (Lakens, 2017). Equivalence testing implies conducting two one-sided tests, of which, however, only the result of the less extreme test is relevant for interpretation and in reporting (Lakens, 2017).

The baseline and dropout analyses were not preregistered, but it was preregistered to test the superiority of the BSCMI (using one-sided tests as did the original study). However, we switched to two-sided tests as the effect turned out to be in the opposite direction in the present data. Additionally, the exploratory reanalysis of the original study data with standard parametric and nonparametric statistical tests also suggested no significant differences between the “mindful” and “mindless” groups either (see the “Results” section).

Preregistered moderation analyses of trait meditative mindfulness on the intervention effect were waived and are not reported. The publicly available dataset of the original study was also reanalyzed, using the same statistical methods as for the dataset of Study 1. The results are reported as exploratory analyses.

In total, 977 participants started the survey, of whom 368 (38%) dropped out before group allocation. 591 participants were allocated to either the control or the intervention condition, exceeding the planned sample size of n = 564. Sociodemographic information and baseline comparisons between the two groups are provided in Online Resource 3. The majority of participants were around 30 years of age, were female, and had secondary or tertiary education. Groups did not differ in these variables. In total, data from 479 participants were investigated post intervention (see Table 1 for descriptive statistics). The baseline variables did not predict dropout after group allocation (χ² = 1.38, df = 3, p = 0.71, Nagelkerke R² < 0.01). The participant flow is shown in Fig. 1.

Cognitive bias mean scores did not differ between the intervention and control groups (Welch’s t-test: t =  − 0.39, df = 461.66, p = 0.70; U test: z = 0.18, p = 0.86; see Table 2 for group means and effect sizes).

There were no differences between the intervention and control groups in the original study data (neither in the full set of 22 items nor in the subset of 11 items used in Study 1; Table 2). There were no significant differences between the two groups in any of the tasks, neither in Study 1 nor in the original study data. The results of the single-item analyses are presented in Online Resource 4.

Welch’s equivalence test was significant, t =  − 2.91, df = 461.66, p = 0.002, indicating that the difference between groups was smaller than the threshold equivalent to correctly solving one additional item in a rational manner.

Recruiting was performed as in Study 1, this time, 12 different research assistants supported data collection. Data were collected between April 6 and May 11, 2023.

The sample size for Study 2 was determined via two approaches. For a conservative estimate, the standardized mean difference in the cognitive-bias scores between the “mindful” and the “mindless” conditions of the original study data (d = 0.33) and their sample sizes (ns = 32 and 38) were used to conduct a safeguard power analysis (Perugini et al., 2014) with a target power of 80% and a 95% confidence interval. This approach yielded a (nearly) infinite total n. Thus, we reverted to the approach of Study 1, which again suggested a total n of 564 (one-sided) or (given the results of the reanalysis of the original data and of Study 1) of 712 (two-sided). As especially this second target sample size appeared excessively large, Study 2 pragmatically opted to simply collect as much data as possible.

Study 2 used all components of the BSCMI administered to the “mindful” group from the original study, thus replicating the original intervention more closely than Study 1. Further, a manipulation check of socio-cognitive mindfulness was implemented, as in the original study. Yet, the replication was also extended, investigating the associations between cognitive biases, trait meditative mindfulness, cognitive reflection, and rationality in an exploratory analysis. For this, the data of Studies 1 and 2 were also combined.

The BSCMI consisted of four tasks that were identical to the ones in the “mindful” condition in the original study: First, participants were presented with “notice new things” task (see Study 1). They were then asked to investigate two computer-generated images which resembled grey noise and state which of the two they would prefer. Next, participants were presented two almost identical hidden-object pictures showing various fruit. They were then asked whether they could identify the missing fruit in one of the pictures. Finally, participants were presented with a standard optical-illusion task (Rubin’s vase) showing two pictures: one two-dimensional black-and-white negative and one three-dimensional colored version. In both pictures one could see either a vase or two opposing faces. Participants were asked which of the two they saw on the two pictures.

The intervention featured, as regards content, the identical elements as the original study. However, the original study placed the “notice new things” task at the end and considered the other three as “warm-up” tasks (yet, without providing any empirical evidence for this). Actually, the placing of the “notice new things” task at the beginning of the intervention was an unintentional deviation that only became clear after the data collection of Study 2 was completed. However, Study 2 still allowed testing the effect of the full intervention of the original study, albeit in a different order.

Randomization was performed as in Study 1 and blinding was performed as in Study 1, as well. However, this time participants were told to “choose the answer that is appropriate for you” in the cognitive-bias tasks.

The same 11 cognitive-bias tasks as in Study 1 were used, but three tasks on loss aversion, hyperbolic discounting, and emotional attachment were added (Items 1, 2, and 9) and one task on cognitive reflection (Item 20) was dropped (see Tables S1 and S2). The item on cognitive reflection was dropped, because it was also contained in the Expanded Cognitive Reflection Test (see below) that was presented prior to group allocation in Study 2. Three cognitive-bias tasks of the original study were thus added (a) to compensate for the loss of the cognitive reflection item and (b) because pilot testing suggested that further tasks could be added without making the trial too long. Items 4, 5, 7, 15, 16, 18, 19, and 21 of the original study thus were excluded from Study 2. The reliability of the cognitive biases used in Study 2 was ω = 0.48.

The Langer Mindfulness Survey (LMS; Pirson et al., 2018; ω = 0.84) was used for a manipulation check and was presented immediately after the cognitive-bias tasks as in the original study. Maymin and Langer (2021) presented LMS scores as evidence of the specific effects of the intervention in their “mindful” condition on socio-cognitive mindfulness, compared to their other two conditions. Notably, the term “manipulation check” was not used in the original study. However, the application of the LMS conceivably served this purpose, which is why it was also used in the present study and under the header of the term “manipulation check.” The LMS consists of 14 items that measure socio-cognitive mindfulness with three subscales (novelty-seeking, novelty-producing, and engagement). Participants are asked whether they agree with a statement (e.g., “I like to investigate new things”) on a 7-point scale (0 = I totally disagree to 6 = I totally agree). The LMS was translated into German using the parallel-blind technique (Behling & Law, 2000).

Other psychological scales and tests included in the research project were used in an exploratory analysis to investigate their associations with cognitive-bias-task performance, beyond socio-cognitive mindfulness. The Expanded Cognitive Reflection Test (ECRT; Toplak et al., 2014; ω = 0.76) and a short form of the Rational Experiential Inventory (REI-S24; Norris & Epstein, 2009; ω = 0.88) were presented prior to group allocation. The ECRT consists of seven reasoning items with open (Items 1 to 6) or single-choice (Item 7) response formats. The items test the tendency to suppress an intuitive response (coded 0 = incorrect) and engage in further reflection to solve the items correctly (coded 1 = correct). The total ECRT score is a predictor of performance on rational thinking tasks. The REI-S24 consists of 24 items and assesses two trait reasoning styles (Rationality and Experientiality, each divided into further subscales) with a 5-point response scale (1 = definitely false to 5 = definitely true). Only the 12 items pertaining to Rationality were presented in Study 2.

Further, a German short form of the Five-Facet Mindfulness Questionnaire (FFMQ; Burzler et al., 2019; ω = 0.82) was administered to the participants in tandem with the other measures. It consists of 23 items and assesses five facets of trait meditative mindfulness (Observe, Describe, Actaware, Nonjudge, Nonreact) with 4 items each (7 for Nonreact) with a 5-point response scale (1 = never or very rarely true to 5 = very often or always true). Meditation practice was rated on a 7-point scale (0 = never, 1 = not regularly, 2 = at least once per month, 3 = once per week, 4 = twice per week, 5 = three times per week, 6 = four times per week or more often). The ECRT and the REI-S24 were translated into German using the parallel-blind technique (Behling & Law, 2000).

As in Study 1. Additionally, participants in the intervention group were excluded if they did not provide answers to all tasks of the intervention or completed all four intervention tasks in less than 30 s. Study 2 provided feedback to participants for each survey page, highlighting unanswered items. Participants could then intentionally either skip the item(s) in question or fill out missing items, before going to the next survey page. This substantially reduced dropout rates in the intervention group, compared to Study 1 (see the “Results” section). For the manipulation check on the group level, the mean difference between the intervention and control groups in total LMS scores was evaluated with a Welch t-test (both for the Study 2 data and the data of the original study). The manipulation check analysis was not preregistered and is presented as an exploratory analysis.

The associations of cognitive-bias-task performance with other psychological variables and sociodemographic characteristics were investigated in an exploratory linear regression analysis, using age, gender, education, meditation practice, and trait meditative mindfulness as predictors in an analysis of the combined datasets of Study 1 and Study 2, and adding trait rationality and cognitive reflection in an analysis of the data from Study 2 only.

Performing the preregistered Student t-test to further check on the results of the Welch t-test was deemed unnecessary and waived, which was a deviation from the preregistered analysis plan.

In total, 623 participants started the survey, of whom 294 (47%) dropped out before group allocation. 335 participants were allocated to either the control or the intervention condition. The participant flow is shown in Fig. 1. Sociodemographic information of the sample and baseline comparisons between groups are provided in Online Resource 3. The participants were socio-demographically similar to those of Study 1 and control and intervention groups did not differ from one another. In total, data from 299 participants were investigated post intervention (see Table 3 for descriptive statistics). The baseline variables did not predict dropout after group allocation (χ² = 2.75, df = 3, p = 0.43, Nagelkerke R² = 0.02).

Cognitive bias mean scores did not differ between the intervention and the control groups (Welch t-test: t =  − 1.46, df = 310.77, p = 0.17; U test: z = 1.48, p = 0.14; Table 2).

The manipulation check indicated a very small mean difference in total LMS scores, favoring the control group, which was not statistically significant (Welch’s t test: t =  − 1.10, df = 285.82, p = 0.27; Table 2). In the original study data, the mean difference in total LMS scores favored the intervention group, but was not significant there either (Table 2); the intervention and control groups also did not differ in cognitive-bias mean scores of the present 13-item selection. The results of the single-item analyses are presented in Online Resource 4. There were, again, no significant differences between the two groups of Study 2 in any of the tasks, except in the availability heuristic, favoring the control group. However, given the number of conducted statistical tests, this could also have been a type 1 error. Correcting for the number of tests (i.e., 13; applying Bonferroni correction), this difference was no longer statistically significant (p = 0.20). The equivalence test was not significant, t =  − 0.71, df = 294.59, p = 0.07, suggesting neither the presence nor absence of a minimally meaningful effect.

The first exploratory regression analysis with the other psychological variables and sociodemographic characteristics (Table 4) in the combined samples of Study 1 and 2 suggested that male and younger participants had higher cognitive-bias scores. It further suggested that (1) more frequent meditation practice was associated with lower task performance (2) and higher trait meditative mindfulness with higher task performance (zero-order correlations and further information on the variables involved are presented in Online Resource 4). This pattern of results was broadly replicated in the second exploratory regression analysis of the Study 2 data (age, p = 0.08; meditation practice, p = 0.05), with the exception that trait meditative mindfulness lost its association with performance, once rationality and cognitive reflection were included in the model. Higher scores in the added predictors were positively associated with better performance. Trait meditative mindfulness and rationality were highly positively correlated (Online Resource 4).