Introduction
Most of our everyday actions have multiple sensory consequences. For example, when we type, we experience the mechanical contact with the keys (tactile modality), the sound of the keystroke (auditory modality) and we see the resulting change on the screen (visual modality). Contiguity and contingency between the actions and their sensory effects results in the formation of representational links between the actions and their effects (Elsner & Hommel,
2001). Such connections lead to mutual influences between motor and sensory processes. On one hand, actions modulate the sensory processing of stimuli caused by them or coinciding with them (see e.g. Horváth,
2015; Hughes et al.,
2013) but motor control processes are also affected by modulations of their action effects (Hommel,
2009; Kunde et al.,
2004; Neszmélyi & Horváth,
2018). These action-effect connections also lead to the sense of agency (Haggard & Chambon,
2012; Haggard & Eitam,
2015). By characterizing changes in the perception and processing of a sensory event as a function of the eliciting action, or by characterizing changes in motor parameters of an action as a function of the elicited stimulus, it is possible to gain insight into the nature of such action-effect representations. To this end, in the current study, we investigated how constant delays between simple pinching actions and their auditory consequences influenced the force application patterns of these actions. We also explored whether the pattern of action effect-related motor adaptation was related to individual differences in sub-clinical schizotypy and Big Five personality factors.
The perception of self-generated stimulation is extensively studied in normal (e.g., Cullen et al.,
2009; Horváth,
2015; Weiskrantz et al.,
1971) as well as clinical (e.g., Ford et al.,
2014) samples. In addition, recent studies also demonstrated alterations of various action parameters as a function of the elicited sensory effects (e.g., Pfister et al.,
2014,
2023). Previous studies showed that during simple, brief interactions with tactile devices (pinching, tapping, or pressing), participants apply less force when these actions consistently elicit sounds, than in conditions without auditory action effects (Horváth et al.,
2018; Neszmélyi & Horváth,
2017,
2018; Volosin & Horváth,
2022). In such cases the elicited sound might be regarded as feedback about the successful interaction with the device, thus participants can lower the applied force. In contrasts, when the auditory feedback is absent and only the tactile feedback is available, there is a higher uncertainty whether the interaction was successful, and participants tend to use more force to ensure the action’s success (Neszmélyi & Horváth,
2017). This suggests that action-induced auditory effects – even when they are not directly task-relevant – are used to optimize the motor parameters of subsequent actions.
The idea that sensory action effects play a central role in the planning and execution of actions fits well with the framework of ideomotor theories such as the Theory of Event Coding (TEC) (Hommel et al.,
2001; Hommel,
2019). According to TEC, motor and sensory features are automatically integrated into episodic representations referred to as
event files. Recent studies indicated that continuous features of actions, like duration or force can be bound into the event file (Varga et al.,
2022,
2024), which raises the possibility that sensory features integrated into the event file can have an influence on these features: Ideomotor theories suggest that motor patterns are selected by activating the sensory effects associated with the action. Adding a sound or any other sensory consequence to an action results in a richer event file, which allows more opportunities to represent the action. The weighting of the features in an event file may depend on several factors (e.g., task relevance; other associated features). In the absence of an auditory action effect, it was hypothesized that the tactile feature will be dominant, whereas the presence of an auditory action effect may allow prioritization of the auditory features, and the weighting of the tactile dimension will be less pronounced. Although the ideomotor approach focuses primarily on the selection of motor plans based on already experienced action-effect conjunctions, the idea of optimization could also fit into this framework: Due to the differences between tactile and auditory effects in the information content of the stimuli (only auditory stimuli provide distinctive feedback on action success) and in their connection to motor features (continuous vs. “all-or-nothing”), the dominance of auditory or tactile features might be also reflected in different action control mechanisms.
Action-effect related motor adaptation may be brought about in several (not mutually exclusive) ways and may depend on different types of action representations. The sound elicited by the operation of the response device may well be regarded as feedback on the success of the action by the participants, which allows a “strategic” optimization: the participant may set a target force level that ensures successful interaction with the device while reducing work (i.e., conserving energy). For interactions longer than about 100 ms, there is also evidence for “on-the-fly” adjustments, that is, following force application onset, participants may respond to the onset of the tone, and stop force exertion (or start to release the device) earlier than initially planned (Cao et al.,
2020; Varga et al.,
2022).
Although theoretically action-effect contingency should provide the necessary conditions for action-effect related motor adaptation, temporal contiguity (proximity) of actions and action effects seems to have decisive influence on whether the type of action optimization described above actually occurs. In a series of experiments conducted by Neszmélyi and Horváth (
2018), participants performed voluntary pinching actions with the only constraint that a uniform between-action interval (BAI) distribution between 2 and 6 s should be produced, that is, no strict adherence to a regular temporal pattern was required. Action-effect delays were manipulated blockwise: sinusoidal tones were presented with 0, 50, 100, 200, 400, 800 or 1600 ms delays or no auditory event followed the actions (motor condition). The results showed that force gradually increased as a function of action-effect delay until about 200 ms, however, forces for delays longer than 200 ms did not differ from forces measured in the motor condition. These results suggest that although the auditory feedback contributes to the optimization of motor performance, the causal action-effect relationship is not sufficient per se, as there is a short temporal window in which optimization can occur (Neszmélyi & Horváth,
2018), and this time interval may otherwise not be necessarily accessible for conscious cognition (Aschersleben & Prinz,
1997; Elijah et al.,
2016). Neszmélyi and Horváth (
2018) speculated that the short window may reflect an automatic action-effect integration (binding) process. Again, it is important to note that this hypothetical process could be one of several mechanisms that contribute to the formation of action-effect representations, as other types of experimental paradigms provided evidence for action-effect integration at longer (1 s) intervals (Elsner & Hommel,
2004).
The importance of temporal contiguity in action-effect integration is also highlighted in studies relying on more complex tasks such as playing instruments (e.g., in playing tone sequences on the piano), where short delays (100–400 ms: Couchman et al.,
2012; 250 ms: Finney,
1997; 180 ms: Gates et al.,
1974) led to less accurate timing, and higher number of errors in comparison to the immediate auditory feedback condition. Moreover, when longer delays were also included (up to 1050 ms), performance was disrupted strongest at 270 ms action-effect delay (Gates & Bradshaw,
1974). Action synchronization to metronomes also slowed and became more variable when auditory feedback was presented with a delay (Pfordrehser & Dalla Bella,
2011; Pfordrehser & Palmer,
2002,
2006). The deteriorated performance was also correlated with reduced sense of agency for delayed sensory feedback (for a review see Rohde & Ernst,
2016; Wen,
2019), that is, participants felt to a lesser degree that their action led to the auditory effects. When participants were explicitly asked to rate the perceived degree of agency after every trial, trials with action-effect delay decreased the sense of agency both for more complex (producing musical sequences: Couchman et al.,
2012) and for simpler actions (actions resulting in visual change on the screen: Haering & Kiesel,
2015; Minohara et al.,
2016).
Forming connections between actions and their sensory effects is impacted in certain psychiatric or neurological conditions. In schizophrenia, a subset of symptoms such as delusions or third-person hallucinations (American Psychiatric Association,
2013) reflect difficulties in distinguishing self-generated and externally generated events (Frith & Done,
1988). Several studies demonstrated that patients with schizophrenia showed lower levels of sensory suppression in the processing of self-generated tones (Ford et al.,
2014) and their own speech (Ford et al.,
2007; Whitford et al.,
2011), which is generally interpreted as a reflection of the disruption of action-related sensory predictions (for reviews see: Abram et al.,
2022; Bansal et al.,
2018; Whitford,
2019). Although less conspicuous, psychosis-like experiences and motor dysfunctions may also occur in the non-clinical population, as well as in first-degree relatives of schizophrenic patients without impairing everyday functioning (Hirjak et al.,
2018; Mason,
2015; Meehl,
1962,
1990; Oestreich et al.,
2016). Thus it seems reasonable to hypothesize that individual differences in forming action-effect representations may correlate with non-clinical schizotypy or schizotypal personality (a cluster of personality traits characterized by possible presence of unusual perceptions, magical or odd thinking, inappropriate affect, cognitive disorganization, social withdrawal and anhedonia based on the feeling of paranoia, as well as discomfort with close relationships, American Psychiatric Association,
2013; Mason,
2015; Ross et al.,
2002). Indeed, when contrasting sensory suppression of self-generated sounds in individuals with high and low levels of schizotypy, a pattern similar to that obtained by contrasting patients with schizophrenia and healthy controls was found (Oestreich et al.,
2015,
2016). In contrast, while the amount of sensory suppression was not associated with schizotypal traits, the peri-motor representation of sense of agency (i.e., readiness potential) was decreased in individuals with high schizotypy, suggesting lower reliance on the efference copy of the motor commands (Luzi et al.,
2024). Behavioral studies demonstrated that when the delay between actions and a resulting tone was varied from 100 to 900 ms, high schizotypal participants showed lower sense of agency in the 100 ms condition but not at intervals of 200 ms or longer (Pan et al.,
2021). This is in line with results of Luzi et al. (
2024) who found no association between schizotypy and sense of agency at action-effect delays of 200, 500 and 800 ms. Utilizing a shorter time scale (from 0 to 135 ms in 15 ms steps), the high schizotypal group started to experience decreased sense of agency significantly earlier than the low schizotypal group (Asai & Tanno,
2008). These results indicate that despite the presence of action-effect contingency, high schizotypal individuals’ ability to recognize the sensory consequences of their own actions is reduced for shorter action-effect delays. Furthermore, individuals with high level of schizotypy reported lower sense of agency when they had to judge whether a cursor was moved by them or by the experimenter (Asai & Tanno,
2007), and they showed abnormal response patterns to other-produced actions (Itaguchi et al.,
2018).
Besides schizotypy, other personality traits might also influence how action-effect links are processed. One might speculate, for example, that people who have a reduced sense of control in a general sense perceive the relationship between their actions and the consequences of these actions as weaker than those characterized by a higher sense of control. Since several explanations of action optimization rely on recognizing the connection between action and effect, such differences in personality might be also reflected in the use of auditory action effects in action control. Schwarz et al. (
2022) demonstrated that the Big Five personality trait framework (Digman,
1990) can capture differences in the perception of the action-effect connection: In an environment characterized by dynamically changing action-effect contingency (from 50 to 100%), participants with higher levels of the Big Five Neuroticism and those with lower level of Big Five Openness traits were experiencing lower level of sense of agency. It is an open question, however, whether explicit agency judgements, investigated by Schwarz et al. (
2022), are also reflected in other phenomena related to the action-effect connection, such as action optimization: When participants had to judge the time of their actions and of the resulting effects, no relationship was found between intentional binding and any of Big Five personality factors (Galang et al.,
2021). It is important to note that beside exploring connections between similar concepts on the levels of personality and motor control, the assessment of Big Five traits might also be relevant for the connection between schizotypy and action optimization, which is a central theme of our study. In addition to the notion that the continuum of schizotypy can be considered as a personality trait which might be present in the normal, non-clinical population (Mason,
2015; Ross et al.,
2002), schizotypy was also associated with general personality dimensions. Specifically, it was demonstrated that the factors of the Big Five personality trait framework (see e.g., Digman,
1990) were not only correlated with schizotypal traits (Asai et al.,
2011) and schizophrenia symptoms, but also shared a common taxonomy, suggesting that schizotypal and normal personality reflect the same underlying dimensions (Cicero et al.,
2019).
Based on the literature presented above, we conducted a conceptual replication of the study by Neszmélyi and Horváth (
2018) with a finer temporal resolution and larger sample. In the current study, participants were instructed to pinch a force sensitive device every 3 s while the time interval between actions and auditory feedback increased gradually, block-by-block from 0 to 560 ms in 70 ms steps. Due to this systematic progression of action-effect delays, the present paradigm provided participants an opportunity to learn the action-sound association and thus adapt to the task over time, thereby providing an upper estimate for the temporal limit of action-effect integration reflected by action-effect related motor adaptation.
We also attempted to individually characterize the duration of the window in which action-effect related motor adaptation occurred by fitting a sigmoid function on the force as a function of action-effect delay. In the study of Neszmélyi and Horváth (
2018), the gradual increase of force with the action-effect delays until 200 ms was demonstrated on group-level, however, the temporal window of integration was not defined individually. In the present study, fitting a sigmoid to individual data allows the characterization of each participant with their own temporal window of integration, opening up the possibility to get a deeper insight into individual differences in motor optimization of actions. Based on this assumption, we also tested the hypothesis that this integration window will be shorter in participants with higher levels of schizotypy (as measured by the Oxford-Liverpool Inventory of Feelings and Experiences – O-LIFE-questionnaire, Mason et al.,
2005, see below), suggesting that these participants rely on the sensory consequences of their actions for a shorter time. That is, we hypothesized that higher levels of schizotypy will be associated with a lower capacity to exploit delayed auditory feedback to optimize actions which might be linked to the altered perception of self-generated stimuli (Oestreich et al.,
2015,
2016), changes in sense of agency (Asai & Tanno,
2007,
2008) or even with unusual perceptions (Mason,
2015; Ross et al.,
2002). Because of the similar taxonomy of schizotypal and general personality dimensions, we also explored potential associations between Big Five personality traits (as measured by the Facet5 questionnaire, Buckley & Williams,
2002, see below) and the temporal patterns of action-effect related motor optimization.
Discussion
The aim of the present study was to investigate the effect of action-effect delay on motor optimization for a simple pinching action. We tested whether the size of the action-effect related motor optimization window was correlated with trait schizotypy, and explored its potential relationship with Big Five personality traits. Confirming previous studies (e.g., Horváth,
2024; Horváth et al.,
2018; Neszmélyi & Horváth,
2017,
2018; Volosin & Horváth,
2022), in comparison to the motor condition, participants exerted less force when pinching the FSR elicited a sound. Our main finding is that even with 100% contingency between actions and tones, the level of force gradually increased as a function of action-effect delay, confirming results of previous studies (Cao et al.,
2020; Neszmélyi & Horváth,
2018) in a larger sample, demonstrating that in similar tasks, action-effect contingency (causal action-effect relationship) on its own does not result in motor optimization – temporal contiguity is also needed (Neszmélyi & Horváth,
2018). The hypothesized correlation with schizotypy, however, was not found, in fact, Bayes Factors provided some support for the absence of correlation. Similarly, no significant correlations with Big Five personality traits were found.
Whereas Neszmélyi and Horváth (
2018, Experiment 1), found no significant force differences for tone delays of 400, 800 or 1600 ms in comparison to actions with no auditory consequences, in the current study all delays (up to 560 ms) resulted in significantly lower forces than in the motor condition. This may be attributed to higher power (the sample size in Neszmélyi & Horváth,
2018 was
N = 28), but also to the potential effect of the fixed presentation order with systematically increasing delays (whereas Neszmélyi & Horváth,
2018, used random block order) that might have allowed participants to utilize the action-contingent sounds as feedback to some degree. The force difference between the motor and the delay conditions may also indicate a strategy, or habit specifically related to the absence of stimulation in the motor condition. The motor condition differs from the delay conditions in that there is no unequivocal feedback from the device that signals that the action was registered. Because of this, participants’ strategies, habits, assumptions (or knowledge) about the operation of the device may become dominant among the processes that shape force application and may contribute more markedly than in the delay conditions (Horváth,
2024). In the present study, participants were exposed to a relatively long period of actions resulting in (delayed) tone elicitation (9 blocks: about 23 min not counting breaks), thus, they might have strongly compensated for the “sudden” lack of auditory feedback in their final experimental block.
In Experiment 2 in Neszmélyi and Horváth (
2018), the authors attempted to map the delay-related force change in the 0 to 200 ms range in 50 ms steps, but failed to find a delay that could be characterized as an inflection point. In the current study the similarly fine-grained delay manipulation in the 0-560 ms range allowed the fitting of a sigmoid curve to the force data in 88% of our sample, which provided a group-mean inflection point latency of about 290 ms, with considerable variability (see Table
2). Comparisons between force differences between adjacent delay conditions (i.e. an approximation of the derivative of the force-delay function) in the whole sample (
N = 116) provided further support for the inflection point being at around the 280 ms action-sound delay. Given the systematic, block-by-block increase of delay during the experiment, which allowed considerable time to adapt to the action-tone relationship, this estimate may well be inflated, but even so, the time interval in which action-effect related motor adaptation occurred was relatively short, which is compatible with the assumption that information on action-effect contingency has to be available within a short time to be usable for the optimization of subsequent movements. This result fits well with the idea of intentional weighting of different action effects within event-files (Memelink & Hommel,
2013): when a sensory action consequence occurs earlier, a stronger action-effect association can be formed, and the event file that includes the given consequence becomes available at an earlier timepoint during the sequence of repetitions – that is, participants can immediately rely on the sound to represent the action instead of enhancing tactile, or proprioceptive re-afference by applying more force. This temporal constraint on action optimization is also compatible with the notion that at longer intervals participants experience lower sense of agency (Rohde & Ernst,
2016; Wen,
2019), in other words, for sense of agency temporal contiguity is also required.
Given that previous studies (e.g., Horváth et al.,
2018) demonstrated that action duration significantly correlated with applied force (i.e. reaching larger peak forces takes longer), one may also argue that the increase in force may reflect participants’ tendency to exert force till they hear a sound, or that they match the peak force latency to the onsets of the elicited tones. Because tactile re-afference is maximal when peak force is reached, and thus the perceptual moment of the tactile interaction may be at the peak latency (Du et al.,
2017; Kunde et al.,
2004), it could be argued that in the present paradigm, participants may pinch the device so that the point of maximum tactile re-afference – that is, the force peak – would be as close to the sound onset as possible. The same line of thought could be also valid for the moment when the device is actually released (action offset) as well. Such an effect could be partially driven (1) by continuous force adjustments, which have been reported in experiments (Varga et al.,
2022) with randomly (50%) occurring sound action-effects. In these experiments force applications were shorter for actions with a sound effect than for actions without. The systematic presentation order of the delays in the present experiments may compel participants to pursue such a strategy by gradually increasing force. A somewhat different speculation on the mechanism driving the increase in force is related to multimodal integration. (2) Based on sensorimotor synchronization studies (Huntley et al.,
2024) one may speculate that participants may extend their actions, and thus delay the perceptual moment of the tactile interaction in order to reduce the temporal separation of the tactile and the sound events, thereby furthering the formation of integrated tactile-sound events. Since these events are highly contingent, representing these as unitary events would provide a more accurately structured event representation. If participants indeed extended their actions until the appearance of the sound, the pinch duration and peak force difference between adjacent conditions would be equal to 70 ms, reflecting the experimental manipulation of the action-effect delay. Our exploratory analyses of pinch durations and peak force latencies showed however, that the between-condition difference in pinch duration or peak force latency were always smaller than the corresponding difference in delay (70 ms, see Fig.
7). That is, the observed force application pattern cannot be fully explained by participants matching peak force or offset latency to the onset of the sound.
The expected relationships between the questionnaire data and the experimental variables were not confirmed – no significant correlations with the respective scales of the O-LIFE questionnaire were found. Indeed, the Bayes Factors, which were lower than 0.33 (sometimes described as “moderate” evidence, see e.g. Lee & Wagenmakers,
2013, but see Rouder et al.,
2009), provide some support for the lack of association between the level of schizotypy and the size of the time window of action optimization. This result is compatible with the notion that in contrast with action-effect contingency, the contiguity factor manipulated in the present study may not play a substantial role in schizotypy, and might also suggest that action optimization and sense of agency rely on – at least partly – different mechanisms. However, the lack of effect may also be due to the low signal-to-noise ratio provided by the procedure, as well as the uncertainty of the measuring device’s reliability reflected by poor Cronbach’s alpha values for Introvertive Anhedonia and Impulsive Nonconformity (between 0.5 and 0.6). A further reason might be that while most of the studies clearly distinguished a high and a low schizotypy group (e.g., Asai et al.,
2008; Asai & Tanno,
2007; Oestreich et al.,
2015,
2016; Pan et al.,
2021), we utilized the O-LIFE scales’ information fully as continuous variables. Since studies demonstrating significant correlational relationship between schizotypy and cognitive domains utilize sample sizes of several hundred participants (e.g., Arzy et al.,
2011; Chen et al.,
1997; Lányi et al.,
2024), the present study may be underpowered.
The lack of significant correlation (reported in detail in the Supplementary Material) between the size of the window for action optimization and the Big Five traits fits to the results of Galang et al. (
2021) who did not find any association between any of the Big Five factors and sense of agency in a Libet clock task in a university sample similar size as in the present study (
N = 80). In contrast, Schwarz et al. (
2022) demonstrated that higher level of Neuroticism and lower level of Openness were associated with lower level of sense of agency in a much larger sample (
N = 491). The exploratory analysis suggested that the size of the force optimization window and reliance on the sound action-effect were mostly associated with the Facet5 Affection (Agreeableness) and Control (Conscientiousness) factors, respectively. These associations suggest that force application patterns in the present paradigm could be well influenced by the social and task-related characteristics of the paradigm (see also Kiss et al., submitted). These may reflect the individual’s response to social demand characteristics (Orne,
1962,
2009) represented by the experimenter (Support, Trust and Altruism - Affection subscales) and performance-related demand characteristics implied by the task itself (Responsibility and Discipline – Control subscales). That is, participants may start applying more force at a shorter delay (i.e. at lower levels of uncertainty during the progression of the experiment), or apply more force despite the presence of reliable feedback provided by the immediately elicited sound.
To sum up, our results show that action-effect related motor adaptation occurs for simple pinching force exertions only within a short time frame with an upper estimate of about 290 ms – with consistent action-effect delays within this window, participants exert less force when their voluntary actions result in a sound. This confirms and extends previous studies (Cao et al.,
2020; Neszmélyi & Horváth,
2018). However, our hypothesis about a potential relationship between this temporal limit and the schizotypy trait was not confirmed.