Examining the Psychometric Properties of the Compassionate Engagement and Action Scales (CEAS) in the UK General Population
Simone Lindsey 1,*, Syd Hiskey 2, Chris Irons 3, Leanne Andrews 1
School of Health and Social Care, University of Essex, Wivenhoe Park, CO4 3SQ, United Kingdom
The Oaks Hospital, Colchester, Essex, CO4 5XR, United Kingdom
Balanced Minds, Edinburgh and London, United Kingdom
* Correspondence: Simone Lindsey
Academic Editor: Gerhard Litscher
Special Issue: Compassion Focused Therapy (CFT) - New Insights and Outcomes
Received: July 06, 2021 | Accepted: August 29, 2022 | Published: September 07, 2022
OBM Integrative and Complementary Medicine 2022, Volume 7, Issue 3, doi:10.21926/obm.icm.2203039
Recommended citation: Lindsey S, Hiskey S, Irons C, Andrews L. Examining the Psychometric Properties of the Compassionate Engagement and Action Scales (CEAS) in the UK General Population. OBM Integrative and Complementary Medicine 2022; 7(3): 039; doi:10.21926/obm.icm.2203039.
© 2022 by the authors. This is an open access article distributed under the conditions of the Creative Commons by Attribution License, which permits unrestricted use, distribution, and reproduction in any medium or format, provided the original work is correctly cited.
Research has shown that compassion – whether to others, from others or to oneself – can be associated with improved well-being and mental health. Most measures to date focus on one of these flows of compassion, with only one recently published scale tracking all three. The primary aim of the current study was to investigate the psychometric properties of The Compassionate Engagement and Action Scales (CEAS) in a UK sample. The secondary aims were to investigate the level of compassion, as measured by the CEAS, in this population and its relationship to psychological distress and positive affect. This study employed a quantitative methodology with a longitudinal design, using an online questionnaire to collect data. 315 participants completed the CEAS alongside a number of self-report questionnaires. Confirmatory factor analysis was used to confirm the factor structure of the CEAS. The CEAS was found to have good psychometric properties as assessed by a number of statistical analyses of validity and reliability. With the removal of four items, a three factor model of compassion towards self, others and receiving compassion was found to be the best fitting model in this sample. In addition, three separate two factor models (to assess the structure of engagement and action) for each flow of compassion were conducted and found to produce good fit. On the whole, the three CEAS subscales were found to have good internal consistency and test-retest reliability. Moderate positive correlations between each subscale confirmed convergent validity. Overall, the general population reported higher levels of giving compassion to others than receiving compassion or being self-compassionate. Self-compassion was found to be the strongest predictor of psychological distress and positive affect. The CEAS is a psychometrically robust measure of compassion which can be used in research and clinical practice. Attention is likely needed to a single item relating to distress tolerance, with further study needed to explore the extent to which this construct might differ in clinical and non-clinical populations.
Compassion; self-compassion; psychometric; measure
Over the past twenty years, there has been a rapid increase in research focusing on training our minds in and for compassion. This has included work on compassion for others, compassion from others, and self-compassion, with studies consistently finding that the purposeful cultivation of compassion has a range of beneficial outcomes [1,2,3].
With this interest has brought a focus on how compassion is measured, with a variety of scales developed to try to capture this. Amongst others, this has included the Compassionate Love Scale , The Compassion Scale , the Self-Compassion Scale  and the Relational Compassion Scale . The content of such scales are in large part dependent upon how compassion is defined, and what theories underpin and guide its conceptualisation.
Discussions regarding the competing ways to measure different components of compassion [1,8,9] therefore remain ongoing and naturally vary in accordance with the way differing groups view and define the construct.
A relatively recent addition to the measurement of compassion is the Compassionate Engagement and Action Scales (CEAS: ). These scales are anchored in the theory underpinning Compassion Focused Therapy (CFT; ), an evidence based psychological approach for both clinical  and non-clinical populations . CFT is anchored in an evolutionary model, where compassion is hypothesised as emerging from an evolved, caring motive. This approach centres on the idea that all motives have stimulus response algorithms underpinning them. From this perspective compassion is believed to have evolved from mammalian parental caretaking behaviour , as a result of an algorithm that enhances our sensitivity to the needs and potential suffering of our young (stimulus) and promotes actions to react accordingly (response). Hence, for Gilbert , compassion is defined as a sensitivity to the suffering of ourselves and others, with a commitment to try to alleviate and prevent it.
Accordingly, the CEAS features items linked to this stimulus-response algorithm. To be sensitive to suffering, a number of competencies are felt necessary to enable engagement with distress (e.g. care for well-being through a willingness to pay attention to suffering, sensitivity to distress, related emotional distress tolerance, sympathy, empathic understanding and non-judgement). To wisely respond to distress, another set of competencies are suggested to be key (e.g. through our ability to attend, be mindful and imagine ways to be helpful, reason, behave compassionately, utilise our bodies to regulate our minds and allow appropriate feelings).
Whilst compassion measures have traditionally focused on one flow of compassion (commonly, compassion to others, or self-compassion), the CEAS – linked to important process aspects of CFT – looks at compassion as a relational process which has three flows: compassion for others, compassion from others, and self-compassion. To the best of our knowledge, other than the Fears of Compassion Scale , existing measures of compassion do not explore this important feature of compassion (i.e. that it can flow in these ways), which can have differing relationships to mental health symptoms . Such processes can be important especially, for example, within psychotherapy where sometimes it may not be self-compassion that is the issue to explore but instead it might be a client’s openness to compassion from others and the concept of trust within relationships that we need to work with. Such processes therefore highlight the need for measures that consider compassion to be a dynamic process rather than a unitary or static construct.
In keeping with the aforementioned definition of compassion within CFT, i.e. as linked to an algorithm of stimulus sensitivity (engagement) and response (action), each of the CEAS subscales can also be separated into two further subscales that track these important components [10,14].
In the development of the CEAS, Gilbert et al.  found that the three flows of compassion were significantly correlated with each other, with the strongest relationship between compassion from others and self-compassion. However, whilst the three flows are inter-related [17,18] in complex ways, they can exist in different yet related social mentalities  that are felt to organise not only our minds but also our experience of the minds of others (with subsequent goal-based emotions, cognitions, and behaviors) in differing ways.
For instance, if in a care eliciting/seeking social mentality one may seek safeness, protection or reassurance from another, while also viewing them as a source of care. Linked threats and fears in this state may then be related to concerns over the withdrawal, withholding, or simple unavailability of their care. In a similar vein, if in a care-giving mentality one may provide protection, etc., view others as requiring this and fear a potential overwhelm state in relation to their current need or one’s inability to provide adequate care. Recent meta-analytic work  exploring possible fears of compassion relating to these three flows shows that fears of self compassion and fears of compassion from others demonstrate the strongest associations with shame, self-criticism and depressive symptoms. Moreover, clinical populations exhibit a stronger relationship between fears of self-compassion and mental health difficulties in comparison to non-clinical groups.
Other social mentalities include co-operation (i.e. seeing oneself of value to others, seeing others as valuing one’s contributions to others and fearing being cheated or unappreciated), competition (i.e. seeing oneself and others in terms of inferior-superior dynamics, with fears of being subordinated or lessened by others) and sexual (i.e. seeing oneself and others as both desiring and desirable and with fears of rejection by others). Research suggests each mentality may have an impact on various processes involved in human distress .
1.1 The CEAS Within Existing Research
In terms of its existing use as a research tool, the CEAS has subsequently featured in student and general population sample studies from Australia and Singapore [20,21] and in a study of those caring for sufferers of lung cancer from Taiwan . It has been translated into Japanese  and also Slovak  for use among general adult populations, while a Swedish version  was constructed specifically for adolescents. In each case it has demonstrated good to excellent psychometric qualities in terms of scale internal reliabilities.
In terms of the psychometric qualities of the measure among adult populations, the original CEAS  demonstrated an adequate factor structure and reliabilities (ranging between 0.72 and 0.90) as well as good convergent validity. The authors conducted separate factor analytic work by sample, utilising an exploratory factor analysis with their UK student sample and a confirmatory factor analysis with their Portuguese and USA samples.
Subsequent, albeit limited, work exploring the measure with additional samples has shown that the CEAS is a valid tool. For example, confirmatory factor analytic work by Murfield et al.  using a sample of English-speaking family carers of older adults found that their results were largely consistent with the two-factor solutions proposed for the three flows of compassion, with an acceptable fit and good reliabilities. Some issues were noted with the internal reliability of the engagement dimension and there was a weak factor loading for an item measuring distress tolerance. Despite this the authors concluded that the use of the CEAS among this group was promising, with further work needed to explore distress tolerance as a competency with compassion.
Cunha et al.  have recently used the CEAS to explore the three flows of compassion among a sample of Portuguese adolescents. Using confirmatory factor analyses they showed that the factor structure of their version was similar to the adult version. The CEAS-A demonstrated higher-order factor models encompassing two first/s-order factors in either dimension (i.e. engagement and action) as well as good construct validity, reliability, and temporal stability.
By way of factor analytic explorations of the measure among adult populations from different language groups, Asano et al.  found single rather than two-factor structures for each scale related to the three flows of compassion, indicating a different structure from the original scale . Halamova et al.’s  study among Slovak adults concluded that their results were hard to compare to previous research as their study utilised different factor analytic methods. They did however, note that the flows of compassion to others and compassion from others seemed to display similar structures to previous studies. The compassion to the self factor was unreliable and so could not be used for measuring this aspect of the construct.
In a similar vein, work by Kleissen  in an unpublished Masters thesis, using a sample of the Dutch public failed to confirm the factor structure of the scales, which the author noted may in part be due to the limited sample size obtained (n = 195) and that the model was underidentified and so could not be fully tested using structural equation modelling methods.
Whilst there have been some methodological suggestions put forward for why these structures differ from the original scales another explanation could be the different cultures and social contexts which may impact on how compassion is defined and expressed. Therefore, it would be useful to conduct further structural explorations of these scales within a UK community sample. The only UK samples explored to date are a sample of students  which only utilised exploratory factor analysis and a sample of English speaking carers of older adults . Neither of these samples are likely representative of the UK general population.
1.2 CEAS and Psychological Measures of Distress and Wellbeing
As the CEAS measures three flows of compassion, this has allowed researchers to look at their differing relationships to distress and wellbeing. For example, Gilbert et al.  found that both being sensitive to and being moved by one’s suffering have a complex relationship with depression, anxiety and stress as well as other attributes of compassion (e.g. empathy).
Typically, research into the CEAS demonstrates that depression and anxiety are inversely correlated with compassion from others and self compassion [13,20,21,25] but this relationship does not reach statistical significance for compassion for others. The same pattern is generally true for measures of stress , although among a Japanese sample  compassion from others failed to demonstrate a significant relationship to stress.
CEAS relationships with further measures of functioning have shown that compassion from others and self compassion are associated with primed shame memories and early memories of warmth and safeness , as well as satisfaction with life . Each of the three flows was found to be negatively related to anger among adolescents, while only compassion from others and self compassion were related to strengths and difficulties .
Positive affect relationships were explored by Steindl et al. , who note significant associations between all three flows of compassion and feeling safe, active and relaxed. The exception to this was the relationship between compassion for others and feeling relaxed which did not reach statistical significance, suggesting some independence in the internal state needed to express compassion from the self to those in need. Feeling safe was also noted as correlated to each of the three flows (but most strongly with self compassion) by Steindl et al. .
The CEAS self compassionate engagement subscale and self compassionate action subscales (rather than the three flows) have been shown to significantly correlate with a measure of body compassion and eating disorder attitudes and behaviours .
In terms of clinical change, 12-week compassion focused therapy with university students using a standardised group manual  demonstrated significant improvement in CEAS three flows of compassion over time. A similar result was also shown by Irons & Heriot-Maitland  in a study of 8-week Compassionate Mind Training groups with the UK general population.
Given the above it is clear that more work needs to be conducted to explore the originally obtained factor structure in a general population of English speaking adults, as the studies we have outlined above have tended to validate the CEAS in specific populations. In addition, we sought to further investigate other clinically relevant factors such as psychological distress, positive affect and their relationships to the three flows of compassion in order to explore the convergent validity of the scales.
The main aims of the present study were:
- To further investigate the psychometric properties of the three Compassionate Engagement and Action subscales in a UK adult population.
- To explore the level of compassion as measured by the three Compassion Engagement and Action subscales in this sample.
- To test the structure of the CEAS using Confirmatory Factor Analysis. Four models will be specified to test the three factor structure and the three two factor structure of the three flows of compassion separately.
- To report the internal reliabilities for the subscales identified within the final models.
- To report test re-test reliability.
- To explore the convergent validity of the three flows of compassion with a measure of psychological distress and a measure of positive affect.
- To report the level of compassion as measured by the three flows of compassion.
2.1 Participants and Procedure
Participants were recruited from the general UK population. To be eligible participants had to be over the age of 18 years. No other exclusion criteria were applied. Participants were invited to take part in the online study via a number of methods using convenience and snowball sampling methods. These included email invitations to participate in the study, which contained a brief overview of the study and included a statement which encouraged recipients to forward the email on to others. Information about the study and a link to the questionnaire was also published on social networking websites and forums.
Qualtrics (https://www.qualtrics.com/) was used to develop and host the survey.
A longitudinal within subjects design was employed in order to investigate the test re-test reliability of the CEAS. All of the measures were administered at an initial time point (time 1), and for a sub-sample, the CEAS was re-administered following a three week interval (time 2). A three week follow up period was considered sufficient to remove any memory effects from completing the measure previously but unlikely to be long enough for anyone to have genuinely changed their level of compassion.
A total of 315 participants took part in the study (time 1). All participants from time one were invited to complete the CEAS at time two and 42 participants did so.
From Table 1, it is clear that despite a much smaller sample at time two there were very few proportional differences in the demographics between these two samples. The largest difference is in ethnicity with 6% fewer reporting to be white.
Table 1 Descriptive statistics for participants at time 1 and time 2.
Ethical approval was granted by the University Ethics Committee. Informed consent was obtained by an electronic form prior to access to the questionnaire. Participation in the study could only commence once consent had been obtained.
The survey consisted of questions regarding demographic information as well as four validated questionnaires. The demographics collected were: gender, age, marital status, employment status, level of educational attainment, identified ethnicity, country of birth, country of residence and previous experience or knowledge of mindfulness and/or compassion practices.
2.2.1 The Compassionate Engagement and Action Scales (CEAS, )
The CEAS has three subscales which measure the three orientations of compassion: self-compassion (e.g. “I am motivated to engage and work with my distress when it arises”), the ability to be compassionate towards others (e.g. “I am motivated to engage and work with other peoples’ distress when it arises”) and the ability to receive compassion from key persons in the respondent’s life (e.g. “other people are actively motivated to engage and work with my distress when it arises”).
Each of the three subscales consists of thirteen items (including 3 reversed filler items) and are divided into section 1 and section 2. Section 1 includes six items that are formulated to reflect the six compassion attributes in the CFT model which are required in order to engage with compassion. These are: (1) sensitivity to the suffering of self and others, (2) sympathy, being emotionally attuned to feelings of distress in the self and others, (3) non-judgemental and accepting stance towards the self and others, (4) empathic understanding of our own thoughts and feeling and those of others, (5) tolerating distress as opposed to avoiding it and (6) caring for the well-being of the self and others . Section 1 also contains two reversed filler items.
The second section of each subscale includes four more items which reflect specific compassionate actions to deal with distress. The four compassionate actions are: (1) focusing attention on things that are helpful, (2) thinking and reasoning to bring about a more balanced perspective of what is likely to be helpful, (3) taking actions to relieve distress and (4) creating inner feelings of support, warmth and kindness to deal with distress in a compassionate and helpful way . Section 2 also includes another reversed filler item.
Participants are asked to rate each statement according to how frequently it occurs on a scale of 1 to 10 (1 = “Never”; 10 = “Always”). Total scores for each subscale are computed by summing the score for each item, excluding the reversed filler items. The reversed filler items are not included in the scoring of the measure, as stipulated by the authors, who state that they were only included in order to minimise response bias. Therefore, only ten items are scored from each subscale. The minimum score obtainable on each subscale is 10 and the maximum score is 100 or when broken down into Engagement or Action subscales the minimum score is 6 and 4 (respectively) up to 60 and 40 (respectively).
In the original development study , the scales were found to have good internal consistency (Compassion for others – Engagement: Cronbach’s α = 0.90; Compassion for others – Action: α = 0.94; Compassion from others – Engagement: α = 0.89; compassion from others – Action: α = 0.91; Compassion for self – Engagement: α = i) emotional sensitivity to suffering 0.77 and ii) engagement to suffering 0.72 and Compassion for self – Action: α = 0.90).
2.2.2 Self-Compassion Scale (SCS; )
The SCS is a 26-item scale which assesses six self-evaluative factors. There are three positive factors: self-kindness, common humanity, mindfulness, and three negative factors: self-judgement, isolation and over-identification. Participants indicate how often they engage in these ways of self-relating on a five-point Likert scale ranging from 5 “almost always” to 1 “almost never”. In the original scale publication, the scale was found to have good internal consistency (Cronbach’s α = 0.92), as did the six subscales (with Cronbach’s α ranging from 0.75-0.81). Test re-test reliability was also found to be good after a three week interval for the total score (Cronbach’s α = 0.93) and the six subscale scores (with Cronbach’s α ranging from 0.80-0.88).
2.2.3 Types of Positive Affect Scale (TPAS, )
The Types of Positive Affect Scale has 18 items and was developed to measure the degree to which people experience three different positive emotions (activating, relaxed and safe/content). Respondents are presented with a list of 18 ‘feeling’ words and asked to rate each one on a 5 point Likert scale to indicate how characteristic it is of them (0 = “not characteristic of me” to 4 = “very characteristic of me”). The scale is made up of three subscales, these are: activating positive affect (eight items, e.g. “excited”, “dynamic”, “active”); relaxed positive affect (six items, e.g. “relaxed”, “calm”, “peaceful”) and safeness/contentment positive affect (four items e.g., “safe”, “secure”, “warm). The scale was reported to have good psychometric properties with Cronbach’s alpha scores of 0.83 for activating positive affect and relaxed positive affect, and 0.73 for safeness/contentment positive affect .
2.2.4 Depression, Anxiety and Stress Scale (DASS-21; )
The DASS-21 is a shortened version of the DASS-42 and consists of three subscales, each with seven items, measuring depression, (e.g. “I couldn’t seem to experience any positive feeling at all”), anxiety (e.g. “I was aware of dryness of my mouth”) and stress (e.g. “I tended to over-react to situations”). Participants are asked to rate how much each statement applied to them over the past week, on a 4-point Likert scale ranging from 0-3 (0 = did not apply to me at all; 3 = applied to me very much, or most of the time). The three subscale scores can be summed to create a total score which is an indicator of psychological distress which is how we will utilise the measure for the purposes of this study.
2.3 Statistical Analysis
Data was analysed using version 21 of the IBM Statistical Package for Social and SPSS analysis of moment structures (Amos) 21.0 version
Four confirmatory factor models were specified and estimated using AMOS 21.0 version. The models specified were:
- A three-factor model with factor 1, items relating to self-compassion; factor 2, items relating to receiving compassion and factor 3, items relating to giving compassion to others.
- Three two factor models. Each of the three models reflect a different flow of compassion (self, receive, give). Factor 1 contained items relating to engagement and factor 2 contained items relating to action.
2.3.1 A Maximum Likelihood Method Was Used to Estimate the Parameters
The following indices were selected to assess goodness of fit based on guidelines of Byrne  and Kline . The normed Chi-Square (χ2/df, ), with a value between 2 and 5 indicating good fit , was selected as opposed to the Chi-Square statistic, as the chi-square statistic has been commonly found to reject the model in larger sample sizes .
The approximate fit indices examined were; the Incremental Fit Index (IFI, ), and the Comparative Fit Index (CFI, ), with values greater than 0.90 considered to reflect acceptable model fit. In addition, the Root Mean Square Error of Approximation (RMSEA ) with 90% confidence intervals (90% CI) were reported. A value less than 0.06 is indicative of close fit  and values up to 0.08 indicate reasonable errors of approximation in the population , although there is no definitive agreement on these values. For example, more recently Steiger  has argued that a value closer to 0.07 should be considered the upper limit.
A hierarchical multiple regression was used to assess the convergent validity of self-compassion on psychological distress and positive affect. Due to the potential bias in the model towards the negative SCS subscales it was decided that the regression would be conducted with the removal of the three SCS negative subscales. This enabled an investigation as to how the self-compassion subscales of the CEAS and the SCS positive subscales, which more closely represent self-compassion attributes, performed in the absence of the negative subscales. It has been argued that the negative SCE subscales do not assess the protective function of compassion against mental health distress .
Internal reliability was assessed using Cronbach’s alpha with values between 0.7 and 0.95 indicating good fit . Test re-test reliability was assessed with a bivariate correlation between the initial time point and a repeat administration three weeks later. Reliability scores above 0.70 are considered acceptable .
Correlations and multiple linear regressions were conducted to explore convergent validity. Correlations were interpreted using Cohen’s  indicators of effect size as follows: r < 0.3 small effect, 0.3-0.5 medium effect and >0.5 large effect.
2.4 Data Screening
All data was collected online using Qualtrics (an online survey software programme) and was exported into SPSS version 21. A random sample of data (5%) was checked to ensure accuracy of the data export into SPSS.Data from all outcome variables was plotted using box plots and histograms and any outliers were checked with the original data set for any mistakes in data entry. All outliers were found to have been correctly imported. Of the final 315 sample who had consented for their data to be used, two of the participants had four missing items and therefore further statistical analysis was performed using Listwise exclusion.
All variables were examined by visually inspecting the P-P plots and histograms. There was some deviation of data values from the diagonal line on the P-P plots and the histograms appeared to be slightly negatively skewed for the CEAS self-compassion and compassion towards others subscales. The receiving compassion subscale appeared to be normally distributed. The data at time 2 for the CEAS subscales also appeared to be normally distributed.
When reviewing the absolute values of skewness and kurtosis for the CEAS, none of the subscales were found to violate normal distribution according to the criteria set out by West et al . All other variables (SCS, PAS, DASS-21 & SCBCS) were also assessed and none were found to violate the normal distribution.
In sum, all variables met assumptions of normal distribution and therefore parametric tests were used for all statistical analyses.
3.1 Assessing the Psychometric Properties of the CEAS
3.1.1 Confirmatory Factor Analysis
The three factor CFA with all items included produced a poor model fit. Removal of low loading items (SC item 4 ‘I am emotionally moved by my distressed feelings or situations’, SC item 5 ‘I tolerate the various feelings that are part of my distress’, CtO item 5 ‘I tolerate the various feelings that are part of other people’s distress’ and CfO item 5 ‘Others tolerate my various feelings that are part of my distress’) and use of modification indices to improve fit (i.e. allowing for correlated error terms between some items) resulted in an acceptable model fit.
The normed chi square value was 2.65. The IFI and CFI both had values above 0.90 (IFI = 0.91, CFI = 0.91). The RMSEA was 0.07 (90% confidence interval = 0.07-0.08).
As seen in Figure 1 Standardised coefficients for the Self compassion factor range from 0.440 (item SC_E2) up to 0.895 (item SC_A2). Standardised coefficients for the giving compassion to others factor range from 0.470 (item CtO_E8) up to 0.841 (item CtO_E1). Standardised coefficients for the receiving compassion from others factor range from 0.691 (item CfO_E4) up to 0.902 (item CfO_A5).
Figure 1 Three factor model of the CEAS with standardised coefficients. Note: Items SC_E4, SC_E5, CtO_E5 and CfO_E5 removed to improve the model fit.
Gilbert et al.  propose that in addition to the CEAS being used as three single factor subscales, each ‘flow’ of the CEAS can be further divided into two subscales, representing the two psychologies of the CFT definition of compassion – engagement and action. In order to assess this, three separate factor analyses were conducted, one for each of the three orientations, with action and engagement identified as two latent factors, which items were specified to load on to. Items 1, 2, 4, 5, 6, 8 from section 1 of the orientation subscale were specified to load onto an Engagement factor and items 1, 2, 4 and 5 from section 2 of the subscale were specified to load onto an Action factor.
All models were found to show good fit but for Self-compassion two items with low loading items were removed (SC item 4 ‘I am emotionally moved by my distressed feelings or situations’, SC item 5 ‘I tolerate the various feelings that are part of my distress’). For Compassion to Others one item (CtO item 5 ‘I tolerate the various feelings that are part of other people’s distress’) required removal.
Figure 2, Figure 3 and Figure 4 present the standardised coefficients for the two factor models, one for each flow of compassion. Standardised coefficients for the self compassion Engagement factor range from 0.513 (item SC_E2) up to 0.803 (item SC_E1) while standardised coefficients for the Action factor range from 0.674 (item SC_A5) up to 0.906 (item SC_A2).
Figure 2 Two factor model of Self-Compassion with standardised coefficients. Note: Items SC_E4 and SC_E5 removed to improve the model fit.
Standardised coefficients for the compassion to others Engagement factor range from 0.473 (item CtO_E8) up to 0.869 (item CtO_E1), while standardised coefficients for the compassion to others Action factor range from 0.780 (item CtO_A5) up to 0.880 (item CtO_A1).
Figure 3 Two factor model of Compassion to Others with standardised coefficients. Note: Item CtO_E5 removed to improve the model fit.
Standardised coefficients for the receiving compassion from others Engagement factor range from 0.541 (item CfO_E5) up to 0.813 (item CfO_E1), while standardised coefficients for the receiving compassion from others Action factor range from 0.852 (item CfO_A2) up to 0.910 (item CfO_A5).
Figure 4 Two factor model of Compassion from Others with standardised coefficients.
Table 2 illustrates the model fit indices for each of the two factor models.
Table 2 Model fit indices for all two factor models.
3.1.2 Internal Reliability
Cronbach’s alpha for the three factors within the main model were: Self-compassion = 0.88; Compassion to others = 0.90; and Compassion from others = 0.94.
Cronbach’s alphas for the three two factor models were: for the Self-compassion model the engagement subscale = 0.73 and the action subscale = 0.88; for the Compassion to others model the engagement subscale = 0.80 and the action subscale = 0.90; for the Compassion from others model the engagement subscale = 0.88 and the action subscale = 0.93, indicating good internal reliability for all subscales.
3.1.3 Test Re-Test Reliability
A Pearson’s product-moment bivariate correlation was conducted using the data at time 1 and time 2 for each flow of compassion. All relationships between the first and second administration were found to be significant at p < 0.01, (see Table 3) indicating good test re-test reliability.
Table 3 Test re-test Reliability Scores for the CEAS Subscales.
3.1.4 Convergent Validity
Correlations Between Subscales. A Pearson’s product-moment correlation analysis was conducted on the relationships between the three subscales (see Table 4).
Table 4 Inter-correlations between CEAS Subscales.
All correlations between the three CEAS subscales were statistically significant and positive. The results suggests that the three orientations of compassion are related, however the moderate to small correlations indicate differentiation, which supports the idea that one can be (relatively) high in one orientation of compassion (e.g. for others) but low in another (e.g. from others) and vice-versa.
3.2 The Relationship Between Symptoms of Psychological Distress, Positive Affect and the Three Orientations of Compassion
To determine convergent validity, we explored how the three subscales of the CEAS were related to positive affect (as measured by the TPAS), and psychological distress (as measured by the DASS-21). A Pearson’s product-moment correlation analysis was conducted, with Table 5 showing the correlation coefficients for the CEAS, the TPAS and the DASS-21.
Table 5 Correlations between the CEAS subscales, TPAS subscales and DASS-21 total score.
The results from the bivariate correlations indicated that all three orientations of compassion were significantly related to positive affect, barring compassion to others failing to significantly relate to relaxed positive affect. CEAS self-compassion and receiving compassion from others subscales were significantly, negatively, related to psychological distress, while compassion to others was not.
3.2.1 Multiple Regression with the CEAS Self-Compassion Subscales and SCS Positive Subscales Predicting Psychological Distress
In all regression analyses, step 1 included SCS positive subscales (self-kindness, mindfulness and common humanity) and step 2 included the CEAS two elements of self-compassion (engagement and action). When predicting psychological distress, as measured by the DASS-21 total score, Model 1 (step 1) explained 12.1% of the variance and was significantly better at predicting the DASS-21 than the mean, F (3,311) = 14.20, p < 0.001. Model 2 (step 2) explained an additional 5.1% of the variance (total 17.2% variance), and this represented a significant change (R2 change = 0.05, F change (2,309) = 9.59, p < 0.001). This suggests that the model is a significant fit of the data overall. In the final model, the strongest predictor was CEAS self-compassion action β = -0.268, p < 0.001. None of the other subscales were statistically significant.
3.2.2 Multiple Regression with the CEAS Self-Compassion Subscales and SCS Positive Subscales Predicting Positive Affect
Hierarchical multiple regression was used to assess the predictive value of self-compassion on positive affect. The three subscales of the Positive Affect Scale were used in turn as dependent variables due to the lack of evidence suggesting these should be combined into a total score.
When predicting the TPAS Active subscale, Model 1 (step 1) explained 8.8% of the variance and was significantly better at predicting active positive affect than the mean, F (3,311) = 10.04, p < 0.001. Model 2 (step 2) explained an additional 8.9% of the variance (total 17.8% of the variance), and this represented a significant change, R2 change = 0.09, F change (2, 309) = 16.77, p < 0.001. This suggests that the model is a significant fit of the data overall. In the final model, the strongest predictor was the CEAS Self-compasison ‘action’ (β = 0.351 p < 0.001), all other subscales did not reach significance. The results from this regression analysis indicate the CEAS self-compassion action subscale is the strongest predictor of active positive affect.
When predicting the TPAS Relaxed subscale Model 1 (step 1) explained 22.6% of the variance and was significantly better at predicting relaxed positive affect than the mean, F (3,311) = 30.33, p < 0.001. Model 2 (step 2) explained an additional 2.9% of the variance (total 25.5% of the variance), and this represented a significant change, R2 change = 0.03, F change (2, 309) = 5.99, p < 0.003. This suggests that the model is a significant fit of the data overall. In the final model, the strongest predictor was the CEAS Self-compassion action (β = 0.22, p = 0.001), all other subscales were significantly related to relaxed positive affect (Self-compassion engagement: β = -0.17, p < 0.01; Common humanity: β = 0.18, p < 0.01; Self-kindness: β = 0.15, p < 0.05; Mindfulness: β = 0.16, p < 0.05). The results from this regression analysis indicate the CEAS self-compassion action subscale is the strongest predictor of relaxed positive affect.
When predicting the TPAS Safe subscale Model 1 (step 1) explained 26.4% of the variance and was significantly better at predicting safe positive affect than the mean, F (3,311) = 37.13, p < 0.001. Model 2 (step 2) explained an additional 6.2% of the variance (total 32.6% of the variance), and this represented a significant change, R2 change = 0.06, F change (2, 309) = 14.17, p < 0.001. This suggests that the model is a significant fit of the data overall. In the final model, the strongest predictor was the CEAS Self-compassion action (β = 0.27, p < 0.001), both common humanity (β = 0.22, p = 0.001) and mindfulness (β = 0.16, p < 0.05) were significantly related to safe positive affect. The results from this regression analysis indicate the CEAS self-compassion action subscale is the strongest predictor of safe positive affect.
3.3 Levels of the Three Flows of Compassion in an Adult Community Sample
From the descriptive statistics, presented in Table 6, it can be seen that on average, participants scored highest on the compassion towards others subscale and lowest on the receiving compassion from others subscale.
Table 6 Means and Standard Deviations (SD) of CEAS Subscales.
A paired-samples t-test was conducted in order to assess the reliability of the difference between participants’ scores on giving and receiving compassion. The difference was found to be significant, t (314) = 18.66, p = < 0.001, with a large effect size, r = 0.73, illustrating this population finds it significantly easier to give compassion to others than to receive compassion from them.
The CEAS was developed to provide a psychometrically sound measurement of the three flows of compassion – to others, from others and to the self – as outlined in the CFT model . This study supports the use of the CEAS in this way, proposing its use as a reliable and valid measure of interpersonal and intrapersonal compassion.
In sum, the CEAS was found to have acceptable psychometric properties as assessed by a number of statistical analyses of validity and reliability. On reviewing the CFA fit indices and the parameter estimates together, it can be concluded that the factor structure of the CEAS can be adequately interpreted as a three factor model. However, in our data this was established following the removal of 4 items (SC item 4 ‘I am emotionally moved by my distressed feelings or situations’, SC item 5 ‘I tolerate the various feelings that are part of my distress’, CtO item 5 ‘I tolerate the various feelings that are part of other people’s distress’ and CfO item 5 ‘Others tolerate my various feelings that are part of my distress’).
Why the experience of being emotionally moved by one’s distress or distressing situations should require removal here is unclear. This could feasibly, for example, relate to reduced experiences of distress in our non-clinical sample (i.e. they could have had fewer distressing life events). The frequency and intensity of adverse life events was not assessed as part of our study but lower levels could potentially lead to an understandably reduced secondary response (i.e. being less emotionally moved) by any related upset. A meta response to one’s emotions might then only be elicited, needed or felt in some but not all circumstances. Within clinical treatment as part of CFT therapists would typically seek to explore not only client current expressions of distress but also client emotional response to their distress, such as being emotionally moved by it . Indeed, exploring meta emotions can be a significant element in treatment more generally. Leahy's  emotional schema model, for example, proposes that individuals differ in their interpretations, evaluations, and strategies about their emotional experience. This work identifies fourteen dimensions related to problematic strategies of emotion regulation (on behalf of both clients and therapists), pointing to the complexity of this construct.
Why across the three flows an item around distress tolerance might load insufficiently is puzzling, although issues with this element of the measure have been noted by Murfield et al  in their sample of English speaking family carers. As these authors note, tolerating various feelings as part of one’s distress was a weak loading item across all CEAS scales. Gu et al.  also reported similar issues with distress tolerance (i.e. weak factor loading despite good model fit indices). Their subsequent development of the Sussex-Oxford Compassion Scales  found no issues for newly generated items related to their ‘tolerating’ factor, which suggests the CEAS distress tolerance item issue we have identified along with Murfield et al.  may be at the item rather than the factor/construct level.
In the CEAS distress tolerance is captured by a single item and as with Murfield et al  we would suggest further exploratory work into this issue. In particular, the relationship between the three flows of compassion and tolerating upsetting feeling states is likely a complex one and given its relationship to suffering important to consider in future dedicated qualitative and quantitative research.
Moreover, future work may need to explore the conditions under which distress tolerance is felt a necessary component of compassion in non-clinical samples. Attention to current levels of distress and functioning will be important to consider in terms of the magnitude of differences between clinical and non-clinical groups and whether these shed further light into individual experiences in terms of responding to suffering.
As well as the three factors identified within the CEAS, each flow subscale can be divided into two further factors representing action and engagement subscales. Again, the removal of an item linked to distress tolerance was necessary for both self-compassion and compassion to others in this context. The significance of this would be important to explore as part of the work suggested above.
On the whole, all three CEAS subscales were found to have good internal consistency and test-re test reliability. Moreover, as hypothesised, the CEAS self-compassion and receiving compassion from others scales were found to be negatively related to psychological distress, suggesting that people with higher reported levels of either element of compassion experienced less overall depression, stress and anxiety. This finding supports the growing body of research which has found that one’s ability to give compassion to the self and allow the receipt of it from others is generally negatively related to psychopathology . Similar findings have been noted for components of positive affect, with the ability to give compassion to others also largely relating to positive/safe internal experiences too.
In terms of predicting positive affect, the CEAS self-compassion scale outperformed the SCS in each of the three analyses conducted, suggesting it may be the more informative measure when exploring active, relaxed and safe elements of positive emotions. Moderate positive correlations between each subscale confirmed the measures convergent validity.
Our findings indicate that, on average, the general UK population experienced a moderate level of compassion across all three flows. Participants scored highest on giving compassion to others, followed by self-compassion and lowest on receiving compassion from others. The difference between scores related to giving compassion to others and receiving it from others were statistically significant, while for self-compassion and giving compassion to others it was not. This could be an important distinction between clinical and non-clinical groups, as Kirby et al.  have shown that overall, clinical populations demonstrate significantly stronger associations between mental health difficulties and fears of self-compassion, relative to non-clinical populations. Such fears of receiving compassion, for example, may be understood in connection to research which has shown that there are a number of barriers to both inter and intrapersonal compassion, such as feeling fearful of its presence . For example, for some, receiving compassion from others may arouse feelings of grief and loneliness  and so is understandably avoided. Either way the three flows appear to be partly separate processing systems.
Indeed, in considering the relative levels of compassion endorsed across the three flows in our non-clinical sample, we note that the scores we have obtained are broadly similar to those reported by Murfield et al’s  international sample of family carers. The relative pattern of CEAS scores is less clear in Gilbert et al’s  original non-clinical student samples, indicating that both gender and country of sample are moderating variables.
By way of clinical groups, Fox et al’s  feasibility trial of a standardised group manual for CFT in a USA counselling and psychology services reported lower levels of self compassion compared with higher levels of ability to receive compassion. Improvements in both self compassion and compassion to others, however, were noted over time.
4.1 Strengths and Limitations
This study highlights that the CEAS performs well in comparison to other established measures of compassion. In particular, it provides a helpful way for researchers and clinicians to explore the relative impact of how the different flows of compassion may be associated with distress and wellbeing. Should disparities within the three flows be identified, then a dedicated measure to explore related fears, blocks and resistances could be employed and focused upon within both the non-clinical and clinical populations .
As with any recruitment method that utilises snowball sampling, participation in our study was not based on random selection. Our sample was made up of predominantly higher-educated, employed females, which is however consistent with previous online research [54,55,56]. Therefore, the results of this study may not be fully representative of the general UK population. In addition, it will be helpful for future research to examine levels of distress within non-clinical groups, which might indicate subgroups with a clinical population profile.
Online surveys are one of the preferred methods of conducting research due to their many advantages, such as efficiency, low-cost, flexibility and global reach . Many factors can influence related response rates and our survey was designed to be as short, easy to complete and convenient as possible. For example, it could be completed on a mobile phone, tablet or other computer device. Lengthy surveys have been found to reduce response rates  and therefore the number of items included in the survey was limited. This meant that other variables, such as self-reassurance, adverse life event history et. were not assessed, which could have provided further knowledge about factors related to compassion.
4.2 Clinical Implications
Our findings support the continued clinical utility of cultivating compassion and suggest that higher levels of self-compassion and receiving compassion are typically related to lower levels of psychological distress and higher levels of positive affect. The CEAS self-compassion subscale was the strongest predictor of psychological distress and positive affect, suggesting that helping the general population to cultivate self-compassion may be important in protecting against mental health difficulties and enhancing overall well-being. In turn, this could reduce the number of those in the general population who go on to seek/need clinical services.
The current research supports the use of the CEAS as a robust measure of compassion and one which can make a unique contribution to the field, due to its ability to capture the three flows.
The findings from this study may be useful in guiding clinicians and researchers to formulate and understand difficulties people may have in the different flows of compassion, inform public health interventions which may be beneficial in addressing these and consider the potential utility of employing the flows of compassion as overall change measures (both in and outside of psychological therapy).
We would like to thank the participants of this study for their time and effort without which this study would not have been possible.
Dr S Lindsey Project development, data collection, data analysis, report writing; Dr S Hiskey Project development, data collection, report writing; Dr C Irons Project development, report writing; Prof L Andrews Project development, data collection, data analysis, report writing.
The authors have declared that no competing interests exist.
- Gilbert P. Compassion: Definitions and controversies. In: Compassion. London: Routledge; 2017. pp. 3-15. [CrossRef]
- Kirby JN, Tellegen CL, Steindl SR. A meta-analysis of compassion-based interventions: Current state of knowledge and future directions. Behav Ther. 2017; 48: 778-792. [CrossRef]
- Carter CS, Bartal IBA, Porges EC. The roots of compassion: An evolutionary and neurobiological perspective. In: The Oxford handbook of compassion science. New York: Oxford University Press; 2017. pp. 173-188. [CrossRef]
- Sprecher S, Fehr B. Compassionate love for close others and humanity. J Soc Pers Relat. 2005; 22: 629-651. [CrossRef]
- Martins D, Nicholas NA, Shaheen M, Jones L, Norris K. The development and evaluation of a compassion scale. J Health Care Poor Underserved. 2013; 24: 1235-1246. [CrossRef]
- Neff KD. The development and validation of a scale to measure self-compassion. Self Identity. 2003; 2: 223-250. [CrossRef]
- Hacker T. The relational compassion scale: Development and validation of a new self rated scale for the assessment of self-other compassion. Glasgow: University of Glasgow; 2008.
- Mascaro JS, Florian MP, Ash MJ, Palmer PK, Frazier T, Condon P, et al. Ways of knowing compassion: How do we come to know, understand, and measure compassion when we see it? Front Psychol. 2020; 11: 547241. [CrossRef]
- Strauss C, Taylor BL, Gu J, Kuyken W, Baer R, Jones F, et al. What is compassion and how can we measure it? A review of definitions and measures. Clin Psychol Rev. 2016; 47: 15-27. [CrossRef]
- Gilbert P, Catarino F, Duarte C, Matos M, Kolts R, Stubbs J, et al. The development of compassionate engagement and action scales for self and others. J Compassionate Health Care. 2017; 4: 4. [CrossRef]
- Gilbert P. Compassion: Conceptualisations, research and use in psychotherapy. Hove: Routledge; 2005.
- Craig C, Hiskey S, Spector A. Compassion focused therapy: A systematic review of its effectiveness and acceptability in clinical populations. Expert Rev Neurother. 2020; 20: 385-400. [CrossRef]
- Irons C, Heriot-Maitland C. Compassionate mind training: An 8-week group for the general public. Psychol Psychother. 2021; 94: 443-463. [CrossRef]
- Gilbert P. The origins and nature of compassion focused therapy. Br J Clin Psychol. 2014; 53: 6-41. [CrossRef]
- Gilbert P, McEwan K, Matos M, Rivis A. Fears of compassion: Development of three self-report measures. Psychol Psychother. 2011; 84: 239-255 [CrossRef]
- Kirby JN, Day J, Sagar V. The 'flow' of compassion: A meta-analysis of the fears of compassion scales and psychological functioning. Clin Psychol Rev. 2019; 70: 26-39. [CrossRef]
- Hermanto N, Zuroff DC, Kopala-Sibley DC, Kelly AC, Matos M, Gilbert P, et al. Ability to receive compassion from others buffers the depressogenic effect of self-criticism: A cross-cultural multi-study analysis. Pers Individ Differ. 2016; 98: 324-332. [CrossRef]
- Hermanto N, Zuroff DC. The social mentality theory of self-compassion and self-reassurance: The interactive effect of care-seeking and caregiving. J Soc Psychol. 2016; 156: 523-535. [CrossRef]
- Gilbert P. Psychotherapy for the 21st century: An integrative, evolutionary, contextual, biopsychosocial approach. Psychol Psychother. 2019; 92: 164-189. [CrossRef]
- Steindl SR, Matos M, Creed AK. Early shame and safeness memories, and later depressive symptoms and safe affect: The mediating role of self-compassion. Curr Psychol. 2021; 40: 761-771. [CrossRef]
- Steindl SR, Yiu RXQ, Baumann T, Matos M. Comparing compassion across cultures: Similarities and differences among Australians and Singaporeans. Aust Psychol. 2020; 55: 208-219. [CrossRef]
- Hsieh CC, Yu CJ, Chen HJ, Chen YW, Chang NT, Hsiao FH. Dispositional mindfulness, self-compassion, and compassion from others as moderators between stress and depression in caregivers of patients with lung cancer. Psychooncology. 2019; 28: 1498-1505. [CrossRef]
- Asano K, Kotera Y, Tsuchiya M, Ishimura I, Lin S, Matsumoto Y, et al. The development of the Japanese version of the compassionate engagement and action scales. PLoS One. 2020; 15: e0230875. [CrossRef]
- Halamová J, Kanovský M, Pačutová A, Kupeli N. Randomised controlled trial of an online version of compassion mind training in a nonclinical sample. Eur J Psychol. 2020; 16: 262-279. [CrossRef]
- Henje E, Rindestig FC, Gilbert P, Dennhag I. Psychometric validity of the compassionate engagement and action scale for adolescents: A Swedish version. Scand J Child Adolesc Psychiatr Psychol. 2020; 8: 70-80. [CrossRef]
- Murfield J, Moyle W, O'Donovan A, Ware RS. Validity of the compassionate engagement and action scales with family carers of older adults: Confirmatory factor analyses. Int Psychogeriatr. 2021; 33: 373-383. [CrossRef]
- Cunha M, Galhardo A, Gilbert P, Rodrigues C, Matos M. The flows of compassion in adolescents as measured by the compassionate engagement and action scales. Curr Psychol. 2021. doi: 10.1007/s12144-021-02097-5. [CrossRef]
- Kleissen Y. Further validation of The Compassionate Engagement and Action Scales (TCEAS). Enschede: University of Twente; 2016.
- de Carvalho Barreto M, Ferreira C, Marta-Simões J, Mendes AL. Exploring the paths between self-compassionate attributes and actions, body compassion and disordered eating. Eat Weight Disord. 2020; 25: 291-297. [CrossRef]
- Fox J, Cattani K, Burlingame GM. Compassion focused therapy in a university counseling and psychological services center: A feasibility trial of a new standardized group manual. Psychother Res. 2021; 31: 419-431. [CrossRef]
- Gilbert P. The compassionate mind: A new approach to life challenges. London: Constable and Robinson Ltd. 2009.
- Gilbert P, McEwan K, Mitra R, Franks L, Richter A, Rockliff H. Feeling safe and content: A specific affect regulation system? Relationship to depression, anxiety, stress, and self-criticism. J Posit Psychol. 2008; 3: 182-191. [CrossRef]
- Lovibond PF, Lovibond SH. The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995; 33: 335-343. [CrossRef]
- Byrne BM. Structural equation modeling with AMOS: Basic concepts, applications, and programming. 2nd ed. New York: Taylor & Francis Group; 2010.
- Kline RB. Principles and practice of structural equation modeling. 3rd ed. New York: Guilford; 2011.
- Wheaton B, Muthen B, Alwin DF, Summers GF. Assessing reliability and stability in panel models. Sociol Methodol. 1977; 8: 84-136. [CrossRef]
- Tabachnick BG, Fidell LS, Ullman JB. Using multivariate statistics. 5th ed. Boston: Person; 2007.
- Bollen KA, Long JS. Introduction. In: Testing structural equation models. Newbury Park: Sage; 1993.
- Bollen KA. A new incremental fit index for general structural equation models. Sociol Methods Res. 1989; 17: 303-316. [CrossRef]
- Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990; 107: 238-246. [CrossRef]
- Steiger JH. Understanding the limitations of global fit assessment in structural equation modeling. Pers Individ Differ. 2007; 42: 893-898. [CrossRef]
- Hu Lt, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling. 1999; 6: 1-55. [CrossRef]
- MacCallum RC, Browne MW, Sugawara HM. Power analysis and determination of sample size for covariance structure modeling. Psychol Methods. 1996; 1: 130-149. [CrossRef]
- Muris P, Petrocchi N. Protection or vulnerability? A meta-analysis of the relations between the positive and negative components of self-compassion and psychopathology. Clin Psychol Psychother. 2017; 24: 373-383. [CrossRef]
- Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007; 60: 34-42. [CrossRef]
- Barker C, Pistrang N, Elliott R. Research methods in clinical psychology: An introduction for students and practitioners. 2nd ed. West Sussex: John Wiley & Sons; 2002. [CrossRef]
- Cohen J. A power primer. Quant Methods Psychol. 1992: 112, 155-159. [CrossRef]
- West SG, Finch JF, Curran PJ. Structural equation models with nonnormal variables: Problems and remedies. In: Structural equation modeling: Concepts, issues, and applications. Thousand Oaks: Sage; 1995. pp. 56-75.
- Gilbert P. An introduction to compassion focused therapy in cognitive behavior therapy. Int J Cognit Ther. 2010; 3: 97-112. [CrossRef]
- Irons C. The compassionate mind approach to difficult emotions: Using compassion focused therapy. London: Robinson; 2019.
- Leahy RL. Emotional schema therapy: A meta‐experiential model. Aust Psychol. 2016; 51: 82-88. [CrossRef]
- Gu J, Cavanagh K, Baer R, Strauss C. An empirical examination of the factor structure of compassion. PLoS One. 2017; 12: e0172471. [CrossRef]
- Gu J, Baer R, Cavanagh K, Kuyken W, Strauss C. Development and psychometric properties of the Sussex-Oxford Compassion Scales (SOCS). Assessment. 2020; 27: 3-20. [CrossRef]
- Curtin R, Presser S, Singer E. The effects of response rate changes on the index of consumer sentiment. Public Opin Q. 2000; 64: 413-428. [CrossRef]
- Goyder J, Warriner K, Miller S. Evaluating socio-economic status (SES) bias in survey nonresponse. J Off Stat. 2002; 18: 1-12.
- Singer E, Van Hoewyk J, Maher MP. Experiments with incentives in telephone surveys. Public Opin Q. 2000; 64: 171-188. [CrossRef]
- Evans JR, Mathur A. The value of online surveys. Int Res. 2005; 15: 195-219. [CrossRef]
- Sheehan KB. E-mail survey response rates: A review. J Comput Mediat Commun. 2001; 6: JCMC621.