Over and above the initial findings of the Study of Australian Leadership (SAL), the manner in which the study was designed has a number of broader methodological contributions to offer.
Given the size of the study and the diversity of organisations required, designing an effective methodology to examine the relationship between workplace leadership and performance is perhaps one of our greatest achievements. Some of the strengths of SAL are:
- Depth: The study includes multiple touch points across the spectrum of leadership roles within organisations. SAL takes a holistic approach to surveying leaders, everyone from CEO’s through to non-supervisory employees at the local supermarket and everyone in between. This is the most comprehensive snapshot of leadership ever undertaken in Australia and possibly anywhere in the world.
- Breadth: The study reaches across professions and industries. With the notable exception of the agriculture sector, the study has included significant numbers of respondents from every sector of the Australian economy; for-profit and not-for-profit; services and manufacturing; primary, secondary and tertiary industries. This breadth has allowed us to draw out the commonalities and interrogate the differences with an unprecedented level of clarity and confidence.
- Rigour: In contrast to the many publications which purport to address the topic of leadership, every claim, every assumption and every question that underlies this study has been rigorously justified and tested by a team of 13 researchers over 3 years and is grounded in prior academic research.
- Innovation: In the process of developing the surveys, we designed and built a robust, inclusive, innovative and broad measure of workplace performance, which we believe will be an important benchmark for future comparative research and analysis.
Measuring workplace performance
In order to find a valid and reliable performance measure for Australian workplaces, three important issues were considered:
- Inclusiveness: ‘Performance’ means different things for different workplaces. Sales and profits are relevant for commercial enterprises, but have little meaning for non-profit and government entities. SAL includes all types of workplaces. To generalise about Australian workplace performance, a measure was needed that includes the outcomes relevant in non-profit and government settings.
- Subjectivity: Some performance measures rely on rankings or assessments of change. For instance, leaders may be asked how their workplace performs when compared to competitors. Such measures depend on the quality of respondents’ information, which is difficult to assess. Workplace leaders may not know how well their competitors are performing in certain areas, such as customer satisfaction. Subjective performance measures are more reliable when information is available to respondents, and when there is a reasonable expectation that they are well-informed.
- Framing: Performance is complicated by expectations. This issue applies even when using quantifiable measures, such as sales. Imagine that Firm 1 increases its sales by 5% and Firm 2 by 3%. Firm 1 might be regarded as the superior performer. But what if Firm 1’s chief executive officer had forecast an 8% sales increase? Now, the 5% growth result seems disappointing. Performance metrics are more informative when they relate outcomes to expectations.
These considerations informed the assessment of workplace performance: the key measure used here is success in meeting targets. This is inclusive, as it encompasses multiple targets. Only those targets that apply for a particular workplace are used to assess its performance. Workplace leaders’ views about meeting targets are well-informed, as they are evaluating their own workplace. And success in meeting targets takes account of expectations. Success and failure are not assessed in absolute terms, but relative to the goals each workplace has set.
Introducing a new performance scale we like to call OTAS-9
OTAS-9 is an Overall Target Attainment Score measured through 9 items.
To measure workplace performance in SAL the Overall Target Attainment Score (OTAS-9) was developed. Workplace leaders were asked about ‘the extent to which your workplace exceeded, met, or did not meet’ nine separate targets:
- Volume of sales.
- Unit labour costs.
- Total costs.
- Return on investment.
- Customer or client satisfaction.
- Labour productivity.
- Labour turnover.
For each item, workplace leaders answered on a three-point scale:
- Did not meet target.
- Met target.
- Exceeded target.
Workplace leaders could say that a particular target was not applicable or not measured in their workplace, and could also ‘prefer not to say’ for any item. Generally, items referring to more universal performance targets (e.g. customer satisfaction) had more responses than those referring to narrower targets (e.g. profits or return on investment).
The OTAS-9 combines workplace leaders’ responses to all the questions about meeting targets. An average score for each workplace is calculated using whichever of the nine performance targets were measured and applicable.
To understand which workplace attributes are associated with high and low performance, the OTAS-9 values are divided into four groups of equal size (quartiles) and then grouped into categories. Workplaces in the bottom quartile are ‘low’ performers, the next two quartiles are ‘middle’ performers, and those in the top quartile are ‘high’ performers.
What the OTAS-9 allows us to do is make comparisons of workplace performance across sectors and industries and will provide a relative reference point for future research. As SAL includes all types of workplaces, the OTAS-9 is critical in allowing us to generalise about Australian workplace performance.
We can also use the individual OTAS-9 scores in other statistical analyses to identify patterns and relationships with other characteristics that we have measured. This also allows us to account for other effects that might be due to industry differences or other such factors. For example, we can run analyses to see if the quality of a workplace’s management practices relates to OTAS-9, above and beyond such other factors.