Measure what matters:Evaluating the impact of public administration on citizensA mixed methods approach to developing and testing the Experience Effectiveness (XE) framework in Australia and Canada
Bibliographic record
Abstract
Public administration has an essential role in enabling citizen compliance behaviour. When public administration is effective, it persuades citizens of its value and fairness. It also supports those attempting to respond appropriately and ensures they are rewarded by easy and seamless services that are transparent and unobtrusive. An effective public sector is also economically proficient and guarantees that those who attempt to evade their responsibilities are unsuccessful. In contrast, when the administration is ineffective it obfuscates the rules and becomes an obstacle that dutiful citizens struggle to comprehend and overcome. The administration becomes an enemy to be feared rather than a friend to be respected.No holistic evaluation of the public sector has been conducted that measures both the impact of its actions on citizens’ lives as well as its ongoing performance. This thesis outlines the design and application of a novel framework for measuring and evaluating the effectiveness of experiences or services created by the public sector. The framework provides two multi-dimensional evaluation tools: the Full Experience Effectiveness (XE) Measurement Scale and the XE Expert. These tools enable the evaluation of both citizen experiences and the impacts of public sector administration. The framework details a typology for describing and documenting citizen designs for evaluation. The core dimensions of XE relate to the design of the experience in terms of products, processes, service and emotions. It also includes viewpoints of the experience from multiple perspectives such as the citizen, the service delivery staff (public sector employees), administrative specialists and related service intermediaries.The approach used in this research is novel in that it draws from multiple domains to construct and measure XE including: (i) the field of new public administration which provides insight into the role of citizens in co-producing the service experience; (ii) services marketing which provides extensive understanding of customer satisfaction for service development; (iii) human factors which suggest practical methods to design systems that anticipate and prevent human error; and (iv) human-centred design which creates ownership and advocacy by engaging the people in the design of services that they will have to use.The research utilised a pragmatic bricolage of methods. The mixed methods approach follows a pluralist view and utilises the iterative, collaborative approach well-established in action research.The research program involved three phases. Phase 1 included the development of the XE Measurement Scale prototype through a comprehensive review of existing measures and evaluation methods including an expert review of the proposed scale and its subsequent refinement. Phase 2 applied the XE Measurement Scale prototype in an Australian context (starting a small business with a tax compliance focus). During this phase, I refined the prototype, added more components and enhanced the model for operationalisation. In Phase 3, I applied the refined XE Framework, which included a full evaluation scale (Full XE Measurement) and the XE Expert evaluation tool. The application of the full suite of tools evaluated six projects impacting citizens in the Canadian public sector. The environment for design and evaluation was also assessed to determine the influence of the organisation culture and context on the outcome of the delivered citizen experiences.Throughout this research, the objective was to provide a practical method for identifying and measuring the factors that impact citizen interactions with public services. The creation of a method to evaluate the effectiveness of the service provided by a public administrator is a responsibility that governments must take seriously to improve compliance and increase productivity. An objective measurement approach of public administration service, unfettered by political influence, is currently unavailable but would be of great benefit to the community. The adoption of the evaluation framework presented in this thesis has the potential for significant impact in all areas of public administration requiring goodwill, adherence to obligations and mutual respect. An evaluation framework encourages self-assessment but also enables the community to hold the bureaucracy accountable, redistribute control and initiate public participation in new forms of collaborative government.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".