1
ADVANCE QUEENSLAND
EVALUATION FRAMEWORK
Advance Queensland Evaluation Framework – July 2019 ii
ACRONYMS
ABS
Australian Bureau of Statistics
AQ
Advance Queensland
ATO
Australian Taxation Office
BERD
Business Expenditure on Research and Development
BLADE
Business Longitudinal Analysis Data Environment
CBA
Cost Benefit Analysis
DDG
Deputy Director-General
DG
Director-General
DITID
Department of Innovation, Tourism Industry Development, and the Commonwealth Games
FDI
Foreign Direct Investment
GDP
Gross Domestic Product
GBAORD
Government Budget Appropriations or Outlays for Research and Development
GOVERD
Government Expenditure on Research and Development
GSP
Gross State Product
HERD
Higher Education Expenditure on Research and Development
OECD
Organisation for Economic Co-operation and Development
R&D
Research and Development
ROGS
Report of Government Services
STEM
Science, Technology, Engineering and Mathematics
UNDP
United Nations Development Programme
Advance Queensland Evaluation Framework – July 2019 iii
TABLE OF CONTENTS
1. Introduction ................................................................................................................................................ 1
1.1 Purpose .............................................................................................................................................. 1
1.2 Scope ................................................................................................................................................. 1
1.3 Review ............................................................................................................................................... 2
2. Overview .................................................................................................................................................... 3
2.1 Advance Queensland ........................................................................................................................ 3
2.2 Evaluation .......................................................................................................................................... 9
3. Approach to evaluation of the Advance Queensland initiative and programs ......................................... 12
3.1 Principles for evaluation of the Advance Queensland initiative and programs ............................... 12
3.2 A system-wide approach ................................................................................................................. 13
3.3 Levels of evaluation ......................................................................................................................... 14
3.4 Types of evaluation .......................................................................................................................... 15
3.5 Logic frameworks and models ......................................................................................................... 16
3.6 Evaluation plans .............................................................................................................................. 17
3.7 Priority evaluations .......................................................................................................................... 18
3.8 Supporting programs of work........................................................................................................... 18
4. Evaluation methodologies ........................................................................................................................ 20
4.1 Evaluation domains and methodologies .......................................................................................... 20
4.2 Considerations in selecting methodologies ..................................................................................... 22
5. Evaluation indicators/measures ............................................................................................................... 24
5.1 Indicator categories ......................................................................................................................... 24
5.2 Macro/system indicators .................................................................................................................. 24
5.3 Whole-of-initiative implementation and performance measures ..................................................... 26
5.4 Program outcomes and measures .................................................................................................. 27
6. Data strategy ............................................................................................................................................ 28
6.1 Identifying data requirements .......................................................................................................... 28
6.2 Collecting and/or retrieving data ...................................................................................................... 28
6.3 Managing data ................................................................................................................................. 29
6.4 Existing data sources ....................................................................................................................... 30
6.5 Data limitations ................................................................................................................................ 31
7. Governance.............................................................................................................................................. 32
7.1 Roles and responsibilities ................................................................................................................ 33
7.2 Risk and issue management ........................................................................................................... 35
8. Stakeholder engagement and communication ........................................................................................ 38
Advance Queensland Evaluation Framework – July 2019 iv
8.1 Communication plan ........................................................................................................................ 38
8.2 Dissemination of evaluation findings ............................................................................................... 39
9. Evaluation resources ............................................................................................................................... 40
9.1 Internal evaluation capability/capacity ............................................................................................. 40
9.2 Financial resources .......................................................................................................................... 40
9.3 External/independent resources ...................................................................................................... 41
Appendix 1 – Glossary of key terms ................................................................................................................ 42
Appendix 2 – Macro measures/system indicators ........................................................................................... 44
A2.1 Currently available macro measures and system indicators ........................................................... 44
A2.2 Additional measures that could be explored or developed .............................................................. 49
Appendix 3 – National Innovation Datasets..................................................................................................... 52
A3.1 National Innovation Map .................................................................................................................. 52
A3.2 BLADE ............................................................................................................................................. 52
A3.3 Innovation Insights Database .......................................................................................................... 53
Advance Queensland Evaluation Framework – July 2019 v
FOREWORD
Advance Queensland is an investment by the Queensland Government of over half a billion dollars,
designed to foster innovation and build a more diversified Queensland economy, creating jobs now and for
the future.
The initiative is a whole-of-government agenda designed to impact all aspects of Queensland’s innovation
system, from inspiring Queenslanders to innovate, enabling the discovery of new breakthroughs and
connecting innovators across boundaries, to promoting investment in Queensland ideas and supporting local
companies to grow.
Advance Queensland encompasses the delivery of a suite of programs and activities, led by multiple
agencies across the Queensland Government to deliver a state made for innovation designed to prosper
now and in the future. A state where ideas matter, collaboration takes us further faster and local innovation
spurs productivity, creates growth and improves the quality of life for all Queenslanders.
These programs and activities are wide ranging and are designed to deliver outcomes that contribute to the
Advance Queensland vision. Co-designed with industry and based on international evidence of what works,
Advance Queensland includes modest programs such as provision of scholarships and small grants; to
ambitious multi-million dollar investments that are re-shaping Queensland’s future.
Advance Queensland is led by an Advance Queensland Strategic Leadership Group, chaired by the Minister
for Innovation. An Advance Queensland Steering Committee, chaired by the Director-General of the
Department of Innovation, Tourism Industry Development, and the Commonwealth Games (DITID) provides
whole-of-government oversight. The Advance Queensland Implementation Unit, established in DITID,
provides whole of government coordination of the initiative.
Framework for Advance Queensland
Advance Queensland Evaluation Framework – July 2019 vi
Advance Queensland Handbook
This document forms part of the Advance Queensland Handbook – a suite of frameworks, strategies, guides
and tools which document the approach to managing the initiative.
The Advance Queensland Handbook provides a comprehensive guide to achieving a consistent approach to
planning, implementation and evaluation of Advance Queensland activities within participating agencies.
Key elements of the Handbook include:
1. Advance Queensland Policy Framework – outlines the rationale and overarching aims of Advance
Queensland
2. Advance Queensland Organising Framework – outlines the programs and activities that contribute
to Advance Queensland goals and objectives, and processes for program initiation, approval and
closure
3. Advance Queensland Governance Arrangements – outlines the whole-of-initiative governance
arrangements for oversight of Advance Queensland
a. DITID Advance Governance Arrangements – outlines the departmental governance
arrangements for oversight of Advance Queensland
4. Advance Queensland Reporting Framework – outlines reporting requirements and mechanisms
5. Advance Queensland Evaluation Framework – outlines the approach and high-level strategy for
evaluation
a. Advance Queensland Evaluation Plan – outlines the key evaluations to be undertaken
6. Advance Queensland Grants Management Framework – provides guidance for grants programs
and processes
7. Advance Queensland Risk and Issue Management Strategy – describes the specific risk and
issue management techniques and standards to be applied
8. Advance Queensland Budget Guidelines – describes budget management processes
9. Advance Queensland Sponsorship Strategy – provides an overview of the sponsorship
governance arrangements and the approach and high-level strategy for sponsorship
a. Advance Queensland Sponsorship Guidelines – provides guidance on the assessment,
approval and management of sponsorships supported by the initiative
10. Advance Queensland Communications and Events Strategy – provides a framework for
coordinating communication activities and major events
Advance Queensland represents is a significant investment by the Queensland Government to transform the
Queensland economy through innovation for the benefit of Queenslanders. It is appropriate to safeguard this
investment with the commensurate program management, governance and evaluation measures.
Advance Queensland Evaluation Framework – July 2019 1
1. Introduction
1.1 Purpose
This document outlines the approach and high-level strategy for evaluating the Advance Queensland
initiative, programs and activities, and provides guidance on appropriate evaluation methodologies,
measures and data.
The Advance Queensland Evaluation Framework is designed to ensure a coordinated approach to reviewing
implementation of Advance Queensland and measuring its outcomes and to inform future investment and
policy direction.
This Framework is consistent with the Queensland Government Program Evaluation Guidelines (2014)
1
, and
draws on examples of best practice from other jurisdictions, including:
• The Magenta Book: Guidance for Evaluation
2
• CSIRO’s Impact Evaluation Guide
3
• BetterEvaluation: International collaboration around improving evaluation practice and theory
4
• Report on Government Services – Approach to Performance Measurement
5
• Better Practice Guide: Successful Implementation of Policy Initiatives
6
It has also been informed by work undertaken by Deloitte Access Economics, commissioned in 2017 by the
(then) Department of Science, Information Technology and Innovation.
1.2 Scope
The framework is focused on Advance Queensland programs but notes the influence of external factors and
the indirect contribution of other government policies and programs.
This framework provides:
• an overview of:
o the Advance Queensland initiative’s vision, strategies, objectives, themes and programs/activities
o evaluation within the context of Queensland Government programs
• the approach that will be taken to evaluate the Advance Queensland initiative and associated
programs/activities:
o key principles underpinning evaluation methodology and activities
o adopting a system-wide approach to identify collective impact
o the levels at which and types of evaluations that will be conducted
o the development of logic frameworks for key elements of the initiative
o planning and prioritisation of evaluation activities .
• evaluation domains and methodologies relevant to the Advance Queensland initiative
• a data strategy for collecting and managing data relevant to the Advance Queensland initiative
1
Queensland Treasury (Queensland Government) (2014). Queensland Government Program Evaluation Guidelines
https://s3.treasury.qld.gov.au/files/qld-government-program-evaluation-guidelines.pdf
2
HM Treasury (2011). The Magenta Book: Guidance for Evaluation.
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf
3
CSIRO (2015). Impact Evaluation Guide. https://www.csiro.au/en/About/Our-impact/Evaluating-our-impact
4
BetterEvaluation. http://www.betterevaluation.org/
5
Productivity Commission, (2017). Report on Government Services – Approach to Performance Measurement.
http://www.pc.gov.au/research/ongoing/report-on-government-services/2017/approach/performance-measurement
6
Australian National Audit Office (Australian Government) (2104): Better Practice Guide: Successful Implementation of
Policy Initiatives https://www.anao.gov.au/work/better-practice-guide/successful-implementation-policy-initiatives
Advance Queensland Evaluation Framework – July 2019 2
• governance arrangements including roles and responsibilities, risk and issue management
• guidance on stakeholder engagement, communication and reporting of evaluation findings.
1.2.1 Exclusions and other evaluation resources
This framework focuses on the approach and high-level strategy for evaluating the Advance Queensland
initiative. It does not articulate the schedule for the evaluations planned to be undertaken – this is included in
the Advance Queensland Evaluation Plan
7
.
While this framework provides guiding principles and specifies some requirements regarding the
development of specific evaluation plans and reports, it does not include a process for the development of
those documents.
The Queensland Government Program Evaluation Guidelines
8
provide a framework to guide agencies in the
development, design and implementation of measureable programs, interventions, initiatives, services or
trials.
1.3 Review
This document will be regularly reviewed and updated as required, especially after substantial changes to
the Advance Queensland initiative.
7
Queensland Government. Advance Queensland Handbook (unpublished)
8
https://www.treasury.qld.gov.au/economy-and-budget/queensland-economy/evaluating-government-programs/
Advance Queensland Evaluation Framework – July 2019 3
2. Overview
2.1 Advance Queensland
The Advance Queensland initiative encompasses the delivery of a comprehensive suite of programs and
activities, led by multiple agencies across Queensland Government.
The Advance Queensland framework outlines how the delivery of each individual program contributes to
something that is more than the sum of its parts — how the programs work together to achieve the Advance
Queensland vision. Conceptually this forms an integrated relationship between the individual programs, their
objectives and their contribution to the overarching initiative’s strategies and vision.
Figure 2.1 – Advance Queensland framework
2.1.1 Advance Queensland Vision
The vision of Advance Queensland positions Queensland as a leader in the knowledge economy, creating
jobs both now and for the future. It seeks to empower our best entrepreneurs, innovators and researchers,
and help translate their ideas into commercial success and social benefit. Creating jobs now and the jobs of
the future requires having the right environment for businesses to thrive - Advance Queensland fosters and
creates an innovation system that enables this to occur.
2.1.2 Advance Queensland Strategies and Objectives
Five key strategies have been identified to implement the vision for Advance Queensland and guide the
design and implementation of all Advance Queensland programs.
Advance Queensland Evaluation Framework – July 2019 4
These strategies, described in detail in the Advance Queensland Policy Framework
9
, are:
• Supporting culture – engaging the community in innovation, inspiring current and future generation
to be creative, develop ideas, work together and identify innovation potential.
• Building capability – maintaining a strong research base to support entrepreneurship, business
commercialisation, creativity and the creation and uptake of new technologies; increase the uptake
and level STEM skills in the community.
• Fostering collaboration – building networks and partnerships across organisations, sectors and
disciplines to help spark creativity and ideas, diffuse existing knowledge, and increase the translation
of ideas into outcomes.
• Increase investment – facilitating access to capital to fund the translation of ideas to outcomes, and
assisting innovators to be “investment ready”.
• Scaling for jobs and growth – empowering businesses and key industries to benefit from
accelerating technological disruption and access changing global chains, unlocking the potential of
small businesses, high grown firms and regions to innovate and develop.
To meet the vision of the Advance Queensland initiative, the strategies collectively target weaknesses in the
existing innovation system and, as a result, create economic and social value which would not have
otherwise occurred.
The Advance Queensland objectives specify key elements to be achieved across all programs, and align
with a particular strategy.
Table 2.1 – Advance Queensland Strategies and Objectives
Strategy
Objective
Supporting
culture
SC1 – Increase
innovation awareness
and engagement
SC2 – Increase
entrepreneurialism
Building
capability
BC1 – Increase
innovation capability
BC2 – Develop, attract
and retain talented
people (including STEM
skills)
Fostering
collaboration
FC1 – Build sustainable
partnerships to deliver
outcomes
FC2 – Increase local
and international
networks
9
Queensland Government. Advance Queensland Handbook (unpublished)
Advance Queensland Evaluation Framework – July 2019 5
Strategy
Objective
Increase
investment
II1 – Grow pipeline of
investable products
/services
II2 – Build access to
capital
Scaling for jobs
and growth
SJ1 – Expedite
commercialisation
SJ2 – Increase
economic benefits from
innovation (including
jobs)
2.1.3 Advance Queensland Programs and Activities
The Advance Queensland initiative is made up of a wide portfolio of programs and activities, delivered by a
range of Queensland Government agencies.
Types of programs and activities delivered under the Advance Queensland include:
• Grants – funding provided to defined entities for a specific purpose or project under a structured
program which includes an application, assessment, decision and funding agreement process
• Partnerships – Financial contribution to one-off strategic projects or organisations to support unique
opportunities
• Competitions – a contest in which people or companies take part in order to win a defined end-prize
• Procurement – obtaining goods or services in a fair and equitable manner that aligns with Advance
Queensland strategic goals
• Events – an event for external participants that is funded by and or/supports Advance Queensland
aims, objectives or programs
• Sponsorships – provision of financial support for an external event or activity
• Foundations and administrative activities – activities to support the delivery and governance of
the initiative.
2.1.4 Advance Queensland Organising Framework and Themes
The Advance Queensland Organising Framework
10
was developed to provide a single point of truth on
programs and activities that contribute to Advance Queensland goals and objectives, and a basis for
reporting on the progress and performance of the initiative.
The Organising Framework outlines all programs and activities, and clarifies and confirms their key attributes
including status, program type, funding arrangements and lead agency.
Programs are grouped under five key themes: Inspire, Discover, Connect, Invest and Grow. Program
foundations and administrative activities are grouped under a sixth category.
10
Queensland Government. Advance Queensland Handbook (unpublished)
Advance Queensland Evaluation Framework – July 2019 6
Figure 2.2 – Advance Queensland themes
While individual programs may contribute to one or more of the Advance Queensland strategies and
objectives (see section 2.1.5), each theme is aligned to a primary strategy and associated objectives.
Table 2.2 – Advance Queensland Themes and alignment to Advance Queensland strategies and objectives
Theme
Description
Aims
Primary Strategies/
Objectives
Inspire Queenslanders
to engage with science
and technology, be
entrepreneurial and take
their ideas to the world
Programs aligned to this
theme promote a
transformational agenda
that aims to inspire
Queenslanders to
embrace new ways of
thinking and working,
and to back themselves
and their ideas
Programs aligned to this
theme aim to:
• ignite the innovation
spirit of
Queenslanders
• inspire the
entrepreneurs of the
future
• celebrate and
support those
having a go.
Supporting culture:
• SC1 – Increase
innovation
awareness and
engagement
• SC2 – Increase
entrepreneurialism
Discover new solutions
to improve everyday
lives in Queensland
through programs that
foster current and future
talent and enable
researchers and
Industry to solve local
and global challenges.
Programs aligned to this
theme aim to inspire and
mobilise Queensland
communities – including
parents, families,
startups and business –
to prepare themselves
and their children for the
jobs of the future.
Programs aligned to this
theme aim to:
• attract and retain
world class talent
• enable researchers
and industry to
solve global
challenges
• prepare
Queenslanders for
the jobs of
tomorrow.
Building capacity:
• BC1 – Increase
innovation capability
• BC2 – Develop,
attract and retain
talented people
(including STEM
skills)
Advance Queensland Evaluation Framework – July 2019 7
Theme
Description
Aims
Primary Strategies/
Objectives
Connect
Queenslanders to world-
leading local and
international innovators
through programs that
encourage collaboration
and build the
entrepreneurial and
innovation ecosystem.
Programs aligned to this
theme are designed to
develop, attract and
retain the talent needed
to support a strong and
vibrant knowledge
economy. They are
designed to connect
Queenslanders with
local and international
collaborators, innovators
and investors.
Programs aligned to this
theme aim to:
• build innovation
networks that spark
opportunities to
connect, learn and
partner
• create global
connections for
talent, markets and
opportunities
• increase
collaboration
between industry,
researchers and
startups.
Fostering
collaboration:
• FC1 – Build
sustainable
partnerships to
deliver outcomes
• FC2 – Increase
international
networks
Invest in Queensland
innovation through
programs to encourage
seed funding, venture
capital and deal-flow,
and foster emerging
industries.
Programs aligned to this
theme are helping
Queensland businesses,
researchers and
innovators build their
skills and businesses to
take their ideas to
market, access finance;
scale and become
investment-ready; and
increase the level of
deal flow.
Programs aligned to this
theme aim to:
• attract local and
global investment
and investors into
Queensland
innovation
• help innovators to
become market and
investment ready
• improve service
delivery through
innovation.
Increase investment:
• II1 – Grow pipeline
of investable
products/services
• II2 – Build access to
capital
Advance Queensland Evaluation Framework – July 2019 8
Theme
Description
Aims
Primary Strategies/
Objectives
Grow the
competitiveness of our
businesses, industries
and regions through
programs to accelerate
growth and unlock new
markets and
opportunities
Programs aligned to this
theme are unlocking
new opportunities for
traditional industries and
creating new industries.
Programs expand
venture capital funding
available to
Queenslanders, and
provide targeted funding
assistance to address
gaps in key strategic
opportunities for the
state.
Programs aligned to this
theme aim to:
• harness innovation
to create
opportunities for
traditional and
emerging industries
• unlock the potential
of small business
and regions to
innovate
• accelerate
development of our
high growth firms
(scale up faster)
• establish
government as a
lead customer and
innovator.
Scaling for jobs and
growth:
• SJ1 – Expedite
commercialisation
• SJ2 – Increase
economic benefits
from innovation
(including jobs)
2.1.5 Program objectives and intended outcomes
Each program within the Advance Queensland initiative has a number of expected outputs and outcomes,
and will contribute to one or more of the Advance Queensland strategies and objectives (see Figure 2.3).
Figure 2.3 – Links between program outcomes and AQ objectives and strategies – bottom up
Advance Queensland Evaluation Framework – July 2019 9
Conversely, each Advance Queensland strategy and objective is supported and achieved through the
outcomes and impacts of multiple individual programs (see Figure 2.4).
Figure 2.4 – Links between program outcomes and AQ objectives and strategies – top down
The relationship between individual program inputs, outputs, outcomes and the broader Advance
Queensland strategies and objectives are detailed in individual program theories or logic models (see
section 3.5 for further detail).
2.2 Evaluation
Evaluation is the systematic, objective process of understanding how a policy or other intervention was
implemented, what effects it had, for whom, how and why.
Evaluation activities can occur before, during or after implementation, and may include an assessment of the
appropriateness, relevancy, process, effectiveness and/or efficiency of a program.
This section gives an overview of how and when to evaluate within the context of Queensland Government
programs.
2.2.1 Why evaluate?
Evaluation is the foundation for effective, evidence-based policy and continual
improvement of Advance Queensland.
Evaluation delivers on the Queensland Government’s commitment to accountability, transparency and
achieving value for money on investments and policy initiatives.
Evaluation can determine whether programs are operating as planned and on track to deliver intended
objectives. It communicates to program managers what elements of a program are working effectively and
highlights areas that require improvement. Incorporating this feedback into program delivery can improve
efficiency and optimise return on investment.
Advance Queensland Evaluation Framework – July 2019 10
Evaluation is particularly important for innovation programs, such as Advance Queensland, which are
intended to disrupt business as usual. Evaluating a program targeting innovation can signal to government
the appropriateness and cost-effectiveness of this disruption.
Given the complexity of an innovation system, and the breadth of the Advance Queensland program, such
interventions may have unintended impacts elsewhere in the economy. Effective evaluation allows these
impacts to be captured, accounted for, and addressed as part of the program’s ongoing implementation.
Equally, Advance Queensland programs may spur unintended positive impacts to be further maximised
including knowledge spill overs, as well as potential cumulative benefits of well-designed programs targeting
common outcomes.
Evaluation also creates an evidence base to support continued investment in programs that work and to
refine innovation policy over time.
2.2.2 When to evaluate
Evaluation planning and activity should be part of program design and delivery.
Evaluation planning is best embedded into the program development and design stage to ensure efficiency
and effectiveness in the delivery of outcomes.
Early evaluation planning can enhance a program’s design by clarifying the program’s goals, objectives and
desired outcomes, activities and key deliverables; and how these will be measured, including appropriate
data sources. The identification and collection of baseline data at the commencement of the program will
strengthen evaluative activities and findings.
Ongoing feedback from evaluations conducted during program implementation can be used to refine data
collection, program design and delivery. At a broader policy level, evaluation findings should be used to
inform future program development and drive improvement in program design and delivery.
Short-term and medium-term outcomes can be measured and evaluated during program implementation,
and at the closure of program funding rounds. Longer-term outcomes, such as wider economic and social
impacts may require a study of program impact to be undertaken a significant time after project and program
completion.
Figure 2.5: Incorporating evaluation into program development and implementation
Advance Queensland Evaluation Framework – July 2019 11
2.2.3 Queensland Government Program Evaluation Guidelines
The Queensland Government Program Evaluation Guidelines (2014)
11
outline a broad set of principles that
are expected to underpin the planning and implementation of evaluations for programs funded by the
Queensland Government.
The guidelines outline the minimum requirements expected to be met for the planning, implementation and
management of program evaluations and are intended as a resource for those responsible for developing,
designing and implementing programs.
The Queensland Government Program Evaluation Guidelines require that all evaluations of public sector
programs will:
• Specify criteria for determining the success of the program
• Focus on the key issues that will inform decision making
• Use a systematic and evidence-based approach to assess performance
• Be reliable, useful and relevant to decision makers and stakeholders
• Be timely.
Ideally, evaluation should be built into program design and have the following features:
• A clear, considered evaluation plan and, where relevant, a well-drafted terms of reference
• Clearly defined roles and responsibilities
• Strong stakeholder engagement
• Evaluation milestones timed to be able to inform decision making
• Strategies in place to compensate for any potential deficiencies in evaluation design, data collection
and analytical methods
• Checks and balances in place to ensure validity of evaluation findings
• Clear, transparent reporting that outlines methods, assumptions and key findings.
2.2.4 Other relevant Queensland Government frameworks and principles
All evaluations should also be consistent with the principles and approaches outlined in other relevant
Queensland Government frameworks, including:
• Queensland Government Performance Management Framework
12
– provides a mechanism to
help strengthen public sector accountability, adopting a holistic approach to performance
management directed at a whole-of-Government, ministerial portfolio, agency and individual level.
The PMF focusses on three key aspects of public sector performance management: planning,
measuring and monitoring performance, and public reporting.
• Project Assessment Framework
13
– used across government to ensure a common, rigorous
approach to assessing projects at critical stages in their lifecycle, from the initial assessment of the
service required, through to delivery.
11
Queensland Treasury (Queensland Government) (2014). Queensland Government Program Evaluation Guidelines
https://s3.treasury.qld.gov.au/files/qld-government-program-evaluation-guidelines.pdf
12
https://www.forgov.qld.gov.au/manage-government-performance
13
https://www.treasury.qld.gov.au/growing-queensland/project-assessment-framework/
Advance Queensland Evaluation Framework – July 2019 12
3. Approach to evaluation of the Advance Queensland initiative and
programs
This section outlines the key elements of the approach that will be taken to evaluate the Advance
Queensland initiative and associated programs/activities, including:
• key principles underpinning evaluation methodology and activities
• adopting a system-wide approach to identify collective impact
• the levels at which evaluations will be conducted (micro, meso and macro)
• the types of evaluations that will be undertaken (formative, process, effectiveness and efficiency)
• the development of logic frameworks for key elements of the initiative and evaluations plans
• prioritisation of evaluations.
3.1 Principles for evaluation of the Advance Queensland initiative and programs
There are eight core principles that underpin the evaluation methodology. All Advance Queensland
evaluation activities will adhere to these principles as closely as is practical.
Figure 3.1 Principles for evaluation of Advance Queensland
Advance Queensland Evaluation Framework – July 2019 13
3.2 A system-wide approach
Innovation does not occur in isolation. Interaction is central to the innovation process, with change in one
part of the system often causing or catalysing change in another. This may be synergistic, with a
combination of programs working together to deliver desired outcomes, or unintended, where one program
impacts on another in a manner that was not anticipated.
Given the complex nature of innovation systems, evaluating innovation programs using a systems approach
can identify their collective impact and help with recognition of gaps or disconnects in the system that
decrease the overall effectiveness of program investments
14
.
Another characteristic of a suite of programs like Advance Queensland is individual programs with common
goals may affect the same indicators. Both these factors can make attribution of impacts to individual
programs challenging, which makes outcome evaluation a difficult task. Evaluating at a system-wide level
can side step this problem.
Figure 4.1: An overview of the Queensland Innovation System
14
Woothuis, Lank Woolthuis, Lankhuizen & Gilsing (2005). A system failure framework for innovation policy design
Advance Queensland Evaluation Framework – July 2019 14
3.3 Levels of evaluation
Evaluation of the Advance Queensland initiative and associated programs/activities occurs at three different
levels:
• Micro-level evaluations – for single programs or activities, with priority given to those that are that are
more complex and/or have significant funding, impact, risk or profile.
• Meso-level evaluations – for groups or clusters of programs which have common processes, audiences
and/or objectives, and therefore could be most efficiently and effectively evaluated together.
• Macro-level evaluations – incorporating all programs and activities, taking into account interlinkages
between programs and outcomes at a system-wide level.
This approach will ensure that evaluative consideration is given to key aspects of the initiative, as well as
making better use of limited resources by consolidating evaluation activity where it makes sense to do so.
Figure 4.2: Levels of evaluation of Advance Queensland initiative and programs/activities
The level of evaluation at which a specific program will be evaluation will be determined by the Evaluation
Sub-Committee, and endorsed by the Advance Queensland Steering Committee, and will be based on an
assessment of key attributes of programs, including:
• Program objectives and scope – including number and scope of program outcomes, linkages to
Advance Queensland strategies and objectives and priorities in the Building Our Innovation Economy -
Advance Queensland Strategy
• Data requirements – including type and availability of data required to support evaluation
• Complexity and risk – including interdependencies with other programs and functional areas
• Funding and value for money – including budget and expected cost of delivering outputs and
achieving outcomes
• Profile or nature of the program – program type and scale, level of scrutiny program is expected to
attract
• Target participants – number and type of innovation system participants targeted by the program
• Governance and stakeholders – number of government departments and stakeholders involved in
program delivery
• Timeframes for implementation and expected impacts – when program outputs are expected to be
delivered, implementation completed, and outcomes and impacts realised
• Evidence base – opportunity or requirement to build the evidence base for program leaders or decision-
makers to rely upon when assessing the development, implementation, continuation or cessation of the
program or other similar programs.
Micro-level
evaluations
•Evaluates an individual
program or activity (with
priority to those with
significant funding,
impact, risk or profile).
Meso-level
evaluations
•Evaluates groups of
programs that have
similar processes, target
groups, and/or
objectives.
Macro-level
evaluations
•Evaluates Advance
Queensland as whole,
incoroprating all
programs and activities.
Advance Queensland Evaluation Framework – July 2019 15
3.4 Types of evaluation
Evaluations of the Advance Queensland initiative and associated programs/activities will be designed to
answer four broad classes of questions:
• What is the need for the program/activity, and how is it best designed for maximum benefit?
• How was the program/activity implemented and/or delivered?
• What difference did the program/activity make? Was it effective?
• Did the benefits of the program/activity justify the costs? Was it efficient?
The specific objectives for each evaluation will vary according to the type of program (or group of programs)
being evaluated, the stage of program implementation, the existing body of evidence/availability of data, and
stakeholder expectations.
3.4.1 Formative (ex-ante) – design and clarification
Formative evaluations are typically undertaken at the beginning of a program that is relatively large or
contributes to an ongoing policy commitment.
This approach can be used:
• when developing a new program or refining an existing program, and/or
• to compare actual performance against what was originally intended.
Formative evaluations help to assess whether the program will address an identified need, inform the design
of the program, identify key environmental elements that may influence the program’s success, make clear
the theory or change that the program is based on and what outcomes the program is aiming to achieve.
Formative evaluation often involve qualitative methods of inquiry, and key evaluation questions are generally
more open and lead to exploration and clarification of aims, objectives and processes.
3.4.2 Process – implementation and delivery
Process evaluations assess whether a policy is being implemented as intended and what, in practice, is felt
to be working more or less well, and why.
This approach can be used:
• when refining an existing program to inform and improve program design;
• during implementation to inform and improve implementation processes, and/or
• at the end of program to determine design or program elements that contributed to the program’s
success, and inform future program development.
Process evaluations can employ a wide range of data collection and analysis techniques, covering multiple
topics and participants. Process evaluations often include the collection of qualitative and quantitative data
from different stakeholders, as well as organisational and administrative information.
3.4.3 Effectiveness – impact/outcome
Impact evaluations assesses the outcomes of a program against its short term, medium, or long term goals.
This kind of evaluation is useful to answer the question “has our program worked?”
This approach can be used:
• at key points during implementation to assess achievement of intermediate outcomes and inform
improvements to program design and/implementation approach;
• at the end of program to determine the outcomes achieved; and/or
• at points in time after the program has been completed to assess longer-term impacts, including wider
economic and social impacts.
Advance Queensland Evaluation Framework – July 2019 16
A comprehensive approach to outcome evaluation also assesses any unintended impacts of a program, as
well as the “counterfactual” or what would have happened in the absence of the program.
Experimental, quasi-experimental, or non-experimental methods are the main approaches used to attribute
observable impacts on relevant indicators to a program’s interventions.
3.4.4 Efficiency – value for money
This approach is used to assess the program’s value for money, or the efficiency in achieving the outcome/s.
For the purposes of program evaluation, are three types of efficiencies:
• Technical efficiency – the program was delivered at the lowest possible cost
• Allocative efficiency – the program is delivered to the areas of greatest needs, and provides the
types of outputs and outcomes that recipients value most (for the given set of resources)
• Dynamic efficiency – the program continues to improve over time, by finding better or lower cost
ways to achieve outcomes.
3.5 Logic frameworks and models
3.5.1 Logic frameworks
A logic framework shows how program activities are understood to contribute to a series of intermediate
outcomes that then produce the intended long-term outcomes.
The development of a logic framework is recommended by the Queensland Government Program Evaluation
Guidelines
15
in order to develop an understanding of how the program works (or is intended to work), what it
is trying to achieve (in terms of measurable objectives), and why (the underlying policy program).
Development of a program logic framework provides:
• a tool to create a dialogue and shared understanding of the program’s goals, objectives and desired
outcomes, activities and key deliverables
• clarity of the cause-and-effect relationship between program activities, outputs and immediate,
intermediate and ultimate outcomes
• a tool to identify and assess the plausibility of the assumptions made
• a method to evaluate and strengthen program design
• a hierarchy of outcomes expected at different times scales
• a framework to monitor and evaluate performance, including an outline of data requirements,
collection methods and analysis techniques.
3.5.2 Logic models
Different types of diagrams can be used to represent a program theory. These are often referred to as logic
models, as they show the overall logic of how the intervention is understood to work.
There are many ways of developing and representing logic models. The methods chosen will likely depend
on the program (or cluster of programs) being assessed or evaluated, however all models should consider:
• Need or driver(s) – why the program is required
• Objectives: what the program aims to achieve and why
• Inputs: the resources needed to operate the program (labour, materials etc.)
• Activities: processes, tools, events, technology and actions integral to program implementation
• Outputs: direct products of program activities (such as types of services to be delivered)
• Short-term outcomes: such as changes in awareness, knowledge, skills, and attitude
15
Queensland Treasury (Queensland Government) (2014). Queensland Government Program Evaluation Guidelines
https://s3.treasury.qld.gov.au/files/qld-government-program-evaluation-guidelines.pdf
Advance Queensland Evaluation Framework – July 2019 17
• Medium-term outcomes: such as changes in behaviour
• Long-term outcomes: such as wider economic, environmental and social impacts.
• Alignment and contribution to broader objectives and frameworks.
Table 3.1 – Sample of methods to represent logic models
Model type
Method
Represented as:
Logframe
Considers the relationships between
available resources, planned activities,
and desired changes or results.
A matrix of key program attributes which
may include:
o Goal, Purpose, Outputs, Activities
o Summary, Indicators, Data sources,
Assumptions
Outcomes
Hierarchy
Demonstrates a series of outcomes
leading up to the final impacts of a
project
Can be shown as a series of “boxes and
lines” or a table showing outcomes to be
achieved at different temporal scales
Results Chain
(pipeline model)
Represents a program as a linear
process with inputs and activities at the
front and long-term outcomes at the end
A series of boxes:
inputs –> activities –> outputs –> outcomes
–>impacts
3.5.3 Advance Queensland logic frameworks
Individual logic frameworks are developed for key elements of the Advance Queensland initiative, including:
• Micro-level frameworks – for significant programs or activities including those that are that are more
complex and/or have significant funding, impact, risk or profile (i.e. subjects of micro-level evaluations)
• Meso-level frameworks – for groups or clusters of programs which have common processes,
audiences and/or objectives, and therefore could be most efficiently and effectively evaluated together
(i.e. subjects of meso-level evaluations)
• Macro-level framework – incorporating all programs and activities, taking into account interlinkages
between programs and outcomes at a system-wide level.
The Advance Queensland Framework provided in Section 2.1 of this document, and the Advance
Queensland Organising Framework
16
provides a high-level logic framework for the initiative.
All logic frameworks and models are to be reviewed and updated as required to ensure they remain accurate
and fit for purpose.
3.6 Evaluation plans
Evaluation plans are developed for each individual evaluation to set out the proposed details of an evaluation
– what will be evaluated, how and when.
Evaluation plans are specific to each evaluation, but generally include:
• what is to be evaluated (the ‘evaluand’)
• the logic framework
• the purpose/s of the evaluation
• key evaluation questions
• when the evaluation will be conducted
• who will conduct the evaluation
• what resources are required and available
16
Queensland Government. Advance Queensland Handbook (unpublished)
Advance Queensland Evaluation Framework – July 2019 18
• what data will be collected, how and when, how data will be analysed,
• who are the key stakeholders and how they will be engaged
• how and when results will be reported.
3.7 Priority evaluations
A system-level approach is used to prioritise the number, frequency, level and type of evaluation activity to
ensure efficient allocation of resources when evaluating the Advance Queensland portfolio.
Advance Queensland programs are identified as a priority evaluation by implementing agencies through the
Evaluation Sub-Committee, and endorsed by the Advance Queensland Steering Committee based on an
assessment of key attributes of programs (refer to Section 3.3) and the following key considerations include:
• what evaluation activity needs to be prioritised and the reasons for prioritisation
• what types of evaluation activity will be undertaken
• what program activities need to be undertaken and finalised in order for evaluation activity to commence
• what data is required to undertake the evaluation and when it will be available
• when program outputs will be delivered, and outcomes and impacts will be realised
• resourcing strategy to undertake evaluation activities, including capacity to undertake concurrent
activities.
The priority evaluations will be regularly reviewed and updated to take into account changes to evaluation
priorities, capacity and capability.
Cross-cutting themes
Outcomes for Indigenous, female and regional Queenslanders should be investigated as cross-cutting
themes in all AQ priority evaluations, as appropriate. Rather than specific evaluations focused only on these
groups, including them across the priority evaluations allows for enablers, barriers and outcomes to be
assessed in the context of each specific program. Some programs specifically focus on these groups, and
consequently evaluations of these programs will include a greater investigation of these outcomes.
3.8 Supporting programs of work
To support the approach to Advance Queensland evaluation outlined in the previous sections, two
supporting programs of work have been identified to progress critical issues impacting across the evaluation
activities.
The table below outlines the purpose of the supporting programs of work and the related sections of this
Framework. They are also identified as management strategies to the high-level risks identified in Section
7.2.2.
Advance Queensland Evaluation Framework – July 2019 19
Table 3.2 – Outline of the supporting programs of work
Supporting Program
of Work
Aims
AQ Evaluation
Framework
1. Evaluation capability,
capacity and resourcing
• Confirm the human and financial
resourcing required to adequately
evaluate the Advance Queensland
initiative
• Build the evaluation culture and skills
within implementing agencies, and
• Develop a long-term, staged resourcing
and sourcing strategy for the evaluation of
Advance Queensland.
Section 9 – Evaluation
Resources
2. Evaluation
methodologies, metrics
& data
• Support the assessment, acquisition
and/or development of appropriate
methodologies, metrics and data sets
required to effectively evaluate innovation
performance
Section 4 – Evaluation
Methodologies
Section 5 – Evaluation
indicators/ measures
Section 6 – Data strategy
Advance Queensland Evaluation Framework – July 2019 20
4. Evaluation methodologies
This section outlines key evaluation domains and methodologies for each evaluation type detailed in section
3.4, as well as some general consideration in choosing a methodology.
4.1 Evaluation domains and methodologies
To guide the selection of evaluation methodologies, evaluation domains for each type of evaluation based on
output and outcome indicators in the Report of Government Services (ROGS) Performance Indicator
Framework
17
and possible evaluation questions have been developed.
Table 4.1 – Overview of evaluation types, domains and methodologies.
Evaluation type
Evaluation domains
Possible evaluation
questions
Possible evaluation
methodologies
Formative
• Need – clarify the
need for the program,
and how it addresses
an identified need
• Logic – clarify the
logic or theory of
change underpinning
the program, and
identify key
assumptions
• Objectives – clarify
what it the program is
trying to achieve
• Design – clarify and
test key design
elements
• Context – key
environmental
elements that may
influence the
program’s success
• Target audience –
clarify the groups that
the program aims to
serve
• Appropriateness –
clarify how the
program will meets the
stated objectives and
needs
• What are the
underlying causes of
problem that the
program is looking to
solve?
• What has worked in
solving similar
problems?
• What key features of
similar programs have
contributed to
outcomes?
• How could program
design improve on
past programs?
• What are the
characteristics of the
intended audience
that are relevant to
program design?
• What data needs to be
collected and
how/when will it be
available?
• Literature scan
• Exploratory research
• Case study/interviews
• Program Logics
• Systematic literature
review
17
Productivity Commission, (2017). Report on Government Services – Approach to Performance Measurement.
http://www.pc.gov.au/research/ongoing/report-on-government-services/2017/approach/performance-measurement
Advance Queensland Evaluation Framework – July 2019 21
Evaluation type
Evaluation domains
Possible evaluation
questions
Possible evaluation
methodologies
Process
(Implementation
and delivery)
• Fidelity – the extent
to which the program
has been delivered as
intended or planned
• Reach – the extent to
which the program
has been adopted by
key stakeholders and
the extent to which
target groups have
been adequately
reached
• Governance – the
extent to which
governance
arrangements support
implementation
• Is the initiative/
program being
implemented as
intended?
• What design or
features influenced
variation in
implementation?
• How appropriate are
the processes
compared with quality
standards?
• Are potential
participants being
reached and/or
engaging in the
program as intended?
• Program reports
• Expert review
(process)
• Program monitoring
documentation
• Semi-structured
interviews/surveys
with program team
and participants
Effectiveness
(Outcome/
Impact)
• Effectiveness – the
extent to which the
program delivers on
stated objectives
• Access – how easily
the target audience
can access the
program or service
• Appropriateness –
how well the program
meets the needs of
stakeholders
• Quality – the extent to
which a service is
suited to its purpose
and conforms to
specifications
• Unintended impacts
– any unexpected
impacts
• To what extent does
the program address
an identified need/
deliver on the
intended objectives?
• How satisfied are the
target audience and/or
key stakeholders with
the program and its
accessibility?
• What short, medium
or long term outcomes
have been observed?
• Would outcomes have
been achieved without
intervention?
• What is the sustained
change because of
this program?
• What benefits would
be absent if this
program had not been
implemented?
• What unintended
outcomes/impacts
(positive/negative)
were produced?
• Quasi-experimental
• Non-experimental
• Expert review
• Realist evaluation
• Regression
econometrics
• Semi-structured
interviews/surveys
with program team
and participants
Advance Queensland Evaluation Framework – July 2019 22
Evaluation type
Evaluation domains
Possible evaluation
questions
Possible evaluation
methodologies
Efficiency
(Value for money)
• Technical efficiency
– whether program
was delivered at the
lowest possible cost
• Allocative efficiency
– whether the program
provides the types of
outputs/outcomes that
recipients value most
(for the given set of
resources)
• Dynamic efficiency –
whether the program
continues to improve
over time, by finding
better or lower cost
ways to achieve
outcomes
• Was the program
delivered at the lowest
possible cost?
• Is the program
providing good value
for money?
• What has been the
ratio of costs to
benefits)?
• Rapid cost-benefit
analysis
• Full cost-benefit
analysis
• Cost effectiveness
analysis
4.2 Considerations in selecting methodologies
There is no one size fits all approach to choosing a methodology: selection should be program specific,
taking into account the motivation and objectives of evaluating the program, data requirements and
availability, resourcing and the complexity of the evaluation methodology.
The table below provides an overview of the evaluation methodologies listed in Section 4.1 and their relative
complexity.
Table 4.2 – Evaluation methodologies and relative complexity.
Straightforward
More Complex
Formative
• Literature scan
• Targeted literature
review
• Exploratory research
• Case study/interviews
• Program Logics
• Systematic literature
review
Process
(Implementation
and delivery)
• Program report
• Program monitoring
(e.g. health check
report)
• Expert review (process)
• Semi structured
interviews
Effectiveness
(Outcome/
Impact)
• Qualitative
assessment of
program impact based
on interviews or
surveys
• Non-experimental
• Expert review
(outcome)
• Regression/econometric
• Experimental
• Quasi-experimental
Advance Queensland Evaluation Framework – July 2019 23
Straightforward
More Complex
Efficiency
(Value for
money)
• Rapid CBA
• Cost effectiveness
• Full economic CBA
Given the nature Advance Queensland programs, it is unlikely that any of the more complex methodologies
would be required.
Advance Queensland Evaluation Framework – July 2019 24
5. Evaluation indicators/measures
Measuring innovation is complex and many of the impacts of programs delivered by Advance Queensland
will not be apparent for many years.
Therefore success will be measured through a combination of:
• Macro/system indicators aligned to Advance Queensland strategies and objectives
• Whole-of-initiative implementation and performance measures
• Bespoke indicators for individual programs
5.1 Indicator categories
Indicators relevant to the evaluation of innovation programs and initiatives can be broadly grouped into the
following categories:
• Intangible outcomes – unmeasurable and are identified as outcomes such as maintaining
relationships or networks.
• Tangible outcomes – can be measured either qualitatively or quantitatively.
• Qualitative indicators –can’t be articulated in a numeric form.
• Quantitative indictors – outcomes that can be represented numerically, either monetised or non-
monetised.
o Non-monetised indicators – include examples such as volume or percentage change;
o Monetised indicators – can be deemed financial or economic
- Financial indicators relate to direct financial impacts
- Economic indicators relate to the impact on the overall economy (GSP or GDP), with a
relevant example identified as, education related government expenses as a share of GSP.
5.2 Macro/system indicators
Recognising the range of external influences and the difficulty of establishing direct causality, macro-level
indicators will be used to determine progress towards the Advance Queensland vision.
Examples of high-level system indicators aligned to the Advance Queensland strategies and objectives are
provided below. This is not intended to be an exhaustive list but rather provides initial high level guidance to
support the identification of indicators for evaluation activities. Further details, along with additional measures
that could be explored or developed are provided at Appendix 3.
Table 5.1 – Currently available macro measures/ system indicators aligned to AQ Strategies and Objectives
AQ Strategy
AQ Objective
System Indicator/ Macro Measure
Supporting
culture (SC)
SC1 – Increase
innovation
awareness and
engagement
Increased awareness of science in Queensland community
Increased awareness of science in regional Queensland
Perceptions of innovation activity (including Advance
Queensland)
SC2 – Increase
entrepreneurial-ism
Value of new and follow-up investment in investee companies
Business entry and exit rates
Count of co-working spaces, startup incubators and
accelerators
Increase in Queensland’s share of tech startups
Advance Queensland Evaluation Framework – July 2019 25
AQ Strategy
AQ Objective
System Indicator/ Macro Measure
Building
Capacity (BC)
BC1 – Increase
innovation capability
Gross expenditure on R&D as a share of GSP including
business expenditure on R&D as well as higher education
Scholarly output per 1000 population
Share of scholarly output in top 1% (or 10%) most cited
publications
Share of SMEs with new to market (world) products
BC2 – Develop,
attract and retain
talent including
STEM
Year 12 students studying identified science, technology and
mathematics subjects
STEM literacy scores (including National Assessment Program
(NAP) – Science Literacy scores)
Proportion of Queenslanders with a non-school qualification
Count of university, TAFE and research institutions per 1,000
population
Fostering
Collaboration
(FC)
FC1 – Build
sustainable
partnerships to
deliver outcomes
Share of businesses conducting innovative activity
Share (%) of Queensland scholarly outputs with international
co-authorship
FC2 – Increase
international
networks
Percentage of R&D financed abroad for Higher Education
Expenditure on R&D (HERD)
Increase
investment
(II)
II1 – Grow pipeline of
investible products
and services
New capital expenditure attracted to Queensland (measured
as growth in business capital expenditure)
Spend on innovation by businesses as a proportion of GSP
More business investment in R&D
Increased investment in research
II2 – Build access to
capital
Value of venture capital by investee company head offices as
a share of GSP
Scaling for
jobs and
growth (SJ)
SJ1 – Expedite
commercialisation
Nil measures readily available
SJ2 – Increase
economic benefits
from innovation
(including jobs)
Jobs driven by Advance Queensland programs
Increase knowledge based jobs in Queensland
Advance Queensland Evaluation Framework – July 2019 26
5.3 Whole-of-initiative implementation and performance measures
As part of the whole-of-initiative reporting arrangements outlined in the Advance Queensland Reporting
Framework
18
, all Advance Queensland program managers are required to provide regular reports on a suite
of key implementation and performance measures.
These measures are listed below, along with relevant evaluation domains, and may be used as key data
sources for process, effectiveness and efficiency evaluations at all levels (micro, meso and macro).
Table 5.2 – Whole-of-initiative measures and relevant evaluation domains
Type
Measure
Relevant evaluation domain/s
Implementation
Program status
• Programs launched
• Rounds opened/closed
• Events held (incl. regional
events)
• Milestones and key
activities
• Fidelity – the extent to which the program has
been delivered as intended or planned
• Reach – the extent to which the program has
been adopted by key stakeholders and the
extent to which target groups have been
adequately reached
• Access – how easily the target audience can
access the program or service
Implementation
Program budget
• Expenditure
• Funds contractually
committed
• Fidelity – the extent to which the program has
been delivered as intended or planned
• Technical efficiency – whether program was
delivered at the lowest possible cost
• Dynamic efficiency – whether the program
continues to improve over time, by finding
better or lower cost ways to achieve outcomes
Performance
Innovators reached
• Applications received
• Attendance at events
• Fidelity – the extent to which the program has
been delivered as intended or planned
• Reach – the extent to which the program has
been adopted by key stakeholders and the
extent to which target groups have been
adequately reached
• Access – how easily the target audience can
access the program or service
• Appropriateness – how well the program
meets the needs of stakeholders
• Allocative efficiency – whether the program
provides the types of outputs/outcomes that
recipients value most (for the given set of
resources)
18
Queensland Government. Advance Queensland Handbook (in development)
Advance Queensland Evaluation Framework – July 2019 27
Type
Measure
Relevant evaluation domain/s
Performance
Innovators supported
• Recipients of grants,
prizes and opportunities
(incl. regional recipients,
female recipients
• Effectiveness – the extent to which the
program delivers on stated objectives
• Reach – the extent to which the program has
been adopted by key stakeholders and the
extent to which target groups have been
adequately reached
• Access – how easily the target audience can
access the program or service
• Appropriateness – how well the program
meets the needs of stakeholders
Performance
Funds leveraged
• External investment
leveraged
• Effectiveness – the extent to which the
program delivers on stated objectives
• Technical efficiency – whether program was
delivered at the lowest possible cost
• Dynamic efficiency – whether the program
continues to improve over time, by finding
better or lower cost ways to achieve outcomes
Performance
Jobs supported
• New jobs reported
• New jobs forecast
• Effectiveness – the extent to which the
program delivers on stated objectives
• Quality – the extent to which a service is
suited to its purpose and conforms to
specifications
Further details about these reporting measures, including data definitions, can be found in the Advance
Queensland Reporting Framework
19
.
5.4 Program outcomes and measures
As outlined in section 2.1.5, each program within the Advance Queensland initiative has a number of
expected outputs and outcomes, and will contribute to one or more of the Advance Queensland strategies
and objectives.
Program outputs, outcomes and the alignment to broader Advance Queensland strategies and objectives
should be detailed in individual program theories or logic models (see section 6 for more detail).
Program measures and indicators will likely include a combination of:
• System indicators aligned to the relevant Advance Queensland strategies and objectives
• Whole-of-initiative implementation and performance measures
• Individual and bespoke indicators.
19
Queensland Government. Advance Queensland Handbook (in development)
Advance Queensland Evaluation Framework – July 2019 28
6. Data strategy
A data strategy is a plan for collecting and managing data to provide evidence-based answers to the
evaluation questions. Data requirements, availability, and approach to collection should be considered and
planned.
6.1 Identifying data requirements
Key questions to ask when assessing data for use in an evaluation include
20
:
• What data needs to be gathered to give reliable and consistent measurement against policy’s
objectives?
• What additional data should be collected to meet the policy maker’s requirements for feedback on
the policy and to support any planned evaluations?
• Who will be in charge of gathering data?
• What are the key timeframes for collection?
• How will the data be gathered, transferred, stored and disposed of?
• What considerations required for appropriate privacy and security?
• What format is the data required in?
• How will the data be verified to ensure it is accurate and consistent with the relevant requirements?
A data matrix linking evaluation questions to qualitative and quantitative data sources is a useful tool to guide
the next steps in performing a program evaluation. Indicators corresponding to the Program Logic should be
described and mapped to the available data sources.
A data matrix for a process evaluation for the grant program example is show in the table below
Table 6.1 – Example data matrix
Evaluation
Domain
Evaluation
Questions
Indicators
Data Sources
Fidelity
Has the program been
implemented as
planned?
• Rounds opened/closed
• Proportion of program
budget expended and
contractually committed
• Number and proportion of
funded projects on track
for completion
• Program data
• Program
performance reports
Appropriateness
How well does the
program meet
stakeholders’ needs?
Reported satisfaction from:
• Grant recipients
• Supervisors
• Industry partners
• Implementation team
• Surveys
• Semi-structured
interviews
6.2 Collecting and/or retrieving data
There are five main sources for collecting and/or retrieving data
21
:
• information from individuals;
20
Evaluation Guidance Note, Scottish Enterprise, 2008. https://www.scottish-enterprise.com/
21
BetterEvaluation. http://www.betterevaluation.org/
Advance Queensland Evaluation Framework – July 2019 29
• information from groups;
• observation;
• physical measurements; and
• existing records and data.
Data collection methods will depend on the identified need. It is important to consider the type of information
required and how it will be analysed before data collection options is selected. It is also recommended that,
where possible, more than one option is selected in order to ensure multiple data sources and perspectives.
Table 6.2 – Data collection options (adapted from BetterEvaluation)
Data source
Data collection options
Information from
individuals
• Interviews
• Questionnaires or surveys
o Email
o Face-to face
o Internet
o Telephone
• Mobile data collection
• Stories and case studies
• Opinion polls
Information from
groups
• Interviews/Focus group discussions
• Debriefs or “After Action Reviews”
• Delphi Study
• SWOT analysis
• World Cafe
Observation
• Field Trips
• Non-participant observation
• Participant observation
Physical measurements
• Geographical
o Demographic mapping
o Geotagging
Existing records and
data
• National datasets and ‘big data’
• Official statistics and reports published by government agencies or other
public bodies
• Previous evaluations and research
• Individual program data, records and reports
Further information about these data collection options is available at
http://www.betterevaluation.org/en/plan/describe/collect_retrieve_data
6.3 Managing data
Data quality assurance – the processes and procedures that are used to ensure data quality – is an essential
component of data management. Using poor quality data may result in inaccurate or inappropriate decisions
about policies and programs. Data quality assurance should be built into each step in the data cycle − data
collection, aggregation and reporting, analysis and use, and dissemination and feedback.
Advance Queensland Evaluation Framework – July 2019 30
Ensuring data quality extends to checking and ‘cleansing’ datasets using standardised procedures, as well
as presenting the data appropriately in the evaluation report so that the findings are clear and conclusions
can be substantiated. Often, this involves making the data accessible so that they can be verified by others
and/or used for additional purposes such as for synthesising results across different evaluations.
Key aspects of data quality
22
include:
• Validity – the degree to which the data measure what they are intended to measure
• Reliability – data are collected consistently; definitions and methodologies are the same when doing
repeated measurements over time
• Completeness – data are complete (i.e., no missing data or data elements)
• Precision – data have sufficient detail
• Integrity – data are protected from deliberate bias or manipulation for political or personal reasons
• Availability – data are accessible so they can be validated and used for other purposes
• Timeliness – data are up-to-date current and available on time.
6.4 Existing data sources
The following section provides an overview of some of the key existing data sources that may be used to
evaluate the Advance Queensland initiative. This is not intended to be an exhaustive list, as additional data
sets will continue to be developed and made available.
6.4.1 National Innovation datasets
The following datasets have been developed by the Commonwealth Department of Industry, Innovation and
Science with additional data from the Australian Bureau of Statistics (ABS) and other domestic and
international sources:
• Business Longitudinal Analysis Data Environment (BLADE) – a series of integrated, linked
longitudinal datasets combining administrative data from the Australian Taxation Office (ATO) with
primary survey data on more than two million actively trading Australian businesses
• National Innovation Map – visually presenting new business creation, expenditure on research and
development, patenting activity and trademarking activity for each statistical region in Australia
23
• Innovation Insights Database – released as part of the annual Australian Innovation System
Report to provide greater information and understanding to policy makers, academics and other who
are interested in Australia’s innovation history
24
Further information on these datasets is provided at Appendix 4.
6.4.2 Program records and reports
In addition to the implementation and performance measures required for whole-of-initiative reports (see
section 5.3), individual programs will have a range of data that can be used in evaluation.
This includes, but is not limited to:
o Literature reviews and needs assessments
o Baseline data
o Program guidelines, logic frameworks, and work plans
o Budget and procurement documents
22
BetterEvaluation. http://www.betterevaluation.org/
23
https://industry.gov.au/Office-of-the-Chief-Economist/Publications/AustralianIndustryReport/Industry-Innovation-
Map.html
24
https://industry.gov.au/Office-of-the-Chief-
Economist/Publications/AustralianInnovationSystemReport2017/index.html
Advance Queensland Evaluation Framework – July 2019 31
o Minutes of meetings and other governance documents
o Funding applications
o Recipient milestone and final progress reports
o Program performance reports
o Case studies and media releases.
6.4.3 Other existing data sources
Other data sources which may be used in the evaluation of innovation programs include:
• Australian Bureau of Statistics (ABS)
25
• Australian Government reports
• Elsevier information and analytics
26
• Independently published surveys and reports
• Queensland Government reports
• University surveys and reports.
High-level system indicators aligned to the Advance Queensland strategies and objectives are provided at
Appendix 3. Two lists are provided:
1. System indicators and macro measures currently available
2. Additional measures that could be explored or developed
6.5 Data limitations
All data sets mentioned are constructed using different methodologies and all have their limitations. For
example, there are limitations in using patents and trademarks as indicators of regional innovation because
innovation could have happened at a location other than where patent holder resides and patents do not
necessarily equal products. Nevertheless, there are few other indicators that provide a superior measure of
innovation.
Data limitations all broadly surround whether the data set is complete and appropriate for use in the context
of the evaluation.
It is also important to ensure validation of externally reported benefits from program participants. For
instance, if data to support the measurement of innovation benefits comes from program participants it is
vital to ensure this self-reporting data is ground-truthed in reality so that it provides reliable evidence to
inform robust evaluations.
25
http://www.abs.gov.au/
26
https://www.elsevier.com
Advance Queensland Evaluation Framework – July 2019 32
7. Governance
As a number of agencies share responsibilities for the implementation of Advance Queensland initiatives, a
coordinated approach for monitoring, review and evaluation is required.
The table below outlines the relationship of Advance Queensland evaluation to the Advance Queensland
Governance Arrangements
27
.
Table 7.1 – Advance Queensland Governance Arrangements relationship to Advance Queensland evaluation
Title
Role statement/s
Evaluation responsibilities
Advance Queensland
Leadership Group
• Provides strategic direction
for the Advance Queensland
initiative
• Consider significant evaluation findings
to inform the development policy
options and future directions for
Advance Queensland
Advance Queensland
Steering Committee
• Providing leadership and
oversight in the delivery of
Advance Queensland
initiative, programs and
activities
• Approve Advance Queensland
Evaluation Framework, Strategies and
Plan
• Ensure a coordinated whole-of-
government approach to evaluation of
the Advance Queensland initiative
• Endorse/ approve major evaluation
reports.
• Review key evaluation reports.
• Address strategic and directional risks
and issues relating to evaluation, as
escalated by the Evaluation Sub-
Committee
Evaluation Sub-
Committee
• Provides oversight of
evaluation activities
• Endorse the Advance Queensland
Evaluation Framework, Plan and
guidance material
• Review, update and oversee the
implementation of the Advance
Queensland Evaluation Plan
• Review and endorse evaluation plans
and reports
• Provide guidance and support on
evaluation principles and practices
• Address risks and issues as relevant to
monitoring and evaluation and escalate
when appropriate
Advance Queensland
Implementation Unit
• Provides whole-of-initiative
coordination through:
o management strategies
and guidance material
• Commission and provide oversight of
key macro-level evaluations (as
outlined in the Evaluation Plan)
• Provide standardised and routine
reports on the implementation and
performance of Advance Queensland
27
Queensland Government. Advance Queensland Handbook (unpublished)
Advance Queensland Evaluation Framework – July 2019 33
Title
Role statement/s
Evaluation responsibilities
• maintenance and provision of
key program information and
data
• Coordinate the collection and
dissemination of key macro/system
level data sets
• Provide guidance and support on
evaluation principles and practices
Implementing
agencies and
program managers
• Planning, implementation,
monitoring, evaluation and
reporting on Advance
Queensland programs and
activities
• Develop program logics and evaluation
plans for meso-groups and priority
programs (as outlined in the Evaluation
Plan)
• Conduct or commission evaluations.
• Conduct data collection, analysis and
validation
• Provide regular implementation and
performance reports
Program recipients
• Plan, deliver and report on
Advance Queensland projects
and activities as per contacts
• Provide regular implementation and
performance reports
• Contribute to data collection(e.g.
through surveys and interviews
Universities,
research institutes
and collaboration
partners
• Conduct data collection,
analysis and validation
• Provide data, information and reports
that may be used to evaluate the
Advance Queensland initiative
7.1 Roles and responsibilities
Effective evaluation requires clear governance so that key points of accountability are defined and
documented, and stakeholders understand their roles and responsibilities, including leadership for
evaluation, who has ultimate responsibility for the evaluation activity, who is responsible for undertaking
evaluation activities.
The following table outlines the key tasks or activities and the responsible stakeholders.
Table 7.2 – Key tasks and activities in planning and implementation of Advance Queensland evaluation
Task or activity
Responsible stakeholder
Description of role
Lead the development of
whole-of-initiative
evaluation frameworks,
plans and guidance
material
• Advance Queensland
Implementation Unit, DITID
• AQ Evaluation Sub-
Committee
• AQ Steering Committee
• Develop, endorse and/or approve:
- AQ Evaluation Framework
- AQ Evaluation Plan
- Other materials as appropriate
Provide oversight of all
evaluation activities
• AQ Evaluation Sub-
Committee
• Review, update and oversee the
implementation of the AQ Evaluation
Plan
• Review and endorse evaluation
plans and reports
• Provide guidance and support on
evaluation principles and practices.
Advance Queensland Evaluation Framework – July 2019 34
Task or activity
Responsible stakeholder
Description of role
Commission and provide
oversight of key macro-
level evaluations
• Advance Queensland
Implementation Unit, DITID
• Department of the Premier
and Cabinet
• Queensland Treasury
• Develop program logics and
evaluation plans
• Conduct or commission evaluations
• Conduct data collection, analysis
and validation
• Engage stakeholders and
communicate evaluation findings.
Undertake/commission
meso-level and priority
micro-level evaluations
• Implementing agencies and
program managers
• Develop program logics and
evaluation plans
• Conduct or commission evaluations
• Conduct data collection, analysis
and validation
• Engage stakeholders and
communicate evaluation findings.
Coordinate the collection
and dissemination of key
macro/system level data
sets.
• Advance Queensland
Implementation and Policy
Units, DITID
•
Advance Queensland Evaluation Framework – July 2019 35
7.2 Risk and issue management
Governance arrangements should also consider risks and issues, including ethical considerations of
conducting evaluations.
7.2.1 Management of risks and issues
Risks and issues associated with the evaluation of Advance Queensland initiative, programs and activities
should be managed in accordance with the Advance Queensland Risk and Issue Management Strategy
28
.
At the highest level, this involves:
• Identifying what may happen (risks) /has happened (issues) and record in the risk/issue register
• Analysing the consequences/likely impact
• Evaluate and develop treatment options
• Treat by implementing agreed management responses
• Communicate and consult with relevant stakeholder
• Monitor and review the effectiveness of management responses and remaining risk/issue levels.
The Advance Queensland Evaluation Sub-Committee is responsible for identifying and addressing high-level
risks and issues relating to monitoring and evaluation. The AQIU will undertake an analysis and evaluation of
the proposed risks/issues, and provide to the AQ Evaluation Sub-Committee for consideration. If accepted,
the risks and issues will be recorded in the relevant register and presented to the AQ Evaluation Sub-
Committee as a standing agenda item at each meeting.
Where appropriate, significant high-level risks and issues relevant to monitoring and evaluation of AQ
programs and activities will be escalated to the AQ Steering Committee, and to the AQ Strategic Leadership
Group.
Program level risks and issues, including those relating to monitoring and evaluation of programs, are to be
managed locally by the program team as per the risk management process within the implementing agency.
Significant program level risks and issues relating to monitoring and evaluation may be escalated to the AQ
Evaluation Sub-Committee via the Secretariat, or raised directly at meetings.
Table 7.3 – Roles and responsibilities for managing evaluation risks and issues
Governance group
Relevant roles and matters for consideration
AQ Strategic
Leadership Group
• Resolving strategic and directional issues between departments delivering
AQ programs and activities
• Considering significant strategic risks and issues.
AQ Steering
Committee
• Addressing strategic and directional issues between departments delivering
AQ programs and activities
• Addressing program risks and issues as escalated by implementing
departments
• Considering significant operational risks and issues.
28
Queensland Government. Advance Queensland Handbook (in development)
Advance Queensland Evaluation Framework – July 2019 36
Governance group
Relevant roles and matters for consideration
AQ Evaluation
Sub-Committee
• Considering risks and issues relevant to monitoring and evaluation of AQ
programs and activities
• Addressing evaluation risks and issues as escalated by implementing
departments
• Providing advice to the AQ Steering Committee on significant strategic and
directional issues relating to monitoring and evaluation of AQ programs and
activities.
Implementing
Agencies
• Resolving strategic and directional issues within departments delivering AQ
programs and activities
• Defining the acceptable risk profile thresholds for the program and related
activities
• Addressing program risk and issues as necessary and escalate when
appropriate.
7.2.2 High-level risks and issues
While identification and management of risks and issues is an iterative process, the following have been
identified as current high-level risks relevant to the evaluation of all Advance Queensland activities.
These risks will be assessed and managed through the progression of two supporting programs of work.
Table 7.4 – High-level risks relevant to all Advance Queensland activities
Risk
Risk description
Management strategy
Financial resourcing
There is a risk that there is inadequate
financial resourcing to complete evaluation
activities required to adequately evaluate
the Advance Queensland initiative.
This may impact the number of evaluation
activities completed and the quality and/or
independence of the evaluations.
Risk to be assessed and
managed through Supporting
Program of Work 1 – Evaluation
capability, capacity and
resourcing
Evaluation capability,
capacity
There is a risk that there is inadequate
capability/capacity to undertake evaluation
activities required to adequately evaluate
the Advance Queensland initiative.
This may impact the number of evaluation
activities completed and the quality of the
evaluations.
Risk to be assessed and
managed through Supporting
Program of Work 1 – Evaluation
capability, capacity and
resourcing.
Data/data quality
There is a risk that there is/will be
inadequate data/ data quality to undertake
evaluation activities required to adequately
evaluate the Advance Queensland initiative.
This may impact the quality of evaluations
activities and outcomes.
Risk to be assessed and
managed through Supporting
Program of Work 2 – Evaluation
methodologies, metrics and data.
Advance Queensland Evaluation Framework – July 2019 37
7.2.3 Ethical and cultural considerations
Planning, implementation and reporting of evaluations must also consider the potential risk of harm to people
participating in the evaluation, whether as informants or as evaluators. The types of harm can range from
loss of privacy or benefits to program participants, damage to vulnerable groups, or physical or mental harm
to informants or researchers.
During the evaluation design step, it is critical to identify:
• Whether external ethics review is required
• If there vulnerable or culturally distinct groups involved
• Consent and privacy issues regarding data and information.
Sources of advice for evaluations to consider ethical and cultural considerations include:
• Australasian Evaluation Society – Guidelines for the Ethical Conduct of Evaluation
29
.
• Australian Institute of Aboriginal and Torres Strait Islander Studies – Guidelines for Ethical Research
in Australian Indigenous Studies
30
.
• The Information Privacy Act 2009 and the Office of the of the Information Commissioner
31
29
Australasian Evaluation Society (2013). Guidelines for the Ethical Conduct of Evaluation.
https://www.aes.asn.au/images/stories/files/membership/AES_Guidelines_web_v2.pdf
30
Australian Institute of Aboriginal and Torres Strait Islander Studies (2012) Guidelines for Ethical Research in
Australian Indigenous Studies https://aiatsis.gov.au/sites/default/files/docs/research-and-guides/ethics/GERAIS.pdf
31
https://www.oic.qld.gov.au/about/privacy
Advance Queensland Evaluation Framework – July 2019 38
8. Stakeholder engagement and communication
Commitment and cooperation from stakeholders is key to a successful evaluation. Evaluation plans should
include a stakeholder consultation and communication plan with a view to ensuring stakeholder ‘buy in’ and
ownership of evaluation activity.
Consideration should be given to who will be responsible for engaging stakeholder groups and ensuring they
have a clear understanding of their respective responsibilities.
Consideration should also be given to how and when evaluation findings will be disseminated.
8.1 Communication plan
A communication plan helps with managing communications during an evaluation, ensuring that the
messages are consistent and the right messages are disseminated to the appropriate audiences at an
appropriate time.
When devising a communication plan, consideration of audience is key. Different evaluation audience
require different levels of detail and mediums for communicating evaluation results. The most appropriate
forum or method of communicating the results should be chosen for each stakeholder group, and
communication methods should be determined during the planning phase of the evaluation activity.
Communication plans should:
• Determine the desired outcomes of communication, including
32
:
o Boosting awareness of the program and its outcomes
o Outlining the rationale of a program to stakeholders, to bolster understanding
o Encouraging action (from funders, stakeholders, industry or other target audiences)
• Identify the audiences required to achieve desired outcomes, including
33
:
o Staff internal to the program
o Broader management and staff members of a department
o Key stakeholders (both internal and external, e.g. industry members, program participants,
funding bodies)
• Establish governance over communication
o Who is in charge of each aspect of communication?
o Is there sensitive information involved; if so, how will the associated risks be managed?
• Outline content, timeframes and channels for communication required to achieve the previously
identified outcomes
34
o What needs to be communicated?
o When does it need to be communicated?
o How will it be communicated?
• Establish mechanisms for two-way communication (i.e. how stakeholders are able to provide
feedback, and how this feedback is to be incorporated).
The communication plan should be considered within the context of the program and throughout the program
– rather than an afterthought. If communication around a particular outcome is desired, it is necessary that
data or information informing progress toward achieving the outcome is collected over the course of the
program and evaluation.
32
Myers, P. and Barnes, J. (2004). Sharing Evaluation Findings: Disseminating the Evidence
33
Rural Health Innovations. (2016). A Guide to Writing a Program Evaluation Plan
34
BetterEvaluation. (2016). Communication plan
Advance Queensland Evaluation Framework – July 2019 39
8.2 Dissemination of evaluation findings
For evaluation findings to be useful they must be shared. The final step in performing a high quality
evaluation is effectively communicating the findings of evaluation activity to the appropriate audiences.
The goal of communications will be audience dependent. For the staff of an individual program, sharing
evaluation findings might inform program re-design. For government, evaluation may act as feedback and
inform the development of future policy. For the general public, the goal of communication might be raise
awareness for the effectiveness and success of an Advance Queensland initiative.
Communicating evaluation findings with the public, where they are non-sensitive in nature and it is deemed
appropriate to do so, strengthens engagement and builds awareness of the program.
Evaluation findings can be disseminated through a broad range of internal and external channels, and
communication needs to be tailored in each instance to ensure that the appropriate messages are conveyed.
Table 8.1 – Examples of dissemination strategies for key participants in the innovation system
Queensland
community
Startups and
Entrepreneurs
Business and
Industry
Investors
Government
Universities
and R&D
Future
Innovators
Formal evaluation
reports
Summary
evaluation reports
Learn and share
workshops
Webinars
Social media
Other interactive
online content
Advance Queensland Evaluation Framework – July 2019 40
9. Evaluation resources
Evaluation can be a long-term process and requires adequate resources, including:
• dedicated human resourcing to coordinate and oversee evaluation activities and conduct internal
evaluation activities
• financial resources to procure independent externals services to conduct evaluations, develop/verify
appropriate evaluation metrics and methodologies, and provide training and skills development
• financial resources to acquire appropriate data sources and acquire/enhance data management
systems.
The Supporting Program of Work 1 – Evaluation capability, capacity and resourcing has been established to:
• confirm the human and financial resourcing required to adequately evaluate the Advance Queensland
initiative
• build the evaluation culture and skills within implementing agencies
• develop a long-term, staged resourcing and sourcing strategy for the evaluation of Advance
Queensland.
9.1 Internal evaluation capability/capacity
9.1.1 Implementing agencies
All implementing agencies are responsible for undertaking and/or commissioning evaluation activities. The
evaluative effort required by each agency will be dependent upon the scope and nature of the programs
administered by the agency.
9.1.2 Evaluation guidance and support
The following Queensland Government governance groups and work units provide additional evaluation
guidance and support for Advance Queensland evaluation activities:
Advance Queensland Evaluation Sub-Committee – provides guidance and support on evaluation
principles and practices, providing strategic advice on resourcing strategies for evaluation activities
• Advance Queensland Implementation Unit, Innovation Division, DITID – provides whole-of-
initiative coordination through management strategies and guidance material; commissioning and
oversight of macro-level and other key evaluations and standardised and routine implementation and
performance reports.
• Innovation Policy Unit, Innovation Division, DITID – coordinates the collection and dissemination
of key macro/system level data sets
• Queensland Government Statistician’s Office – provides a range of statistical services to support
stakeholders’ evidence base for policy evaluation and performance.
9.2 Financial resources
9.2.1 Whole-of-initiative
In 2015-16, funding of $1.1 million was allocated for evaluation of the Advance Queensland initiative. This
funding is utilised for core evaluation activities, including (but not limited to):
• development and review of key evaluation frameworks, plans and guides
• commissioning independent macro-level evaluations
• acquisition of key data sets
• development of suitable and accurate innovation metrics and methodologies.
Through the 2018-19 budget process, a further $1 million was provided to enable the implementation of the
first tranche of priority evaluation activities, in line with recommendations provided by Deloitte Access
Advance Queensland Evaluation Framework – July 2019 41
Economics. This funding has been used to form a Cross-Agency AQ Evaluation Fund, managed by the
AQIU, with allocations overseen by the Evaluation Sub- Committee in line with agreed priorities.
Appropriate allocations from the Fund to priority evaluations will be determined based on the level of
evaluation (refer Section 3.3) and relative complexity (refer Section 4.2).
9.2.2 Departmental/program specific
All implementing agencies are responsible for appropriately identifying and managing financial resources for
evaluation activities at a departmental and program level. Queensland Treasury advise that agencies should
consider appropriate allocations for evaluation activities when requesting funding for new or continuation of
existing programs.
9.3 External/independent resources
External resources may be required to undertake evaluation activities, particularly where evaluation is
sensitive, complex or requires specific technical expertise.
External resources may also provide independent peer review of evaluation plans and activities.
Additionally, collaboration partners may provide data, information and reports or innovation metrics and
methodologies that may be used to evaluate the Advance Queensland initiative.
Sources of external evaluation resources include:
• Consultants and contractors
• Australian Evaluation Society
• Universities, research institutes and collaboration partners.
Advance Queensland Evaluation Framework – July 2019 42
Appendix 1 – Glossary of key terms
The following glossary has been adapted from the Queensland Government Program Evaluation
Guidelines
35
and provides definitions of key terms as they apply to the evaluation of Advance Queensland.
Term
Definition
Advance
Queensland
Advance Queensland is a flagship Queensland Government initiative, designed to
promote innovation and build a stronger and more diversified Queensland
economy, creating jobs now and for the future.
Counterfactual
An estimate of what would have happened in the absence of the policy and
associated programs.
Effectiveness
The extent to which a program is responsible for a particular outcome or
outcomes. To ascertain effectiveness requires consideration of other potential
influences on the outcomes of interest and the counterfactual (what would have
happened in absence of the program).
Efficiency
The extent to which a program is delivered at the lowest possible cost, to the
areas of greatest need, and continues to improve over time by finding better or
lower cost ways to deliver outcomes.
Evaluation
The systematic, objective process of understanding how a policy or other
intervention was implemented, what effects it had, for whom, how and why.
Evaluation activities can occur before, during or after implementation, and may
include an assessment of the appropriateness, relevancy, process, effectiveness
and/or efficiency of a program.
Evaluation process
The steps involved in planning and conducting an evaluation, and disseminating
evaluation findings.
Foundations and
administrative
activities
Activities undertaken to support the delivery and governance of the Advance
Queensland initiative, including secretariat support to key governance groups,
grant administration and assessment, supporting and promoting significant
events, programs, and achievements, sponsorship activities.
Impact evaluation
Assessment of whether the program was effective in meeting its objectives and
achieving its ultimate goals.
Monitoring
A systematic and ongoing process of collecting, analysing and using information
about the progress of development activities over time, to help guide activities and
improve programs, projects and initiatives.
Outcomes
The short, medium and/or long term results generated as a direct result of the
delivery of a program (i.e. what difference the program made). Possible outcomes
of programs can include changes in awareness, knowledge, skills, attitude and
behaviour, as well as economic environmental and social impacts. For the
purposes of the guidelines, the terms outcomes and impacts are used
interchangeably.
35
Queensland Treasury (Queensland Government) (2014). Queensland Government Program Evaluation Guidelines
https://s3.treasury.qld.gov.au/files/qld-government-program-evaluation-guidelines.pdf
Advance Queensland Evaluation Framework – July 2019 43
Term
Definition
Outputs
The services or facilities provided as a result of a program’s processes or
activities. Outputs capture what the program does and who it reaches, rather than
what difference the program made (i.e. outcomes).
Objective
Key elements to be achieved across all Advance Queensland programs, aligned
to a particular strategy.
Policy
A statement of Government intent in relation to an issue, which can be
implemented through the use of policy instruments, such as laws, advocacy,
monetary flows and direct actions. The development and implementation of
programs is one way that Government can act in response to a policy decision.
Program
A discrete set of activities created in response to an identified need and/or
targeting weaknesses in the innovation system, create economic and/or social
value. Types of programs delivered under the Advance Queensland initiative
include grants, partnerships, competitions, procurement, events and
sponsorships.
Program design
The process undertaken to develop a program prior to program implementation.
Program design will often include development of an implementation plan,
consideration of resource or training requirements, and agreement on a
governance structures. Program design should also include the development of
an evaluation plan. Program design is also commonly referred to as program
development or program planning.
Program logic
A method to assist program design. It depicts the logic or pathways through which
the programs processes (inputs, activities and outputs) are intended to achieve
the desired outcomes. Logic models can assist in understanding how the program
is intended to work, what it is trying to achieve and why. Program logic is also
commonly referred to as program theory or service logic.
Strategy
A plan of action to achieve the vision of Advance Queensland by targeting
weaknesses in the innovation system and creating economic and social value
which would otherwise not have occurred.
Theme
Categories under which Advance Queensland programs are grouped for
governance and reporting purposes. While individual programs may contribute to
one or more of the Advance Queensland strategies and objectives, each theme is
aligned to a single strategy and associated objectives.
Advance Queensland Evaluation Framework – July 2019 44
Appendix 2 – Macro measures/system indicators
The following tables provide examples of high-level system indicators aligned to the Advance Queensland
strategies and objectives. This is not intended to be an exhaustive list but rather provides initial high level
guidance to support the identification of indicators for evaluation activities.
Two lists are provided:
1. System indicators and macro measures currently available
2. Additional measures that could be explored or developed
A2.1 Currently available macro measures and system indicators
The following table provide a list of currently available macro measures and system indicators aligned to
Advance Queensland strategies and objectives.
The measures have been selected based on relevance to the objective, availability of the data and likelihood
for ongoing collection. The tables also provide the latest available data for each measure to illustrate the way
in which each measure is expressed and provided a baseline of current performance.
Table A3.1 – Currently available macro measures and system indicators
Objective
System Indicator/
Macro Measure
Source(s)
Baseline
(latest available period)
Strategy: Supporting culture (SC)
SC1 – Increase
innovation
awareness and
engagement
Increased awareness
of science in
Queensland
community
• Queenslanders’
Perceptions and
Attitudes to Science
(Office of the
Queensland Chief
Scientist, 2016)
• Three in four (74%)
Queenslanders are somewhat
or very interested in science
• 72% believe science is critical
for our economy
• Interest levels are higher
amongst males (79%)
compared to females (70%)
• Those aged 18-24 years are
less interested in science
than other age groups (65%)
Increased awareness
of science in regional
Queensland
• Queenslanders’
Perceptions and
Attitudes to Science
(Office of the
Queensland Chief
Scientist, 2016)
• Interest in science across the
whole state:
o Greater Brisbane/Gold
Coast/Sunshine Coast
(75%)
o Darling Downs (72%)
o Northern/ Mackay (75%)
o Far North Metro (76%)
o Fitzroy/Wide Bay/Burnett
(69%)
o Remote/Outback
Queensland (66%)
Advance Queensland Evaluation Framework – July 2019 45
Objective
System Indicator/
Macro Measure
Source(s)
Baseline
(latest available period)
Perceptions of
innovation activity
(including Advance
Queensland)
• Queenslanders’
Perceptions &
Attitudes to
Innovation (Colmar
Brunton, 2017)
• 94% of Queenslanders are
somewhat or very interested
in innovation.
• 90% of Queenslanders feel
innovation positively impacts
on themselves and the state.
SC2 – Increase
entrepreneurial-
ism
Value of new and
follow-up investment
in investee
companies
• ABS 5678.0 –
Venture Capital and
Later State Private
Equity Australia
• $286 million in 2015-16
(18.5% of national total)
Business entry and
exit rates
• ABS 8165.0, Counts
of Australian
Businesses,
including Entries
and Exits
• Survival rate (60.2% as at
June 2016; national is 62.1%)
• Entry rate (14.6% in 2015-16;
national is 14.6%)
• Exit rate (12.7% in 2015-16;
national is 12.3%)
Count of co-working
spaces, startup
incubators and
accelerators
• The Fetch – Startup
Incubators and
Accelerators in
Australia
• Counts (data accessed via
The Fetch on 10/1/18):
o Count of co-working
spaces: 23
• Count of startup incubators
and accelerators: 7
Increase in
Queensland’s share
of tech startups
• Startup Muster –
Annual Report
• 20.8% of founders in 2017
Strategy: Building Capacity (BC)
BC1 – Increase
innovation
capability
Gross expenditure on
R&D as a share of
GSP including
business expenditure
on R&D as well as
higher education
• ABS 8104.0 –
Research and
Experimental
Development,
Businesses
• BERD intensity (BERD/GSP)
0.62% in 2015-16
• ABS 8111.0,
Research and
Experimental
Development,
Higher Education
Organisations
• HERD intensity (HERD/GSP)
0.5% in 2014
Advance Queensland Evaluation Framework – July 2019 46
Objective
System Indicator/
Macro Measure
Source(s)
Baseline
(latest available period)
• ABS 8109.0,
Research and
Experimental
Development,
Government and
Private Non-Profit
Organisations
• GOVERD intensity
(GOVERD/GSP) 0.1% in
2014-15
• Health of
Queensland
Science and
Innovation Report
by Office of
Queensland Chief
Scientist
• GERD intensity (GERD/GSP)
1.5% in 2013
Scholarly output per
1000 population
• Elsevier SciVal
• 3.83 in 2017
Share of scholarly
output in top 1% (or
10%) most cited
publications
• Elsevier SciVal
• 1.6% (2017 top 1%)
• 15.6% (2017 top 10%)
Share of SMEs with
new to market
(world) products
• ABS 8158.0 –
Innovation in
Australian Business
NB: Queensland
specific data not
currently available
• 0-4 persons: 7.7%
• 5-19 persons: 8.8% in 2014-
15
BC2 – Develop,
attract and
retain talent
including STEM
Year 12 students
studying identified
science, technology
and mathematics
subjects
• Office of
Queensland Chief
Scientist (Sourced
from: Queensland
Curriculum and
Assessment
Authority)
• 90,508 in 2016
STEM literacy scores
(including National
Assessment Program
(NAP) – Science
Literacy scores)
• NAP Sample
Assessment
Science Literacy
Public report
(ACARA, 2015)
• 398 mean score in 2015
Proportion of
Queenslanders with
a non-school
qualification
• ABS Census of
Population and
Housing
• 54.5% in 2016
Count of university,
TAFE and research
institutions per 1,000
population
• Office of the Chief
Economist, National
Innovation Map
• 0.04 per thousand population
(non-recurring)
Advance Queensland Evaluation Framework – July 2019 47
Objective
System Indicator/
Macro Measure
Source(s)
Baseline
(latest available period)
Strategy: Fostering Collaboration (FC)
FC1 – Build
sustainable
partnerships to
deliver
outcomes
Share of businesses
conducting
innovative activity
• ABS 8166.0 –
Summary of IT Use
and Innovation in
Australian Business
NB: Queensland
specific data not
currently available
• 48.7% in 2015-16
• 45.0% in 2014-15
Share (%) of
Queensland
scholarly outputs with
international co-
authorship
• Health of
Queensland
Science and
Innovation (Office of
the Queensland
Chief Scientist,
2016)
• 52.2% in 2017
• 48.5% in 2015
FC2 – Increase
international
networks
Percentage of R&D
financed abroad for
Higher Education
Expenditure on R&D
(HERD)
• ABS 8111.0 –
Research and
Experimental
Development,
Higher Education
Organisations,
Australia
• $27.1M (11.3% of total
national overseas funding) in
2014
Strategy: Increase investment (II)
II1 – Grow
pipeline of
investible
products and
services
New capital
expenditure attracted
to Queensland
(measured as growth
in business capital
expenditure)
• National Accounts:
State Details –
Queensland
Treasury
• Quarterly growth (2.9% Sept
17)
• Annual growth (9.1% Sept 17)
Spend on innovation
by businesses as a
proportion of GSP
• ABS 8158.0 –
Innovation in
Australian Business
NB: Queensland
specific data not
currently available
• Estimated total expenditure
spent (by Australian
businesses) on innovation in
2014-15 is between $26B to
$30B (approx. 2% of GDP)
More business
investment in R&D
• ABS 8104.0,
Research and
Experimental
Development,
Businesses
• BERD intensity (BERD/GSP)
0.62% in 2015-16
Advance Queensland Evaluation Framework – July 2019 48
Objective
System Indicator/
Macro Measure
Source(s)
Baseline
(latest available period)
Increased investment
in research
• ABS 8111.0 –
Research and
Experimental
Development,
Higher Education
Organisations
• HERD intensity (HERD/GSP)
0.5% in 2014
• ABS 8109.0 –
Research and
Experimental
Development,
Government and
Private Non-Profit
Organisations
• GOVERD intensity
(GOVERD/GSP) 0.1% in
2014-15
II2 – Build
access to capital
Value of venture
capital by investee
company head
offices as a share of
GSP
• ABS 5678.0 –
Venture Capital and
Later Stage Private
Equity, Australia
• $1,1B in 2015-16 or 0.3% of
GSP
Strategy: Scaling for jobs and growth (SJ)
SJ1 – Expedite
commercialisati
on
Nil measures readily
available
SJ2 – Increase
economic
benefits from
innovation
(including jobs)
Jobs driven by
Advance Queensland
programs
• Advance
Queensland
Program Data
• 9,426 as at 30 September
2017
Increase knowledge
based jobs in
Queensland
• Health of
Queensland
Science and
Innovation (Office of
the Queensland
Chief Scientist,
2016)
• 244,000 Queenslanders
worked in knowledge-based
occupations (10.4% of total
workforce, up from 7.6% in
May 2001)
Advance Queensland Evaluation Framework – July 2019 49
A2.2 Additional measures that could be explored or developed
In addition to the available macro measures and system indicators, listed above, a range of additional
measures may also exist or could be developed to further quantify the impact of Advance Queensland. The
following tables provide a list of potential measures and is provided as initial high level guidance to support
the future identification of additional indicators for evaluation purposes.
Table A3.2: Additional measures that could be explored or developed
Objective
System Indicator/ Macro Measure
Possible source(s)
Strategy: Supporting culture
SC1 – Increase
innovation
awareness and
engagement
Increased participation in innovation
initiatives (including Advance
Queensland)
• Program/recipient data
Feedback of participants attending
innovation events
• Program/recipient data
Feedback of partners hosting sessions at
innovation events
• Program/recipient data
SC2 – Increase
entrepreneurialism
Value of funding raised by Queensland
startups
• Program/recipient data
• VC databases or monitoring
services
Survival rate of startups assisted
• Program/recipient data
• Startup databases or monitoring
services
Early stage entrepreneurship activity and
increased number of entrepreneurs
• Program/recipient data
• Startup databases or monitoring
services
Value added of new startup operators
attracted to the State
• Program/recipient data
• Startup databases or monitoring
services
Proportion of participants engaged in
entrepreneurial activities
• Program/recipient data
Strategy: Building Capability
BC1 – Increase
innovation
capability
Number of global and national innovation
awards
• Program/recipient data
• Media monitors
Percentage of population working in
knowledge intensive industries (as a ratio
of the labour force)
• ABS 6291.0 – Labour Force
NB: Requires an agreed definition
of ‘knowledge intensive industries’
Share of innovators in key industry
sectors including emerging knowledge
intensive industries
• ABS 8158.0 – Innovation in
Australian Business
NB: Requires an agreed definition
of ‘knowledge intensive industries’
Share of medium and high technology
industries as a share of GSP
• ABS 5220 – Australian National
Accounts: State Accounts
NB: Requires an agreed definition
of ‘medium and high technology
industries’
Advance Queensland Evaluation Framework – July 2019 50
Objective
System Indicator/ Macro Measure
Possible source(s)
Share of businesses more regularly
innovating (i.e. persistent innovators)
• Requires further investigation
Discovery research spend
• Requires further investigation
BC2 – Develop,
attract and retain
talent including
STEM
Proportion of population workforce holding
STEM field qualifications
• ABS Census of Population and
Housing
NB: Requires an agreed definition
of ‘STEM field qualifications’
Count of individuals per 1,000 knowledge
workers and STEM
• ABS Census of Population and
Housing
• NB: Requires an agreed definition
of ‘knowledge workers’ and ‘STEM’
Increased uptake of STEM careers
• Requires further investigation
Share of workforce that applies complex
skills in their everyday work tasks
• Requires further investigation
Education related government expenses
as a share of GSP
• ABS 5518.0.55.001 - Government
Finance Statistics, Education,
Australia
Aboriginal and Torres Strait Islander
students much more likely to engage with
industry/continue research
• Requires further investigation
Industry/end use partner much more likely
to engage with researchers/graduates
• Requires further investigation
Fellows maintain research careers long
term
• Requires further investigation
Industry/end user partner much more
likely to engage with industry in the future
• Requires further investigation
Strategy: Fostering Collaboration
FC1 – Build
sustainable
partnerships to
deliver outcomes
Ongoing collaborative partnerships after
project completion
• Program/recipient data
Increased commercialisation of research
• Requires further investigation
Number of joint projects between industry,
science and government
• Program/recipient data
• Media monitoring
R&D expenditure of foreign
affiliates/partners of Advance Queensland
programs
• Program/recipient data
FC2 – Increase
international
networks
Proportion of patents with foreign co-
investors
• Intellectual Property Government
Open Data
Foreign Direct Investment (FDI)
investment inflows related to innovation
activities
• Requires further investigation
Exports of knowledge intensive industries
• Requires further investigation
NB: Requires an agreed definition
of ‘knowledge intensive industries’
Increased research outcomes from
international collaborations
• Requires further investigation
Advance Queensland Evaluation Framework – July 2019 51
Objective
System Indicator/ Macro Measure
Possible source(s)
Strategy: Increase investment
II1 – Grow pipeline
of investible
products and
services
Growth in re-investment in
innovation/knowledge precincts
• Requires further investigation
II2 – Build access
to capital
Success stories generating inward
investment related to innovation activities
• Program/recipient data
• Media monitoring
New discoveries are quicker to market
• Requires further investigation
More investment raised by startups
• Program/recipient data
• Startup databases or monitoring
services
Increased startup investment
• Program/recipient data
• Startup databases or monitoring
services
Strategy: Scaling for jobs and growth
SJ1 – Expedite
commercialisation
Total revenue generated by patents per
1000 population
• Requires further investigation
Increasing number of discoveries entering
the innovation system
• Requires further investigation
Translation of research into
commercialisable products
• Requires further investigation
SJ2 – Increase
economic benefits
from innovation
(including jobs)
Multi-factor productivity
• Requires further investigation
Increased retention of wealth capture by
Queensland science and research
• Requires further investigation
Advance Queensland Evaluation Framework – July 2019 52
Appendix 3 – National Innovation Datasets
The following data sources were developed by the Commonwealth Department of Industry, Innovation and
Science are increasingly used to evaluate innovation in Australia. However, the availability of state level data
through these datasets varies and in most instances is not readily available to support the evaluation of state
level programs.
A3.1 National Innovation Map
The National Innovation Map provides easy to understand information on innovation in Australia’s regions.
The interactive map visually presents new business creation, expenditure on research and development,
patenting activity and trademarking activity for each statistical region in Australia. The map lets users tailor
their information by indictor, year or state, territory and suburb. The map highlights innovation hot spots and
regional innovation trends.
Studies using the map show that innovation and entrepreneurship are concentrated in the major metropolitan
areas of Australia, and there is a correlation between research institutions and higher innovation activity and
creation of new businesses. Queensland is performing extremely well in terms of new business entries
especially in regional areas, as shown in the National Innovation Map snapshot in Figure 5.2.
Figure A4.1: Business Entries per 10,000 inhabitants, averaged 2009-2014, by Statistical Area 3 regions
A3.2 BLADE
The Business Longitudinal Analysis Data Environment (BLADE) is a series of integrated, linked longitudinal
datasets that contains administrative data on more than two million actively trading Australian businesses
over the period of 2001-02 to 2012-13. The data set combines administrative data from the Australian
Taxation Office (ATO) with primary survey data to increase the research capacity of businesses to undertake
firm- level evaluations and to offer a robust evidence base for broader policy evaluation and decision making.
BLADE offers decision makers:
• Data on actively trading businesses’ turnover, employment and labour productivity from 2000-2001
to now to analyse firm performance overtime
• Reports on export status, foreign ownership status and innovation status as well as the size and
industry distribution of program participants.
• Trend data on the number of new business entries, business startup rates and the survival rates of
such business over time
Advance Queensland Evaluation Framework – July 2019 53
A3.3 Innovation Insights Database
As part of the annual Australian Innovation System Report, the department of industry, Innovation and
Science release an Innovation Insights Database to provide greater information and understanding to policy
makers, academics and other who are interested in Australia’s innovation history.
A sample of the indictors are presented in Table A5.1, these indicators are compared against all countries in
the OECD, as well as Singapore, China and Taiwan (when data is available).
Table A4.1: Sample of indicators in the Innovation Insights Database
Measurements
Sample of indicators
Outcomes
• GDP per capita relative to the USA (USA = 100) (index)
• Real GDP growth
• UNDP Human Development Index
• Gini coefficient
Innovation and
entrepreneurship
activity
• Percentage of innovation-active large firms
• Proportion of businesses introducing operational/ process innovation
• Innovation Patents by AU residents
International
engagement
• Trade, % of GDP
• Net Foreign Direct Investment Inflows (% of GDP)
• Proportion of patents with foreign co-inventors
• Short term business trips churn
Business collaboration
activity by innovation-
active businesses
• Percentage of innovation-active SMEs collaborating on innovation
• Percentage of innovation-active total businesses with international
collaboration on innovation
Framework conditions
in Australia
• Operating surplus (% of GDP)
• NAB Index of capacity utilisation
• Barrier to innovation: Lack of access to additional funds (% of
respondents)
Education and skills
base
• Proportion of population aged 25-64 attaining tertiary education
• Barrier to innovation: Lack of skilled persons in any location (% of
respondents)
Investment in research
• Business expenditure on R&D (BERD)
• Higher education expenditure on R&D (HERD)
• Government Budget Appropriations or Outlays for R&D (GBAORD)
Research workforce
• PhD graduation rate
• Proportion of international students enrolled in advanced research
programs
• Researchers (% of total labour force)
Research publications
• Share of world publications
• Share of world's top 1% highly cited publications
• Top 1% publications per Bn PPP GERD (excluding BERD)
• Proportion of publications in top 10%
Research
commercialisation
outcomes
• Number of startup companies in which major publicly funded research
• agencies, universities and medical research institutes have an equity
holding