SDI performance and assessment


1	Introduction
2	Why SDI assessment is important
3	The span of performance
4	The depth of performance

1. Introduction

Why SDI assessment is important

The span of performance

The depth of performance

SDI performance and assessment

Is about providing an answer to questions such as:

How is my SDI performing?
How is an SDI doing in comparison with other SDI’s?
Is my SDI performing better than five years ago?
Which SDI is the best SDI in the world?
Why is this the best SDI in the world?
What should I do to make my SDI better?

Learning activity (introduction)

Go to:

http://opendatabarometer.org/ (home page)

https://index.okfn.org/ (“places’ or ‘compare countries’)

Questions

Who is leading?

How is your country doing?

What does this all mean (e.g. leading in what?)?

Where’s the ‘spatial’ component?

Learning activity (introduction)

SDI assessment

SDI assessment is about measuring the performance of an SDI

SDI assessment deals with collecting, interpreting, sharing and using information on the performance of – particular aspects of – an SDI

History of SDI assessment

1. Introduction
2. Why SDI assessment is important
3. The span of performance
4. The depth of performance

Why assessment is important

Information on the performance (of government initiatives and policies) can be used in many different ways

In general, there are three main categories of using performance information:

retrospective achievement: letting decision makers know in comparative terms how their country or organization is performing
prospective direction/priorities: assisting policy makers and practitioners with strategic decision making and future implementation projects
accountability: enabling governments and agencies to be held to account for the resources they have invested

To evaluate
To control
To budget
To motivate

To promote
To celebrate
To learn
To improve

Incorporation of performance information

Incorporation = intentionally importing performance-related data and information in documents and procedures with the potential and purpose of using them

Ensuring/institutionalizing the use of performance information

Examples:
- In the policy cycle: performance information is included in policy preparation, implementation, monitoring and evaluation
- In the contract cycle: performance information is included in the design, implementation and evaluation of contracts between different parties

Example: INSPIRE

INSPIRE Directive art. 17

Network services are necessary for sharing spatial data between the various levels of public authority in the Community. Those network services should make it possible to discover, transform, view and download spatial data and to invoke spatial data and e-commerce services. The services of the network should work in accordance with commonly agreed specifications and minimum performance criteria in order to ensure the interoperability of the infrastructures established by the Member States. The network of services should also include the technical possibility to enable public authorities to make their spatial data sets and services available.

Example: INSPIRE (2)

Implementing Rule on Network Services

1. PERFORMANCE

The response time for sending the initial response to a Discovery service request shall be maximum 3 seconds in normal situation.
For a 470 Kilobytes image (e.g. 800 × 600 pixels with a colour depth of 8 bits), the response time for sending the initial response to a Get Map Request to a view service shall be maximum 5 seconds in normal situation.

Normal situation represents periods out of peak load. It is set at 90 % of the time.

2. CAPACITY

The minimum number of served simultaneous requests to a discovery service according to the performance quality of service shall be 30 per second.

The minimum number of served simultaneous service requests to a view service according to the performance quality of service shall be 20 per second.

3. AVAILABILITY

The probability of a Network Service to be available shall be 99 % of the time.

1. Introduction
2. Why SDI assessment is important
3. The span of performance
4. The depth of performance

Assessment is about performance

Many different meanings of performance:

Between disciplines: geomatics, spatial planning, economics, social sciences, managerial sciences
Within one discipline: no consensus on the meaning of SDI performance

To create some clarity, a distinction can be made between

The span of performance
The depth of performance

Span of performance

Horizontal dimension of performance

Refers to performance in terms of

1. inputs that are processed in activities,

2. these activities result in outputs (products or services)

3. which finally (should) lead to certain outcomes

Span of SDI performance

‘Performance’ of SDI can be defined and measured in various ways:

Readiness of the infrastructure
Organizational maturity of the infrastructure
Suitability of the geoportal
Level of implementation of different components
Availability of geographic data
Application of principles of open data
Number of users of geographic data
Use of geographic data in society
Social benefits of the infrastructure
Economic benefits of open geographic data

Span of SDI performance

Thinking in terms of ‘input – output – outcome - impact’ to create some clarity

Input: technological and non-technological components of an SDI

Output: availability and accessibility of spatial data

Outcome: use of spatial data by public administration, businesses, citizens, …

Impact: internal and external benefits of spatial data

SDI assessment

Span of SDI performance

‘Performance’ of SDI can be defined and measured in various ways:

Readiness of the infrastructure
Organizational maturity of the infrastructure
Suitability of the geoportal
Level of implementation of different components
Availability of geographic data
Application of principles of open data
Number of users of geographic data
Use of geographic data in society
Social benefits of the infrastructure
Economic benefits of open geographic data

Readiness

Data

Use

Impact

Notes for slide 21 21
It can be seen that many of the ways SDIs are assessed (i.e. the scope of the assessment and the indicators used) are linked to different elements of the span of performance (i.e. readiness, data, use and impact)
Notes for slide 22 22
It can be seen that many of the ways SDIs are assessed (i.e. the scope of the assessment and the indicators used) are linked to different elements of the span of performance (i.e. readiness, data, use and impact)
Notes for slide 23 23
It can be seen that many of the ways SDIs are assessed (i.e. the scope of the assessment and the indicators used) are linked to different elements of the span of performance (i.e. readiness, data, use and impact)
Notes for slide 24 24
It can be seen that many of the ways SDIs are assessed (i.e. the scope of the assessment and the indicators used) are linked to different elements of the span of performance (i.e. readiness, data, use and impact)
Notes for slide 25 25
It can be seen that many of the ways SDIs are assessed (i.e. the scope of the assessment and the indicators used) are linked to different elements of the span of performance (i.e. readiness, data, use and impact)

1. Introduction
2. Why SDI assessment is important
3. The span of performance
4. The depth of performance

Depth of performance

Vertical dimension of performance

Performance can be measured - and managed - at different – (interrelated) levels

Typical levels in public sector performance:

Micro performance is at the level of an organisation
Meso performance is at the level of a policy area
Macro performance is government (or infrastructure) wide

Depth of performance

Apply the input – output – outcome scheme at the three levels of an individual organisation, a policy field and the consolidated government-wide level

Need to match input, activities, output and outcome/effects at micro, meso, and macro level

Depth of SDI performance

Also the performance of SDIs can be measured at different levels:

Data sets
Organizations
Policy chains and policy areas
Government wide
Country wide (government + private sector + other sectors)
Globally

Example: INSPIRE

Learning activity (conclusion)

Source: Grus, 2010

Questions:

Which of the countries included in the table on the right has the best SDI? And why?

The assessment approaches used in the table apply a different span of performance. What does this mean? How could this affect the results of the assessment?

The assessment approaches could also apply a different depth of performance. What does this mean? How could this affect the results of the assessment?

What could be other reasons why different assessments of the same SDIs provide different results?

Reference list

Crompvoets, J., Rajabifard, A., Van Loenen, B., & Fernández, T. D. (2008). A multi-view framework to assess SDIs. Wageningen University, RGI.

Giff, G. A., & Crompvoets, J. (2008). Performance indicators a tool to support spatial data infrastructure assessment. Computers, Environment and Urban Systems, 32(5), 365-376.

Vancauwenberghe, G., Valečkaitė, K., Van Loenen, B., & Donker, F. W. (2018). Assessing the Openness of Spatial Data Infrastructures (SDI): Towards a Map of Open SDI. International Journal of Spatial Data Infrastructures Research, 13, 88-100.