[1.1] Welcome

Preamble

Disclaimer

• IPRs All intellectual property rights relating to the content of the course, its structure and chapter headings, the figures and illustrations, the written and oral statements, as well the conceptual design of the exercises and examples, are with the authors.

• Lineage A first version of this course was built based on the PhD work of S Lang and teaching material for courses on remote sensing delivered by S Lang and T Blaschke between 2002 and 2006. The present version 2.0 is based on material from University courses taught by S Lang and D Tiede, including an online OBIA course from 2010 onwards. The course also contains conceptual material originating from Master theses done at Z_GIS in the field of OBIA, notably from F Albrecht and M Hagenlocher. The latest update and conversion to reveal.js has been accomplished with support of A Schlagbauer and V Streifeneder between 2020 and 2022, as a contribution to the EO4GEO Skills Alliance.

• Usage The course shall be used by students to self-achieve the given learning objectives. Using the material in the broader academic context is encouraged, when properly cited.

• Reuse and reproduction The online material of this course (including slide-deck and explaining text) has been released under a creative-commons share-alike principle. All rights remain with the authors. Please contact us for granting access to the Git repository of this course.

• Liability Authors do not accept any liability for misuse or inappropriate usage of the described techniques, nor for whatsoever failure when applying these in academic, professional, or any other context.

[1.2] Learning objectives

Learning objectives

• This course aims at introducing the field of object-based image analysis providing a comprehensive overview of the techniques and theoretical background.

• Specific learning objectives include:

1. Remember how the approach of object-based image analysis has emerged from bridging remote sensing and geoinformatics

2. Understand the scientific foundations (hierarchy theory, human perception, knowledge representation, computer vision) for gaining comprehensive background knowledge for applying the tools.

3. Apply various tools and techniques in geographical applications and related fields.

4. Compare OBIA methods to pixel-based remote sensing concepts (e.g. multispectral classification, accuracy assessment) and understand how the latter needs to be adapted

5. Practice OBIA workflows in a dedicated software environment mimicking real-world scenarios.

Course structure

• The course contains both theoretical background and practical exercises:

• Theoretical chapters

  • Oral explanations, commented slide sequences, and various sources of other material

  • Designed as a foundation for handling the challenges of practical exercises.

• Practical exercises

  • Self-explanatory with step-by-step instructions for getting acquainted with relevant software and tools.

  • Designed to demonstrate potential and limitations of OBIA and to develop a critical thinking towards the presented routines.

[1.3] Content overview

Table of contents

Assignments

• Course assignments are based on three components
1. Answering conceptual/theoretical questions
2. A (short) literature review
3. The completion (and documentation) of practical exercises

40%

20%

40%

[1.4] Objects in our life

Objects in our life

Why are numbers not considered objects? When and how do these numbers turn into meaningful representations of real-world units?

Objects in our life

Objects in our life

• Forest stands
• Different forest structures including spacious or closed forests, multiple tree canopy layers
• Scale-specific delineation of stands



Different forest stand types in the Bavarian Forest national park, Germany

expert delineation

automated delineation

Tiede et al. 2004, Lang et al. 2006

[1.5] Example: landscape objects

Landscape objects

Landscape objects

Landscape objects

Landscape objects

Landscape objects

Landscape and objects

[1.6] OBIA - Information update

OBIA information update

OBIA information update

• Some composite objects that cannot be directly delineated, but they need to be ‘modelled’ .
• Here a biotope-complex class “mixed arable land” is modelled
• This intermediate stage shows the constituting elements used
• From a user’s point of view, this product is ‘use-less’ , as the information is up-to-date, but not, provided in the required form.

OBIA information update

• Now the ‘mixed-arable land’ object is fully valid (from user’s point of view)
• It is composed by a reasonable mix of grassland and crop fields
• It obeys the average size constraints (2 ha)
• It is fully congruent with admin boundaries (her: cadaster)

Quotation

[1.7] What is / what does OBIA

What is/ What does OBIA?

• Bridging methods and techniques from (formerly distinct) realms GIS and Remote Sensing
So is it a paradigm shift ?
• Yes in the sense of addressing research questions more efficiently (more intelligently?)
Scarcity in data? Not really, but need for adapted methods, e.g. for validation of OBIA results

(what we would like to achieve) to represent complex scene content to best describe the imaged reality best and to understand a maximum of the respective content, to extract it and to convey it to users or researchers.
(how we do it) OBIA combines spatial concepts with signal processing for (semi‐)automated image analysis, that works on objects, rather than on pixels

[1.8] Two related pillars...

Two related pillars...

• Image segmentation and image classification
• Both of them use algorithms that require specific parameterization
• to (semi‐)automatically perform specific processing tasks
• Segmentation
provide units (image objects), which can be labelled, assigned to classes or used for further class modelling
By grouping neighboring pixels to larger wholes , according to their similarity, sensu (bottom-up) regionalization
Key challenge: to provide image objects which match scale of operation

Two related pillars...

• Classification
• The assignment of objects to classes via labelling or class modelling
We differentiate between feature extraction (e.g. single dwellings or trees) and wall-to-wall classification.
Full representation of a scene content is the actual aim of image understanding



Feature extraction (dwellings, left) vs. wall-to-wall classification (right)

Segmentation/ Classification

• A cyclic process ...

[1.9] (Very) High Resolution Data

(Very) High Resolution Data

• Very high spatial resolution (VH(S)R) data today: sub-meter, up to 50 cm.
Higher spatial resolution usually accompanied by higher radiometric (and often higher spectral) resolution
Increasing resolution means increasing complexity
Experience from interpreters as important as in air-photo interpretation, but to be incorporated into rules and algorithms

(Very) High Resolution Data

(Very) High Resolution Data

(Very) High Resolution Data

Sensor resolution hits the scale of human activity

Field of potential applications is increasing

More and more detailed object levels can be represented

Complexity increases as well

(Very) High Resolution Data

(Very) High Resolution Data

Quotation

[2.1] Visual Perception

What is / what does OBIA?

Visual Perception

Visual Perception

Visual Perception

Image context

contrast black/white and shape: "black line in white area"

contrast b/w and shape (elongated and acute): "stripes resembling a zebra pattern"

mainly shape (pattern suppressed): "chair with zebra pattern "

[2.2] Early Vision

Early Vision

• Conceptual framework of Marr (1982)

• Three-leveled structure of visual information processing

  • Computational level (purpose and strategy of perception)

  • Algorithmic level (implementation)

  • Hardware level (physical realization)

Marr's Paradigm

• Computational level

  • Visual processing in stages (“sketches”)

  • Provide more detailed information

  • 2D representation of a scene

    Raw primary sketch:
    grey shades and colour tones

    Full primary sketch:
    blobs and edges, ‘place tokens’

Algorithmic level

• Scale-space analysis

• multi-scale image segmentation

• class modelling etc.

Recognizing objects

What makes a 1-year old recognize various instances of a strawberry?

What makes us recognize objects where there are none …?

Recognizing objects

We perceive a shoulder ... despite the fact there is no distinct boundary!

🡇

Expert 'filter'

Objects and gestalt

• Gestalt approach (Wertheimer, Koffka, etc.)

• Ehrenfels criterion: a gestalt is more than the mere sum of its parts (emergent properties)

  • ‘Laws’ of perceptual organization

    Factor of good gestalt

    Factor of proximity (granularity)

    Factor of good continuation

  Those laws do not explain how perception actually works, but help to predict, how structures are perceived

Orchard problem

[2.3] Role of experience

Role of experience

A river has

• spectral properties: 'blue' (as water)

• in addition specific form / shape: elongated, quasi 'linear'

A municipal park has

• spectral properties: ‘green’ (as vegetation)

• in addition specific spatial context: surrounded by urban settlement

From Definiens, 2004

Quotation

Role of experience

• Human vision is well adapted for complex image interpretation tasks

Experience built up since early childhood

But human vision is challenged when dealing with remote sensing imagery:

• Applying an overhead view

• Dealing with spectral characteristics beyond the visual spectrum

• Working with unfamiliar scales and resolutions

🢂 Experience is an important prerequisite for skillful and successful interpretation

Different views of the castle in Salzburg, Austria

Role of experience

• Personal experience

  • depending on age

  • etc.

Small kids see dolphins in a bottle

Role of experience

• Professional experience

• e.g. spectral signature

  • We know spectral profile of geographic features

  • We know what different colours mean, including ‘false colours’ (IR …)

Role of experience

Quantisation

[2.4] Pixel-scape & object-scape

Pixel-scape

• Pixel

  • 'picture element'

  • integrated signal - depending on GSD

  treated individually, no matter where located

Pixel-scape

• Pixel-based classification process

Raw image

🢂

Feature space

🢂

Classified image

Problems

• Spectral values belong to more than one information class

• No spatial relationship used in classification

• Pixel artificial spatial unit

• 'Artifacts' (salt-and-pepper effect)

Pixel-scape

• Limitations of pixel-based analysis

• considering

  • Color (spectral reflectance in n Bands

  • 'Space' in the sense of texture (given environment, e.g. 3*3 pixels)

• but not ...

  • Form & shape

  • Neighborhood & context

  • Hierarchical levels

Object-scape

Manual delineation (‚Bog‘)

Pixel-based classification

🡿

Object-based classification

🡾

Object-scape

• Relation between target objects and spatial resolution

  • Increasing importance of VHR EO data

  • High level of detail provides extended set of target classes

  • Addressing these target classes in a Landsat-imagery would fail

• Quotation

  • Hay et al. 2003

Object-scape

• Integration of Remote Sensing and GIS

  • GIS users are 'used to' polygon environment

  • Aggregation of information (highly textured images like VHR data or radar)

  • Modelling of scale-specific ecological processes through multi-scale representation

• Quotation

  • Blaschke & Strobl 2001

Object-scape

[2.5] Visual delineation vs. machine-based segmentation

Visual interpretation vs. automated delineation

Visual interpretation vs. automated delineation

• Problems occurring with visual delineation visually (may be solved via segmentation):

  • selection of appropriate generalization level

  • individual delineations

  • placement of boundaries when there is a gradual transition

• Problems that challenge machine-based segmentation:

  • Delineating conceptual boundaries (e.g. the 'outline' of an orchard, see above)

• Quotation

  • Campbell 2002

Visual interpretation vs. automated delineation

Visual interpretation vs. automated delineation

Visual interpretation vs. automated delineation

Visual interpretation vs. automated delineation

Visual interpretation vs. automated delineation

[3.1] Multi-scale representation

Multi-scale representation

• Assumptions
World in its complexity is hierarchically structured
This complexity can be decomposed and structured in scaled representations
Objects are interlinked on different scales (spatial concepts required)
Multi‐scale segmentation mimics our way of perception

Multi-scale representation

Multi-scale representation

• Human eye
• Represents reality in various levels simultaneously
• Image segmentation
• Representation in various levels sequentially

Multi-scale representation

• Human eye
• Represents reality in various levels simultaneously
• Image segmentation
• Representation in various levels sequentially
• Links between between levels via object hierarchy ( scale‐adaptive or scale‐specific )
• How to hit the appropriate scale domain?

[3.2] Scale

Scales

Two inherent scale domains
The entire face (including the recognition of a person)
The constituting parts (eyes , forehead, nose, beard, collar)
But not: single pixels (that’s why we twinkle our eyes)

Scales

• Scale
• Def: “the period of time or space over which signals are integrated [...] to give message ” (Allen & Starr, 1982)
Here: Refers to the size of objects in reality or in a representation of it (e.g. a map or a satellite image)
Different objects have different scales
• Every object has an inherent scale
• It appears in a certain range scale
Depending on the elevation of our viewpoint we see certain objects

Grain and Extent

• Relevant range of the scale spectrum for landscape analysis
Grain
• minimum area at which an organism responds comparable to resolution (spatial, spectral, temporal) in an image
Extent
• coarsest scale of spatial heterogeneity extent of the whole scene (total area, bandwidths, covered temporal duration)

Same landscape, same extent, different 'grain size'

[3.3] Hierarchy theory

Holons

[3.4] Hierarchical Patch Dynamics

Decomposability & holarchy

• Landscape as a system
• Consisting of interacting subsystems
• Hierarchical Patch Dynamics Paradigm (HPDP) , Wu 1999
Decomposition
• Separating a system into its components according to their scale and ordering them hierarchically

Quotation

Decomposability & holarchy

• Scaling ladder
• Every portion of land contains objects of many different scales resulting in a series of scales
• Boundaries within the scale spectrum
• Thresholds between ranges of scale are never crisp

Decomposability & holarchy

• Subsystems
• Horizontal and vertical coupling
• Self‐assertive tendencies
• rather independent from each other
• Integrative tendencies
• Part‐being of constituting elements

[3.5] Multi-scale imagery

Multiple scales in images

• Definition of fundamental objects in remote sensing images
• Integrated objects vs. aggregates objects
• Interaction of objects within and across scale domains
• What scales should be chosen for the different objects?
• At what scale should hierarchies be established?
Is a pan‐hierarchical representation sufficient or do we need regionalized or class‐specific segmentation?

[3.6] Scale- space analysis

Scale-space analysis

• Linear Scale Space Analysis
• Repetitive smoothing of an image applying a Gauss filter
• Image objects emerge and disappear in a dynamic process
• Multiscale representation
• Stacking of multiple filter layers

Scale- space analysis

• Blob detection
• Dynamc image objects
• 4 dimensions: x & y, reflection, scale
• Visualisation as 2D‐blobs (a) vs. 3D Hyper blob (b)
• Scale-Space Events
• C: Creation
• A: Annihilation
• S,M: Split and Merge
• Application
• E.g. tree crown detection

[4.1] Knowledge Representation - Why?

Why?

• An intuitive, yet experience‐ and culture‐driven concept of our geographical reality.
• While single elements are perceived, likewise the entire image is understood.
• Once understanding the whole scene we assign meaning, maybe even emotion, to it.
• To achieve such level in image analysis (or at least an approximation), we need to make implicit knowledge explicit!

Knowledge Representation

• Knowledge plays a key role in the interpretation‐oriented parts of the remote sensing process chain (Campbell 2001)
Colour form, and arrangement evolve certain parts of our experience and knowledge
Object hypotheses tested and verified against what we see
Matching to stored information

[4.2] Experience and Learning

Experience and Learning

• Training feeds knowledge through formalized learning
Artificial intelligence (AI)
Procedural knowledge vs. structural knowledge
Knowledge organizing systems (KOS), formal concept analysis (FCA)
Semantic nets
Artificial neural networks
Neuron‐like machines
Adaptive vector coding with tuned weights weights and synapses
But otherwise black box system and limited transparency

Experience and Learning

• Natural computing (Binning et al. 2002)
Parallel nature of natural thinking and computing
Expert systems with net‐like character
Self‐organizing, semantic, and self‐similar network (Triple‐S)
Fractal machine with excitements and thresholds (activation)
Inheritance concept
Unit delineation and labelling in cyclic manner

[4.3] Production Systems

Rule-based production system

• Rule‐based approach
• Logical inference (if … then)
• Knowledge needs to be encoded in rules
• Transparency of intelligent system
Challenges
• How to encode implicit knowledge into rules?
• How we guarantee guarantee integrity of such rule‐sets?
Procedural and semantic dimension
Considering vagueness in class assignment ➞ fuzzy approach

[4.4] Fuzzy Rule Sets

Fuzzy rule sets

• We would all consider freshly brewed tea as something “hot”
Eventually the tea gets cold … just when exactly and at which temperature?
We do not know precisely, but:
We all have a concept of “warm”, “cold”
We simplify a gradual phenomenon and categorize it
Boundaries between these concepts are not crisp
The category becomes clearer with increasing distance from boundaries

Fuzzy rule sets

• Vagueness in class description
Imprecise human thinking and vague (linguistic) class descriptions
Uncertainty in sensor measurements
Class mixtures due to limited (spatial and spectral) resolution
Potential Potential of fuzzy rule sets
Express each object’s membership to more than one class
Probability of class assignments

Fuzzy Rule Sets

• The fuzzy set (A):
• Is a certain subset of values of the whole range of an object feature X (e.g. NIR‐band)
• Represents an object feature class (e.g. forest) within one object feature
• Replace boolean logic (“false“ and “true“) of the membership value μ by acontinuous continuous range of [0, …, 1]
• Define membership function μ(x)
• Assigning to every object feature value x a membership value μ
• If μ > 0, then x belongs to the fuzzy set A
• Relation between object feature and classification

⇒ Choice and parameterisation of the membership function influence the quality of the classification

⇒ Introducing expert knowledge

Fuzzy rule sets

• Fuzzy rule “if – then” for assigning an object to a class
• If feature value x (of the object) is member of the fuzzy set (e.g. associated with the class forest), the image object is a member of the land-cover forest
• Combination of fuzzy sets to create advanced fuzzy rules
• Operator “AND” – Minimum operation
• Operator “OR” – Maximum operation
• Operator “NOT” – inversion of a fuzzy value: returns 1-fuzzy value
• Fuzzy rule‐base (combination of the fuzzy rules of all classes) delivers a fuzzy classification
• Every object has a tuple of return values assigned to it with the degrees of membership to each class/degrees of class assignment
• Since these values are possibilities to belong to a class, they don’t have to add up to 1 (unlike probabilities)

Fuzzy rule sets

• Comparison of membership degrees
• Reliability of class assignment
• The higher the degree of the most possible class, the more reliable is the assignment
• Stability of classification
• Stable classification for differences between highest membbership value and other values
• Equal membership degrees
• High values – reliability for both classes: classes cannot be distinguished distinguished with the provided provided classification
• Low values – unreliable classification (use threshold of a least required membership degree to ensure quality of classification)
• Defuzzification
• Maximum membership degree of fuzzy classification used as crisp classification

[4.5] Image Understanding

Image understanding

a. Definition

Image understanding UI is a process leading to the description of the image content (= reconstruction of an image scene) (Prinz, 1994)

b. Extent of UI

Reaching from signals (image data) to a symbolic representation of the scene content

c. Conditions for IU

Outcome depends on the domain of interest of the interpeter, defined by:
• Underlying research question
• Specific field of application
• A priori knowledge and experience of the interpreter

d. Output description

• Description of real-world objects and their relationship in the specific scene
• Resulting in thoroughly described features (not mere listing and labelling of features)

e. Knowledge input

Process is driven by
• Utilisation of procedural knowledge
• Transformation of structural knowledge

f. Involved disciplines

• Image preprocessing
• Pattern recognition
• Artificial intelligence

IU and OBIA

IU and OBIA

IU and OBIA

IU and OBIA

[4.6] Object categories

Object categories

• Lang, S., F. Albrecht, S. Kienberger & D.Tiede (2010): Object validity for operational tasks in a policy context. Journal for Spatial Science, 55(1),9-22.
• Implications of OBIA image understanding and basic objct ontology in general

Object categories

• Bona Fide objects
• With 'natural' boundaries
• Correspond to local physical discontinuities
• Perceived by people in more or less the same way

• Composite Objects
• Real-world modelled objects
• Correspond to functional homogeneity
• Perceived by experts in similar way (convention)

• (Conceptual) Fiat objects
• With 'un-natural' boundaries
• Correspond to no genuine heterogeneity
• Objects not obvious in the landscape

cf. Lang et al. 2010

Object categories

[5.1] Brief History

Brief history

• 1980s
• First milestone: Haralick & Shapiro 1985 “Image segmentation techniques"
• Major driver: industrial computer vision
• 1990s
• Discovery of image segmentation in remote sensing
• Challenge: multi‐spectral, geo‐referenced data
• 2000s+
• Advent of VHSR data boosted development
• Key role in semi‐automated analysis of EO data (e.g., Copernicus information layers)

Source: unkown

[5.2] What is segmentation?

What is segmentation?

• Segments
• Portions of a linear object (such as a road)
• Pieces of a wooden snake, limbs of a chain, etc.
• Segmentation of linear geographical features
• Road segments
• 1‐dimensional discrete georeferencing (milestone, km‐post)
• ‘Dynamic segmentation’ (according to address data)

Wooden snakes have segments and so have roads. Segments from exit to exit have irregular lengths, segments from milestone to milestone regular ones.

What is segmentation ?

• But let's move to 2-D (or 3-D) geographical space
Principles of 2D (areas) segmentation
• Space is cut into pieces
• Pieces should be of similar size
• Entire space covered
• No gaps or overlaps
Perfect 2D 'segmentation'
• Hierarchical systems of administrative units (states, provinces, districts, ,etc.)
• Number of units >>1
Top-down view

What is segmentation ?

• Regionalisation (bottom-up)
• Think of a...
Re‐organization of Europe towards the …
United Regions of Europe (URE)
Regions should be quite distinct to each other, yet internally internally homogenous
Politicians decide to take ‘natural language’ as the criterion to group neighbouring communities
Some pragmatic decisions to be taken
... better stop here :)



Fictitious example of the ‘United regions of Europe'(illustration usin NUTS-1 level units)

What is segmentation?

• Another example for 'spontaneous' (bottom-up) regionalization
• Students practicing a fire alarm
When arriving in the gym, clusters of students are built
Students cluster according to their class
Chance is very high to meet the majority of students at their class cluster, even if some of them move around
The class cluster will gather around one place, and should not occupy several spaces

[5.3] Image segmentation in RS

Image segmentation in RS

Image segmentation in RS

• Objects against background (matrix) represented on image
• Region (token): neighbouring pixels with similar value

Image segmentation in RS

Image segmentation in RS

• Image segmentation is a form of (dual) regionalisation
• Regionalisation within the feature space and (at the same time)
• Within real space
Level of detail of objects to be resolved depends on pixel size
• 3 to 4 times smaller than objects of interest
• Depending (again) on experience, contextual information, etc.



2-dimensional geographical space

Image segmentation in RS

• How well does segmentation do?
• Good matches, poor matches with real‐world objects
• What about our own object hypotheses ?

Image segmentation in RS

Image segmentation in RS

Segmentation results are challenged by the ultimate benchmark, our visual perception. Conceptual boundaries are hardly to be detected by segmentation algorithms while perceived by the human eye with ease (orchard problem).

Quotation

[5.4] Groups of segmentation

Groups of segmentation

• Pixel-based or histogram-based
• Thresholding techniques
• Segmentation of feature space
• Region-based
• Region growing
• Split and merge
• Edge-based
• Laplace filter, Sobel-operator, representativeness,...
Non-image related
• Non content expecting
• Tiling image with a honeycomb or chessboard structure

Histogram segmentation

• Histogram Thresholding
• Simplest way to accomplish an exhaustive segmentation
• One‐ or multi ‐modal distribution of grey values, threshold has to be determined
• Only works in feature space → 'pseudo-segmentation'

Edge-based segmentation

• A segmentation routine when elongated structures (e.g. roads) separate otherwise homogenous areas
• Edge: clear boundary between homogenousareas, detectable by edge-sensitive algorithms (filter)
• E.g. Lee-Sigma filter

Edge-based segmentation

• Edge detection
• Filtering – smoothing to decrease noise in the image
• Enhancement Enhancement – revealing revealing local changes changes in intensities
• Detection – select edge pixels, e.g. by thresholding
• Closing of gaps / deleting artefacts
• Combining, extending of lines
• Linking the edge pixels to form the region boundaries

Region-based segmentation

• Region growing routine
• Seed cells are distributed over image
• Bottom up (randomly)
• Top‐down (content expected)
• Strategy
• Neighbours (4‐ or 8‐neighbourhood) are included into region if
• They do not belong to another region yet
• A homogeneity criterion H applies
• Two neighbouring regions are unified, if H applies

Watershed segmentation

• Watershed segmentation
• Grey values as relief
• Virtual flooding
• Watersheds separate pools
• Controlled by markers
Split& Merge
• Top down following quadtree principle
• Scene is subdivided into n (usually 4) equally sized parts / cells.
• Cell is maintained as is, if H applies applies, otherwise subdivided again.
• Repeated until H applies
• Merging of neighboring quadtree quadtree cells, if homogenous.

[5.5] Multi-scale segmentation

Multi-scale segmentation

• Segmentation in hierarchical scale domains
• Principles
• Mimic the human eye
• Reflect hierarchical structure of (geographical) reality
• Segmentation layers are interlinked
• Reproducibility and universality
• In eCognition
• region-based, local mutual best fitting appraoch (see below)
• Colour homogeneity
• Form homogeneity
• Strictly hierarchical

Multi-scale segmentation

• Homogeneity
• Central concept in region‐based segmentation
• Regions: maximum internal homogeneity , and
• Maximum external heterogeneity ('difference')
Segmentation ≠ unsupervised classification
Involves a priori spatial aspects
Employs spatial optimization techniques (e.g. compactness)

Multi-scale segmentation

• Recent advances support multi‐resolution segmentation
• Key characteristics
• Several levels of object delineations
• Derived from one single image
• Multi‐resolution means multi‐scale
• But which scales.... ?

2-scale representation of scene with bush encroachment

Multi-scale segmentation

• What are relevant segmentation layers?
• Regionalized hierarchies (Lang,2002)

Multi-scale segmentation

• What are relevant segmentation layers?
• Statistical estimation of scale parameter
• ESP Tool by Dragut et al.

Multi-scale segmentation

Multi-scale segmentation

• SCRM (Size constrained region merging) (G.Castilla)
• The desired mean size of output polygons (DMS ‐ in hectares)
• The minimum size required for polygons, or minimum mapping unit (MMU ‐ in hectares)
• The maximum allowed size (MAS ‐ in hectares)
• The minimum distance between vertices in the vector layer, or minimum vertex interval (MVI ‐in meters)
• eCognition
• Scale Parameter (average size of generated objects)
• Colour homogeneity (spectral similarity)
• Shape homogeneity homogeneity (compactness (compactness of form)

Multi-scale segmentation

• Bottom up region merging technique
• Starting with each pixel being a region
• A pair of regions is merged into one region, each merge having a merging cost (degree of fitting)
• Objects are merged into bigger objects as long as the cost is below a ‘least degree of fitting’ (scale parameter) = the merge fulfils the homogeneity criterion
• Starting points for merging distributed with maximum distance
• Pair wise clustering process considering smallest growth of heterogeneity
• Establishing segmentation levels on several scales using different scale parameters

(e.g. 2nd level based on 1st level: larger scale parameter results in larger objects consisting of the objects of the 1st level)

Multi-scale segmentation

• Decision heuristics
• Finding an adjacent object B for an arbitrary object A for merging them
• Fitting: when the homogeneity criterion is fulfilled
• Best fitting: when the homogeneity citerion is fulfilled, and the merge between B and A produces the best degree of fitting compared to the merge between A and any other adjacent object A

Multi-scale segmentation

• Decision heuristics (cont.)
• Local mutually best fitting: find the best fitting object B for the object A, then find the best fitting object C for the object B
• Confirm that object C is the object A, otherwise take B for A and C for B and repeat the procedure
• Find best fitting pair of objects in the local vicinity of A following the gradient of homogeneity
• Global mutually best fitting:merge the pair of objects for which the homogeneity criterion is fulfilled fulfilled best in the whole image
• Distributed treatment order
• Use starting points with maximum distance to all other points treated before (treatment order defined over pixels or segments)

Multi-scale Segmentation

Definition of the degree of fitting

• Colour and shape homogeneity are weighted against each other
• Compactness Compactness and smoothness smoothness make up the shape homogeneity and are weighted against each other

Multi-scale segmentation

• Segmentation parameters
• 'Scale parameter'(- relative average size)
• Compactness and smoothness

Multi-scale segmentation

• Domain-specific segmentation
• Within ‚image domains‘, i.e. broader classes such as forest or open land
• Independent segmentation parameters control specific segmentation within domains

[5.6] Object Features

Object features -overview

layer values

• mean
• std-dev

geometrical properties

• size, shape, ...

textural properties

• layer value texture (e.g. mean of sub objects: std-dev)
• shape texture (e.g. directions of sub objects)

hierarchical properties

• number of higher levels
• number of super or sub objects



relations to classes of...

• neighbour objects
• sub objects (relative area of...)
• super objects

membership to...

[5.7] Adaptive parcel-based segmentation

Adaptive-parcel based segm.

• Modified per-parcel approach
• Based on digital cadastre data and multispectral image data (e.g. SPOT)
• Depending on internal characteristics, objects are retained, aggregated or split apart
• Finally: spectrally homogenous elementary units ( as basis for e.g. class modelling process)



➨

Adaptive-parcel based segm.

• Three cases
• (1) boundaries retained: parcel corresponds to one, single homogenous image object, no change or updatae of the geometry required
• (2) boundaries removed: parcels are merged because of internal homogeneity, change of geometry
• (3) boundaries introduced: a single parcel is spectrally heterogeneous and split according to spectral behavior, change of geometry (mainly forest)

[6.1] Strengths of object-based classification

Object-based classification - why and how?

• Additional information can be utilized
• shape, texture, relationship to neighbours
• 'object-relationship modelling' (Burnett & Blaschke 2003)
Classification
• Based on samples (training objects)
• Based on rules (advanced object feature database query)
'Click and classify'
• Direct labeling of well delineated objects
• Easy re-classification

Object-based classification - why and how?

• Overcoming noise through segmentation
• Especially useful at VHSR data
• Improved signal/noise ratio
• Adaptable image resolution
Decreasing number of units
• Lesser units to be classified ( in comparison to number of pixels)
• But increasing complexity in class description

Object-based classification - why and how?

• Recall the Yin-Yang analogy
• Classification interlinked with segmentation
• Intermediate pre-classification, before performing next step of segmentation (e.g. domain-specific segmentation)

[6.2] Sample- vs. rule-based classification

Sample- vs. rule-based

• Sample-based classification
• Define class membership by similarity to selected samples (training objects)
• Sample has to be representative for its class
• Use features clearly distinguishing the sampled class from other classes
• Attention: only a few of potential fetaures ma be distinctive
Classifier
• E.g. nearest neighbour Classifier
• Object will be assigned to the class whose samples are closest in feature space distance
Useful approach, if knowledge about the scene's content is limited

Sample- vs. rule-based

• Rule-based classification
• Heuristics are encoded in a set of rules
• Define a class by single rule on one feature or by several rules on several features
Hierarchical relations of classes
Rules can address different kinds of features
• Object features
• Class-related features
Fuzzy or crisp rule definition
Advantages compared to sample-based cassification
Incorporation of expert knowledge in the classification
Formulation of complex class description
Transparency (especially compared to neutral networks)
Transferability

[6.3] Fuzzy vs. crisp rule-based classification

Fuzzy vs. crisp rule-based classification

• Sharp boundaries vs. gradual changes
• Especially under natural conditions
• Fuzzy rule sets account for this uncertainty



Clear-cuts in a forest are less ambiguous, both in terms of boundaries and class assignment

[6.4] Class hierarchy

Class hierarchy

• Classes are embedded in a hierarchical heritage scheme
• (1) Feature-based inheritance
child classes inherit all descriptive features from their parent classes
not confined to spectral values
(2) Semantic inheritance
• classes can be grouped semantically
• belong to the same logical parent class
Semantic inheritance may turn into feature-based when additional explicit information is provided

Cognition network

[6.5] Class-related features

Class-related features

• Classifying an object based on the classification of other objects
• classified label functions as a contextual feature, e.g. a green area is classified as urban park because it is embedded in urban area
• Classified object acts as context feature for other objects

[6.6] Class modelling

Class modelling

Class modelling

• Mire complex wit 'non-a-priori' units
• Entire bog consists of a complicated pattern of vegetation patches with sub-units if ecological relevance
• Sub-units form habitats that show a certain degradation stage
• In manual interpretation: habitats are delineated and aggregated by experienced interpreter (generalization)
• Difficult to automate, since we deal with 'double complexity'
• Spatial structure of habitats can be categorized ( structural signatures), but not standardised

Class modelling

• Two aspects
• Minimum percentage of bushed
• Distribution of bushes

Quotation

Class modelling

• Bona fide objects
• With 'natural' boundaries
• Correspond to local physical discontinuities
• Perceived by people in more or less the same way

• Composite Objects
• Real-world modelled objects
• Correspond to functional homogeneity
• Perceived by experts in similar way (convention)

• (Conceptual) Fiat objects
• With 'un-natural' boundaries
• Correspond to no genuine heterogeneity
• Objects not obvious in the landscape

Lang et al. 2010

Class modelling

• Demand profile and policy scope
• Biotope complexes as basic units for regional planning purpose
• BIMS (Biotope information and management system)

Class modelling

• Establishing rule sets for the description of the structural composition of biotope complexes
• Consideration of minimum size of biotope compexes (e.g. 4ha, 2ha)
• Additional hint layers
• realization in Cognition Network Language (CNL)

Class modelling

Class modelling

• Thematic conditioning (functionally homogenous units, minimum size)
• 31,698 biotope complexes were delineated for the whole Stuttgart Region
• Average size: 11.5 ha

[7.1] Definitions

Definitions

• Accuracy
• Degree of correctness of a classification result, agreement of image classification with reality
• Qantitative quality measure judging the validity of results in scientific investigations, that also ...
• Describes (technical) usability of generated thematic maps in practical applications.
• Error
• Discrepancy between the thematic map and the situation on the ground (Foody, 2002)
• Accuracy assessment
• The process of assessing and quantifying the accuracy of digital image classification; Image analysis is not completed until the accuracy of the outcome is assessed (Lillesand & Kiefer 2000)
• Comparison of pixels or polygons in a remote sensing‐derived classification (map to be evaluated) and independent evaluation data (reference map)

[7.2] Non-site accuracy assessment

Non-site specific accuracy assessment

• Comparing area percentages of classes as occuring in the analysed image and the reference map
• "Non-site specific" means the actual spatial distribution of classes is not considered

[7.3] Site specific accuracy assessment

Site specific accuracy assessment

• Agreement of categories in classified image and reference map at specific locations
• based on site‐by‐site comparison (pixels or objects)
• assumption: each location is occupied by one (and only one) class
• Calculation of error matrix as an aggregation of localised errors

[7.4] Error Matrix

Error matrix

• Percentage correct (overall accuracy)
• Error of omission ⇔ commission
• Evaluating an error from two different viewpoints
• Error of omission: correct class not recognised by the classification process (false exclusion)
• Error of commission:assignment to the wrong class (false inclusion)
• Producer's accuracy ⇔ Consumer's accuracy
• Accuracies of individual categories
• Producer's accuracy: Error in the responsibility of the
  producer, in other words: percentage of pixels correctly
  classified by the producer.
• Consumer's / User's accuracy: Error in the eyes of the
  consumer, in other words: percentage of pixels that a
  user can expect to be correct.

Error matrix

• Kappa coefficient (K_hat)



• Explanation of the formula:
• Observed agreement = overall accuracy
• Expected agreement = sum of the products of the consumer’s accuracy (CA) and the producer’s accuracy (PA) of each class

Error matrix

• Limitations
• Error matrix is not a standardised measure
• Many different variations exist
• Sampling design and sample size can limit the meaning of the measures
• Types of error
• Disagreement of labels or just misalignment?
• Reliability of reference data
• 'Ground truth' data might be error-prone as well!
• Other remote sensing data, when being used as a surrogate for ground reference data, should have a higher spatial resolution.
• When visual delineation is used as a reference, make sure this process is conducted independently and you are aware of its quality.
• Spatial distribution of error not represented

[7.5] Object-based accuracy assessment

Towards object-based accuracy assessment

• Semantic assessment of object‐based classification
• Site specific approach (based on object centroids or whole objects)
• Separating objects from training process (if so) as an independent test set
• Optional: small test areas where all occuring objects are checked.
• Geometric assessment of image objects
• Checking classified object against visual delineation
• Quantitative assessment with spatial overlay techniques
• Challenges of object‐based accuracy assessment
• No full geometrical fit of reference objects and objects to be evaluated due to different ways of boundary generation.
• When using fuzzy classification rules the categorical class assignments are not necessarily mutually exclusive
• Accuracy is a matter of geometric and semantic agreement

Towards object-based accuracy assessment

• Comparing an OBIA classification with manual delineation

[7.6] Object fate analysis (OFA)

Spatial agreement

• Quantitative assessment: spatial overlay with tolerance
• Classified object has the same extent as reference object (“stable object”)
• Reference object is an aggregate of several classified objects
• Complex geometry with non-overlapping objects
• Characterisation of object relationship ("object fate")
• Good objects
• Expanding objects
• Invading objects

Object fate analysis

Object fate analysis

Object loyalty (OL)

• Similarity measures:
• Proportion of overlapping area
• Number of overlapping objects
• Similarity measures:
• Proportion of overlapping area
• Number of overlapping objects

➔ Locating systematic errors (e.g. shift in geolocation etc.)

[7.7] Object validity

What means object validity?

• "Degree of fitness for an operational task in a policy context." (Lang et al. 2010)
• From crisp boundaries to more complex objects
• As we leave the domain of crisp, well defined bona fide objects (Smith, 1995), the binary decision between correct and false labeling begins to vanish.
• Fuzzification of a class assignment relates to the semantic ambiguity but not (necessarily) to the delineation.
• The domain of fiat objects requires more flexible concepts to address these.
• We suggest the term validity to address the appropriateness of an object delineation.

Object validity

• Automated object generation - consequent, but not valid?
• Spatial inappropriateness and scale mismatch
• Different object representations
• Terrestrial survey (red outline)
• Visual interpretation (green outline)
• Automated extraction (yellow outline)
• Error band (buffer)

Object validity

Object validity

• Conditioned information
  • Aggregated class: Mixed arable land (not single fields)
  • Object outlines follow cadastral boundaries
  • Minimum size: ~ 4 hectares

[8.1] Integrated geons

Complex geospatial phenomena

[8.2] Spatial composite indicators

Meta (composite) indicators

Meta (composite) indicators

Operationalising vulnerability