Object-based Image Analysis - An Introduction

Object-based Image Analysis (OBIA)

An introductory course

Assoc-Prof Dr Stefan LANG

[1] Why spatial image analysis? | [2] Regions and image objects | [3] Image segmentation | [4] Knowledge representation | [5] Class modelling

University of Salzburg, Department of Geoinformatics, 2021/21

[1] Why spatial image analysis?

Space and spatial image analysis

space https://www.youtube.com/watch?v=c7OO3qCfH9Y
View from ISS to Earth. Depending on observation distance and resolution we perceive 'objects' in a certain scale. Here we see the eye of a hurricane. © S. Lang (PLUS) / Photograph: A. Gerst (ESA)

Spatial image analysis - bridging remote sensing and GIS

conceptualbridge conceptualbridge

Geographical OBIA (GEOBIA) as a new paradigm in image analysis

GEOBIA paradigm
A growing literature body and a suite of international conference supporting the idea of (GE)OBIA as a paradigm. © S. Lang (PLUS)

GEOBIA - trans-application potential

GEOBIA applications
In a variety of application domains, OBIA may be used in bridging between geospatial models and imaged representations. © S. Lang (PLUS)

GEOBIA provides real-world representations

RealityModel
OBIA mimicking our conceptual understanding of reality, facilitates image understanding and complex representations of the real-world. © S. Lang (PLUS)

Space over colour?

Spatial properties (structure) are as important as spectral properties (colour) in image analysis

space over color
Once structural features make us understand there are agricultural fields shown on an image, we start exploring the type of fields. © S. Lang (PLUS)

Boundaries in images

Image data neither contain nor store boundaries.
However, boundaries can be extracted from image data using image segmentation techniques. They 'occur' in several scales and can be used for better image understanding.
boundaries
Different spatial information derived from images and other continous datasets (e.g. DEM) © B. Riedler, S. Lang (PLUS)

Real-world objects on images

Chiemsee
Real-world features can be characterised by spatial properties and spatial relations. © D. Tiede (PLUS)

Spatial properties of objects

spatialAspects
Spatial (i.e. geometrical, textural and contextual) aspects that can be addressed, even on grey-scale image. © S. Lang (PLUS)

Extended set of target classes in spatial image analysis

OBIA enables to address target classes, which are defined spatially. There are basically two groups, namely geometrically defined classes (e.g. lake vs. river) and spatio-relational classes (e.g. an island, an urban park, or composite objects such as a residential area)
spatialImageAnalysis
OBIA expands the set of target classes as compared to pixel-based analysis. Geometric or spatio-relational classes can be derived. © S. Lang, D. Tiede (PLUS)

[2] Regions and image objects

Spatial auto-correlation

According to Tobler's First Law of Geography, which states that "near things are more related than distant ones"), neighbouring pixels tend to be similar. Boundaries can be detected at discontinuities or gradients between these groups of similar pixels. Boundaries form objects on distinct levels (scales)
spatial principle spatial principle
A simple image matrix is converted to random colours (left). When using graduated grey shades it reveals a familiar pattern (right). © S. Lang, PLUS

Spatial auto-correlation and grouping

Spatial auto-correlation is a key spatial principle in image analysis. We rely on this principle when grouping pixels according to their similarity. Without spatial autocorrelation the number of grouping options would be arbitrary. However, there is also ambiguity in the grouping process.
samesimilar
By spatial autocorrelation neighboring pixels can be grouped. It is, however, a matter of optimisation and ambivalence ('ill-posed'). © S. Lang, PLUS

Spatial auto-correlation and boundaries

varianceRegions
Despite the similarity of neighbouring pixels, we eventually encounter gradients. Those gradients we can identify as boundaries. © S. Lang, PLUS

Regions

  • Region = a spatially contiguous area with common attributes or uniform behaviour (internal ‘homogeneity’)

  • Regionalisation = spatial classification, i.e. classification under spatial constraints (contiguity of an attribute in space)

  • Why does it work? Principle of spatial auto-correlation (“1st law of Geography”, W Tobler, 1970)

Regions
XXX (© S. Lang, PLUS)

Image Objects

imageObjects
XXX (© S. Lang, PLUS)

Image Objects

image_objects.png
image_objects.png
XXX (© XXX)

What means homogeneous?

not a trivial task...

  • Challenges in generating regions

  • They need to be spatially coherent
    They do not exist a priori

  • Defining Homogeneity criterion H

  • Variance* of pixels within a region
    Spectral distance (SD) of pixels
    Vectors in n-dimensional feature space


*as one (statistical) measure to operationalize homogeneity

Regions
XXX (© S. Lang, PLUS)

[3] Multispectral Segmentation

Principles

Which image objects are we able to perceive?

  • From 252 digital numbers (14 x 18 image matrix) to ‘Abraham Lincoln’ (one individual person)

What do images evoke in our visual context and our brain?

  • Trying to ignore pixels, to identify familiar structures
  • Object hierarchy (here: exactly 2)
  • Semantic enrichment
  • Combination of experience and knowledge
  • Regions
    Image segmentation happens in our human brain. © S. Lang (PLUS)

    Grey-scale Segmentation

    imageObjects
    A boundary can be found along gradients between pixel values. © S. Lang (PLUS)

    Multispectral Segmentation

    imageObjects
    The same principle applies in multispectral images. © S. Lang (PLUS)

    Segmentation Challenges

    Conceptual boundaries in images
    Fiat boundaries, according to convention, expertise, etc.
     See lesson “Class modelling”
    image_challenges.png image_challenges.png
    Some boundaries in images appear distinctively, others are more conceptual. © S. Lang (PLUS)

    Segmentation Challenges

    Conceptual boundaries in images
    Complex classes, often defined by spectrally heterogeneous, but functionally homogenous units Segmentation rarely provides these units directly Cyclic approach needed (class modelling) Expert-based (object validity)
    image_challenges.png image_challenges.png
    Semi-automatically delineated complex biotope types. Left: open orchard meadows; Right: mixed arable land and grassland area (© S. Lang, PLUS)

    Image Segmentation

    Local pixel neighborhood

    • Spatial autocorrelation among pixels
    • neighbouring pixels tend to have similar values
    • But not everywhere!

    Regions
    Homogeneity and gradients in a small portion of an image matrix. © S. Lang (PLUS)

    Segmentation and classification

    Two interrelated methodological pillars (Lang 2008)

    (1) segmentation / regionalisation for nested, scaled representations ('object scape')

    • Segmentation of different types of continuous data
    • Optimizing segmentation (algorithms and parameterization) in terms of usability, theoretical foundation, etc.
    • Systematic comparison of segmentation results

    (2) rule-based classifiers based on spectral and geometrical properties as well as spatial relationships

    • Ontology driven classification, advanced sets of target classes
    • Process of OBIA not necessarily sequential (i.e. as described here), rather sequential, iterative (class modelling)
    • Smooth transition to methods to further characterise and analyse classification results (e.g. landscape metrics)

    segmentation_classification
    Yin-Yang allegory of segmentation and classification in OBIA. © S. Lang (PLUS)

    Multi-scale segmentation

    Segmentation in several hierarchical scales

    scale-specific vs. scale-adapted segmentation

    multiscale_segmentation
    Hierarchical, i.e. multiscale, representation. © S. Lang (PLUS)

    Multi-scale segmentation


    Representations in various nested levels
    Each object 'knows' its neighbours and hierarchical level

    multiscale_segmentation
    Schematic illustration of multi-scale segmentation. © F. Albrecht, S. Lang (PLUS)

    Multi-scale segmentation

    multiscale_segmentation
    Multi-scale segmentation in two hierarchical levels. © S. Lang (PLUS)

    [4] Knowledge representation

    Why knowledge representation?

    We all have a concept of “cold”, “warm”, “hot”

    • In remote sensing and image analysis a lot of expert knowledge exists
    • Such knowledge is implicit or explicit
    • The latter can be formalised and encoded in rules, (e.g. NDVI > 0.2 => vegetation)
    knowledge_representation
    Knowledge is used in remote sensing to convert primary data (images) into GIS-ready secondary data. © S. Lang (PLUS)

    What type of knowledge can be used?

    Knowledge in image interpretation: visual skills are complemented by explicit knowledge, enriched by experience and training

    Structural knowledge

    • how concepts within a domain are interrelated
    • links between image objects and real-world geographical features

    Procedural knowledge

  • specific computational functions, algorithms
  • Encoding of structural knowledge, e.g. by a set of rules … or by a set of representative samples

  • knowledge_representation
    Structured knowledge (screenshot of a ruleset). © S. Lang (PLUS)

    Some facts we can take for granted...

    • Physical laws and principles (e.g. Stefan Boltzmann‘s Law : The hotter a body, the more energy is emitted; Wien‘s Displacement Law: The hotter a body, the shorter the wavelength of the maximum radiation)
    • Different land surface types show specific spectral profiles
    • Spatial autocorrelation
    • ...
    others
    Laws and theories we can trust in image analysis. © S. Lang (PLUS)

    Others we know from experience...

    Heuristics describing certain geographical phenomena are encoded in a set of rules. Rules can address all kinds of features.

    Example soccer field = a special kind of grassland patch with certain size and shape.

    • Grassland appears “red” in a false-colour band combination
    • How does a football field look like on Sentinel-2?
    • Object features (color, shape, size, etc.)
    • Object-to-neighbour relationships
    • Hierarchical relationships>
    others
    Heuristics of a soccer field translated into a rule-set © S. Lang (PLUS)

    Fuzzy (rule) set

    We all have a concept of “cold”, “warm”, “hot”

    • We discretise a gradual phenomenon by categorizing it
    • Boundaries between these concepts are not crisp
    • The category under concern gets more stable with increasing ‘distance’ from boundaries
    • Fuzzy sets: a way to model / operationalise vague knowledge

    fuzzy_rule_sets
    Fuzzyfication of class boundaries (example water temperature). © S. Lang (PLUS)

    Class hierarchy (in eCognition)

    Classes are embedded in a hierarchical heritage scheme
    (1) Feature-based inheritance

    • child classes inherit all descriptive features from their parent classes
    • Not confined to spectral values

    (2) Semantic inheritance

    • Classes can be grouped semantically
    • Belong to the same logical parent class
    • Semantic inheritance may turn into feature-based when additional explicit information is provided

    class_hierarchy
    Feature-based and semantic grouping of classes. © S. Lang (PLUS)

    What is semantic querying?

    Example: Monitoring of water quality in swimming lakes over three seasons
    Image search criteria: [(0) sensor type], (1) time window , (2) geographical focus (AOI), (3) cloud cover , (4) thematic focus , (5) monitoring period.

    • Use standard metadata (location) and cloud mask
    • Thematic focus: requires a fundamental (i.e. low level semantic) concept of the existence and representation of (swimming) lakes.
    • Temporal aspect: (constancy or change) implies the notion for changes in critical indicators, e.g. by indices in IR reflectance recorded by multi-temporal imagery

    class_hierarchy
    Semantic information stacked in a data cube through time © D. Tiede (PLUS)

    Data to information

    How to measure the information in an image?

    • Histogram (distribution of values)
    • Distribution of colours / colour levels
    • Image content (based on classification)

    histogram
    Histogram of NIR band of Landsat satellite data. © S. Lang (PLUS)

    Data to information (cont.)

    Information measurement requires information units
    (bins, classified pixels, objects, etc.)

    information
    Information-as-a-thing (measurable) vs information-as-interpretation (semantic). © S. Lang (PLUS)

    Quantifiable information content

    Shannon's diversity is a measure of composition (information content).
    • Image content can be measured based on classification
    • Indices: Diversity H, Dominance, Evenness
    • Key elements: categorised units, number and percentages of classes

    information
    where P = percentage, i = class 1 … i

    information
    Shannon's Diversity measures the information content of a landscape. © S. Lang, PLUS

    Knowledge organising systems (KOS)

    Structural knowledge can be organized in knowledge organizing systems (KOS)

    • Realised as graphic notations such as semantic nets or frames
    • Semantic knowledge representation
    • using inheritance concept, e.g. ‘is part of’, ‘is more specific than’, ‘is an instance of’

    Semantic net

    • Control over existing connections, once established
    • Transparency and operability

    KOS
    XXX (© XXX)

    Classification schemes

    classification_schemes
    XXX (© S. Lang, PLUS)

    Classification schemes / taxonomies


    Corine Land Cover (CLC)

    • Standard EU mapping scheme (used by European Environmental Agency EEA) initiated in 1985 (reference year 1990)
    • Updates produced in 2000 and 2006, with the latest 2012 update under production
    classification_schemes
    XXX (© S. Lang, PLUS)

    Classification schemes

    classification_schemes
    XXX (© XXX)


    CLC consists of an inventory of 44 classes. It uses a Minimum Mapping Unit (MMU) of 25 ha for areal phenomena and a minimum width of 100 m for linear phenomena

    Classification schemes


    Land Cover Classification System (FAO-LCCS)

    • capable to record any land cover type, independent of specific applications and/or geographical areas
    • overcome problems associated with the interpretation of different land cover class definitions (Mucher et al. 2014)
    • uses a set of independent diagnostic criteria strictly based on vegetation physiognomy and structure rather than establishing land cover classes based on terminology (Kosmidou et al. 2014)
    • comprehensive land cover characterization, regardless of mapping scale, data collection method or geographic location.
    classification_schemes
    (© S. Lang, PLUS)

    Classification schemes

    classification_schemes
    (© S. Lang, PLUS)

    Spectral libraries


    Spectral libraries represent spectral reflectance and physical properties of different land cover types

    spectral_libraries simplified model!
    (© S. Lang, PLUS)

    [5] Class modelling

    Composite objects

    composite_objects
    (© S. Lang, PLUS)

    Composite objects


    Level of landscape elements

    • Spectrally homogenous, correspond to agricultural fields, road segments, blocks of houses etc.
    Composite objects
    • Contain elements (building blocks)
    • Are spectrally heterogeneous, but functionally homogenous
    • Geons (Lang et al. 2014)
    composite_objects
    (© S. Lang, PLUS)

    Class modelling


    Ambition: to delineate and classify composite objects in a complex real-world scene

    • To represent the scene in at least in two hierarchical levels
    • To delineate homogenous elementary units on lower level(s)
    • To establish lateral and vertical relationships among objects (object-relationship modelling, ORM)
    • To characterise the composition of target classes based on the arrangement of elementary units
    classification_schemes
    (© Lang & Tiede 2015, PLUS)

    Class modelling

    class_modelling
    (© Lang & Tiede 2015, PLUS)

    Class modelling


    A strategy to approach composite classes by applying multi-scale segmentation and spatial modelling

    • The target classes represent functionally homogenous units – often spectrally heterogeneous –, as being composed by sub-units
    • The delineation of target units may be ambiguous (but inter-subjectively agreeable – thus non-arbitrary)
    • The spatial properties of composite classes (or objects) can be described and their structural arrangements of building blocks (elements) can be characterised
    classification_schemes
    (© Tiede et al. 2010, Lang & Tiede 2015)

    Class modelling

    class_modelling

    Multiscale class modelling


    “Multiscale” is implicit in the term class modelling

    • A complex class (composite object) consists of several functional parts in a specific arrangement, such as:
    • ‘mixed arable land’ composed by a mix of agricultural and grassland fields
    • ‘residential area’ composed by family houses, gardens, shopping malls, etc.
    • Complex classes are hierarchically structured (‘body plan’)
    multiscale_class_modelling
    XXX (© XXX)

    Multiscale class modelling

    multiscale_class_modelling
    (© S. Lang, PLUS)

    Multiscale class modelling


    Spectral signatures of elements

    • Spectral profiles are created by charting the object mean of representative class samples
    • Profiles can be translated in rules
    • Fuzzy rules can be applied to translate uncertainty in class descriptions
    multiscale_class_modelling
    multiscale_class_modelling
    XXX (© XXX)

    Multiscale class modelling


    Defining an object’s body-plan

    • A higher-level objects consists of …
    • x% of object type A
    • y% of object type B
    • z% of …

    Critical reconditions

    • Elementary units need to be classified (‘Lego blocks’)
    • Outline of composite object needs to be delineated
    multiscale_class_modelling
    XXX (© S. Lang, PLUS)

    Multiscale class modelling


    Specify set of target classes

    • Define class descriptors
    • Including non-target classes
    • Perform classification

    multiscale_class_modelling
    multiscale_class_modelling
    XXX (© S. Lang, PLUS)

    Multiscale class modelling


    Composite object (yellow outline) consists of several components (sub-objects)

    • Create customized feature
    • Specify relative area of classified sub-objects
    multiscale_class_modelling
    XXX (© S. Lang, PLUS)

    References


    Lang, S., Hay, G., Baraldi, A., Tiede, D., & Blaschke, T. (2019). GEOBIA achievements and spatial opportunities in the era of big Earth observation data. ISPRS International Journal of Geo-Information, 8, pp. 474-483.
    doi: 10.3390/ijgi8110474


    Lang, S., & Tiede, D. (2015). Geospatial data integration in OBIA – implications of accuracy and validity. In P. Thenkabail (Ed.), Remote Sensing Handbook (Vol. Vol I - Land Resources: Monitoring, Modeling, and Mapping, pp. 295-316). New York: Taylor & Francis

    Blaschke, T., Hay, G. J., Kelly, M., Lang, S., Hofmann, P., Addink, E., . . . Tiede, D. (2014). Geographic Object-based Image Analysis: a new paradigm in Remote Sensing and Geographic Information Science. ISPRS Journal of Photogrammetry and Remote Sensing, 87(1), pp. 180-191
    doi: 10.1016/j.isprsjprs.2013.09.014


    Lang, S. (2008). Object-based image analysis for remote sensing applications: modeling reality – dealing with complexity. In T. Blaschke, S. Lang & G. J. Hay (Eds.), Object-Based Image Analysis - Spatial concepts for knowledge-driven remote sensing applications (pp. 3-28). Berlin: Springer
    doi: 10.1016/j.isprsjprs.2013.09.014


    Tiede, D., Lang, S., Albrecht, F., & Hölbling, D. (2010). Object-based class modeling for cadastre constrained delineation of geo-objects. Photogrammetric Engineering & Remote Sensing, 76(2), pp. 193-202. doi:10.14358/PERS.76.2.193
    Retrieved from: http://essential.metapress.com/content/15u6x2189425705l/