Overview of this page

This page is a fast-navigation toolset for getting information about modalities, centered around three modality taxonomy tree representations and a list of generator modalities. All modalities are described following a common template.

Modalities are different ways of representing and exchanging information among people and between people and computers. Modalities have very different properties that make them apt for very different purposes of information representation and exchange.

Original taxonomy structure [back to top, Tree 1, Tree 2, Tree 3, List]

The static graphics conceptual diagram below (Tree 1) shows a taxonomy of all modalities in the media of graphics, acoustics and haptics, organised into linguistic, analogue, arbitrary, and structure modalities.

Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.

Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, the generic level launches 20 numbered modality families whose descendancy lines are being kept track of using simple alphanumeric notation.

Tree 1 [back to top, Tree 1, Tree 2, Tree 3, List]

Super level Generic level Atomic level Sub-atomic level
Linguistic 1. Static analogue graphic elements
modalities »  2. Analogue acoustic elements
3. Analogue haptic elements 4a. Sign language discourse
4b. Sign language lab.-keyw. 5a1. Typed text
4. Dyn. analogue graphic elements »  4c. Sign language notation 5a2. Hand-written text
5a. Static text »  5b1. Typed lab.-keyw.
5. Static graphic language »  5b. Static labels/keywords »  5b2. Hand-written lab.-keyw.
5c. Static notation »  5c1. Typed notation
6a. Spoken discourse 5c2. Hand-written notation
6. Acoustic language »  6b. Spoken lab.-keyw.
6c. Spoken notation
7a. Text
7. Haptic language »  7b. Labels/keywords
7c. Notation
8a. Dynamic text
8. Dynamic graphic language »  8b. Dynamic lab.-keyw.
8c. Dynamic notation
8d. Visual spoken discourse
8e. Visual spoken lab.-keyw.
8f. Visual spoken notation
Analogue 9. Analogue static graphics »  9a. Images
modalities »  9b. Maps
9c. Compositional diagrams
9d. Graphs
9e. Conceptual diagrams
10. Analogue acoustics »  10a. Images »  10a1. Gesture » 
10b. Maps 10a2. Expression
10c. Compositional diagrams
10d. Graphs
10e. Conceptual diagrams
11. Analogue haptics »  11a. Images »  11a1. Gesture » 
11b. Maps 11a2. Expression
11c. Compositional diagrams 11a3. Body action
11d. Graphs
11e. Conceptual diagrams
12. Analogue dynamic graphics »  12a. Images »  12a1. Gesture » 
12b. Maps 12a2. Expression
Arbitrary 13. Arbitrary static graphics 12c. Compositional diagrams 12a3. Body action
modalities »  14. Arbitrary acoustics 12d. Graphs
15. Arbitrary haptics 12e. Conceptual diagrams
16. Arbitrary dynamic graphics
Structure 17. Static graphic structure
modalities 18. Acoustic structure
19. Haptic structure
20. Dynamic graphic structure

Taxonomy tree organised by medium [back to top, Tree 1, Tree 2, Tree 3, List]

The taxonomy tree below (Tree 2) has the same contents as the original taxonomy above but a different structure because it has been organised into modalities belonging to the media of graphics, acoustics, and haptics, respectively.

Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.

Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, the generic level launches 20 numbered modality families whose descendancy lines are being kept track of using simple alphanumeric notation.

Tree 2 [back to top, Tree 1, Tree 2, Tree 3, List]

Super level Generic level Atomic level Sub-atomic level
Graphic 1. Static analogue linguistic elements
modalities »  4. Dyn. analogue linguistic elements »  4a. Sign language discourse
4b. Sign language lab.-keyw. 5a1. Typed text
4c. Sign language notation 5a2. Hand-written text
5. Static graphic language »  5a. Static text »  5b1. Typed lab.-keyw.
5b. Static labels/keywords »  5b2. Hand-written lab.-keyw.
5c. Static notation »  5c1. Typed notation
8. Dynamic graphic language »  8a. Dynamic text 5c2. Hand-written notation
8b. Dynamic labels/keywords
8c. Dynamic notation
8d. Visual spoken discourse
8e. Visual spoken lab.-keyw.
8f. Visual spoken notation
9. Analogue static graphics »  9a. Images
9b. Maps
9c. Compositional diagrams
9d. Graphs
9e. Conceptual diagrams 12a1. Gesture » 
12. Analogue dynamic graphics »  12a. Images »  12a2. Expression
12b. Maps 12a3. Body action
13. Arbitrary static graphics 12c. Compositional diagrams
16. Arbitrary dynamic graphics 12d. Graphs
17. Static graphic structure 12e. Conceptual diagrams
20. Dynamic graphic structure
Acoustic 2. Analogue linguistic elements
modalities »  6. Acoustic language »  6a. Spoken discourse
6b. Spoken lab.-keyw.
6c. Spoken notation
10. Analogue acoustics »  10a. Images »  10a1. Gesture » 
10b. Maps 10a2. Expression
10c. Compositional diagrams
14. Arbitrary acoustics 10d. Graphs
18. Acoustic structure 10e. Conceptual diagrams
Haptic 3. Analogue linguistic elements
modalities »  7. Haptic language »  7a. Text
7b. Labels/keywords
7c. Notation
11. Analogue haptics »  11a. Images »  11a1. Gesture » 
11b. Maps 11a2. Expression
11c. Compositional diagrams 11a3. Body action
15. Arbitrary haptics 11d. Graphs
19. Haptic structure 11e. Conceptual diagrams

Sub-atomic-level extensions [back to top, Tree 1, Tree 2, Tree 3, List]

The taxonomy sub-tree below (Tree 3) extends the gesture part of the taxonomy to sub-sub-atomic level.

Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.

Legendum: the abbreviation lab.-keyw. means labels or keywords. An arrow (» ) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, descendancy lines are being kept track of using simple alphanumeric notation.

Tree 3 [back to top, Tree 1, Tree 2, Tree 3, List]

Super level Atomic level Sub-atomic level Sub-sub-atomic level
12a1a. Baton
Graphic 12a. Dynamic graphic images »  12a1. Gesture »  12a1b. Deictic
modalities »  12a2. Expression 12a1c. Emblem
12a3. Body action 12a1d. Iconic
12a1e. Metaphoric

10a1a. Baton
Acoustic 10a. Acoustic images »  10a1. Gesture »  10a1b. Deictic
modalities »  10a2. Expression 10a1c. Emblem
10a1d. Iconic
10a1e. Metaphoric
11a1a. Baton
Haptic 11a. Haptic images »  11a1. Gesture »  11a1b. Deictic
modalities »  11a2. Expression 11a1c. Emblem
11a3. Body action 11a1d. Iconic
11a1e. Metaphoric

Generator modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Generator modalities are those that (i) have been used to generate the numbered modalities in Trees 1, 2, and 3, that is, all modalities at generic, atomic, sub-atomic, and sub-sub-atomic level, and which (ii) do not themselves appear as numbered modalities in those trees. The reason generator modalities exist is that each numbered modality has been systematically generated from a series of distinctions between basic properties, like static/dynamic or graphic/acoustic/haptic. Each basic property defines a class of generator modalities.

For example, to generate modality 6a, spoken discourse, we need the concepts of acoustics as a medium, of linguistic modalities, of static and dynamic modalities, and of discourse modalities. None of these appear as numbered modalities in Trees 1, 2 and 3, and only some of them appear at super level in Trees 1 and 2. Rather, they are what the numbered modalities are made of. Thus, the parent of spoken discourse is acoustic language or speech (modality 6) which can be generated from the acoustic medium, the linguistic modalities, and the static/dynamic distinction. To generate spoken discourse from acoustic language, we need an additional distinction among modalities, i.e., the one between discourse, labels/keywords, and notation.

The List below shows all generator modalities. Clicking on a taxonomy item provides information about the modality as described in the template for describing modalities.

When looking up a modality from one of Trees 1, 2, or 3, or from the List, the modality's generator modalities, if any, are shown in the first entry, i.e., Inherits properties from.

Legendum: An arrow (->) following a modality name indicates that the modality has descendants at the next level. Same or similar background colour across two levels shows parent-offspring relationship. Moreover, descendancy lines are being kept track of using simple alphanumeric notation.

List [back to top, Tree 1, Tree 2, Tree 3, List]

Template for describing modalities [back to top, Tree 1, Tree 2, Tree 3, List]

All modalities are described following a common template. The current template has five entries:

Inherits properties from:

Specific properties:

Aptitude:

Descendants:

Multimodal combination:

Modality descriptions [back to top, Tree 1, Tree 2, Tree 3, List]

Super level [back to top, Tree 1, Tree 2, Tree 3, List]

Not a modality. The highest level of abstraction for distinguishing between modalities. Different categorisations exist at this level, resulting in the differently structured taxonomies shown above.

Generic level [back to top, Tree 1, Tree 2, Tree 3, List]

Not a modality. The second highest level of abstraction for distinguishing between modalities. This level currently includes members from 20 numbered modality families, many of which have descendants at lower levels.

Atomic level [back to top, Tree 1, Tree 2, Tree 3, List]

Not a modality. The third highest level of abstraction for distinguishing between modalities. This level currently includes members from 9 numbered modality families, some of which have descendants at lower levels.

Sub-atomic level [back to top, Tree 1, Tree 2, Tree 3, List]

Not a modality. The fourth highest level of abstraction for distinguishing between modalities. This level currently includes members from 4 numbered modality families, some of which have descendants at lower levels.

Sub-sub-atomic level [back to top, Tree 1, Tree 2, Tree 3, List]

Not a modality. The fifth highest level of abstraction for distinguishing between modalities. This level currently includes members from 3 numbered modality families.

Acoustic modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Acoustic modalities,

Aptitude: Some example conclusions are that acoustic modalities

Descendants:

Multimodal combination: Acoustic modalities,

Analogue modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Analogue modalities,

Aptitude: Analogue modalities

Descendants: see Tree 1. At generic level, the analogue sub-tree is split between static and dynamic analogue graphics, while such a split has not been found necessary at this point for the media of acoustics and haptics. Analogue acoustic representation is mostly dynamic, analogue haptic mostly static, whereas graphic representation is far more evenly balanced between the static and the dynamic. At atomic level, the analogue sub-tree splits into 4 times 5 modalities of the same type. Among these types, images are by far the broadest and most comprehensive form of representation, whereas maps, compositional diagrams, graphs, and conceptual diagrams have been invented and developed for specific purposes in each case, and originally for the graphics medium.

Multimodal combination:

Arbitrary modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Arbitrary modalities,

Aptitude: Arbitrary modalities,

Descendants: see Tree 1. Arbitrary modalities (i) have descendants at generic level only, and (ii) only the graphics daugthers have been split into static and dynamic. (i) is because arbitrary modalities appear to be relatively simple in each medium. (ii) is due to the assumption that, so far, the largest number of arbitrary representations by far have been created in the graphic medium, most of the time as static representations but also sometimes as dynamic representations.

Multimodal combination:

Baton gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image, gesture.

Specific properties: Baton gesture modalities

Aptitude: Baton gesture modalities

Descendants: See Tree 3 which distinguishes between baton gestures in the media of graphics, acoustics, and haptics.

Multimodal combination:

Body action modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image.

Specific properties: Body action modalities

Aptitude: Body action modalities

Descendants: see Tree 3. Body action is largely haptic and visual (or graphic), since it is quite limited how much we can act acoustically without those actions being acts of communication in the acoustic medium. Apart from screams that shatter glass, the closest we come to acoustic body action may be the groans and puffs we produce during physically exerting body action.

Multimodal combination:

Compositional diagram modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic.

Specific properties: Compositional diagram modalities

Aptitude: Compositional diagram modalities

Descendants: See Tree 1 at atomic level under analogue modalities. The current version of the taxonomy distinguishes between compositional diagrams in three media, and between static and dynamic graphic compositional diagrams.

Multimodal combination:

Conceptual diagram modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic.

Specific properties: Conceptual diagram modalities

Aptitude: Conceptual diagram modalities

Descendants: See Tree 1 at atomic level under analogue modalities. The current version of the taxonomy distinguishes between conceptual diagrams in three media, and between static and dynamic graphic conceptual diagrams.

Multimodal combination:

Deictic gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image, gesture.

Specific properties: Deictic gesture modalities

Aptitude: Deictic gesture modalities

Descendants: See Tree 3 which distinguishes between deictic gesture in the media of graphics, acoustics, and haptics.

Multimodal combination:

Diagram modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic.

Specific properties: Diagram modalities

Aptitude: Diagram modalities

Descendants: See Tree 1 at atomic level under analogue modalities. The current version of modality theory distinguishes between two types of diagram, compositional diagram and conceptual diagram.

Multimodal combination:

Discourse modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, static/dynamic.

Specific properties: Discourse modalities

Aptitude: Discourse modalities

Descendants: See Tree 1 under linguistic modalities. The current version of modality theory distinguishes between spoken-acoustic, spoken-visual, and sign language discourse, but does not at this point have a separate entry for the much more rare modality of haptic discourse.

Multimodal combination:

Dynamic modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Dynamic modalities,

Aptitude: Dynamic modalities

Descendants: See all trees. Note that a modality tree could be made which classifies all modalities at super level as being either static or dynamic. However, we haven't made such a tree representation of the modality taxonomy.

Multimodal combination:

Emblem gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image, gesture.

Specific properties: Emblem gesture modalities

Aptitude: Emblem gesture modalities,

Descendants: See Tree 3 which distinguishes between emblem gesture in the media of graphics, acoustics, and haptics.

Multimodal combination:

Expression modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image.

Specific properties: Expression modalities

Aptitude: Expression modalities

Descendants: See Tree 1 under analogue modalities. The current version of the modality taxonomy distinguishes between graphic, acoustic, and haptic expression at sub-atomic level.

Multimodal combination:

Gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image.

Specific properties: Gesture modalities

Aptitude: Gesture modalities,

Descendants: See Tree 1 under analogue modalities. The current version of the modality taxonomy distinguishes between graphic, acoustic, and haptic gesture at sub-atomic level, and - see Tree 3 - between five types of gesture per medium at sub-sub-atomic level.

Multimodal combination:

Graph modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic.

Specific properties: Graph modalities

Aptitude: Graph modalities

Descendants: See Tree 1 at atomic level under analogue modalities. Graphs are of many types although no agreed-upon typology has been established yet. The terminology in the field is rather confusing and heavily influenced by the traditional prevalence of graphic graphs, in particular 2D static graphic graphs. Possible types include (1) bar graphs which represent relatively small numbers of individual quantities and enable comparison and the spotting of relationships; (2) pie graphs which represent small numbers of parts of a whole with a view to comparing them in terms of quantity; (3) line graphs ("fever graphs", "curve charts") in which a line connects a series of discrete values collected over time or otherwise, enabling the spotting of trends; and (4) scatter graphs ("scatter plots", "dot charts") showing the distribution of discrete data points and enabling the spotting of flow, profiles, trends, history and projections of time-related and other data. Each of (1) through (4) have several sub-types in addition to existing in both 2D and 3D. All types and sub-types, it would seem, can be realised in both graphics and haptics, whereas acoustics poses problems for most of them, problems which so far have not been addressed much.

Multimodal combination:

Graphic modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Graphic modalities

Aptitude: Graphic modalities,

Descendants: See Tree 2. At generic level, graphic modalities split into linguistic, analogue, arbitrary, and structure varieties, all of which are described separately in their static and dynamic forms.

Multimodal combination:

Hand-written language modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static graphic language/dynamic graphic language.

Specific properties: Hand-written language modalities

Aptitude: Hand-written language modalities,

Descendants: See Tree 1 under linguistic modalities, sub-atomic level, which shows the static hand-written modalities text, labels/keywords, and notation (modalities 5a2, 5b2, 5c2). The corresponding dynamic hand-written modalities are not shown in the current version of the taxonomy but could be easily added as modalities 8a2, 8b2, 8c2. In general, the use of dynamic hand-written language is not recommended, because the issue of legibility of hand-written language gets compounded by the issue of not having freedom of perceptual inspection.

Multimodal combination:

Haptic modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Haptic modalities,

Aptitude: Haptic modalities

Descendants: See Tree 2. At generic level, haptic modalities split into linguistic, analogue, arbitrary, and structure varieties. No strong reason has been found for generating separate modalities for static and dynamic haptic representations of any kind. Each haptic modality in the current version of the taxonomy includes both static and dynamic instances of the modality.

Multimodal combination:

Iconic gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image, gesture.

Specific properties: Iconic gesture modalities

Aptitude: Iconic gesture modalities

Descendants: See Tree 3 which distinguishes between iconic gesture in the media of graphics, acoustics, and haptics.

Multimodal combination:

Image modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic.

Specific properties: Image modalities,

Aptitude: Image modalities

Descendants: See Tree 1 at atomic level under analogue modalities. Images, both static and dynamic, are an extremely important modality in graphics, which is why the taxonomy distinguishes between static and dynamic image modalities, whereas no distintion is being made yet between static and dynamic haptic and acoustic image modalities. The hypothesis is that both static and dynamic haptic images, and dynamic acoustic images, are extremely important to people as well, which we are gradually realising as we create immersive environments and simulations. It would seem that the distinction between perceptually rich images and sketches is valid across media.

Multimodal combination:

Label/keyword modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, static/dynamic.

Specific properties: Label or keyword modalities

Aptitude: Label/keyword modalities,

Descendants: See Tree 1 under linguistic modalities.

Multimodal combination:

Linguistic modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Linguistic modalities

Aptitude: Linguistic modalities,

Descendants:

Multimodal combination:

Map modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, compositional diagram.

Specific properties: Map modalities

Aptitude: Map modalities,

Descendants: See Tree 1 at atomic level under analogue modalities.

Multimodal combination:

Metaphoric gesture modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, static/dynamic, image, gesture.

Specific properties: Metaphoric gesture modalities

Aptitude: Metaphoric gesture modalities

Descendants: See Tree 3 which distinguishes between metaphoric gesture in the media of graphics, acoustics, and haptics.

Multimodal combination:

Notation modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, static/dynamic.

Specific properties: Notation modalities

Aptitude: Notation modalities

Descendants: See Tree 1 under linguistic modalities. Notation has atomic-level descendants for sign language, audio-visual spoken language, and graphic and haptic written language.

Multimodal combination:

Static modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Static modalities

Aptitude: Static modalities

Descendants: See all Trees.

Multimodal combination:

Structure modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: none.

Specific properties: Structure modalities

Aptitude: Structure modalities

Descendants: See Tree 1. Structure modalities (i) have graphic, acoustic and haptic descendants at generic level only, and (ii) only the graphics have been split into static and dynamic. (ii) is due to the assumption that, so far, the largest variety of structure representations has been created in the graphic medium, most of the time as static representations but sometimes as dynamic representations.

Multimodal combination: Structure modalities

Text modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, static/dynamic.

Specific properties: Text modalities,

Aptitude: Text modalities

Descendants: See Tree 1 under linguistic modalities.

Multimodal combination:

Typed language modalities [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static graphic language/dynamic graphic language.

Specific properties: Typed language modalities

Aptitude: Typed language modalities

Descendants: See Tree 1 under linguistic modalities, sub-atomic level, which shows the static typed language modalities text, labels/keywords, and notation (modalities 5a1, 5b1, 5c1). The corresponding dynamic typed language modalities are not shown in the current version of the taxonomy but could be easily added as modalities 8a1, 8b1, 8c1.

Multimodal combination:

1. Static graphic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, graphic, static.

Specific properties: This entry discusses the special case of static graphic languages that use analogue static graphic elements. The general case of static graphic language is discussed under modality (5). Language does not depend on whether its basic units of expression are analogue or non-analogue. Some written natural languages, however, use or include analogue units of expression as opposed to, e.g., the non-analogue latin letters used in the present text.
Static graphic language using analogue units

Aptitude: Static graphic language using analogue units

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination: See static graphic language using non-analogue units (modality 5).

2. Acoustic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, acoustic, static/dynamic.

Specific properties: This entry discusses the special case of analogue acoustic elements in acoustic languages. The general case of acoustic language is discussed under modality (6). Language does not depend on whether its basic units are analogue or non-analogue. However, many, if not all, spoken natural languages include a subset of words or other elements that have some acoustic similarity to what they represent.
Analogue acoustic linguistic elements

Aptitude: Analogue acoustic linguistic elements,

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination: See spoken language using non-analogue units (modality 6).

3. Haptic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, haptic, static/dynamic.

Specific properties: This entry discusses the special case of analogue elements in haptic languages. The general case of haptic language is discussed under modality (7). Language does not depend on whether its basic units are analogue or non-analogue. However, haptic language might conceivably include words or other units that have some haptic similarity to what they represent.
Haptic language using analogue units

Aptitude: Analogue haptic linguistic elements

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination: See haptic language using non-analogue units (modality 7).

4. Dynamic graphic language using analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, graphic, dynamic.

Specific properties: Dynamic graphic language using analogue units

Aptitude: Graphic sign language

Descendants: This generic-level modality has three descendants in the taxonomy: sign language discourse (4a), sign language labels/keywords (4b), and sign language notation (4c).

Multimodal combination: Graphic sign language,

4a. Graphic sign language discourse [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, discourse.

Specific properties: Discourse is maximum-expressive situated natural language. Graphic sign language discourse

Aptitude: Graphic sign language discourse

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

4b. Graphic sign language labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, labels/keywords.

Specific properties: Graphic sign language labels/keywords

Aptitude: Graphic sign language labels/keywords

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

4c. Graphic sign language notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, analogue, graphic, static/dynamic, dynamic graphic language using analogue units, notation.

Specific properties: Graphic sign language notation

Aptitude: Graphic sign language notation

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5. Static graphic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static.

Specific properties: This entry discusses static graphic language in general, except for the small contribution made under modality (1) on analogue static graphic elements.
Static graphic language

Aptitude: Static graphic language

Descendants: This generic-level modality has three descendants in the taxonomy: text (5a), labels/keywords (5b), and notation (5c).

Multimodal combination:

5a. Static graphic text [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, text.

Specific properties: Text is maximum-expressive non-situated natural language. Static graphic text

Aptitude: Static graphic text

Descendants: This atomic-level modality has two descendants in the taxonomy: static graphic typed text (5a1) and static graphic hand-written text (5a2).

Multimodal combination:

5b. Static graphic labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords.

Specific properties: Static graphic labels/keywords

Aptitude: Static graphic labels/keywords

Descendants: This atomic-level modality has two descendants in the taxonomy: typed labels/keywords (5b1) and hand-written labels/keywords (5b2).

Multimodal combination:

5c. Static graphic notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, notation.

Specific properties: Static graphic notation

Aptitude: Static graphic notation

Descendants: This atomic-level modality has two descendants in the taxonomy: typed notation (5c1) and hand-written notation (5c2).

Multimodal combination:

5a1. Static graphic typed text [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, text, static graphic text, typed language.

Specific properties: Static graphic typed text

Aptitude: Static graphic typed text

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5a2. Static graphic hand-written text [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, text, static graphic text, hand-written language.

Specific properties: Static graphic hand-written text

Aptitude: Static graphic hand-written text

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5b1. Static graphic typed labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords, static graphic labels/keywords, typed language.

Specific properties: Static graphic typed labels/keywords

Aptitude: Static graphic typed labels/keywords

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5b2. Static graphic hand-written labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, labels/keywords, static graphic labels/keywords, hand-written language.

Specific properties: Static graphic hand-written labels/keywords

Aptitude: Static graphic hand-written labels/keywords

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5c1. Static graphic typed notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, notation, static graphic notation, typed language.

Specific properties: Static graphic typed notation

Aptitude: Static graphic typed notation

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

5c2. Static graphic hand-written notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static, static graphic language, notation, static graphic notation, hand-written language.

Specific properties: Static graphic hand-written notation

Aptitude: Static graphic hand-written notation

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

6. Acoustic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, acoustic, static/dynamic.

Specific properties: This entry discusses spoken language, or the speech modality, in general, except for the small and particular contribution made by analogue words (se modality 2). Spoken language

Aptitude: Some example conclusions based on the properties of spoken language and its ancestors in the taxonomy, are that speech

Descendants: This generic-level modality has three descendants in the taxonomy: spoken discourse (6a), spoken labels/keywords (6b), and spoken notation (6c).

Multimodal combination: Spoken language

6a. Spoken discourse [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, discourse.

Specific properties: Spoken discourse

Aptitude: Spoken discourse

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

6b. Spoken labels/keywords. [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, labels/keywords.

Specific properties: Spoken labels/keywords

Aptitude: Spoken labels/keywords

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

6c. Spoken notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, acoustic, static/dynamic, acoustic language, notation.

Specific properties: Spoken notation

Aptitude: Spoken notation

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

7. Haptic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, haptic, static/dynamic.

Specific properties: This entry discusses haptic language in general, both static and dynamic, except for the small contribution made under modality (3) on analogue haptic elements.
Haptic language is of at least three very different types:

Aptitude: Haptic output language (non-situated),

Descendants: This generic-level modality has three descendants in the taxonomy: haptic text (7a), haptic labels/keywords (7b), and haptic notation (7c).

Multimodal combination:

7a. Haptic text [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, text.

Specific properties: Text is maximum-expressive non-situated natural language. Haptic text,

Aptitude: Haptic text

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

7b. Haptic labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, labels/keywords.

Specific properties: Haptic labels/keywords

Aptitude: Haptic labels/keywords

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

7c. Haptic notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, haptic, static/dynamic, haptic output language, notation.

Specific properties: Haptic notation

Aptitude: Haptic notation,

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

8. Dynamic graphic language using non-analogue units [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, dynamic.

Specific properties: Dynamic graphic language

Aptitude: Dynamic graphic writing

Aptitude: Visual speech

Descendants: This generic-level modality has six descendants of two different kinds: (i) dynamic graphic writing, consisting of dynamic text (8a), dynamic labels/keywords (8b), and dynamic notation (8c); and (ii) visual speech consisting of graphic spoken discourse (8d), graphic spoken labels/keywords (8e), and graphic spoken notation (8f).

Multimodal combination:

8a. Dynamic graphic text [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, text.

Specific properties: Dynamic graphic text,

Aptitude: Dynamic graphic text

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic text exists in at least two different varieties, typed and hand-written, cf. static modalities 5a1 and 5a2.

Multimodal combination:

8b. Dynamic graphic labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, labels/keywords.

Specific properties: Dynamic graphic labels/keywords

Aptitude: Dynamic graphic labels/keywords

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic labels/keywords exist in at least two different varieties, typed and hand-written, cf. static modalities 5b1 and 5b2.

Multimodal combination:

8c. Dynamic graphic notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, dynamic, dynamic graphic language, notation.

Specific properties: Dynamic graphic notation

Aptitude: Dynamic graphic notation,

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. However, dynamic graphic notation exists in at least two different varieties, typed and hand-written, cf. static modalities 5c1 and 5c2.

Multimodal combination:

8d. Visual spoken discourse [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken discourse.

Specific properties: Visual spoken discourse

Aptitude: Visual spoken discourse,

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

8e. Visual spoken labels/keywords [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken labels/keywords.

Specific properties: Visual spoken labels/keywords,

Aptitude: Visual spoken labels/keywords,

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

8f. Visual spoken notation [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: linguistic, graphic, static/dynamic, dynamic graphic language, spoken notation.

Specific properties: Visual spoken notation

Aptitude: Visual spoken notation,

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

9. Analogue static graphics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static.

Specific properties: Analogue static graphics

Aptitude: Analogue static graphics are apt for visualising information in 1D, 2D and 3D (scale model) spatial while preserving full freedom of perceptual inspection, representations serving as snapshots of real or imagined objects, processes, and events or aspects of these. Several snapshots can be sequenced into a cartoon to represent causality and change over time.

Descendants: This generic-level modality has five descendants in the taxonomy: static graphic images (9a), static graphic maps (9b), static graphic compositional diagrams (9c), static graphic graphs (9d), and static graphic conceptual diagrams (9e).

Multimodal combination:

9a. Static graphic images [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static, analogue static graphics, image.

Specific properties: Static graphic images

Aptitude: Static graphic images

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some commonly referred-to prototypes include sketches, photo-realistic images, and semi-abstract images whose specificity has been reduced in order to highlight identity, like the basic shapes of geometry in textbooks. Static graphic image icons are an essential component of the classical WIMP (Windows, Icons, Menus and Pointing) graphical user interface.

Multimodal combination:

9b. Static graphic maps [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static, analogue static graphics, map.

Specific properties: Static graphic maps

Aptitude: Static graphic maps

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. The prototypical static graphic map is the roadmap, a drawn, reduced-scale, birds-eye 2D schematic representation of part of the surface of the earth, showing selected features (or points of interest), such as roads, rivers, mountains and cities, and having been designed to enable travellers to find the right route between geographical locations. Note that this map is typically bi-modal because points of interest are labelled using labels like those of modality (5b).

Multimodal combination:

9c. Static graphic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static, analogue static graphics, compositional diagram.

Specific properties: A diagram is an analytical image. Static graphic compositional diagrams

Aptitude: Static graphic compositional diagrams afford freedom of visual inspection of analytical information about spatial and spatio-temporal decomposition.

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, static graphic maps (modality 9b) are a descendant because maps are a kind of compositional diagram.

Multimodal combination:

9d. Static graphic graphs [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static, analogue static graphics, graph.

Specific properties: Static graphic graphs

Aptitude: Static graphic graphs

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some familiar graph types are line graphs, bar graphs, pie graphs, and scatter plot graphs.

Multimodal combination:

9e. Static graphic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, static, analogue static graphics, conceptual diagram.

Specific properties: Static graphic conceptual diagrams

Aptitude: Static graphic conceptual diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10. Analogue static-dynamic acoustics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic.

Specific properties: Analogue acoustics

Aptitude: Analogue acoustics are apt for auralising information about real or imagined objects, processes, and events or aspects of these.

Descendants: This generic-level modality has five descendants in the taxonomy: acoustic images (10a), acoustic maps (10b), acoustic compositional diagrams (10c), acoustic graphs (10d), and acoustic conceptual diagrams (10e).

Multimodal combination:

10a. Acoustic images [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, image.

Specific properties: Acoustic images

Aptitude: Acoustic images

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Earcons (acoustic image icons) are common in graphical user interfaces as, e.g., user action feedback and alerts.

Multimodal combination:

10b. Acoustic maps [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, map.

Specific properties: Acoustic maps

Aptitude: Acoustic maps

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. We are not aware of any prototypical acoustic maps.

Multimodal combination:

10c. Acoustic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, compositional diagram.

Specific properties: A diagram is an analytical image. Acoustic compositional diagrams

Aptitude: Acoustic compositional diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10d. Acoustic graphs [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, graph.

Specific properties: Acoustic graphs

Aptitude: Acoustic graphs

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. The simplest and most obvious type seems to be a graph that represents changes of a single parameter over time.

Multimodal combination:

10e. Acoustic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue static-dynamic acoustics, conceptual diagram.

Specific properties: Acoustic conceptual diagrams

Aptitude: Acoustic conceptual diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10a1. Acoustic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture.

Specific properties: Gesture is movement of a body part, several body parts, or the body as a whole to convey shared non-linguistic meaning. Acoustic gesture

Aptitude: Acoustic gesture

Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: acoustic batons (10a1a), acoustic deictic gestures (10a1b), acoustic emblems (10a1c), acoustic iconic gestures (10a1d), and acoustic metaphoric gestures (10a1e).

Multimodal combination:

10a2. Acoustic expression [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, expression.

Specific properties: Acoustic expression

Aptitude: Acoustic expression

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy. Candidate classifications should not forget music and song in addition to tone of voice and other acoustic expressions.

Multimodal combination:

10a1a. Acoustic batons [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, baton.

Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Acoustic batons

Aptitude: Acoustic batons

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10a1b. Acoustic deictic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, deictic.

Specific properties: Acoustic deictic gesture

Aptitude: Acoustic deictic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10a1c. Acoustic emblems [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, emblem.

Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Acoustic emblems

Aptitude: Acoustic emblems

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10a1d. Acoustic iconic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, iconic.

Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Acoustic iconic gesture

Aptitude: Acoustic iconic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

10a1e. Acoustic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, acoustic, static/dynamic, analogue acoustics, image, acoustic images, gesture, acoustic gesture, metaphoric.

Specific properties: Metaphoric gesture makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Acoustic metaphoric gesture

Aptitude: Acoustic metaphoric gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11. Analogue static-dynamic haptics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic.

Specific properties: Analogue haptics

Aptitude: Analogue haptics

Descendants: This generic-level modality has five descendants in the taxonomy: haptic images (11a), haptic maps (11b), haptic compositional diagrams (11c), haptic graphs (11d), and haptic conceptual diagrams (11e).

Multimodal combination:

11a. Haptic images [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image.

Specific properties: Haptic images

Aptitude: Haptic images

Descendants: This atomic-level modality has two descendants in the taxonomy: haptic gesture (11a1) and haptic body action (11a2).

Multimodal combination:

11b. Haptic maps [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, map.

Specific properties: Haptic maps

Aptitude: Haptic maps

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Nor is there any prototypical instances yet, like the 2D roadmap is a prototype of a static graphic map (modality 9b). However, a typical haptic map may be expected to include linguistically labelled points of interest, using, e.g., spoken or Braille output labelling, making the map at least bi-modal.

Multimodal combination:

11c. Haptic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, compositional diagram.

Specific properties: A diagram is an analytical image. Haptic compositional diagrams

Aptitude: Haptic compositional diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, haptic maps (modality 11b) are a descendant because maps are a kind of compositional diagram.

Multimodal combination:

11d. Haptic graphs [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, graph.

Specific properties: Haptic graphs

Aptitude: Haptic graphs

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. Some familiar graph types are line graphs, bar graphs, pie graphs, and scatter plot graphs.

Multimodal combination:

11e. Haptic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, conceptual diagram.

Specific properties: Haptic conceptual diagrams

Aptitude: Haptic conceptual diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1. Haptic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture.

Specific properties: Gesture is movement of a body part, several body parts, or the body as a whole to convey shared meaning non-linguistically. Haptic gesture

Aptitude: Haptic gesture

Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: haptic batons (11a1a), haptic deictic gestures (11a1b), haptic emblems (11a1c), haptic iconic gestures (11a1d), and haptic metaphoric gestures (11a1e).

Multimodal combination:

11a2. Haptic expression [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, expression.

Specific properties: Haptic expression

Aptitude: Haptic expression

Descendants:

Multimodal combination:

11a3. Haptic body action [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from:analogue, haptic, static/dynamic, analogue haptics, image, haptic images, body action.

Specific properties: All representation and exchange of information, like speaking or showing a graph, requires physical action by humans or computers. Body action is physical action that does not issue any of the representations described in the taxonomy, from linguistic to arbitrary representations. Haptic body action

Aptitude: Haptic body action

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1a. Haptic batons [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, baton.

Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Haptic batons

Aptitude: Haptic batons

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1b. Haptic deictic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, deictic.

Specific properties: Haptic deictic gesture

Aptitude: Haptic deictic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1c. Haptic emblems [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, emblem.

Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Haptic emblems

Aptitude: Haptic emblems

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1d. Haptic iconic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, iconic.

Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Haptic iconic gesture

Aptitude: Haptic iconic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

11a1e. Haptic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, haptic, static/dynamic, analogue haptics, image, haptic images, gesture, haptic gesture, metaphoric.

Specific properties: Metaphoric gesture makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Haptic metaphoric gesture

Aptitude: Haptic metaphoric gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12. Analogue dynamic graphics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic.

Specific properties: Analogue dynamic graphics

Aptitude: Analogue dynamic graphics

Descendants: This generic-level modality has five descendants in the taxonomy: dynamic graphic images (12a), dynamic graphic maps (12b), dynamic graphic compositional diagrams (12c), dynamic graphic graphs (12d), and dynamic graphic conceptual diagrams (12e).

Multimodal combination:

12a. Dynamic graphic images [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image.

Specific properties: Dynamic graphic images

Aptitude: Dynamic graphic images

Descendants: This atomic-level modality has three descendants in the taxonomy: visual facial expression (12a1), visual gesture (12a2), and visual body action (12a3). Dynamic graphic image icons are appearing in graphical user interfaces.

Multimodal combination:

12b. Dynamic graphic maps [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, map.

Specific properties: Dynamic graphic maps - a kind of dynamic graphic compositional diagram (modality 12c) -

Aptitude: Dynamic graphic maps

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12c. Dynamic graphic compositional diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, compositional diagram.

Specific properties: A diagram is an analytical image. Dynamic graphic compositional diagrams

Aptitude: Dynamic graphic compositional diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy. In fact, however, dynamic graphic maps (modality 12b) are a descendant because maps are a kind of compositional diagram.

Multimodal combination:

12d. Dynamic graphic graphs [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, graph.

Specific properties: Dynamic graphic graphs

Aptitude: Dynamic graphic graphs

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12e. Dynamic graphic conceptual diagrams [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, conceptual diagram.

Specific properties: Dynamic graphic conceptual diagrams

Aptitude: Dynamic graphic conceptual diagrams

Descendants: This atomic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1. Graphic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture.

Specific properties: Gesture is movement of any body part, several body parts, or the body as a whole to convey shared meaning non-linguistically. Graphic gesture

Aptitude: Graphic gesture

Descendants: This sub-atomic level modality has five descendants in the current version of the taxonomy: graphic batons (12a2a), graphic deictic gestures (12a2b), graphic emblems (12a2c), graphic iconic gestures (12a2d), and graphic metaphoric gestures (12a2e).

Multimodal combination:

12a2. Graphic expression [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, expression.

Specific properties: Graphic expression

Aptitude: Graphic expression

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a3. Graphic body action [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, body action.

Specific properties: All representation and exchange of information, like speaking or showing a graph, requires physical action by humans or computers. Body action is physical action that does not issue any of the representations described in the taxonomy, from linguistic to arbitrary representations. Graphic body action

Aptitude: Graphic body action

Descendants: This sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1a. Graphic batons [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, baton.

Specific properties: A baton is a rhytmic movement, typically of the hand-arm, that typically accompanies speech for visible emphasis. Graphic batons

Aptitude: Graphic batons

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1b. Graphic deictic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, deictic.

Specific properties: Graphic deictic gesture

Aptitude: Graphic deictic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1c. Graphic emblems [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, emblem.

Specific properties: Emblems are gestures that are more or less equivalent to conventional linguistic expressions. Graphic emblems

Aptitude: Graphic emblems

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1d. Graphic iconic gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, iconic.

Specific properties: Iconic gesture is like very sketchy analogue rendering of objects and events. Graphic iconic gesture

Aptitude: Graphic iconic gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

12a1e. Graphic metaphoric gesture [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: analogue, graphic, dynamic, analogue dynamic graphics, image, dynamic graphic images, gesture, dynamic graphic gesture, metaphoric.

Specific properties: Metaphoric gesture typically makes a very sketchy analogue rendering of some source domain and wants it interpreted as being about some different target domain. Graphic metaphoric gesture

Aptitude: Graphic metaphoric gesture

Descendants: This sub-sub-atomic level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

13. Arbitrary static graphics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: arbitrary, graphic, static.

Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like creating a blue team and an orange team prior to playing some game. Arbitrary static graphics

Aptitude: Arbitrary static graphics

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

14. Arbitrary static-dynamic acoustics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: arbitrary, acoustic, static/dynamic.

Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like agreeing to whistle once if everything is calm and three times if there is danger. Arbitrary acoustics

Aptitude: Arbitrary acoustics

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

15. Arbitrary static-dynamic haptics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: arbitrary, haptic, static/dynamic.

Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already. Arbitrary haptics

Aptitude: Arbitrary haptics

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

16. Arbitrary dynamic graphics [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: arbitrary, graphic, dynamic.

Specific properties: Arbitrary representation consists in assigning meaning ad hoc to something that does not have conflicting shared meaning already, like creating a blue team and an orange team prior to playing some game. Arbitrary dynamic graphics

Aptitude: Arbitrary dynamic graphics

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

17. Static graphic structure [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: structure, graphic, static.

Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Static graphic structure

Aptitude: Static graphic structure

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

18. Static-dynamic acoustic structure [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: structure, acoustic, static/dynamic.

Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Acoustic structure

Aptitude: Acoustic structure

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

19. Static-dynamic haptic structure [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: structure, haptic, static/dynamic.

Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Haptic structure

Aptitude: Haptic structure

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

20. Dynamic graphic structure [back to top, Tree 1, Tree 2, Tree 3, List]

Inherits properties from: structure, graphic, dynamic.

Specific properties: Modality structure structures space and time in ways that people perceive. These structures may be highlighted or made explicit by adding semi-abstract geometric objects, such as points, lines, frames, grids, or boxes, temporal order and pauses, etc. Dynamic graphic structure

Aptitude: Dynamic graphic structure

Descendants: This generic-level modality has no descendants in the current version of the taxonomy.

Multimodal combination:

Taxonomy modifications compared to book version [back to top, Tree 1, Tree 2, Tree 3, List]

Modality theory is work in progress. Preparation of the Modalities part of this website has led to the following extensions and modifications of the taxonomy presented in the book: