Intelligence Augmentation Framework

This is part 2 of the Revisiting Augmenting Human Intellect series, where I break down Douglas Engelbart's Augmenting Human Intellect, the classic paper on how computers can improve human intellectual effectiveness.

In this post, we will explore the framework Doug used to break down the intelligence augmentation problem, including the nature of intelligence, the tools we use to augment it, and three principles for optimizing our capabilities.

For an introduction to this series and the background of Augmenting Human Intellect, start with Part 1.

Because this series is still evolving, I'd love to hear your suggestions, feedback, or questions! Feel free to send me a message via twitter, facebook, or email.

You can also subscribe to get new posts when they come out.

Table of Contents

• The Source of Intelligence
• Four Classes of Augmentation Means
• H-LAM/T system
• First Principle: Basic Capability Repertoire Hierarchy
• Second Principle: Executive and Direct-Contributive Capabilities
• Third Principle: Synergetic Structuring
• The Relationship between Synergetic Structuring and Augmentation Means
• Understand Symbol and Concept through Semiotics

The Source of Intelligence

Imagine someone from 1824 suddenly teleported to 2024 and using Uber on an iPhone for the first time. They might wonder:

1. How does this shiny glass (when the word screen hasn't acquired its modern meaning yet) know where I'm touching?
2. How does this small box know the maps and where other cars are?
3. How does the driver know where I am and the best route to pick me up?
4. How can I pay with just numbers from a metal card?
5. And why is it so expensive when it suddenly rains?

A small box capable of performing all these complex tasks seems too magical. It must possess god-like, omniscient intelligence¹, right? Perhaps that half-bitten apple logo on the back is from an advanced extraterrestrial civilization with all this unfathomable power to gather and process information?

The misconception of intelligence

To augment any intelligence, we need to know the source of the intelligence first. When we see a computer system performing complex tasks, we often attribute its intelligence to a single, mysterious power.

However, if we examine this system closely, we'll find that its intelligence actually comes from layers of simple functions instead of a single inherent intelligence capability.

Hierarchy of functional components

The source of intelligence is distributed across a hierarchy of functional processes that breaks down complexity through layers of organization.

When we request an Uber ride, thousands of different functional components work together. Uber's app communicates with its servers, which are composed of thousands of small software components called microservices². Each microservice performs a specific function and communicates with others.

For example, one component uses algorithms to determine the closest driver, while another considers real-time traffic data and optimal routes. You can see their architecture graph below to understand the number of these components and how they interact.

These software components can be broken down into at least six more layers of hierarchy. For example:

1. At the top level, a software system architect designs how each component fits into the overall architecture.
2. At the second level, a software engineer implements the component using high-level programming languages like Java or Python.
3. At the third level, the compiler translates high-level code into low-level machine code.
4. At the fourth level, a hardware systems architect designs how different hardware components (CPU cores, RAM, SSDs, etc.) work together.
5. At the fifth level, a hardware engineer designs the internal organization of these components.
6. At the sixth level, an electrical engineer deals with the foundational electronic components like transistors, capacitors, and integrated circuits.

Artificial Intelligence's Intelligence

Similarly, for an artificial intelligence (AI) system like a self-driving car, what enables an autonomous vehicle to drive isn't one big machine learning model that processes inputs from cameras and uses a mysterious power to make decisions like turning right or accelerating as output.

Instead, the system's intelligence comes from multiple machine learning (ML) models working together in a hierarchical structure. For instance, there will be ML models for object detection, object tracking, prediction, decision-making, and many other components for non-ML functions like data processing.

Beyond the software and hardware aspects, each social, life, or physical phenomenon also derives from a hierarchy of organized functions. Human intelligence, for instance, ultimately comes from the characteristics of individual nerve cells. Even remembering the first thing we did after waking up today (an episodic memory) requires multiple parts of our brain, like the prefrontal cortex (PFC) and hippocampus, to work together.

Now that we've demystified the source of intelligence within a system, let's explore the components of enhancing the capabilities of groups of humans as systems to solve complex problems.

Four Classes of Augmentation Means

Doug Engelbart defined four fundamental classes of "augmentation means" that enhance human capabilities:

Artifact

Artifacts are physical objects and digital tools³ designed to enhance human capabilities to manipulate physical things or symbols.

For example, a hammer is an artifact that helps us drive nails, and Google Docs software is an artifact that expands our capacity to edit and organize text.

Language

Languages are systems of symbols and concepts we use to understand, think, and communicate.

For instance, spoken and written English is one of our most common languages. Other forms of symbolic representation, like mathematical notation or musical scores, are also examples of languages.

Methodology

Methodologies are organized methods, procedures, or strategies we use to achieve specific goals and solve problems.

For example, a recipe is a methodology for cooking a specific dish, and the scientific method is a methodology for investigating and understanding the natural world.

Training

Training is the process of learning and practicing the skills needed to use artifacts, languages, and methodologies effectively.

For instance, learning to ride a bicycle requires training to develop the physical coordination needed to use the artifact effectively. Similarly, learning a new programming language requires training to understand its syntax and concepts.

H-LAM/T system

The four classes of augmentation means are interconnected. We use language to communicate about artifacts and methodologies, and training is essential to use them effectively. Together, these augmentation means form a system that enables us to understand and interact with the world.

Let's recap the goal of Augmenting Human Intellect:

Use computers to enhance humans' ability to solve more complex problems together.

To achieve this, Doug proposed treating "a group of trained humans together with their artifacts, language, and methodology" as an entire system, which he called the H-LAM/T system (Human using Language, Artifacts, Methodology / Training).

This allows us to approach the augmentation of collective intelligence as an optimization problem for a unified system.

There are three principles of optimizing this H-LAM/T system.

First Principle: Basic Capability Repertoire Hierarchy

The first principle of optimizing the H-LAM/T system is identifying the fundamental building blocks of human intellect and the tools we create. These building blocks are the basic capabilities we use to interact with the world and solve problems. These capabilities exist in two domains:

1. Human domain: Our natural abilities like thinking, remembering, and using our senses.
2. Artifact domain: The capabilities of the tools we create, like computers and pencils.

To identify basic capabilities, we can break down complex abilities in both domains through a recursive decomposition process:

1. Start with a complex ability.
2. Ask if this ability can be improved with any augmentation means.
3. If yes, break it down into simpler components.
4. If no, we've found a basic capability.
5. Repeat until reaching abilities that cannot be effectively improved.

For example, writing an essay in the human domain can be broken down into typing, which involves pressing keys, ultimately requiring finger movements (a basic human capability). In the artifact domain, writing involves editing information, which can be broken down into storing data (a basic artifact capability).

This decomposition reveals two classes of basic capabilities within each domain:

1. Used Capabilities: Abilities actively utilized in our daily problem-solving and tool use.
2. Unused Capabilities: Potential abilities yet to be utilized or discovered.

By identifying these basic capabilities, we can construct a basic capability repertoire hierarchy. This hierarchical collection of basic capabilities allows us to explore our natural limits and envision new ways to augment our abilities.

Furthermore, by decomposing both domains, we can leverage the combined capabilities of humans and artifacts to optimize our collective problem-solving.

Second Principle: Executive and Direct-Contributive Capabilities

The second principle of optimizing the H-LAM/T system is understanding capabilities' distinct roles within a hierarchical network. These roles are divided into two primary classes: executive and direct-contributive.

Executive Class Capabilities

Executive capabilities are responsible for planning, comprehending, and executing processes. They include high-level decision-making, strategy formulation, and overall supervision.

The executive class delegates tasks to executive classes in lower levels or direct-contributive classes.

Direct-contributive Class Capabilities

Direct-contributive capabilities are organized by the executive class and directed towards the realization of specific tasks within the capabilities hierarchy.

For example, when we pick up and answer a phone, deciding to pick up is an executive capability, and physically picking up the phone is a direct-contributive capability.

Executive Class Automation

Capabilities vary across different levels in the H-LAM/T system. Lower-level capabilities are more automatic, while higher-level ones require conscious effort and planning.

Humans need organization and management to divide finite capabilities between executive and direct-contributive activities. For complex problems, the executive class must frequently update its plans as new options emerge and our comprehension evolves.

While humans often prefer to retain control over high-level decision-making, it can be helpful to let computers automate some executive functions. This can free up our time and cognitive resources from being misallocated to the wrong direct-contributive processes.

Therefore, technological augmentation should not only enhance our direct capabilities but also optimize the management and allocation of our cognitive resources.

Third Principle: Synergetic Structuring

The third principle of optimizing the H-LAM/T system is synergetic structuring, which refers to the organization of elements that produces an effect greater than the sum of their individual effects. Synergism occurs when the entire structure is more effective than the sum of its parts.

This principle is fundamental to building sophisticated capabilities from basic ones. In Doug's words, it is "our most likely candidate for representing the actual source of intelligence".

Synergetic structuring applies to five interconnected types of structures within the H-LAM/T system:

Mental Structuring

Mental structuring is the internal organization of conscious and unconscious mental images that provide intuition, judgment, and inference.

For example, a chess player recognizing patterns on the board relies on mental structuring to anticipate and plan moves, demonstrating more effective strategies than considering each piece in isolation.

Concept Structuring

Concept structuring is the capacity to use concepts as tools for mental processes and combine basic concepts into complex ideas for problem-solving.

For instance, we can use mathematical operations to solve equations, and combining the concepts of subtraction and multiplication allows us to solve more complicated equations.

Symbol Structuring

Symbol structuring is the structured representations of concepts that enable communication and externalization of thought.

For instance, a flowchart uses symbols like arrows, circles, and rectangles to represent the steps in a process, their sequence, and their inputs/outputs. The periodic table in chemistry uses letters and numbers to organize elements and demonstrate relationships between their properties.

Process Structuring

Process structuring is the organization and execution of processes to manipulate symbols and coordinate actions toward specific goals efficiently.

For example, in the agile project management framework, tasks are typically structured into fixed-length sprints with planning, standup, and retrospective meetings to ensure the project can be delivered on time.

Physical Structuring

Physical structuring is the physical construction and arrangement of artifacts in the system, representing the tangible aspects of augmentation means.

For instance, a car engine assembly arranges individual parts into a functional system. Bret Victor's Communal computing for 21st century science is a novel example where projectors and laser pointers are arranged to turn the whole biomolecular lab space into a computer.

Five types of structures are interdependent

These five types of structuring are interdependent and cyclic. Each type relies on and influences the others, creating a dynamic system that continually enhances human intellectual capabilities.

For example, improvements in process structuring through advanced artifacts like computers can lead to better concept and mental structuring, which in turn can further enhance process structuring.

The Relationship between Synergetic Structuring and Augmentation Means

Having explored the four classes of augmentation means and the five types of synergetic structuring, let's review them together to understand their relationship.

Physical structuring to Artifact

Physical structuring directly relates to the artifact component in our augmentation means, focusing on the construction of tools and technologies.

Process structuring to Methodology

Process structuring represents the methodology component in our augmentation means but extends beyond it. It involves not only the organization of processes but also their execution. Therefore, process structuring serves as both the theoretical framework for how things should be done and the actual implementation of doing them.

Symbol and concept structuring to Language

Symbol structuring and concept structuring together form the language component in our augmentation means. Symbol structuring deals with the symbolic representation of a language, while concept structuring organizes the internal representation of ideas and their relationships in a language.

Mental structuring to Symbol and Concept

Though not directly mapped to a specific augmentation means, mental structuring forms the foundation of the entire system.

We can view mental structuring as the space where we find concepts in the form of conscious or unconscious feelings, scenes, and images humans have experienced. Concepts, therefore, serve as the medium of exchange between mental structures and their symbolic expression.

Because symbol structuring and concept structuring are crucial to language, our primary tool for thought, let's explore their relationship from a different angle.

Understand Symbol and Concept through Semiotics

I found studying semiotics, the study of signs, helpful to better understand the relationship between Symbol and Concept.

Augustine, sometimes credited as the founder of semiotics, wrote in De Doctrina Christiana to explain what is a sign:

“A sign is a thing which causes us to think of something beyond the impression the thing itself makes upon the senses. [...] All instruction is either about things or about signs; but things are learnt by means of signs."

Signifier and Signified

Ferdinand de Saussure, a linguist and one of the most influential figures in the development of semiotics, defined a sign as being composed of two parts:

1. Signifier: the form which the sign takes.
2. Signified: the concept which the sign represents.

Together, the signifier and signified make up the sign, the smallest unit of meaning that can be used to communicate.

For example, the word "Open" on a shop doorway is a sign consisting of the English word "Open" as the signifier and the concept that the shop is open for business as the signified.

Using Saussure's Dyadic Model, we can roughly think of concepts as the signified and symbols as the signifier⁴. Concepts and symbols together form the unit for us to think and communicate with each other.⁵

Take written communication as an example. When writing, the writer encodes concepts (signified) in their head into symbols (signifier) on the paper. Conversely, the reader decodes the symbols (signifier) into concepts (signified) when reading.

Charles Sanders Peirce's Three types of signs

While Saussure mainly focused on the linguistic part of semiotics and the arbitrary nature of the linguistic sign, Charles Sanders Peirce, another key figure in semiotics, focused on a broader range of signs.

1. Indexical sign: Based on causality or correlation, like smoke indicating fire.
2. Iconic sign: Based on resemblance, like portraits, cartoons, scale models, onomatopoeia.
3. Symbolic sign: Based on conventions (which are arbitrary and must be learned), like language, numbers, morse code, and national flags.

These categories show that the relationship between concepts and symbols is not one-to-one. For example, the concept of fire can be represented by the word "fire" (symbolic), a fire emoji (iconic), a burned object (indexical), or even another word "flame" (symbolic).

There can be one-to-many, many-to-one, and many-to-many mappings between concepts and symbols. This complex interplay of meanings is crucial for human thinking, as Peirce put it: "We think only in signs".

In the next post, I will explain how better mapping between concepts and symbols enhances our thinking. We will explore the four elements of symbol manipulation, three ways modern computers "think," the distinction between Intelligence Amplification and Intelligence Augmentation, and finally, the three directions for augmenting human intellect. Stay tuned!

Thanks to Angelica Kosasih for reading the draft of this and giving feedback.