Registered AR: The (Next) Future of Human-Computer Interaction

Ever since humans began using tools, we have been designing them to fit our bodies so we can physically hold, swing, and manipulate them more easily. As our tools became more mechanically complex, such as the machinery of the industrial revolution, we designed them so that our physical interactions maximized output. Our technology grew more advanced still, and we leaped from systems which augment our physical capabilities to those which augment our cognitive ones. With the advent of computers, our technology was capable of processing data in volumes and at speeds far exceeding anyone's ability, just as we made machines which could physically outperform us. Over the last several decades, we have seen remarkable advancements in computational systems such that they are seemingly approaching the ability to make decisions for themselves.

Throughout history, there has been a critical, symbiotic balance between the technology we create and our ability to use it effectively. By looking back at this historical arc, we can see these milestones not only as turning points in the field of human factors engineering, but also as fundamental shifts which have redefined how we use technology to change our world. This begs the question: what is the next frontier of human-computer interaction?

Claude E. Shannon wrote a seminal paper (1948) that defined mathematical models for how information is transmitted between people and/or systems. His work, also known as “Information Theory,” was a watershed moment for the field of human factors engineering. Some have said it created the framework by which we understand and express how people’s minds interact with technology (Hancock, 2016): in other words, how someone sees, hears, or feels information from a display, interprets it, and ultimately makes a decision and performs an action.

Warren Weaver was a contemporary of Shannon, and the two collaborated to publish an article which made Information Theory accessible to a wider audience (1949). By describing its concepts in commonly understood language, Weaver’s adaptation also lays the foundation for applying the theory to other domains--as we have seen with its adoption by the human factors engineering community. A central theme of the article builds on the original work to define "three levels of communications problems" which must be overcome for a system to be useful. The transmitted information must be:

1. Accurate: not lost or corrupted. This is referred to as the technical problem.
2. Meaningful: you cannot act on information that is out of context or nonsensical. This is the semantic problem.
3. Effective: this is the ultimate reason information was sent—to “affect conduct in the desired way.” This is the effectiveness problem.

These three problems provide a roadmap for past, current, and future human-computer interactions.

Past. Researchers have shown how Graphical User Interfaces (GUIs) are the solution to the first problem: technical, the accuracy with which information is transmitted (Bao et al., 2011; Comber and Maltby, 1996; Comber, 2002; Comber, 2010). GUIs have replaced many text-based interactions because they are more robust. Text commands are brittle--if you misspell a command, it doesn’t work--whereas GUIs can be intuitively manipulated in multiple ways to achieve a desired effect. But this research has focused on defining solutions to the first problem, without yet identifying solutions for the second or third.

Current. Augmented Reality (AR) systems overlay digital objects on your view of the real world, as seen through a display you hold or wear. There are two primary classes of AR: that in which digital elements are placed regardless of the real world (called unregistered), and registered AR, that in which placement with the respect to the real world is critical (Porter and Heppelmann, 2017). The potential of AR comes from its ability to blend the real and digital worlds, but its value is largely defined by how accurately and for how long this registration is held. For example, consider the the difference between displaying the next step of your written checklist versus showing you exactly what button to press on a control panel or where to place your drill on a sheet of metal.

Registered AR is the solution to Weaver's second communication problem: semantic, the precision with which transmitted information conveys meaning. By providing digital information precisely where and how users need it, they do not need to refer back to other resources or translate instructions into action. Because it is easier to access and interpret information, workload is reduced, tasks take less time to learn and perform, and fewer errors are made.

Future. If registered AR solves the semantic problem, that leaves the third and final level: effectiveness, how effectively the received meaning affects conduct in the desired way. The future conclusion of human-computer interfaces is a direct connection to the brain which closes the loop between what the system is attempting to convey and your resulting perception, interpretation, and action. That future may be closer than we think given the growing body of Brain-Computer Interface (BCI) work. I’m not sure how I feel about this. As a human factors engineer, I’m thrilled...but also terrified.

We will always have GUIs: for some information, it can be the most accurate, meaningful, and effective means of communication. But registered AR, with its ability to precisely blend information with the real world is a true paradigm shift. As the technology which enables it continues to advance, now is the time to lay the theoretical foundations and supporting research which will guide us as we enter this exciting age.

--Eric

References:

Bao, J., Basu, P., Dean, M., Partridge, C., Swami, A., Leland, W., & Hendler, J. A. (2011). Towards a theory of semantic communication. Paper presented at the Network Science Workshop (NSW), 2011 IEEE.

Comber, T., & Maltby, J. R. (1996). Investigating layout complexity.

Comber, T. (2002). An analysis of entropy, redundancy and complexity in graphical user interfaces.

Comber, T. (2010). An analysis of entropy, redundancy and complexity in graphical user interfaces.

Hancock, P. (2016). Origins and Overview of Human Factors. https://peterhancock.ucf.edu/class/; http://peterhancock.ucf.edu/wp-content/uploads/sites/175/2016/12/Origins-of-Human-Factors-Lecture.doc

Porter, M. E., & Heppelmann, J. E. (2017). Why every organization needs an augmented reality strategy. Harvard Business Review, 6, 46-57.

Shannon, C. E. (1948). A mathematical theory of communication. Bell system technical journal, 27(3), 379-423.

Shannon, C. E., & Weaver, W. (1949). Recent contributions to the mathematical theory of communication. The mathematical theory of communication, 1, 1-12.

This work is not sponsored by any corporate or government entity. Any opinions, findings and conclusions or recommendations expressed in this material are solely those of the author.