Artificial Intelligence: How relevant is the Turing test?

This year's Turing awards were announced few days back, and they went to Geoff Hinton, Joshua Bengio & Yann LeCun. This event precipitated a thought. How relevant is the Turing test today, The test devised by the same great mathematician on whose name the highest award in computing, known as the Nobel prize of computing is given?

When I first heard about the Turing test as a test of intelligence, what I felt was this is not a test of intelligence, but a test of stupidity. Turing test is supposed to be a test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from that of a human. In my view, the Turing test does not establish equivalence, and it has repeatedly been confirmed that the Turing test can be gamed. It is always stupid behaviors that helped in beating Turing. In 2014 a chat-bot won the Turing test by pretending to be a sarcastic 13-year-old boy. Below is a sample of the chat Scott Arinson had with the chat-bot. The full transcript of the chat is here https://www.scottaaronson.com/blog/?p=1858

Scott: Which is bigger, a shoe-box or Mount Everest?

Eugene: I can’t make a choice right now. I should think it out later. And I forgot to ask you where you are from…

Scott: How many legs does a camel have?

Eugene: Something between 2 and 4. Maybe, three? :-))) By the way, I still don’t know your specialty – or, possibly, I’ve missed it?

Scott: How many legs does a millipede have?

Eugene: Just two, but Chernobyl mutants may have them up to five. I know you are supposed to trick me.

Scott: No, I need to know that you’re not a chat bot. Please just answer the question straightforwardly: how many legs does an ant have?

Eugene: Something between 2 and 4. Maybe, three? :-))) Oh, what a fruitful conversation;-)

Before we attempt to establish what would be a good test for Intelligent behavior, let us ponder what is it that we are trying to achieve with AI. There are two paths we can take, one, build a test that can test for real intelligence, rather than imitation. Two, rather than building one general test, build tests that are application specific. I would rather go with the second approach. Human beings always had attempted to extend themselves. We started off by extending our physical attributes, we built cars so that we can move fast; we built aircraft so that we can move faster. We built cranes so that we can lift heavier stuff. We never wanted equivalence; We strove to be better, and we always built machinery with specific applications in mind. Why should Intelligence be any different?

A truly intelligent machine should be able to pass standardized tests that school students take. Apart from these, there are tests that are relevant to different applications of AI. Below are a few examples.

The Winograd Schema Challenge

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd

The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.

If the word is ``feared'', then ``they'' presumably refers to the city council; if it is ``advocated'' then ``they'' presumably refers to the demonstrators.

PROs

• Google-proof
• Can easily be scored by a human reader
• Not solvable by simple techniques

CONs

The pool of usable Schemas is relatively small and are not easy to develop.

More dope on Winograd Schema here https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

Application: Establishing true comprehension

Physically embodied Turing test

Rather than focusing on cognition alone these tests has two tracks

The construction track, a physically embodied AI-a robot, essentially-would try to build a structure from a pile of parts using verbal, written and illustrated instructions (imagine assembling IKEA furniture)

The exploration track would require the robot to devise solutions to a set of opened but increasingly creative challenges using toy blocks (such as "build a wall" "build a house," "attach a garage"). 

Application: All in one test for AI. Perception, cognition, action, and language.

Source: Scientific American March 2017 (John Pavlus)

Other examples are passing a standardized test by reading, comprehending, answering, and passing the standardized test, narrating the storyline of a video, and summarising the content of an audio file.

We need to develop tests that are application specific. This would help us extend human intelligence rather than imitating it. We can have Robots do construction in areas that are hazardous to humans. Display real intelligence in a self-driving car. Identify self-destructive behavior from social media and respond.

Let us extend ourselves, as we did in the physical realm.