My research is in the area of reinforcement learning, where an agent is learning to solve a sequential decision making task. In it, the agent takes an action from state, which leads it to some new state and provides it some reward. The outcome of the actions can be noisy or random. For example, when you’re driving your car and you make a decision to take a particular turn, there is some noise in the outcome (you can not predict deterministically what will happen). Instead, there may be construction, traffic, pedestrians, etc which cause your trip down this road to take longer or shorter than you would expect. You can’t predict exactly the number of seconds it will take. However, this randomness, at its core, is really the result of partial observability. Partial observability is where you can only observe some part of the world to make your prediction. In this example, you only have access to what you see at the intersection and the info you get from the internet, radio, tv, and your phone. With this information, you cannot predict the time the road will take precisely and instead there is some noise to it. However, if you knew what every other driver was thinking and the route and timings they would take, along with what the weather would do, what the exact construction plans were, etc, then you could conceivably make this prediction accurately (or a computer could). So you can view this decision making task as having noise or being partially observable. This goes on at all levels, but at some point you would need knowledge of every atom and molecule in the system (which can’t be observed), so its easier to approximate things as being noisy. But as we get better and better at observing such things, we can make more deterministic predictions that consider all the factors that we previously ignored as “noise.”

A lot of interviews I’ve seen with sports stars recently has made me think of this. I’ve seen quite a few instances (like this one: http://msn.foxsports.com/nfl/story/buffalo-bills-stevie-johnson-blames-god) of some seemingly random event happening, and the athlete’s response is to credit it to God. Instead of believing in randomness, they believe the world is partially observable and God is the hidden observation that can not be seen. Random events aren’t random, but instead are things that God did, and of course we couldn’t observe God’s decision making process. Anyway, I thought this connection was interesting, I’m not sure if its better to think there’s some force deciding all outcomes or to think that some things do just happen randomly. But it’s an interesting question: Is God simply an non-observed feature of the world state?

I just read a fascinating edge talk by Stanislas Dehaene on consciousness. He’s a neuroscientist studying the “easy” problem of consciousness, meaning he’s studying how conscious awareness arises in the brain, not how we get this subject perception of the world. He does some pretty interesting experiments, using MRI to study people’s brain’s as they consciously and subliminally perceive various stimuli. He comes the conclusion that consciousness is when multiple areas of the brain begin to communicate and synchronize on one concept or stimulus. Then it becomes conscious and you can think about, process it in different areas of the brain, etc. Here’s what he says:

This idea is relatively simple, and it is not far from the one that Daniel Dennett proposed when he said that consciousness is “fame in the brain”. What I propose is that “consciousness is global information in the brain” — information which is shared across different brain areas. I am putting it very strongly, as “consciousness is”, because I literally think that’s all there is. What we mean by being conscious of a certain piece of information is that it has reached a level of processing in the brain where it can be shared.

Because it is sharable, your Broca’s area (or the part of it involved in selecting the words that you are going to speak) is being informed about the identity of what you are seeing, and you become able to name what you are seeing. At the same time, your hippocampus is perhaps informed about what you have just seen, so you can store this representation in memory. Your parietal areas also become informed of what you have seen, so they can orient attention, or decide that this is not something you want to attend to… and so on and so forth. The criterion of information sharing relates to the feeling that we have that, whenever a piece of information is conscious, we can do a very broad array of things with it. It is available.

It’s pretty interesting that the main purpose of consciousness may just be so that different parts of the brain can use the information. Suddenly we’re able to think about, visualize, memorize and talk about some particular thing. We can decide to pay attention to it or not. I thought the “all or nothing” principle was particularly interesting:

Another important feature that I have briefly mentioned already is the all-or-none property. You either make it into the conscious workspace, or you don’t. This is a system that discretizes the inputs. It creates a digital representation out of what is initially just a probability distribution. We have some evidence that, in experiments where we present a stimulus that is just at threshold, subjects end up either seeing it perfectly and completely with all the information available to consciousness — or they end up not having seen anything at all. There doesn’t seem to be any intermediate state, at least in the experiments that we’ve been doing.

Having a system that discretizes may help solve one of the problems that John Von Neumann considered as one of the major problems facing the brain. In his book The computer and the brain, Von Neumann discusses the fact that the brain, just like any other analog machine, is faced with the fact that whenever it does a series of computations, it loses precision very quickly, eventually reaching a totally inaccurate result in the end. Well, maybe consciousness is a system for digitizing information and holding on to it, so that precision isn’t lost in the course of successive computations.

I thought the analogy to the computer here was particularly interesting. In robotics, there are plenty of times where we have an uncertain estimate of what is going on. For example in robot localization, we have some distribution over possible places we might be located. But its really hard to determine what to do based on a range of places we might be. So we typically take a single location, the most likely one, and determine behavior based on this. It’s hard to process a large uncertain analog symbol, so we reduce it to a discrete point that we are. It seems like we’re losing valuable information by doing this, but perhaps this is exactly what consciousness is doing. It has to pick out one interesting thing to process at a time, because it’s too difficult to actively do that with many things at once.

I’ve been looking at a lot of cool robot videos lately, so I thought I’d post some here. First the cool walking robots from Boston Dynamics, then some biology inspired ones from Festo.

Here’s the BigDog robot from Boston Dynamics if you haven’t seen it.

And here’s their humanoid PETMAN, which is the first humanoid I’ve seen that does heel-to-toe walking.

From Festo robotics, a bunch of cool ones. First, robotic penguins for swimming or flying.

Robotic air jellyfish and stingray.

And the Festo humanoid robot.

It seems like people are really stuck to this idea that we’re unique. Perhaps they lose something if they feel we aren’t special. Back in the day, it was that we were at the center of the universe, and anyone who said otherwise was a heretic. Now we’re in the midst of some people saying we have to be unique from the animals, we couldn’t have evolved from them. And then there’s intelligence. We have to be the only ones who are intelligent right? According to Searle, even if we have a machine that behaves and acts indistinguishably from a human, it can’t be intelligent - it doesn’t have our secret brain stuff. Well I’ve got news for everyone. We’re are unique. Each one of us is different in subtle or not so subtle ways. But we’re not special, other creatures can be intelligent, other planetary bodies can have things orbiting around them, other planets can have life. Accept it.

This is going to lead directly to skynet: Halted ’03 Iraq Plan Illustrates U.S. Fear of Cyberwar Risk. As the military starts writing these algorithms to try to attack other’s computer networks and infiltrate their security, and then start working on ones to defend against such attacks, and then algorithms to outsmart the defensive ones…. this could easily lead to skynet. We could have single algorithms/agents in charge of the military infrastructure and able to adapt and learn. Hopefully John Connor is out there somewhere…

Microsoft recently demoed Project Natal at E3. It looks really impressive. Here’s a video. And here’s part of an article from Wired:

Later, Microsoft producer Kudo Tsunoda, who’d demoed this game to us, showed us what was going on “behind the scenes.” He brought up an image that showed what Natal’s two cameras could see. The standard RGB camera in the unit, he says, was watching the positions of our skeletons. Sure enough, crisp solid stick-figures appeared on screen, representing me and Baker. As we moved, the stick figures on screen matched our movements. It wasn’t picking up our individual finger movements, but it could tell when we bent our wrists, picked up our legs, tilted our heads, etc., not to mention tracking us as we moved around the room.

Meanwhile, an infrared camera was tracking our distance away from the television. As we moved forward, our outlines turned red, as we moved back they shimmered blue. By combining both of these sets of data, Natal knows exactly where you are and what you’re doing. And it knows when I step behind Baker, or if he walks in back of me, and it still tracks, separately, the parts of both of us that it can see.

Seriously, the motion capture we used at the hospital required you to wear a bunch of reflective balls, used 12 cameras placed around the room, needed constant calibration, and still would mess up, and miss markers sometimes. If this thing really works this well, it’s really impressive. I really want to see it in a room with lots of people and varying lighting conditions.

As a grad student studying artificial intelligence, it might be useful to know what exactly intelligence is.  No one can really define it.  We know that we have it.  We know that tables and chairs do not.  Everything in between is up for grabs.  I’m going to discuss what some historical views on intelligence are and what I think it might be.

The classic definition of intelligence in AI is based on the Turing Test.  This is a test created by Alan Turing to determine if something is intelligent.  You have a person and a computer in separate rooms.  In a third room is a judge.  He chats with both of them over a computer.  If he can’t tell which one is the person, then the computer is intelligent.  Basically the computer is considered to be intelligent if it can imitate a person in conversation.  If it doesn’t have this ability then its not intelligent.  This test can be extended to a more complicated task, say if you can’t tell a person and a computer/robot apart in every day life, then the computer/robot would be intelligent.

Then there are a class of people who believe that having intelligent behavior is not enough.  They say its critical what is going on to make that intelligent behavior.  Usually they say that having computer circuits create such behavior is not intelligent.  Some of them say that only having an exact human brain create the behavior is intelligence.  The classic argument from this side is John Searle’s Chinese Room.  There is a man in a room.  He receives pieces of paper with weird symbols on them.  He has a set of directions of what to do when he receives these pieces of paper, which generally involves him creating new weird symbols and passing them back out.  The symbols are Chinese and he is actually answering someone’s questions with someone in Chinese this way. Searle’s main argument is that the man does not know Chinese even though he can act like he does this way.  The room also does not speak Chinese.  Searle goes on to say that even if the man and directions were replaced by a fully accurate simulation of the human brain it would not be intelligent, you need the magic stuff of the brain to be intelligent.  Another point to make is that this example is utterly impossible, it would not be possible to create a set of directions to converse fluently in Chinese.

Another argument from a more computer science perspective is that of Simon and Newell.  They put forth the ‘Physical Symbol System Hypothesis’, which states that “A physical symbol system has the necessary and sufficient means for general intelligent action.”  They define a physical symbol system as any system which physical patterns (symbols) that are related somehow (in a system).  Basically, anything.  They later go on to say that one of the signs of intelligence is avoiding exponential search (Many problems can be reduced to searching through a space of possible solutions.  If the search space is exponential, then it is impractical or impossible to search it exhaustively.  Avoiding this exponential search requires some intelligence in deciding how to avoid it).  This is a pretty interesting idea.  I often feel that one of the key things to do to be successful is to know how much time to put into something and when its not worthwhile anymore, which seems somewhat related to this idea of avoiding exponential search.

Recently, our reading group read a paper by Ned Block that also discussed this topic.  Block is part of the camp that says that the internal processing that creates the intelligent behavior is critical to whether something is intelligent.  He provides a counter-example to the Turing Test.  A computer is loaded with a database of every conceivable conversation that is less than an hour in length.  Then when taking the Turing Test, it looks up a conversation that matches up to the current sentence, and speaks the next sentence in that conversation.  He says that this machine is clearly not intelligent, and therefore intelligent behavior is not enough.  He does allow that computers could be intelligent, but they would have to be doing something better than this, such as learning or adapting.  Another student in the group brought up a great point that the Turing Test is fine if you say that it has to be a feasible machine that passes for a human.  Both this counterexample and the Chinese room are not feasible agents (the number of hour long conversations is much greater than the number of particles in the universe).  Any feasible agent (reasonable memory and computation power) that passes the Turing Test must be doing something intelligent to be acting that way with limited resources.  This is an interesting definition that lets the Turing Test stand and does put a restriction on the internal processing to be ‘feasible’, which any actually realized agent would be.

So where do I stand?  I’m not sure.  Intelligence may be defined on some scale by behavior.  More complex or more efficient behaviors come from more intelligent beings.  I’m not sure that a definition with a restriction on the internal processing of a being will work.  Intuitively I don’t think an agent that is simply looking up a conversation in its database like above is intelligent.  An agent that is learning or adapting seems to me to be more intelligent.  But depending on what level of abstraction you look at, even we aren’t doing anything that exciting.  Sure, you say that we’re understanding what is said to us, thinking of a response, using our memories, and responding.  Which sounds intelligent. But at its basest level, its just chemicals and ions flowing back and forth in our brain, following the inexorable laws of physics.  Which sounds very unintelligent.  The same thing applies in the case of AI.  You can write some cool learning algorithm that adapts and learns how to behave in some environment over time.  But of course how it learns was written by you and is pre-determined and the entire course of what it would do in that environment could probably be predicted if you knew all the variables.  So I don’t think any definition of intelligence that involves the internal processing going on could ever work, because every agent simply has very boring particles following the laws of physics at the core of their internal processing.

In summary, I think a definition of intelligence has to be about producing intelligent behavior and not about the internal processing that creates it.  The Turing Test seems pretty reasonable if you restrict it to feasible agents.  And I still think algorithms that let machines learn and adapt are more exciting that simply programming in a static solution to a computer (I prefer to think of things at the learning abstraction level than the level of particles following the laws of physics).

In our reading group this week, we read John Searle’s classic paper “Minds, Brains, and Programs.” In the paper, he makes the claim that digital computers doing symbol manipulation cannot have intelligence. His motivating example is what he calls the Chinese Room Experiment. Basically, a man is inside a room. He is given a set of instructions to follow when he receives papers full of Chinese symbols. By following the instructions, he creates a second set of symbols which he returns. Although he does not personally understand Chinese, to an observer outside the room, they have just received fluent answers to questions in Chinese. Searle claims that this shows that intelligence cannot be created in a digital computer (the man) with a program (the instructions) because in this example the man clearly does not understand Chinese. I would argue that the room as a whole understands Chinese, as its the whole system that understands it (you wouldn’t expect a computer to be intelligent without its program). He argues that the system as a whole cannot have intelligence, and that some subsystem must have it. But the same systems rule applies to the brain, the brain as a whole has intelligence, but you can’t ascribe the intelligence to any subsystem of the brain such as a neuron or brain region.

The other main thing that bothers me about Searle’s essay is that he claims that human intelligence can never be replicated on a digital computer because a computer does not have the same “causal properties” as the brain. He claims that our intentionality is somehow created by these “causal properties” inherent to the brain that cannot be replicated in a computer. He makes no claim as to what they are or why they can not be replicated. I find it highly unlikely that our intentionality is due to some specific physical feature of the brain as opposed to an emergent property of the coalition of neurons in our brain. It seems like Searle is searching for some little homonculi in our brain that is directing things, and that we can’t recreate that homonculi in a computer. But there is no little man inside our brains.

One interesting point that Searle brings up is the separation of mind and brain. The separation of mind and brain may not be the same as the separation of a program and a computer. While a program can be run on any number of equivalent computers, the mind may be dependent on specific properties of the brain, or may be integrated in the brain rather than being some program that can be applied on top of the brain. I can agree that there are probably some aspects of the brain that are essential to mind, although I don’t see why we won’t be able to simulate those somehow in a computer.

Here’s a ridiculous story predicting that we’ll be marrying robots by 2050.  Not only do they predict that, but they predict that Massachusetts will be the first state to legalize it.  I certainly see the sex robots coming very soon - the sex industry always takes advantage of new technology before anyone else.  But marriage with robots?  I suppose if robots are actually intelligent by 2050 and seem like equals to humans then maybe its possible.  But if not, then its like asking if people will be marrying their cars or computers.  A little absurd, I think.

I just got back from RoboCup in Atlanta, GA (ok, I actually got back 3 weeks ago, but I’m just getting around to writing about it). RoboCup is basically the world cup of robot soccer. Our team is UT Austin Villa, and we compete in the legged league, where we play four on four soccer using Sony Aibo robots. All the teams have the same robots, so its mainly a programming competition. There are lots of difficult problems to solve, such as vision, localization, movement, behaviors, and communication. I’ve worked a lot on the localization, which means having the robot figure out where it is from the landmarks that it sees. I also worked a lot on the goalie, and then a little bit on many many other aspects of the team.

RoboCup was pretty crazy. We were in the ballroom of the Fox Theater, which was built in the 1920s. In the ballroom they had three soccer fields set up, and there was a side room where all the teams had their own tables. Every day we were there from 7 am (when it opened) to 10 pm (when they kicked us out). Everyone is there trying to fix things at the last minute and making adjustments after games. The team from Bowdoin College, Northern Bites, was especially relentless, you would always hear them having discussions about minute details of the robot soccer strategy and constantly trying to improve.

Our first game was against Carnegie Mellon, where my advisor went to school. So it was a big rivalry. Tekin and I got to the venue when it opened at 7 am to try to fix the many problems we saw the night before but we ended up losing to Carnegie Mellon 4-2 (video). We had some shots on goal but missed most of them wide of the post. We played a team from Japan, FC Twaves, in the afternoon. We were hoping to win this one, but they scored first. Then we had a problem where our robot got switched to the other team and it scored on our own goal. So we were down 2-0 to this team with a few minutes left in the half but luckily we managed to score three goals in about four and a half minutes and we won 4-2. The start of this video is right after their 2^nd goal and shows us (in blue) scoring three goals in four and a half minutes. This game was stressful, Tekin and I were both pretty concerned when they took a 2-0 lead, but thankfully we came back to win.

After playing pretty poorly in the two games on the first day, Tekin and I worked pretty hard that night and the next morning to try to fix things. We improved ball-grabbing, ball following, ball vision, shooting, and communication and strategy, among other things. The next day, the team looked the best I had seen it. If you compare the videos from the first two games and the videos from these three, you will see a significant difference. We beat the Baby Tigers from Osaka by a score of 5-0 and then beat SPQR by 5-0 as well. Baby Tigers has a ridiculous kick where they climb over the ball and shoot it out from behind (video). Our third game was against the German Team, who are one of the best teams, and won the German Open this year. They had won 3 of their 4 earlier games by 10-0 scores (after 10-0 the game is called). We managed to hold them to 3-0, which I was pretty excited about. My goalie looked really good during the game, making some sweet saves. A few people who worked on goalies from other teams came up to me afterward to tell me how good it looked, which was really cool. Here is the video from the first and second halves of the game against the German Team. UT is in red in the first half and blue in the second half.

Winning two of our three second round games meant that we made it to the quarterfinals, which is better than UT did last year. Considering there were only three of us at RoboCup (most teams had 5-12 people), I was pretty happy. We played Wright Eagle from China in the quarterfinal match. We were really slow in the first half because of some network issues so they went up 4-0. In the second half that was fixed and they only scored one goal. The entire second half consisted of the ball being right around and in front of our goal, and they were repeatedly taking shots and my goalie kept making nice saves. My goalie looked great, but I don’t think we ever had the ball on their half of the field. Here is the video from this game. Even though we lost in the quarterfinals, I was pretty happy with our performance.

Watching the championship games was pretty exciting. After talking to and hanging out with all the guys on the other teams all week I was definitely hoping some teams would win more than others. Plus all of the final four teams were really good so it was really entertaining to watch. Carnegie Mellon won 3^rd place over Wright Eagle in a game that went to overtime and took 8 penalty shots before CMU prevailed. The championship was between Northern Bites from Maine and NU Bots from Australia. Northern Bites won 5-1. Here is the video of the championship game.

It was cool to see most of the teams get really into the games. TecRams from Mexico and Araibo from Japan both decided to cheer for Carnegie Mellon because CMU came out of the same first round group as them. During CMU’s games they would all be screaming and yelling. Araibo was probably the loudest team there, they were constantly screaming, yelling “GO GO GO” on offense and chanting “Goalie! Goalie!” on defense. I also was head referee for one of their games and it was hard to make calls over all their screaming. I have a great video of Araibo, TecRams, and CMU celebrating a goal during one of Carnegie Mellon’s games.

RoboCup was really cool and a lot of fun. It was a lot of work, not only at RoboCup, but all the work in the months leading up to RoboCup. But it was definitely a lot of fun. It was really cool to hang out with guys from other teams all over the world. One night we went out to dinner with guys from the German Team, Microsoft Hellhounds (Germany), NU Bots (Australia) and Northern Bites (Maine). Good times

Here are some various links to cool stuff. Northern Bites has a blog, which is very detailed and has some interesting stories. I have a set of photos I took at RoboCup along with some other photos I stole along the way. All the videos that we took of RoboCup are available at UT Austin Villa’s website. And here is a link to some photos on Time magazine’s website about RoboCup. Here are sets of random RoboCup photos and videos. Here is the full website with all of the results from the Legged League. Another really cool event at Robocup was the first ever game played with robots and humans, that video is available here.