My research is in the area of reinforcement learning, where an agent is learning to solve a sequential decision making task. In it, the agent takes an action from state, which leads it to some new state and provides it some reward. The outcome of the actions can be noisy or random. For example, when you’re driving your car and you make a decision to take a particular turn, there is some noise in the outcome (you can not predict deterministically what will happen). Instead, there may be construction, traffic, pedestrians, etc which cause your trip down this road to take longer or shorter than you would expect. You can’t predict exactly the number of seconds it will take. However, this randomness, at its core, is really the result of partial observability. Partial observability is where you can only observe some part of the world to make your prediction. In this example, you only have access to what you see at the intersection and the info you get from the internet, radio, tv, and your phone. With this information, you cannot predict the time the road will take precisely and instead there is some noise to it. However, if you knew what every other driver was thinking and the route and timings they would take, along with what the weather would do, what the exact construction plans were, etc, then you could conceivably make this prediction accurately (or a computer could). So you can view this decision making task as having noise or being partially observable. This goes on at all levels, but at some point you would need knowledge of every atom and molecule in the system (which can’t be observed), so its easier to approximate things as being noisy. But as we get better and better at observing such things, we can make more deterministic predictions that consider all the factors that we previously ignored as “noise.”

A lot of interviews I’ve seen with sports stars recently has made me think of this. I’ve seen quite a few instances (like this one: http://msn.foxsports.com/nfl/story/buffalo-bills-stevie-johnson-blames-god) of some seemingly random event happening, and the athlete’s response is to credit it to God. Instead of believing in randomness, they believe the world is partially observable and God is the hidden observation that can not be seen. Random events aren’t random, but instead are things that God did, and of course we couldn’t observe God’s decision making process. Anyway, I thought this connection was interesting, I’m not sure if its better to think there’s some force deciding all outcomes or to think that some things do just happen randomly. But it’s an interesting question: Is God simply an non-observed feature of the world state?

There was an interesting blog post on the NY Times today about an experiment that showed that the decisions of monkeys could be predicted before the monkeys knew they had made the decision. The part I found more interesting connected with my previous post on free will and how the ability to predict our decisions does not have to mean that we don’t have free will:

Let me explain what I mean by way of an example. Imagine we suspend a steel ball from a magnet directly above a vertical steel plate, such that when I turn off the magnet, the ball hits the edge of the plate and falls to either one side or the other.

Very few people, having accepted the premises of this experiment, would conclude from its outcome that the ball in question was exhibiting free will. Whether the ball falls on one side or the other of the steel plate, we can all comfortably agree, is completely determined by the physical forces acting on the ball, which are simply too complex and minute for us to monitor. And yet we have no problem assuming the opposite to be true of the application of the monkey experiment to theoretical humans: namely, that because their actions are predictable they can be assumed to lack free will. In other words, we have no reason to assume that either predictability or lack of predictability has anything to say about free will. The fact that we do make this association has more to do with the model of the world that we subtly import into such thought experiments than with the experiments themselves.

The rest of the blog post was a bit crazy and I think extrapolated a bit much from these experiments, but I think the excerpt above makes clear the argument that we can still have free will even if it turns out we can predict all our decisions by looking at all the neurons, chemicals, signals, inputs, etc to the brain.

I just read “The Brain that Changes Itself” by Norman Doidge. It’s all about brain plasticity and how our brain is constantly changing. Basically all of our brain is maps, like auditory map where different areas respond to different frequencies. Or higher up, different areas respond to different syllables. When we’re young, these maps quickly differentiate, just on inputs, i.e. your syllable map will adjust to the syllables you hear, part of why its difficult to learn a 2nd language when you’re an adult. But they continue to change and adapt as you grow older. If you blindfold someone, their visual cortex starts helping out with other senses almost immediately. When they train a monkey to perform a new task using its index finger, its index finger quickly takes up more of its map. People with brain damage from strokes or other accidents, are able, through training, to regain many skills by re-training other parts of the brain to take over.

One important point that is emphasized in the book is the need to keep your brain plastic. The more often you do some experience, the more learned in the brain it is, the more of the map it will take up, and the harder it will be to learn more stuff. So you should continue changing things up and trying new experiences to keep your brain lively. And the less learning you do, the less plastic your brain may become. So do your brain exercises as you get older!

A significant part of the book focuses on research by Michael Merzenich. He has developed some software programs to help improve people’s brain maps. Due to defects in the brain or body, or the inputs the brain is getting from the environment itself, certain maps may develop abnormally, but can be fixed through training. For example, you can re-train your audio map to properly hear sounds. He has software to do this for thousands of kids who have struggled with language, speaking, listening, and reading, due to this abnormality. He also has software for the elderly. And after the same software proved very helpful with autistic children, he has some interesting theories there as well. Here’s an interesting TED talk by Merzenich.

Random chance. Somehow the big bang happened, the universe was created, stars formed, our star formed, planets formed around the sun, the Earth formed, it was the right distance from the sun, it revolved and rotated at the right speeds, it had a moon and tides, life appeared, it evolved, we arrived, and no man-made or natural disasters have wiped us out (like the dinosaurs). Eventually your grandparents were born, your parents, and you. That’s it. It’s entirely random dumb luck that we (and you) are here. There’s no special meaning, no higher purpose that we’re here for. We just happen to have occurred, we are here for maybe 100 years and then we die.

We die. Think about that for a minute. You will at some point cease to exist. It’s hard to imagine. You try to think about what it will feel like to die… but you can’t. It won’t feel like anything. You won’t exist. You’ll be gone. There is no magic, no afterlife, your consciousness isn’t going to live on or be reincarnated. When life ends, that’s it. Your existence is really really short, make the most of it!

What is the meaning of life? There isn’t one. Life just happens. Is that depressing? On the contrary, I think life is meaningless until you see that it has no meaning. Then ‘life is what you make it’. Life is random and brief, and that is what makes it so beautiful and amazing. So many things had to fall the right way for you to exist and it will be gone so fast… So appreciate it! Savor it! Enjoy it! Do something with it! Do you want to be remembered after you’re gone? Build relationships! Help others! Contribute to society!

I just read a fascinating edge talk by Stanislas Dehaene on consciousness. He’s a neuroscientist studying the “easy” problem of consciousness, meaning he’s studying how conscious awareness arises in the brain, not how we get this subject perception of the world. He does some pretty interesting experiments, using MRI to study people’s brain’s as they consciously and subliminally perceive various stimuli. He comes the conclusion that consciousness is when multiple areas of the brain begin to communicate and synchronize on one concept or stimulus. Then it becomes conscious and you can think about, process it in different areas of the brain, etc. Here’s what he says:

This idea is relatively simple, and it is not far from the one that Daniel Dennett proposed when he said that consciousness is “fame in the brain”. What I propose is that “consciousness is global information in the brain” — information which is shared across different brain areas. I am putting it very strongly, as “consciousness is”, because I literally think that’s all there is. What we mean by being conscious of a certain piece of information is that it has reached a level of processing in the brain where it can be shared.

Because it is sharable, your Broca’s area (or the part of it involved in selecting the words that you are going to speak) is being informed about the identity of what you are seeing, and you become able to name what you are seeing. At the same time, your hippocampus is perhaps informed about what you have just seen, so you can store this representation in memory. Your parietal areas also become informed of what you have seen, so they can orient attention, or decide that this is not something you want to attend to… and so on and so forth. The criterion of information sharing relates to the feeling that we have that, whenever a piece of information is conscious, we can do a very broad array of things with it. It is available.

It’s pretty interesting that the main purpose of consciousness may just be so that different parts of the brain can use the information. Suddenly we’re able to think about, visualize, memorize and talk about some particular thing. We can decide to pay attention to it or not. I thought the “all or nothing” principle was particularly interesting:

Another important feature that I have briefly mentioned already is the all-or-none property. You either make it into the conscious workspace, or you don’t. This is a system that discretizes the inputs. It creates a digital representation out of what is initially just a probability distribution. We have some evidence that, in experiments where we present a stimulus that is just at threshold, subjects end up either seeing it perfectly and completely with all the information available to consciousness — or they end up not having seen anything at all. There doesn’t seem to be any intermediate state, at least in the experiments that we’ve been doing.

Having a system that discretizes may help solve one of the problems that John Von Neumann considered as one of the major problems facing the brain. In his book The computer and the brain, Von Neumann discusses the fact that the brain, just like any other analog machine, is faced with the fact that whenever it does a series of computations, it loses precision very quickly, eventually reaching a totally inaccurate result in the end. Well, maybe consciousness is a system for digitizing information and holding on to it, so that precision isn’t lost in the course of successive computations.

I thought the analogy to the computer here was particularly interesting. In robotics, there are plenty of times where we have an uncertain estimate of what is going on. For example in robot localization, we have some distribution over possible places we might be located. But its really hard to determine what to do based on a range of places we might be. So we typically take a single location, the most likely one, and determine behavior based on this. It’s hard to process a large uncertain analog symbol, so we reduce it to a discrete point that we are. It seems like we’re losing valuable information by doing this, but perhaps this is exactly what consciousness is doing. It has to pick out one interesting thing to process at a time, because it’s too difficult to actively do that with many things at once.

Here’s an interesting question: would you rather live 100 years in the past or 100 years in the future? There are good arguments for both sides. Let me try to make a list of advantages for each below.

The past

• Families are closer together geographically.
• Families spend more time together. No tv or video games to distract them.
• Communities are closer together.
• People aren’t as distracted or dumbed down by technology.
• Life is perhaps more of an adventure? Everything is more difficult, from keeping food safe, to traveling, etc.
• Most work is physical, working the land with your hands.

The future

• People are healthier and live longer (better healthcare).
• It’s easy to travel all around the world (or beyond it?).
• You can stay connected with people from everywhere.
• You can get information quickly and easily.
• We’ll have computer or robots that can perform lots of tasks that people do now.

Personally, I am really curious about the future, what it will be like, what we’ll be like, what technologies we’ll have, what we’ll be able to do, etc. I think the past would be neat as well, as it would be nice to have a tight-knit family and community, and I feel like it could be slightly more adventurous, as just traveling across the country would be a big deal.

This is a crazy thought, but I feel an interesting balance between these things in a few decades would be to live on Mars. It’s still the future and you get all of those things. But I also assume its a very small group, tight-knit, probably without tv and those things, and probably working the land to survive (although probably with machines). That can be another blog post: If you could move to Mars, would you? What if you could never come back?

I’ve been looking at a lot of cool robot videos lately, so I thought I’d post some here. First the cool walking robots from Boston Dynamics, then some biology inspired ones from Festo.

Here’s the BigDog robot from Boston Dynamics if you haven’t seen it.

And here’s their humanoid PETMAN, which is the first humanoid I’ve seen that does heel-to-toe walking.

From Festo robotics, a bunch of cool ones. First, robotic penguins for swimming or flying.

Robotic air jellyfish and stingray.

And the Festo humanoid robot.

This is a pretty interesting article on children learning self-control by playing make believe: Can the Right Kinds of Play Teach Self-Control?. Play can be useful! I wonder if I can put this idea into an artificial agent somehow.

It seems like people are really stuck to this idea that we’re unique. Perhaps they lose something if they feel we aren’t special. Back in the day, it was that we were at the center of the universe, and anyone who said otherwise was a heretic. Now we’re in the midst of some people saying we have to be unique from the animals, we couldn’t have evolved from them. And then there’s intelligence. We have to be the only ones who are intelligent right? According to Searle, even if we have a machine that behaves and acts indistinguishably from a human, it can’t be intelligent - it doesn’t have our secret brain stuff. Well I’ve got news for everyone. We’re are unique. Each one of us is different in subtle or not so subtle ways. But we’re not special, other creatures can be intelligent, other planetary bodies can have things orbiting around them, other planets can have life. Accept it.

Here’s an interesting article on overconfidence and the banking collapse: Cocksure. It’s interesting how overconfidence can ruin you and lead you to bad decisions, yet in some areas, overconfidence is what’s required, when confidence is vital. Here’s an excerpt:

This is what social scientists mean when they say that human overconfidence can be an adaptive trait. “In conflicts involving mutual assessment, an exaggerated assessment of the probability of winning increases the probability of winning,” Richard Wrangham, a biological anthropologist at Harvard, writes. “Selection therefore favors this form of overconfidence.” Winners know how to bluff. And who bluffs the best? The person who, instead of pretending to be stronger than he is, actually believes himself to be stronger than he is. According to Wrangham, self-deception reduces the chances of “behavioral leakage”; that is, of “inadvertently revealing the truth through an inappropriate behavior.” This much is in keeping with what some psychologists have been telling us for years—that it can be useful to be especially optimistic about how attractive our spouse is, or how marketable our new idea is. In the words of the social psychologist Roy Baumeister, humans have an “optimal margin of illusion.”