Learning of Hand Reaching Movement and Hidden Representation

Summary
- Hidden Repesentation -
Iriki et al. reported interesting results regarding the visual receptive field of two kinds of neurons in the parietal cortex of a monkey. A monkey did a task to reach its hand or tool to a target. The receptive field of one kind of neuron was enlarged when the monkey used the tool grasped by its hand. The receptive field of the other type of neurons moved together with its hand even though the hand was hidden under an opaque plate. They discussed those results in relation to high-order cognitive functions such as body image and symbolization.

In this paper, a hypothesis is posited that these neurons contribute to generate the critic output (state evaluation in a given task) and are obtained through reinforcement learning. Thereby, tool use is considered to be the change of link length for simplicity; a layered neural network learns hand reaching by a manipulator based on reinforcement learning. Inputs of the network are visual sensory signals and the state of the manipulator. Outputs are the critic and joint torques as the actor. After learning, the manipulator came to move its hand toward the target on the visual sensor when the target was located within the hand's reach. Both types of neurons observed in experiments of Iriki et al. were found in the hidden layer of the neural network.

- Force Load -
It has been known that when a human moves its hand to a target, the trajectory becomes almost a straight line from the start point to the target. When a viscosity force field is loaded to the hand unexpectedly, it is pulled toward the force direction once and then goes back to the target. However, after the learning in the force field, the trajectory becomes a straight line again, and when the force field is removed, it is pulled toward the opposite direction of the force that was loaded to the hand\cite{Shadmehr}. This is called after-effect. In this paper, a neural network, whose inputs are visual sensory signals and state of manipulator, and whose outputs are joint torques, was trained by reinforcement learning. The effect of the first force field exposure and after-effect could be observed. This means that the system obtains inverse dynamics of its hand and environment in the neural network through reinforcement learning. Further, when the neural network learned with a random force at every trial, it became to control its hand based on feedback control rather than feedforward control.

Reference
7. Katsunari Shibata & Koji Ito:
Hidden Representation after Reinforcement Learning of Hand Reaching Movement with Variable Link Length,
Proc. of IJCNN (Int'l Conf. on Neural Networks) 2003, 2003.7 (to appear)
pdf File (6 pages, 764kB)

6. 柴田克成:
強化学習によって獲得された手先のリーチング運動における軌跡と速度履歴,
第21回計測自動制御学会九州支部学術講演会予稿集, pp. 241--244, 2002.12
(in Japanese)

5. Katsunari Shibata \& Koji Ito:
Effect of Force Load in Hand Reaching Movement Acquired by Reinforcement Learning,
Proc. of Int'l Conf. on Neural Information Processing Systems (ICONIP '02), Vol. 3, pp. 1444-1448, 2002.11
pdf File (5 pages, 200kB)

4. 柴田克成, 杉坂政典, 伊藤宏司:
強化学習によるリーチング運動の獲得,
電子情報通信学会技術研究報告, NC2000-170, pp. 107-114, 2001. 3.
(in Japanese)
pdf File (8 pages, 318kB)

3. Katsunari Shibata, Masanori Sugisaka and Koji Ito:
Hand Reaching Movement Acquired through Reinforcement Learning,
Proc. of 2000 KACC (Korea Automatic Control Conf.), 90rd (CD-ROM), 2000. 10
PDF File (186kB)

2. 柴田 克成, 伊藤 宏司:
Direct-Vision-Based 強化学習に基づく Hand-Eye Coordination の形成,
第12回自律分散システム・シンポジウム資料, pp. 217 - 222, 2000.1.
(in Japanese)
pdf File (6 pages, 470kB)

1. K. Shibata and K. Ito:
Hand-Eye Coordination in Robot Arm Reaching Task by Reinforcement Learning Using Neural Networks,
Proc. of IEEE SMC (Int'l Conf. on Systems, Man, and Cybernetics) '99, Vol.V, pp. V-458 - 463, 1999.10