다리 움직임(보행연구)... 신경회로 'CPG와 반사회로' 2021 Nature

[문형철]

1. nature  
2. scientific reports  
3. articles  
4. article

An optimality principle for locomotor central pattern generators

Download PDF

• Article
• Open access
• Published: 23 June 2021

An optimality principle for locomotor central pattern generators

• Hansol X. Ryu & 
• Arthur D. Kuo 

Scientific Reports volume 11, Article number: 13140 (2021) Cite this article

•  
•  
•  

Abstract

Two types of neural circuits contribute to legged locomotion: central pattern generators (CPGs) that produce rhythmic motor commands (even in the absence of feedback, termed “fictive locomotion”), and reflex circuits driven by sensory feedback. Each circuit alone serves a clear purpose, and the two together are understood to cooperate during normal locomotion. The difficulty is in explaining their relative balance objectively within a control model, as there are infinite combinations that could produce the same nominal motor pattern. Here we propose that optimization in the presence of uncertainty can explain how the circuits should best be combined for locomotion. The key is to re-interpret the CPG in the context of state estimator-based control: an internal model of the limbs that predicts their state, using sensory feedback to optimally balance competing effects of environmental and sensory uncertainties. We demonstrate use of optimally predicted state to drive a simple model of bipedal, dynamic walking, which thus yields minimal energetic cost of transport and best stability. The internal model may be implemented with neural circuitry compatible with classic CPG models, except with neural parameters determined by optimal estimation principles. Fictive locomotion also emerges, but as a side effect of estimator dynamics rather than an explicit internal rhythm. Uncertainty could be key to shaping CPG behavior and governing optimal use of feedback.

초록

다리 움직임에 기여하는

두 가지 유형의 신경 회로가 존재합니다:

피드백이 없어도 리듬적인 운동 명령을 생성하는 중앙 패턴 생성기(CPGs)와

감각 피드백에 의해 구동되는 반사 회로입니다.

central pattern generators (CPGs) that produce rhythmic motor commands (even in the absence of feedback, termed “fictive locomotion”), and 

reflex circuits driven by sensory feedback

각 회로는 단독으로 명확한 역할을 수행하며,

두 회로는 정상적인 움직임 동안 협력하는 것으로 알려져 있습니다.

문제는 제어 모델 내에서 두 회로의 상대적 균형을 객관적으로 설명하는 것입니다.

동일한 명목상 운동 패턴을 생성할 수 있는 무한한 조합이 존재하기 때문입니다.

본 연구에서는

불확실성 하에서의 최적화가

운동을 위해 회로를 가장 효과적으로 결합하는 방법을 설명할 수 있다고 제안합니다.

핵심은

CPG를 상태 추정기 기반 제어의 맥락에서 재해석하는 것입니다:

감각 피드백을 활용해 환경적 및 감각적 불확실성의 경쟁 효과를 최적화하여

사지의 상태를 예측하는 내부 모델입니다.

최적 예측 상태를 활용해 이족 동적 보행의 단순 모델을 구동하는 것을 시연했으며,

이는 운송 에너지 비용이 최소화되고 안정성이 가장 우수한 결과를 보여줍니다.

내부 모델은 고전적 CPG 모델과 호환되는 신경 회로로 구현될 수 있으나,

신경 매개변수는 최적 추정 원리에 의해 결정됩니다.

가상 운동도 발생하지만,

이는 추정기 동역학의 부수적 효과로 나타날 뿐 명시적인 내부 리듬은 아닙니다.

불확실성은

CPG 행동을 형성하고 피드백의 최적 사용을 조절하는 핵심 요소가 될 수 있습니다.

Similar content being viewed by others

Eval‎uating the energetics of entrainment in a human–machine coupled oscillator system

Article Open access04 August 2021

Oscillating latent dynamics in robot systems during walking and reaching

Article Open access19 May 2024

A neuromuscular model of human locomotion combines spinal reflex circuits with voluntary movements

Article Open access17 May 2022

Introduction

A combination of two types of neural circuitry appears responsible for the basic locomotory motor pattern. One type is the central pattern generator (CPG; Fig. 1A), which generates pre-programmed, rhythmically timed, motor commands1,2,3. The other is the reflex circuit, which produces motor patterns triggered by sensory feedback (Fig. 1C). Although they normally work together, each is also capable of independent action. The intrinsic CPG rhythm patterns can be sustained with no sensory feedback and only a tonic, descending input, as demonstrated by observations of fictive locomotion4,5. Reflex loops alone also appear capable of controlling locomotion1, particularly with a hierarchy of loops integrating multiple sensory modalities for complex behaviors such as stepping and standing control6,7. We refer to the independent extremes as pure feedforward control and pure feedback control. Of course, within the intact animal, both types of circuitry work together for normal locomotion (Fig. 1B)8. However, this cooperation also presents a dilemma, of how authority should optimally be shared between the two9.

서론

두 가지 유형의 신경 회로가

기본적인 운동 패턴을 생성하는 데 관여하는 것으로 알려져 있습니다.

하나는

중앙 패턴 발생기(CPG; 그림 1A)로,

사전 프로그래밍된 리듬에 따라 운동 명령을 생성합니다(1,2,3).

다른 하나는

감각 피드백에 의해 유발되는 운동 패턴을 생성하는 반사 회로입니다(그림 1C).

이 두 회로는 일반적으로 함께 작동하지만,

각각 독립적으로 작동할 수 있습니다.

내재적 CPG 리듬 패턴은 감각 피드백 없이도 지속될 수 있으며,

이는 허구적 운동 관찰을 통해 입증되었습니다4,5.

반사 루프만으로도 운동을 제어할 수 있으며1,

특히 다중 감각 모달리티를 통합하는 루프의 계층 구조를 통해

복잡한 행동(예: 걸음걸이 및 서기 제어)을 조절할 수 있습니다6,7.

우리는 이 독립적인 극단을

순수한 피드포워드 제어와 순수한 피드백 제어라고 부릅니다.

물론, 정상적인 운동을 위해 완전한 동물에서는 두 유형의 회로가 함께 작동합니다(그림 1B)8.

그러나 이 협력은 두 회로 간 권한이 최적의 방식으로 어떻게 분배되어야 하는지에 대한 딜레마를 제기합니다9.

Figure 1

Three ways to control bipedal walking.

(A) The central pattern generator (CPG) comprises neural oscillators that can produce rhythmic motor commands, even in the absence of sensory feedback. Rhythm can be produced by mutually inhibiting neural half-center oscillators (shaded circles).

(B) In normal animal locomotion, the CPG is thought to combine an intrinsic rhythm with sensory feedback, so that the periphery can influence the motor rhythm.

(C) In principle, sensory feedback can also control and stabilize locomotion through reflexes, without need for neural oscillators.

The extreme of (A) CPG control without feedback is referred to here as pure feedforward control, and the opposite extreme (C) with no oscillators as pure feedback control. Any of these schemes could potentially produce the same nominal locomotion pattern, but some (B) combination of feedforward and feedback appears advantageous.

이족 보행을 제어하는 세 가지 방법.

(A) 중앙 패턴 발생기(CPG)는 감각 피드백이 없어도 리듬적인 운동 명령을 생성할 수 있는 신경 진동기로 구성됩니다.

     리듬은 상호 억제되는 신경 반중심 진동기(그늘진 원)에 의해 생성될 수 있습니다.

(B) 정상적인 동물 운동에서 CPG는 내재적 리듬과 감각 피드백을 결합하여 주변부가 운동 리듬에 영향을 미칠 수 있도록 한다고 추정.

(C) 원칙적으로 감각 피드백은 신경 진동기 없이 반사 작용을 통해 운동을 제어하고 안정화할 수 있습니다.

(A) 피드백이 없는 CPG 제어의 극단적 형태는 순수 피드포워드 제어라고 하며, 반대 극단인 (C) 진동기 없는 형태는 순수 피드백 제어라고 합니다. 이 중 어떤 방식도 동일한 명목상 운동 패턴을 생성할 수 있지만, 일부 (B) 피드포워드와 피드백의 조합이 유리해 보입니다.

Full size image

The combination of central pattern generators with sensory feedback has been explored in computational models. For example, some models have added feedback10,11,12,13 to biologically-inspired neural oscillators e.g.,14, which employ networks of mutually inhibiting neurons to intrinsically produce alternating bursts of activity. Sensory input to the neurons can change network behavior based on system state, such as foot contact and limb or body orientation, to help respond to disturbances. The gain or weight of sensory input determines whether it slowly entrains the CPG15, or whether it resets the phase entirely16,17. Controllers of this type have demonstrated legged locomotion in bipedal18 and quadrupedal robots19,20, and even swimming and other behaviors21. A general observation is that feedback improves robustness such as against uneven terrain22. And the addition of feedforward into feedback-based control has been used to vary walking speed23, adjust interlimb coordination24, or enhance stability25. However, a disadvantage is that the means of combining CPG and feedback is often designed ad hoc. This makes it challenging to extend the findings from one CPG model to another or to other gaits or movement tasks.

Optimization principles offer a means for a model to be uniquely defined by quantitative and objective performance measures26. Indeed, CPG models have long used optimization to determine parameter values23,27,28. However, the most robust and capable models to date have tended not to use CPGs or intrinsic timing rhythms. For example, human-like optimization models can traverse highly uneven terrain29,30,31 using only state-based control, where the control command is a function of system state (e.g., positions and velocities of limbs). In fact, reinforcement learning and other robust optimization approaches (e.g., dynamic programming32,33) are typically expressed solely in terms of state, and do not even have provision for time as an explicit input. They have no need for, nor even benefit from, an internally generated rhythm. But feedforward is clearly important in biological CPGs, suggesting that some insight is missing from these optimal control models.

중심 패턴 발생기와 감각 피드백의 결합은 계산 모델에서 탐구되어 왔습니다. 예를 들어, 일부 모델은 생물학적으로 영감을 받은 신경 진동자에 감각 피드백을 추가했습니다. 예를 들어,14와 같은 모델은 상호 억제하는 신경 세포 네트워크를 사용하여 내재적으로 번갈아 발생하는 활동 버스트를 생성합니다. 신경 세포로의 감각 입력은 시스템 상태(예: 발 접촉, 사지 또는 신체 방향)에 따라 네트워크 행동을 변경하여 방해 요인에 대응하는 데 도움을 줍니다. 감각 입력의 이득 또는 가중치는 CPG에 천천히 동기화되도록15 또는 위상을 완전히 재설정하도록16,17 결정합니다. 이러한 유형의 컨트롤러는 이족18 및 사족 로봇19,20, 심지어 수영 및 기타 행동21에서 다리 운동을 시연했습니다. 일반적인 관찰 결과, 피드백은 고르지 않은 지형에 대한 견고성을 향상시키는 것으로 나타났습니다22. 또한 피드백 기반 제어에 피드포워드를 추가하여 보행 속도를 변화23시키고, 사지 간 협응력을 조정24하거나 안정성을 향상25시키는 데 사용되었습니다. 그러나 단점은 CPG와 피드백을 결합하는 방법이 종종 임의적으로 설계된다는 점입니다. 이는 한 CPG 모델에서 얻은 결과를 다른 모델이나 다른 보행 방식 또는 운동 과제로 확장하는 것을 어렵게 만듭니다.

최적화 원리는 모델이 정량적 및 객관적인 성능 지표로 유일하게 정의될 수 있는 방법을 제공합니다26. 실제로 CPG 모델은 오랫동안 매개변수 값을 결정하기 위해 최적화를 사용해 왔습니다23,27,28. 그러나 현재까지 가장 견고하고 강력한 모델들은 CPG나 내재적 타이밍 리듬을 사용하지 않는 경향이 있습니다. 예를 들어, 인간과 유사한 최적화 모델은 상태 기반 제어만을 사용하여 극도로 불규칙한 지형을 이동할 수 있습니다29,30,31. 여기서 제어 명령은 시스템 상태(예: 사지의 위치와 속도)의 함수입니다. 실제로 강화 학습과 다른 견고한 최적화 접근법(예: 동적 프로그래밍32,33)은 일반적으로 상태만으로 표현되며, 시간은 명시적인 입력으로 제공되지 않습니다. 이들은 내부적으로 생성된 리듬이 필요하지도, 이로부터 이점을 얻지도 않습니다. 그러나 생물학적 CPG에서 피드포워드가 명확히 중요하다는 점은 이러한 최적 제어 모델에 일부 통찰력이 부족함을 시사합니다.

There may be a principled reason for a biological controller not to rely on state, as measured, alone. Realistically, a system's state can only be known imperfectly, due to noisy and imperfect sensors. The solution is state estimation33, in which an internal model of the body is used to predict the expected state and sensory information, and feedback from actual sensors is used to correct the state estimate. The sensory feedback gains may be optimized for minimum estimation error separate from control. The separation principle33 of control systems shows that state-based control may be optimized for control performance without regard to noisy sensors, and nevertheless combine well with state estimation33. In practice, actual robots (e.g., bipedal Atlas34 and quadrupedal BigDog35) gain high performance and robustness through such a combination of state estimation driving state-(estimator)-based control. In fact, state-estimator control may be optimized for a noisy environment33, and has been proposed as a model for biological systems36,37,38. The internal model for state estimation is not usually regarded as relevant to the biological CPG’s feedforward, internal rhythm, but we have proposed that it can potentially produce a CPG-like rhythm under conditions simulating fictive locomotion39. Here the rhythmic output is interpreted not as the motor command per se, but as a state estimate that drives the motor command. We demonstrated this concept with a simple model of rhythmic leg motions39 and a preliminary walking model40. This suggests that a walking model designed objectively with state estimator-based control might produce CPG-like rhythms that objectively contribute to locomotion performance.

The purpose of the present study was to test an estimator-based CPG controller with a dynamic walking model. We devised a simple state-based control scheme to produce a stable and economical nominal gait, producing stance and swing leg torques as a function of the leg states. Assuming a noisy system and environment, we devised a state estimator for leg states. A departure from other CPG models is that we optimize sensory feedback and associated gains not for walking performance, but for accurate state estimation. The combination of control and estimation define our interpretation of a CPG controller that incorporates sensory feedback in a noisy environment. Moreover, this same controller may be realized in the form of a biologically-inspired half center oscillator14, with neuron-like dynamics. Because the control scheme depends on accurate state information for its stability and economy, we expected that minimizing state estimation error (and not explicitly walking performance) would nonetheless allow this model to achieve better walking performance. Scaling the sensory feedback either higher or lower than theoretically optimal would be expected to yield poorer performance. Such a model may conceptually explain how CPGs could optimally incorporate sensory feedback.

생물학적 컨트롤러가 측정된 상태에만 의존하지 않는 데는 원칙적인 이유가 있을 수 있습니다. 현실적으로 시스템의 상태는 노이즈가 많고 불완전한 센서 때문에 완벽하게 알 수 없습니다. 해결책은 상태 추정33으로, 신체에 대한 내부 모델을 사용하여 예상 상태와 감각 정보를 예측하고 실제 센서로부터의 피드백을 사용하여 상태 추정을 교정합니다. 센서 피드백 이득은 제어와 독립적으로 최소 추정 오차를 위해 최적화될 수 있습니다. 제어 시스템의 분리 원리33은 상태 기반 제어가 노이즈 센서에 무관하게 제어 성능을 최적화할 수 있으며, 그럼에도 불구하고 상태 추정과 잘 결합될 수 있음을 보여줍니다. 실제로 실제 로봇(예: 이족 보행 Atlas34 및 사족 보행 BigDog35)은 상태 추정 기반 제어와 결합된 이러한 방식으로 높은 성능과 견고성을 달성합니다. 실제로 상태 추정기 제어는 잡음이 있는 환경33에 최적화될 수 있으며, 생물학적 시스템의 모델로 제안되었습니다36,37,38. 상태 추정용 내부 모델은 일반적으로 생물학적 CPG의 피드포워드 내부 리듬과 관련이 없다고 여겨지지만, 우리는 가상의 운동을 모방한 조건에서 CPG 유사 리듬을 생성할 수 있음을 제안했습니다39. 여기서 리듬 출력은 모터 명령 자체로 해석되지 않고 모터 명령을 구동하는 상태 추정값으로 해석됩니다. 우리는 간단한 리듬 다리 운동 모델39과 초기 보행 모델40을 통해 이 개념을 입증했습니다. 이는 상태 추정기 기반 제어를 객관적으로 설계한 보행 모델이 보행 성능에 객관적으로 기여하는 CPG 유사 리듬을 생성할 수 있음을 시사합니다.

본 연구의 목적은 동적 보행 모델과 결합된 추정기 기반 CPG 컨트롤러를 테스트하는 것입니다. 우리는 안정적이고 경제적인 기본 보행 패턴을 생성하기 위해 다리 상태에 따라 서기 다리와 스윙 다리 토크를 생성하는 간단한 상태 기반 제어 방안을 설계했습니다. 소음 시스템과 환경을 가정하여 다리 상태를 추정하는 상태 추정기를 설계했습니다. 다른 CPG 모델과의 차이점은 감각 피드백과 관련된 이득을 보행 성능이 아닌 정확한 상태 추정 위해 최적화했다는 점입니다. 제어와 추정의 결합은 소음 환경에서 감각 피드백을 통합한 CPG 컨트롤러에 대한 우리의 해석을 정의합니다. 또한 이 동일한 컨트롤러는 신경 세포 유사 동역학을 갖춘 생물학적으로 영감받은 반 중심 진동자14 형태로 구현될 수 있습니다. 제어 방식의 안정성과 효율성은 정확한 상태 정보에 의존하기 때문에, 상태 추정 오차를 최소화하는 것(보행 성능을 명시적으로 최적화하지 않더라도)이 이 모델이 더 나은 보행 성능을 달성할 수 있을 것으로 예상했습니다. 감각 피드백을 이론적으로 최적화된 수준보다 높거나 낮게 조정하면 성능이 저하될 것으로 예상됩니다. 이러한 모델은 CPG가 감각 피드백을 최적화하여 통합하는 메커니즘을 개념적으로 설명할 수 있습니다.

Results

Central pattern generator controls a dynamic walking model

The CPG controller produced a periodic gait with a model of human-like dynamic walking (Fig. 2A). The model was inspired by the mechanics and energetics of humans41, whose legs have pendulum-like passive dynamics42 (the swing leg as a pendulum, the stance leg as inverted pendulum), modulated by active control. Model parameters such as mass distribution and foot curvature, and the resultant walking speed and step length were similar to human walking. The nominal step length for a given walking speed also minimized the model’s energy expenditure, as also observed in humans43. Acting on the legs were torque commands (T1 and T2, Fig. 2B) from the CPG, designed to yield a periodic gait (Fig. 2C), by restoring energy dissipated with each step’s ground contact collision44,45. The leg angles and the ground contact condition (“GC”, 1 for contact, 0 otherwise; Fig. 2C) were treated as measurements to be fed back to the CPG. Each leg’s states (xxi≜[θi,θ˙i]T) described a periodic orbit or limit cycle (Fig. 2D), which was locally stable for zero or mild disturbances, but could easily be perturbed enough to make it fall (Fig. 2E).

결과

중앙 패턴 발생기(CPG)가 동적 보행 모델을 제어합니다

CPG 컨트롤러는

인간과 유사한 동적 보행 모델(그림 2A)을 기반으로

주기적인 보행 패턴을 생성했습니다.

이 모델은 인간의 기계적 및 에너지적 특성을 참고로 설계되었으며41,

인간의 다리는 진자 같은 수동적 동역학42(스윙 다리가 진자, 스탠스 다리가 역진자)를 보여주며,

이는 능동적 제어로 조절됩니다.

모델 매개변수(질량 분포, 발 곡률)와 결과적인 보행 속도 및 보폭은

인간 보행과 유사했습니다.

주어진 보행 속도에 대한 명목상 보폭은 모델의 에너지 소비를 최소화했으며,

이는 인간에서도 관찰된 현상입니다43.

다리에 작용하는 토크 명령(T1과 T2, 그림 2B)은 CPG에서 생성되며, 각 걸음의 지면 접촉 충돌로 소모된 에너지를 복원하여 주기적인 걸음걸이를 생성하도록 설계되었습니다(그림 2C)44,45. 다리 각도와 지면 접촉 상태(“GC”, 접촉 시 1, 그렇지 않을 때 0; 그림 2C)는 CPG로 피드백되는 측정값으로 처리되었습니다. 각 다리의 상태(xxi≜[θi,θ˙i]T)는 주기적 궤도 또는 한계 주기(그림 2D)를 묘사했으며, 이는 제로 또는 경미한 방해 조건에서 지역적으로 안정적이었지만, 충분히 방해받으면 쉽게 불안정해져 떨어질 수 있었습니다(그림 2E).

Figure 2

Dynamic walking model controlled by CPG controller with feedback.

(A) Pendulum-like legs are controlled by motor commands for hip torques T1 and T2, with sensory feedback of leg angle and ground contact “GC” relayed back to controller. (B) Controller produces alternating motor commands versus time, which drive (C) leg movement θθ. Sensory measurements of leg angle and ground contact in turn drive the CPG. (D) Resulting motion is a nominal periodic gait (termed a “limit cycle”) plotted in state space θ˙ versus θθ. (E) Discrete perturbation to the limit cycle can cause model to fall.

CPG 컨트롤러로 제어되는 피드백 기반 동적 보행 모델.

(A) 진자 모양의 다리는 골반 토크 T1과 T2를 제어하는 모터 명령으로 움직이며, 다리 각도와 지면 접촉(“GC”)의 감각 피드백이 컨트롤러로 전달됩니다. (B) 컨트롤러는 시간에 따라 교대하는 모터 명령을 생성하며, 이는 (C) 다리 움직임 θθ를 구동합니다. 다리 각도와 지면 접촉의 감각 측정값은 다시 CPG를 구동합니다. (D) 결과적인 운동은 상태 공간 θ˙ 대 θθ에서 표시된 명목상 주기적 보행(“한계 주기”라고 함)입니다. (E) 한계 주기에 대한 이산적 교란은 모델이 넘어질 수 있습니다.

Full size image

The resulting nominal (undisturbed) gait had approximately human-like walking speed and step length. The nominal walking speed was equivalent to 1.25 m/s with step length 0.55 m (or normalized 0.4 (gl)0.5 and 0.55 l, respectively; g is gravitational constant, l is leg length). The corresponding mechanical cost of transport was 0.053, comparable to other passive and active dynamic walking models (e.g.,43,46,47).

This controller had four important features for the analyses that follow. First, the gait had dynamic, pendulum-like leg behavior similar to humans41,42. Second, the controller stabilized walking, meaning ability to withstand minor perturbations due to its state-based control. Third, the information driving control was produced by the controller including CPG, whose entire dynamics and feedback gains were objectively designed by optimal estimation principles, with no ad hoc design. And fourth, the overall amount of feedback could be varied continuously between either extreme of pure feedforward and pure feedback (sensory feedback gain L ranging zero to infinity), while always producing the same nominal gait under noiseless conditions. This was to facilitate study of how parametric variation of sensory feedback affects performance, particularly under noisy conditions.

Pure feedforward and pure feedback are both susceptible to noise

The critical importance of sensory feedback was demonstrated with a disturbance acting on the legs (Fig. 3A). Termed process noise, it represents not only disturbances but also any uncertainty in the environment or internal model. For this demonstration, the disturbance consisted of a single impulsive swing leg angular acceleration (amplitude 5 (g/l) at 15% of nominal stride time). The pure feedforward controller failed to recover (Fig. 3A left), and would fall within about two steps. Its perturbed leg and ground contact states became mismatched to the nominal rhythm, which in pure feedforward does not respond to state deviations. In contrast, the feedback controller could recover from the same perturbation (Fig. 3A right) and return to the nominal gait. Feedback control is driven by system state, and therefore automatically alters the motor command in response to perturbations. Our expectation is that even if a feedforward control is stable under nominal conditions (with zero or mild disturbances), a feedback controller could generally be designed to be more robust.

Figure 3

Pure feedforward and pure feedback (left and right columns, respectively) are adversely affected by (A) process and (B) sensor noise. Process noise refers to disturbances from the environment or imperfect actuation, and sensor noise refers to imperfect sensing. Plots show ground contact condition, leg angles, commanded leg torques, and noise levels versus time, including both the nominal condition without noise (dashed lines), and the perturbed condition with noise (solid lines). With an impulsive, process noise disturbance, pure feedforward control tended to fall, whereas pure feedback was quite stable. With sensor noise alone, pure feedforward was unaffected, but pure feedback tended to fall.

Full size image

We also applied an analogous demonstration with sensor noise (Fig. 3B). Adding continuous sensor noise (standard deviation of 0.1 for each independent leg) to sensory measurements had no effect on pure feedforward control (Fig. 3B left), which ignores sensory signals entirely. But pure feedback was found to be sensitive to noise-corrupted measurements, and would fall within a few steps (Fig. 3B right). This is because erroneous feedback would trigger erroneous motor commands not in accordance with actual limb state. The combined result was that both pure feedforward and pure feedback control had complementary weaknesses. They performed identically without noise, but each was unable to compensate for its particular weakness, either process noise or sensor noise. Feedback control can be robust, but it needs accurate state information.

Neural half-center oscillators are re-interpreted as state estimator

We determined two apparently different representations for the same CPG model. This first was a biologically inspired, neural oscillator (Fig. 4A) representation, intended to resemble previous CPG models10,11,12,13 demonstrating incorporation of sensory feedback. As with such models, the intrinsic rhythm was produced with two mutually inhibiting half-center oscillators, one driving each leg (i=1 for left leg, i=2 for right leg). Each half-center had a total of three neurons, one a primary neuron with standard second-order dynamics (states u and v). Its output drove the second neuron (α) producing the motor command to the ipsilateral leg. The third neuron was responsible for relaying ground contact (“c”) sensory information, to both excite the ipsilateral primary neuron and inhibit the contralateral one. Although this architecture is superficially similar to previous ad hoc models, the present CPG rhythmic dynamics were determined objectively by optimal estimation principles.

Figure 4

Locomotion control circuit interpreted in two representations: (A) Neural central pattern generator with mutually inhibiting half-center oscillators, and as (B) state estimator with feedback control. Each half-center has a primary neuron with two states (u and v, respectively), an auxiliary neuron c for registering ground contact, and an alpha motoneuron α driving leg torque commands. Inputs include a tonic descending drive, and afferent sensory data with gain L. State estimator acts as second-order internal model of leg dynamics to estimate leg states θ^ (hat symbol denotes estimate) and ground contact GC^, which drive state-based command T. The estimator dynamics and estimator parameters including sensory feedback L, and thus the corresponding neural connections and weights, are designed for minimum mean-square estimation error. Leg dynamics have nonlinear terms (see “Methods” section) of small magnitude (thin grayed lines).

Full size image

The same CPG architecture was then re-interpreted in a second, control systems and estimation framework (Fig. 4B), while changing none of the neural circuitry. Here, the structure was not treated as half-center oscillators, but rather as three neural stages from afferent to efferent. The first stage receiving sensory feedback signal was interpreted as a feedback gain L (upper rectangular block, Fig. 4B), modulating the behavior of the second stage, interpreted as a state estimator (middle rectangular block, Fig. 4B) acting as an internal model of leg dynamics. Its output was interpreted as the state estimate, which was fed into the third, state-based motor command stage (lower rectangular block, Fig. 4B). In this interpretation, the three stages correspond with a standard control systems architecture for a state estimator driving state feedback control. In fact, the neural connections and weights of the half-center oscillators were determined by, and are therefore specifically equivalent to, a state estimator driving motor commands to the legs.

The two representations provide complementary insights. The half-center model shows how sensory information can be incorporated into and modulate a CPG rhythm. Half-center models have previously been designed ad hoc and tuned for a desired behavior, and have generally lacked an objective and unique means to determine the architecture (e.g., number of neurons and interconnections) and neural weights. The state estimator-based model offers a means to determine the architecture, neural weights, and parameters for the best performance. This half-center model with feedback could thus be regarded as optimal, for producing accurate state estimates despite the presence of noise.

Optimized sensory feedback gain L yields accurate state estimates

We next examined walking performance in the presence of both process and sensor noise, while varying sensory feedback gain L above and below optimal (Fig. 5; result from 20 simulation trials per condition, 100 steps per trial). We intentionally applied a substantial amount of noise (with fixed covariances), sufficient to topple the model. This was to demonstrate how walking performance can be improved with appropriate sensory feedback gain L, unlike the noiseless case where the model always walks perfectly.

Figure 5

State estimation accuracy and walking performance under noisy conditions, as a function of sensory feedback gain. The theoretically optimal sensory feedback gain (normalized gain of 1) yielded best performance, in terms of mechanical cost of transport (mCOT), step length variability, mean time between falls (MBTF), and state estimator error. Normalized sensory feedback gain varies between extremes of pure feedforward (to the left) and pure feedback (to the right), with 1 corresponding to theoretically predicted optimum Llqe∗. Formally, normalized gain is defined as |L|/|Llqe∗|, where |⋅| denotes matrix norm. Vertical arrow indicates best performance (minimum for all measures except maximum for MTBF). For all gains, model was simulated with a fixed combination of process and sensor noise as input to multiple trials, yielding ensemble average measures. Each data point is an average of 20 trials of 100 steps each, and errorbar indicates standard deviation of the trials. Mechanical cost of transport (mCOT) was defined as positive work divided by body weight and distance travelled, and step variability as root-mean-square (RMS) variability of step length. Falling takes time and dissipates mechanical energy, and so mCOT was computed both including and excluding losses from falls (work, time, distance).

Full size image

As expected of optimal estimation, best estimation performance was achieved for the gain L equal to theoretically predicted optimum Llqe∗ (Fig. 5, normalized sensory feedback gain of 1). We had designed Llqe∗ with linear quadratic estimation (LQE) based on the covariances of process and sensor noise. Applying that gain in nonlinear simulation with added noise resulted in minimum estimation error (Fig. 5, bottom). This suggests that a linear gain was sufficient to yield a good state estimate despite system nonlinearities.

The optimal sensory feedback gain also yielded best walking performance. Even though the same state-based motor command function (α in Fig. 4) was applied in all conditions, that function was dependent on accurate state information. As a result, the optimal gain Llqe∗ yielded greater economy, less step length variability, and fewer falls (Fig. 5). This was actually a side-effect of optimal state estimation, because our state-based motor command was not explicitly designed to optimize any of these performance measures. The minimal mechanical cost of transport was 0.077 under noisy conditions, somewhat higher than the nominal 0.053 without noise. Step length variability was 0.046 l, and the model experienced occasional falls, with MTBF (mean time between falls) of about 9.61 g−0.5l0.5 (or about 7.1 steps). This optimal case served as a basis for comparisons with other values for gain L.

Accurate state estimates yield good walking performance

Applying sensory feedback gains either lower or higher than theoretically predicted optimum generally resulted in poorer walking performance (Fig. 5). We expected that any state-based motor command would be adversely affected by poorer state estimates. The effect of reducing sensory feedback gain (normalized gain less than 1) was to make the system (and particularly its state estimate) more reliant on its feedforward rhythm, and less responsive to external perturbations. The effect of increasing sensory feedback gain was to make the system more reliant and responsive to noisy feedback. Indeed, the direct effect of selecting either too low or too high a sensory feedback gain was an increased error in state estimate. The consequences of control based on less accurate state information were more falls, more step length variability, and greater cost of transport. Over the range of gains examined (normalized sensory feedback gain |L|/|Llqe∗| ranging 0.82–1.44), the performance measures worsened on the order of about 10% (Fig. 5). This suggests that, in a noisy environment, a combination of feedforward and feedback is important for achieving precise and economical walking, and for avoiding falls. Moreover, the optimal combination can be designed using control and estimation principles. This testing condition was referred as “reference condition” for the following demonstration.

Amount of noise determines optimal sensory feedback gain L

We next eval‎uated how walking performance and feedback gain would change with different amounts of noise. The theoretically calculated optimum sensory feedback gain L depends on the ratio of process noise to sensor noise covariances48. Relatively more process noise favors higher sensory feedback gain, and relatively more sensor noise favors a lower sensory feedback gain and thus greater reliance on the feedforward internal model. We demonstrated this by applying different amounts of process noise (low, medium, and high) with a fixed amount of sensor noise, eval‎uating the optimal sensory feedback gain, and performing walking simulations with gains varied about the optimum. Noise covariances were set to multiples of the reference conditions (Fig. 5), of 0.36, 1.15, and 2.06 respectively for low, medium, high process noise; sensor noise was 1.15 of reference condition. The ratio of process to sensor noise covariance was thus smaller (low), the same (medium), and larger (high) compared to the reference condition, as was the theoretically optimal gain obtained using linear quadratic estimation (LQE) equation (Fig. 6).

Figure 6

Theoretically optimal sensory feedback gains increase with greater process noise. Effect of three conditions of increasing process noise (L low, M medium, H high) on walking performance as a function of sensory feedback gain. The theoretically optimal gains (vertical lines) led to best performance, as quantified by mechanical cost of transport (mCOT, including falls), step length variability, mean time between falls (MBTF), and state estimator error. (An exception was step length variability, which had a broad and indistinct minimum.) The predicted optimal sensory feedback gains for each noise condition are indicated with vertical lines. Arrows indicate best performance for each measure for each noise condition. Performance is plotted with normalized sensory feedback gain ranging between extremes of pure feedforward (to the left) and pure feedback (to the right), with 1 corresponding to theoretical optimum Llqe∗ of the previous testing condition (Fig. 5). The process noise covariance was set to multiples of the previous reference values: 0.36 for L, 1.15 for M, and 2.06 for H. Sensor noise covariance was set to 1.15 of previous value.

Full size image

With varying noise, good performance was still achieved with the corresponding, theoretically optimal gain (Fig. 6). The theoretically predicted optimal gain increased with greater process noise, and simulation trials yielded minimum state estimation error at that gain (Fig. 6, Estimation RMS error). Accurate state estimation also contributed to walking performance, with a trend of minimum cost of transport and step variability, and maximum time between falls at the corresponding theoretical optimum. An exception was step length variability in the high process noise condition, which had a broad minimum, with simulation optimum at a slightly lower gain than theoretically predicted. Overall, the linear state estimator predicted the best performing gain well, in terms of state estimation error, energy economy, and robustness to falls. These results are consistent with the expectation that accurate state estimation contributes to improved control.

Removal of sensory feedback causes emergence of fictive locomotion

Although the CPG model normally interacts with the body, it was also found to produce fictive locomotion with peripheral feedback removed (Fig. 7). Here we considered two types of biological sensors, referred to as “error feedback” and “measurement feedback” sensors. Error feedback refers to sensors that can distinguish unexpected perturbations from intended movements49. For example, some muscle spindles and fish lateral lines50 receive corollary efferent signals (e.g. gamma motor neurons in mammals, alpha in invertebrates51) that signify intended movements, and could be interpreted as effectively computing an error signal within the sensor itself50. Measurement feedback sensors refers to those without efferent inputs (e.g., nociceptors, golgi tendon organs, cutaneous skin receptors, and other muscle spindles52), that provide information more directly related to body movement. Both types of sensors are considered important for locomotion, and so we examined the consequences of removing either type.

Figure 7

Emergence of fictive locomotion from CPG model. (A) Block diagram of intact control loop, where sensory measurements y and estimation error e are fed into internal model. Motor command T drives the legs and (through efference copy) the internal model of legs. (B) Two models of fictive locomotion, starting with the intact system but with sensory feedback removed in two ways. Error feedback refers to sensors that receive efferent copy as inhibitory drive (e.g., some muscle spindles). Removal of error (dashed line) results in sustained fictive rhythm, due to feedback between internal model and state-based command. Measurement feedback refers to other, more direct sensors of limb state xx. Removal of such feedback can also produce sustained rhythm from internal model of legs and sensors, interacting with state-based command. (C) Simulated motor spike trains show how fictive locomotion can resemble intact. Measurement feedback case produces slower and weaker rhythm than error feedback.

Full size image

These cases were modeled by disconnecting the periphery in two different ways. This is best illustrated by redrawing the CPG (Fig. 4) more explicitly as a traditional state estimator block diagram (Fig. 7A). The state estimator block diagram could be rearranged into two equivalent forms (Fig. 7B) by relocating the point where the error signal is calculated. In the Error feedback model (Fig. 7B, top), the error is treated as a peripheral comparison associated with the sensor, and in the Measurement feedback model (Fig. 7B, bottom), as a more central comparison. The two block diagrams are logically equivalent when intact, but they differ in behavior with the periphery disconnected.

The case of fictive locomotion with Error feedback sensors (Fig. 7B, top) was modeled by disconnecting error signal e, so that the estimator would run in an open-loop fashion, as if the state estimate were always correct. Despite this disconnection, there remained an internal loop between the estimator internal model and the state-based command generator, that could potentially sustain rhythmic oscillations. The case of fictive locomotion with measurement feedback sensors (Fig. 7C Measurement feedback) was modeled by disconnecting afferent signal y, and reducing estimator gain by about half, as a crude representation of highly disturbed conditions. There remained an internal loop, also potentially capable of sustained oscillations. We tested whether either case would yield a sustained fictive rhythm, illustrated by transforming the motor command T into neural firing rates using a Poisson process.

We found that removal of both types of sensors still yielded sustained neural oscillations (Fig. 7C), equivalent to fictive locomotion. In the case of Error feedback (Fig. 7B), the motor commands from the isolated CPG were equivalent to the intact case without noise in terms of frequency and amplitude. In the case of Measurement feedback (Fig. 7B), the state estimator tended to drive estimate θ^ toward zero, and simulations still produced periodic oscillations, albeit with slower frequency and reduced amplitude compared to intact. This is not unlike observations of fictive locomotion in animals53, although our model’s response depends on manner of disconnection. Here, with both types of disconnection, the resulting fictive locomotion should not be interpreted as evidence of an intrinsic rhythm, but rather a side effect of incorrect state estimation.

Discussion

We have examined how central pattern generators could optimally integrate sensory information to control locomotion. Our CPG model offers an adjustable gain on sensory feedback, to allow for continuous adjustment between pure feedback control to pure feedforward control, all with the same nominal gait under perfect conditions. The model is compatible with previous neural oscillator models, while also being designed through optimal state estimation principles. Simulations reveal how sensory feedback becomes critical under noisy conditions, although not to the exclusion of intrinsic, neural dynamics. In fact, a combination of feedforward and feedback is generally favorable, and the optimal combination can be designed through standard estimation principles. Estimation principles apply quite broadly, and could be readily applied to other models, including ones far more complex than examined here. The state estimation approach also suggests new interpretations for the role of CPGs in animal or robot locomotion.

Feedforward and feedback may be combined optimally for state estimation

One of our most basic findings was that the extremes of pure feedforward or pure feedback control each performed relatively poorly in the presence of noise (Fig. 3). Pure feedforward control, driven solely by an open-loop rhythm, was highly susceptible to falling as a result of process noise. The general problem with a feedforward or time-based rhythm is that a noisy environment can disturb the legs from their nominal motion, so that the nominal command pattern is mismatched for the perturbed state. Under noisy conditions, it is better to trigger motor commands based on feedback of actual limb state, rather than time. But feedback also has its weaknesses, in that noisy sensory information can lead to noisy commands.

Better than these extremes is to combine both feedforward and feedback together, modulated by sensory feedback gain L. The relative trade-off between process and measurement noise, described by the ratio of their covariances, determines the theoretically optimal gain. That gain was found to yield the least estimation error in simulation (normalized sensory feedback gain = 1 in Fig. 5). Moreover, variations in covariance produced predictable shifts in the theoretical optimum, which again yielded least estimation error in simulation (Fig. 6). We demonstrated this using relatively simple, but rigorously defined33 linear estimator dynamics, which worked well despite nonlinearities in the motor command and walking dynamics. Still better performance would be expected with nonlinear estimation techniques such as extended Kalman filters and particle filters54.

State estimation may be separated from state-based control

A unique aspect of our approach is the separation of sensory processing from control. We treat sensory information as inherently noisy, and treat the CPG as the optimal filter of noisy sensory information yielding the best state estimate. The control is then driven by that estimate, and may be designed independently of sensors and for arbitrary objectives. In fact, the present state-based motor command (Eq. 6) was designed ad hoc for reasonable performance, without explicitly optimizing any performance measures. Nevertheless, measures such as cost of transport and robustness against falls showed best performance with the theoretically optimal sensory feedback gain L (Figs. 5, 6).

There are several advantages to this combination of control and estimation. First, accurate state estimation contributes to good state-based control, and poor estimation to poor control. For example, imprecise visual information can induce variability in foot placement55. Increased variability in walking has been associated with poorer walking economy56 and increased fall risk57. Second, this is manifested more rigorously as “the separation principle” of control systems design, where state-based control and state estimation may be optimized separately and then combined for good performance33,48. Third, the same state estimator (and therefore most of the CPG) can be paired with a variety of different state-based motor control schemes, such as commands for different gaits, for non-rhythmic movements including gait transitions, or to achieve different objectives such as balance and agility. This differs from ad hoc approaches, where different tasks are generally expected to require re-design of the entire CPG.

Central pattern generators may be re-interpreted as state estimators

Our model also explains how neural oscillators can be interpreted as state estimators (Fig. 4). Previous CPG oscillator models have incorporated sensory feedback for locomotion15,16,17,18,20,21,22,58, but have not generally defined an optimal feedback gain based on mechanistic control principles. We have re-interpreted neural oscillator circuits in terms of state estimation (Fig. 4), and shown how the gain can be determined in a principled manner, to minimize estimation error (Fig. 5) in the presence of noise. Here, the entire optimal state estimator architecture is defined objectively by the dynamics of the body and environment (including noise parameters), with no ad hoc parameters or architecture. This makes it possible to predict how feedback gains should be altered for different disturbance characteristics (Fig. 6). The nervous system has long been interpreted in terms of internal models, for example in central motor planning and control59,60,61 and in peripheral sensors49. Here we apply internal model concepts to CPGs, for better locomotion performance.

This interpretation also explains fictive locomotion as an emergent behavior. We observed persistent CPG activity despite removal of sensors (and either error or measurement feedback; Fig. 7), but this was not because the CPG was in any way intended to produce rhythmic timing. Rather, fictive locomotion was a side effect of a state-based motor command, in an internal feedback loop with a state estimator, resulting in an apparently time-based rhythm (Fig. 7B). Others have cautioned that CPGs should not be interpreted as generating decisive timing cues62,63,64, especially given the critical role of peripheral feedback in timing9,65,66. In normal locomotion, central circuits and periphery act together in a feedback loop, and so neither can be assigned primacy. The present model operationalizes this interaction, demonstrates its optimality for performance, and shows how it can yield both normal and fictive locomotion.

Optimal control principles may be compatible with neural control

This study argues that it is better to control with state rather than time. The kinematics and muscle forces of locomotion might appear to be time-based trajectories driven by an internal clock. But another view is that the body and legs comprise a dynamical system dependent on state (described e.g. by phase-plane diagrams, Fig. 2), such that the motor command should also be a function of state. That control could be continuous as examined here, or include discrete transitions between circuits (e.g.,8), and could be optimized or adapted through a variety of approaches, such as optimal control33, dynamic programming32, iterative linear quadratic regulators67, and deep reinforcement learning31). Such state-based control is capable of quite complex tasks, including learning different gaits and their transitions, avoiding or climbing over obstacles, and kicking balls31,68. But with noisy sensors34,35, state-based control typically also requires state estimation34,54, which introduces intrinsic dynamics and the possibility of sustained internal oscillations. We therefore suggest that models should be controlled by state-based control (e.g. deep reinforcement learning) coupled with state estimation for noisy environments to achieve advanced capability and performance. The resulting combination of state-driven control and estimation might also exhibit CPG-like fictive behavior, despite having no explicit time-dependent controls.

State estimation may also be applicable to movements other than locomotion. The same circuitry employed here (Fig. 4) could easily contribute a state estimate x^ for any state-dependent movements. For example, a different rhythmic39 movement might be produced with a different state-based motor command; a non-rhythmic postural stabilization might employ a reflex-like (proportional-derivative) control; and a point-to-point movement might be produced by a descending command, supplemented by local stabilization. All of these would be equivalent to substituting different gains and interconnections to the state-based motor command (α or T in Fig. 4), while still relying on all of the remaining circuitry (of Fig. 4). In our view, persistent oscillations could be the outcome of state estimation with an appropriate state-based command for the α motoneuron (see “Methods” section). But the same half-center circuitry could be active and contribute to other movements that use non-locomotory, state-based commands. It is certainly possible that biological CPGs are indeed specialized purely for locomotion alone, but the state estimation interpretation suggests the possibility of a more general, and perhaps previously unrecognized, role in other movements.

The present optimization approach may offer insight on neural adaptation. Although we have explicitly designed a state estimator here, we would also expect a generic neural network, given an appropriate objective function, to be able to learn the equivalent of state estimation. The learning objective could be to minimize error of predicted sensory information, or simply locomotion performance such as cost of transport. Moreover, our results suggest that the eventual performance and control behavior should ultimately depend on body dynamics and noise. A neural system adapting to relatively low process noise (and high sensor noise) would be expected to learn and rely heavily on an internal model. Conversely, relatively high process noise (and low sensor noise) would rely more heavily on sensory feedback. A limitation of our model is that it places few constraints on neural representation, because there are many ways (or “state realizations”48) to achieve the same input–output function for estimation. But the importance and effects of noise on adaptation are hypotheses that might be testable with artificial neural networks or animal preparations.

Limitations of the study

There are, however, cases where state estimation is less applicable. State estimation applies best to systems with inertial dynamics or momentum. Examples include inverted pendulum gaits with limited inherent (or passive dynamic46) stability and pendulum-like leg motions39. The perturbation sensitivity of such dynamics makes state estimation more critical. But other organisms and models may have well-damped limb dynamics and inherently stable body postures, and thus benefit less from state estimation. Others have proposed that intrinsic CPG rhythms may have greater importance in lower than higher vertebrates, potentially because of differences in inherent stability69. There may also be task requirements that call for fast reactions with short synaptic delays, or organismal, energetic, or developmental considerations that limit the complexity of neural circuitry. Such concerns might call for reduced-order internal models48, or even their elimination altogether, in favor of faster and simpler pure feedforward or feedback. At the same time, actual neural circuitry is considerably more complex than the half-center model depicted here, and animals have far more degrees of freedom than considered here. There are some aspects of animal CPGs that could be simpler than full state estimation, and others that encompass very complex dynamics. A more holistic view would balance the principled benefits of internal models and state estimation against the practicality, complexity, and organismal costs.

There are a number of other limitations to this study. The “Anthropomorphic” walking model does not capture three-dimensional motion and multiple degrees of freedom in real animals. We used such a simple model because it is unlikely to have hidden features that could produce the same results for unexpected reasons. We also modeled extremely simple sensors, without representing the complexities of actual biological sensors. The estimator also used a constant, linear gain, and could be improved with nonlinear estimator variants. However, more complex body dynamics could be incorporated quite readily, because the state estimator (Fig. 4) consists of an internal model of body dynamics (Eq. 7) and a feedback loop with appropriate gain (L designed by estimation principles), and no ad hoc parameters. Nonlinearities might also be accommodated by methods of Bayesian estimation70, of which optimal state estimation or Kalman filtering is a special case.

Another limitation is that we used a particularly simple, state-based command law, which was designed more for robustness than for economy. Better economy could be achieved by powering gait with precisely-triggered, trailing-leg push-off39, rather than the simple hip torque applied here. However, the timing is so critical that feedforward conditions (low sensory feedback gains, Fig. 5) would fall too frequently to yield meaningful economy or step variability measures. We therefore elected for more robust control to allow a range of feedforward through feedback to be compared (Fig. 5). But even with more economical control or other control objectives, we would still expect best performance to correspond with optimal sensory gain, due to the advantages of accurate state information. We also used a relatively simple model of walking dynamics, for which more complex models could readily be substituted (Eq. 7) to yield an appropriate estimator. But more complex dynamics also imply more complex state-based locomotion control, for which there are few principled approaches. The present study interprets the CPG as a means to produce an accurate state estimate, which could be considered helpful for state-based control of any complexity.

Conclusion

Our principal contribution has been to reconcile optimal control and estimation with biological CPGs. Evidence of fictive locomotion has long shown that neural oscillators produce timing and amplitude cues. But pre-determined timing is also problematic for optimal control in unpredictable situations63, leading some to question why CPG oscillators should dictate timing62,64. To our knowledge, previous CPG models have not included process or sensor noise in control design. Such noise is simply a reality of non-uniform environments and imperfect sensors. But it also yields an objective criterion for uniquely defining control and estimation parameters. The resulting neural circuits resemble previous oscillator models and can produce and explain nominal, noisy, or fictive locomotion. In our interpretation, there is no issue of primacy between CPG oscillators and sensory feedback, because they interact optimally to deal with a noisy world.

Method

Details of the model and testing are as follows. The CPG model is first described in terms of neural, half-center circuitry, which is then paired with a walking model with pendulum-like leg dynamics. The walking gait is produced by a state-based command generator, which governs how state information is used to drive motor neurons. The model is subjected to process and sensor noise, which tend to cause the gait to be imprecise and subject to falling. The CPG is then re-interpreted as an optimal state estimator, for which sensory feedback gain and internal model parameters may be designed, as a function of noise characteristics. The model is then simulated over multiple trials to computationally eval‎uate its walking performance as a function of sensory feedback gain. It is also simulated without sensory feedback, to test whether it produces fictive locomotion.

CPG architecture based on Matsuoka oscillator

The CPG consists of two, mutually-inhibiting half-center oscillators, receiving a tonic descending input (Fig. 4A). Each half-center has second-order dynamics, described by states ui and vi. This is equivalent to a primary Matsuoka neuron with states for a membrane potential and adaptation or fatigue14. Locomotion requires relatively longer time constants than is realistic for a single biological neuron, and so each model neuron here should be regarded as shorthand for a network of biological neurons with adaptable time constants and synaptic weights, that in aggregate produce first- or second-order dynamics of appropriate time scale. The state ui produces an output qi that can be fed to other neurons. In addition, we included two types of auxiliary neurons (for a total of three neurons per half-center): one for accepting the ground contact input (ci, with value 1 when in ground contact and 0 otherwise for leg i), and the other to act as an alpha (αi) motoneuron to drive the leg. We used a single motoneuron to generate both positive and negative (extensor and flexor) hip torques, as a simplifying alternative to including separate rectifying motoneurons.

Each half-center receives a descending command and two types of sensory feedback. The descending command is a tonic input s, which determines the walking speed. Sensory input from the corresponding leg includes continuous and discrete information. The continuous feedback contains information about leg angle from muscle spindles and other proprioceptors71, which could be modeled as leg angle yi for measurement feedback, or error ei for error feedback sensors. The discrete information is about ground contact ci sent from cutaneous afferents72.

The primary neuron’s second-order dynamics are described by two states. The state ui is mainly affected by its own adaptation, a mutually inhibiting connection from other neurons, sensory input, and efference copy of the motor commands. The second state vi has a decay term, and is driven by the same neuron’s ui as well as sensory input. This is described by the following equations, inspired by14 and previous robot controllers designed for rhythmic arm movements (e.g.,73 and walking22):

u˙i+aiui=−bivi+∑j=12−wijqj+∑j=12⁡hijej+∑j=12⁡rijαj(s,vj,cj)+fi(uu,vv,cc)

(1)

qi=g(ui)

(2)

v˙i+ai′vi=qi+∑j=12⁡hij′ej

(3)

where there are several synaptic weightings: decays ai and ai′, adaptation gain bi, mutual inhibition strength wij (weighting of neuron i’s input from neuron j’s output, where wii=0), an output function g(ui) (set to identity here), sensory input gains hij and hij′, and efference copy strength rij. The neuron also receives efference copy of its associated motor command αj(s,vj,cj), which depends on neuron state, descending drive and ground contact. There are also secondary, higher-order influences summarized by the function fi(uu,vv,cc), which have a relatively small effect on neural dynamics but are part of the state estimator as described below (see “Theoretical equivalence” section). The network parameters for such CPG oscillators are traditionally set through a combination of design rules of thumb and hand-tuning, but here nearly all of the parameters will be determined from an optimal state estimator, as described below.

Walking model with pendulum dynamics

The system being controlled is a simple bipedal model walking in the sagittal plane (Fig. 2A). The model features pendulum-like leg dynamics46, with 16% of body mass at each leg (of length l) and 68% to pelvis/torso, which is modeled as a point mass. The leg’s center of mass was located at 0.645 l from the feet, radius of gyration of 0.326 l, and curved foot with radius 0.3 l. The legs are also actively actuated by added torque inputs (the "Anthropomorphic Model,"43), and energy is dissipated mainly with the collision of leg with ground in the step-to-step transition. The dissipation determines the amount of positive work required each step. In humans, muscles perform much of that work, which in turn accounts for much of the energetic cost of walking44.

The walking model is described mathematically as follows. The equations of motion may be written in terms of vector θθ≜[θ1,θ2]T as

M(θθ,GC)θ¨θ¨+C(θθ,θθ˙,GC)θθ˙+G(θθ,GC)=T(s,θθ,GC)

(4)

where M is the mass matrix, C describes centripetal and Coriolis effects, G contains position-dependent moments such as from gravity, GC≜[GC1,GC2]T contains ground contact, and T≜[T1,T2]T contains hip torques exerted on the legs (State-based control, below). The equations of motion depend on ground contact because each leg alternates between stance and swing leg behaviors, inverted pendulum and hanging pendulum, respectively. We define each matrix to switch the order of elements at heel-strikes, so that equation of motion can be expressed in the same form.

At heelstrike, the model experiences a collision with ground affecting the angular velocities. This is modeled as a perfectly inelastic collision. Using impulse-momentum, the effect may be summarized as the linear transformation

[θ˙1+θ˙2+GC1+GC2+]=S(θθ,GC)⋅[θ˙1−θ˙2−GC1−GC2−]

(5)

where the plus and minus signs (‘ + ’ and ‘–’) denote just after and before impact, respectively. The ground contact states are switched such that the previous stance leg becomes the swing leg, and vice versa. The simulation avoided inadvertent ground contact of the swing legs by ignoring collisions until the stance leg reached a threshold (set as 10% of nominal stance leg angle at heel strike), as is common in simple 2D walking models with rigid legs46.

The resulting gait has several characteristics relevant to the CPG. First, the legs have pendulum-like inertial dynamics, which allow much of the gait to occur passively. For example, each leg swings passively, and its collision with ground automatically induces the next step of a periodic cycle46. Second, inertial dynamics integrate forces over time, such that disturbances can disrupt timing and cause falls. This sensitivity could be reduced with overdamped joints and low-level control, but humans are thought to have significant inertial dynamics42. And third, inertial dynamics are retained in most alternative models with more degrees of freedom and more complexity (e.g.,29,31). We believe most other dynamical models would also benefit from feedback control similar in concept to presented here.

State-based motor command generator

The model produces state-dependent hip torque commands to the legs. Of the many ways to power a dynamic walking model (e.g.,43,74,75,76), we apply a constant extensor hip torque against the stance leg, for its parametric simplicity and robustness to perturbations. The torque normally performs positive work (Fig. 2B) to make up for collision losses, and could be produced in reaction to a torso leaned forward (not modeled explicitly here;46). The swing leg experiences a hip torque proportional to swing leg angle (Fig. 2B,C), with the effect of tuning the swing frequency39. This control scheme is actually suboptimal for economy, and is selected for its robustness. Optimal economy actually requires perfectly-timed, impulsive forces from the legs45, and has poor robustness to the noisy conditions examined here. The present non-impulsive control is much more robust, and can still have its performance optimized by appropriate state estimation.

The overall torque command Ti for leg i is used as the motor command αi, and may be summarized as

Ti(s,θi,GCi)=−(kst+μsts)⋅ GCi−(kswθi)⋅(1−GCi)

(6)

where the stance phase torque is increased from the initial value kst by the amount proportional to the descending command s with gain μst. The swing phase torque has gain ksw for the proportionality to leg angle θi.

There are also two higher level types of control acting on the system. One is to regulate walking speed, by slowly modulating the tonic, descending command s (Eq. 6). An integral control is applied on s, so keep attain the same average walking speed despite noise, which would otherwise reduce average speed. The second type of high-level control is to restart the simulation after falling. When falling is detected (as a horizontal stance leg angle), the walking model is reset to its nominal initial condition, except advanced one nominal step length forward from the previous footfall location. No penalty is assessed for this re-set process, other than additional energy and time wasted in the fall itself. We quantify the susceptibility to falling with a mean time between falls (MTBF), and report overall energetic cost in two ways, including and excluding failed steps. The wasted energy of failed steps is ignored in the latter case, resulting in lower reported energy cost.

Noise model with process and sensor noise

The walking dynamics are subject to two types of disturbances, process and sensor noise (Fig. 3). Both are modeled as zero-mean, Gaussian white noise. Process noise nx (with covariance Nx) acts as an unpredictable disturbance to the states, due to external perturbations or noisy motor commands. Sensor or measurement noise ny (with covariance Ny) models imperfect sensors, and acts additively to the sensory measurements y. The errors induced by both types of noise are unknown to the central nervous system controller, and so both tend to reduce performance.

The noise covariances were set so that the model would be significantly affected by both types of noise. We sought levels sufficient to cause significant risk of falling, so that good control would be necessary to avoid falling while also achieving good economy. Process noise was described by covariance matrix Nx, with diagonals filled with variances of noisy angular accelerations, which had standard deviations of 0.015(g/l) for stance leg, 0.16(g/l) for swing leg for the reference testing condition. Sensor noise covariance Ny was also set as a diagonal matrix with both entries of standard deviation 0.1 for the reference testing condition. For the later demonstrations to test the effect of the different amount of noise, we multiplied 1.15 to the sensor noise covariance, and 0.36, 1.15, 2.06 to the process noise covariance. Noise was implemented as a spline interpolation of discrete white noise sampled at frequency of 16 (g/l)0.5 (well above pendulum bandwidth) and truncated to no more than ± 3 standard deviations.

State estimator with internal model of dynamics

A state estimator is formed from an internal model of the leg dynamics being controlled (see block diagram in Fig. 4), to produce a prediction of the expected state x^ and sensory measurements y^ (with the hat symbol ‘^’ denoting an internal model estimate). Although the actual state is unknown, the actual sensory feedback y is known, and the expectation error e=y−y^ may be fed back to the internal model with negative feedback (gain L) to correct the state estimate. Estimation theory shows that regulating error e toward zero also tends to drive the state estimate towards actual state (assuming system observability, as is the case here; e.g.,48). This may be formulated as an optimization problem, where gain L is selected to minimize the mean-square estimation error. Here we interpret the half-center oscillator network as such an optimal state estimator, the design of which will determine the network parameters.

The estimator equations may be described in state space. The estimator states are governed by the same equations of motion as the walking model (Eqs. 4, 5), with the addition of the feedback correction. Again using hat notation for state estimates, the nonlinear state estimate equations are

[θ^˙θ^˙θ^¨θ^¨]=[θ^˙θ^˙M−1(−Cθ^˙θ^˙−G+T)]+L(θθ−θθ^)

(7)

We used standard state estimator equations to determine a constant sensory feedback gain L. This was done by linearizing the dynamics about a nominal state, and then designing an optimal estimator based on process and sensor noise covariances (Nx and Ny) using standard procedures (“lqe” command in Matlab, The MathWorks, Natick, MA). This yields a set of gains that minimize mean-square estimation error (xx−xx^), for an infinite horizon and linear dynamics. The constant gain was then applied to the nonlinear system in simulation, with the assumption that the resulting estimator would still be nearly optimal in behavior. Another sensory input to the system is ground contact GCi, a boolean variable. The state estimator ignores measured GCi for pure feedforward control (zero feedback gain L), but for all other conditions (non-zero L), any sensed change in ground contact overrides the estimated ground contact GC^i. When the estimated ground contact state changes, the estimated angular velocities are updated according to the same collision dynamics as the walking model (Eq. 5 except with estimated variables).

The state estimate is applied to the state-based motor command (Eq. 6). Although the walking control was designed for actual state information (θi,GCi), for walking simulations it uses the state estimate instead:

Ti(s,θ^i,GC^i)=−(kst+μsts)⋅GC^i−(kswθ^i)⋅(1−GC^i)

(8)

As with the estimator gain, this also requires an assumption. In the present nonlinear system, we assume that the state estimate may replace the state without ill effect, a proven fact only for linear systems (certainty-equivalence principle,33,77). Both assumptions, regarding gain L and use of state estimate, are tested in simulation below.

Theoretical equivalence between neural oscillator and state estimator

Having fully described the walking model in terms of control systems principles, the equivalent half-center oscillator model may be determined (Fig. 4B). The identical behavior is obtained by re-interpreting the neural states in terms of the dynamic walking model states,

u1≜θ^˙1,v2≜θ^1,u2≜θ^˙2,v2≜θ^2

(9)

along with the neural output function defined as identity,

qi=g(ui)≜ui.

(10)

In addition, motor command and ground contact state are defined to match state-based variables (Eq. 6):

α1≜T1,α2≜T2,c1≜GC^1,c2≜GC^2.

(11)

The synaptic weights and higher-order functions (Eqs. 1–3) are defined according to the internal model equations of motion (Eq. 7),

[a1w12w21a2]=M−1C

(12)

[b1θ^1−f1(θ^θ^,θ^˙θ^˙,GC^)b2θ^2−f2(θ^θ^,θ^˙θ^˙,GC^)]=M−1G 

(13)

[h′11h′12h21′h22′h11h12h21h22]=L 

(14)

[r11r12r21r22]=M−1

(15)

a′1=a2′=0

(16)

Because the mass matrix and other variables are state dependent, the weightings above are state dependent as well. The functions f1 and f2 are higher-order terms, which could be considered optional; omitting them would effectively yield a reduced-order estimator.

The result of these definitions is that the half-center neuron equations (Eqs. 1–3) may be rewritten in terms of θ^i and θ^˙i, to illustrate how the network models the leg dynamics and receives inputs from sensory feedback and efference copy:

θ^¨i+aiθ^˙i=−biθ^i+∑j=12−wijθ^˙j+∑j=12⁡hijej+∑j=12⁡rijα(s,θ^j,GC^j)+fi(θ^θ^,θ^˙θ^˙,GC^)

(17)

θ^˙i+ai′θ^i=θ^˙i+∑j=12⁡hijej

(18)

The above may be interpreted as an internal model of the stance and swing leg as pendulums, with pendulum phasing modulated by error feedback ej and efference copy of the motor command (plus small nonlinearities due to inertial coupling of the two pendulums).

The result is that the entire neural circuitry and parameters are fully specified by the control systems model. In Eqs. (17–18), all of the quantities except motor command α are determined by a state estimator (Eq. 7) with optimal gains (determined by single Matlab command ‘lqe’.) For example, higher-order terms fi(θ^θ^,θ^˙θ^˙,GC^) are defined by Eq. (13). The only aspect of the system not determined by optimal estimation was the motor command α, equal to the state-based motor command (Eq. 8). This was designed ad hoc to produce alternating stance and swing phases with high robustness to perturbations.

Parametric effect of varying sensory feedback gain L

The sensory feedback gain is selected using state estimation theory, according to the amount of process noise and sensor noise. High process noise, or uncertainty about the dynamics and environment, favors a higher feedback gain, whereas high sensor noise favors a lower feedback gain. The ratio between the noise levels determines the optimal linear quadratic estimator gain Llqe∗ (Matlab function “lqe”). A constant gain was determined based on a linear approximation for the leg dynamics, an infinite horizon for estimation, and a stationarity assumption for noise. In simulation, the state estimator was implemented with nonlinear dynamics, assuming this would yield near-optimal performance.

It is thus instructive to eval‎uate walking performance for a range of feedback gains. Setting L too low or too high would be expected to yield poor performance. Setting L equal to the optimal LQE gain Llqe∗ would be expected to yield approximately the least estimation error, and therefore the most precise control (e.g.78). In terms of gait, more precise control would be expected to reduce step variability and mechanical work, both of which are related to metabolic energy expenditure in humans (e.g.,56). The walking model is also prone to falling when disturbed by noise, and optimal state estimation would be expected to reduce the frequency of falling.

We performed a series of walking simulations to test the effect of varying the feedback gain. The model was tested with 20 trials of 100 steps each, subjected to pseudorandom process and sensor noise of fixed covariance (W and V, respectively). In each trial, walking performance was assessed with mechanical cost of transport (mCOT, defined as positive mechanical work per body weight and distance travelled; e.g.,47), step length variability, and mean time between falls (MTBF) as a measure of walking robustness (also referred to as Mean First Passage Time79). The sensory feedback gain Llqe∗ was first designed in accordance with the experimental noise parameters, and then the corresponding walking performance was eval‎uated. Additional trials were performed, varying sensory feedback gain L with lower and higher than optimal values to test for a possible performance penalty. These sub-optimal gains were determined by re-designing the estimator with process noise ρW (ρ between 10^-4 and 10^0.8, with smaller values tending toward pure feedforward and larger toward pure feedback). This procedure guarantees stable closed-loop estimator dynamics, which would not be the case if the matrix Llqe∗ were simply scaled higher or lower. For all trials, the redesigned L was tested in simulations using the fixed process and sensor noise levels. The overall sensory feedback gain was quantified with a scalar, defined as the L2 norm (largest singular value) of matrix L, normalized by the L2 norm of Llqe∗.

We expected that optimal performance in simulation would be achieved with gain L close to the theoretically optimal LQE gain, Llqe∗. With too low a gain (L=0, feedforward Fig. 1A), the model would perform poorly due to sensitivity to process noise, and with too high a gain (L→∞, feedback Fig. 1C), it would perform poorly due to sensor noise. And for intermediate gains, we expected performance to have an approximately convex bowl shape, centered about a minimum at or near Llqe∗. These differences were expected from noise alone, as the model was designed to yield the same nominal gait regardless of gain L. Simulations were necessary to test the model, because its nonlinearities do not admit analytical calculation of performance statistics.

Eval‎uation of fictive locomotion

We tested whether the model would produce fictive locomotion with removal of sensory feedback. Disconnection of feedback in a closed-loop control system would normally be expected to eliminate any persistent oscillations. But estimator-based control actually contains two types of inner loops (Fig. 7A), both of which could potentially allow for sustained oscillations in the absence of sensory feedback. However, the emergence of fictive locomotion and its characteristics depend on what kind of sensory signal is removed. We considered two broad classes of sensors, referred to producing error feedback and measurement feedback, with different expectations for the effects of their removal.

Some proprioceptors relevant to locomotion, including some muscle spindles and fish lateral lines50, could be regarded as producing error feedback. They receive corollary discharge of motor commands, and appear to predict intended movements, so that the afferents are most sensitive to unexpected perturbations. The comparison between expected and actual sensory output largely occurs within the sensor itself, yielding error signal e (Fig. 7B). Disconnecting the sensor would therefore disconnect error signal e, and would isolate an inner loop between state-based command and internal model. The motor command normally sustains rhythmic movement of the legs for locomotion, and would also be expected to sustain rhythmic oscillations within the internal model. Fictive locomotion in this case would be expected to resemble the nominal motor pattern.

Sensors that do not receive corollary discharge could be regarded as direct sensors, in that they relay measurement feedback related to state. In this case, disconnecting the sensor would be equivalent to removing measurement y. This isolates two inner loops, both the command-and-internal-model loop above, as well as a sensory prediction loop between sensor model and internal model. The interaction of these loops would be expected to yield a more complex response, highly dependent on parameter values. Nonetheless, we would expect that removal of y would substantially weaken the sensory input to the internal model, and generally result in a weaker or slower fictive rhythm.

We tested for the existence of sustained rhythms for both extremes of error feedback and measurement feedback. Of course, actual biological sensors within animals are vastly more diverse and complex than this model. But the existence of sustained oscillations in extreme cases would also indicate whether fictive locomotion would be possible with some combination of different sensors within these extremes.

Code availability

The source code for the simulation, supplementary table & video are available in a public repository at: https://github.com/hansolxryu/CPG_biped_walker_Ryu_Kuo (https://doi.org/10.5281/zenodo.4739744).

References

1.  
2.  
3.  
4.  
5.  
6.  
7.  
8. 1. 1. 1. 1. 1. 1. 1. 1.  
9.