Transfer on Count-based Quadratic Control of Markov Jump Linear Systems with Unknown Transition Probabilities

Rafael Lemes Beirigo¹; Marcos Garcia Todorov¹; André da Motta Salles Barreto²

¹ Laboratório Nacional de Computação Científica - LNCC; ² Laboratório Nacional de Computação Científica - LNCC; Google DeepMind

Resumo

This paper proposes a transfer strategy to accelerate the model-based optimal quadratic control of discrete-time Markov jump linear systems (MJLS) with unknown Markov chain transition probabilities, but complete information of the jump process. Our approach is based upon a recently developed adaptive control strategy that incrementally builds a transition model via maximum-likelihood estimation, based on online measurements of the Markov chain, and uses it to adjust the current policy in a certainty equivalence fashion. Despite the advantages presented by that strategy, it relies on repetitive executions of a subroutine used to obtain the policy by solving the subjacent Riccati equation, which can be computationally expensive. Here we propose an attempt to mitigate this computational cost by transferring intermediate solutions across subsequent calls to the subroutine that solves the Riccati equation during the model improvement steps and policy calculations. The method's performance is illustrated on a numerical example regarding Samuelson's macroeconomic model. The experimental results suggest the method was able to incrementally decrease the computational cost needed to calculate the optimal policy approximations.

Palavras-chave: Markov jump linear systems; adaptive control; learning; transfer