https://www.invenia.ca/wp-content/uploads/22Leveraging-power-grid-topology-in-machine-learning-assisted-optimal-power-flow22.png
Leveraging power grid topology in machine learning assisted optimal power flow
Published at
IEEE-TPS.
Machine learning assisted optimal power flow (OPF) aims to reduce the computational complexity of these non-linear and non-convex constrained optimization problems by consigning expensive (online) optimization to offline training. The majority of work in this area typically employs fully connected neural networks (FCNN). However, recently convolutional (CNN) and graph (GNN) neural networks have also been investigated, in effort to exploit topological information within the power grid. Although promising results have been obtained, there lacks a systematic comparison between these architectures throughout literature. Accordingly, we introduce a concise framework for generalizing methods for machine learning assisted OPF and assess the performance of a variety of FCNN, CNN and GNN models for two fundamental approaches in this domain: regression (predicting optimal generator set-points) and classification (predicting the active set of constraints). For several synthetic power grids with interconnected utilities, we show that locality properties between feature and target variables are scarce and subsequently demonstrate marginal utility of applying CNN and GNN architectures compared to FCNN for a fixed grid topology. However, with variable topology (for instance, modeling transmission line contingency), GNN models are able to straightforwardly take the change of topological information into account and outperform both FCNN and CNN models.
https://arxiv.org/abs/2110.00306
Thomas Falconer, Letif Mones
https://www.invenia.ca/wp-content/uploads/PRACTICAL-CONDITIONAL-NEURAL-PROCESSES-VIA-TRACTABLE-DEPENDENT-PREDICTIONS.png
Practical Conditional Neural Processes Via Tractable Dependent Predictions
Conditional Neural Processes (CNPs; Garnelo et al., 2018a) are meta-learning models which leverage the flexibility of deep learning to produce well-calibrated predictions and naturally handle off-the-grid and missing data. CNPs scale to large datasets and train with ease. Due to these features, CNPs appear well-suited to tasks from environmental sciences or healthcare. Unfortunately, CNPs do not produce correlated predictions, making them fundamentally inappropriate for many estimation and decision making tasks. Predicting heat waves or floods, for example, requires modelling dependencies in temperature or precipitation over time and space. Existing approaches which model output dependencies, such as Neural Processes (NPs; Garnelo et al., 2018b) or the FullConvGNP (Bruinsma et al., 2021), are either complicated to train or prohibitively expensive. What is needed is an approach which provides dependent predictions, but is simple to train and computationally tractable. In this work, we present a new class of Neural Process models that make correlated predictions and support exact maximum likelihood training that is simple and scalable. We extend the proposed models by using invertible output transformations, to capture non-Gaussian output distributions. Our models can be used in downstream estimation tasks which require dependent function samples. By accounting for output dependencies, our models show improved predictive performance on a range of experiments with synthetic and real data.
https://arxiv.org/abs/2203.08775
Stratis Markou, James Requeima, Wessel P. Bruinsma, Anna Vaughan, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/Modelling-Non-Smooth-Signals-with-Complex-Spectral-Structure.png
Modelling Non-Smooth Signals with Complex Spectral Structure
The Gaussian Process Convolution Model (GPCM; Tobar et al., 2015a) is a model for signals with complex spectral structure. A significant limitation of the GPCM is that it assumes a rapidly decaying spectrum: it can only model smooth signals. Moreover, inference in the GPCM currently requires (1) a mean-field assumption, resulting in poorly calibrated uncertainties, and (2) a tedious variational optimisation of large covariance matrices. We redesign the GPCM model to induce a richer distribution over the spectrum with relaxed assumptions about smoothness: the Causal Gaussian Process Convolution Model (CGPCM) introduces a causality assumption into the GPCM, and the Rough Gaussian Process Convolution Model (RGPCM) can be interpreted as a Bayesian nonparametric generalisation of the fractional Ornstein-Uhlenbeck process. We also propose a more effective variational inference scheme, going beyond the mean-field assumption: we design a Gibbs sampler which directly samples from the optimal variational solution, circumventing any variational optimisation entirely. The proposed variations of the GPCM are validated in experiments on synthetic and real-world data, showing promising results.
https://arxiv.org/abs/2203.06997
Wessel P. Bruinsma, Martin Tegnér, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/Wide-Mean-Field-Bayesian-Neural-Networks-Ignore-the-Data-1.png
Wide Mean-Field Bayesian Neural Networks Ignore the Data
Bayesian neural networks (BNNs) combine the expressive power of deep learning with the advantages of Bayesian formalism. In recent years, the analysis of wide, deep BNNs has provided theoretical insight into their priors and posteriors. However, we have no analogous insight into their posteriors under approximate inference. In this work, we show that mean-field variational inference entirely fails to model the data when the network width is large and the activation function is odd. Specifically, for fully-connected BNNs with odd activation functions and a homoscedastic Gaussian likelihood, we show that the optimal mean-field variational posterior predictive (i.e., function space) distribution converges to the prior predictive distribution as the width tends to infinity. We generalize aspects of this result to other likelihoods. Our theoretical results are suggestive of underfitting behavior previously observered in BNNs. While our convergence bounds are non-asymptotic and constants in our analysis can be computed, they are currently too loose to be applicable in standard training regimes. Finally, we show that the optimal approximate posterior need not tend to the prior if the activation function is not odd, showing that our statements cannot be generalized arbitrarily.
https://arxiv.org/abs/2202.11670
Beau Coker, Wessel P. Bruinsma, David R. Burt, Weiwei Pan, Finale Doshi-Velez
https://www.invenia.ca/wp-content/uploads/Machine-learning-assisted-industrial-symbiosis-Testing-the-ability-of-word-vectors-to-estimate-similarity-for-material-substitutions.jpeg
Machine learning-assisted industrial symbiosis: Testing the ability of word vectors to estimate similarity for material substitutions
A challenge of facilitating industrial symbiosis involves identifying novel uses of waste streams that can satisfy the demands of other industries. For these efforts, a variety of characteristics must often be considered. A mine of relevant knowledge has been gathered in resources such as academic journals and patent databases. However, in looking to harness the potential of such data to support facilitation, compiling information on expansive ranges of material properties and technical requirements from a variety of unstructured sources can pose a significant manual effort. To ameliorate this, we demonstrate and evaluate an automated system that, given a large collection of patents and academic articles related to waste valorization, is able to assist with the process of identifying which waste streams could potentially be used as substitute feedstocks. Instead of aiming to measure (potentially thousands of) material properties directly, we use word correlations as a proxy to reflect “common knowledge.” Novel in furthering this approach is the application of word vectors, which have emerged as a promising natural language processing tool. The process employs a machine learning approach where words are represented as high-dimensional vectors which encode latent features related to words that often appear around it. When this approach is assessed by comparing its suggestions to documented cases, the use of vectors shows potential to incorporate latent information in data-based explorations. Further research into how this approach compares, and could be integrated with, established symbiosis development practices will be key to understanding its full potential and drawbacks.
https://onlinelibrary.wiley.com/doi/full/10.1111/jiec.13245
Chris Davis, Graham Aid
https://www.invenia.ca/wp-content/uploads/AbstractDifferentiation.jl-Backend-Agnostic-Differentiable-Programming-in-Julia.png
AbstractDifferentiation.jl: Backend-Agnostic Differentiable Programming in Julia
(Best Poster Award)
No single Automatic Differentiation (AD) system is the optimal choice for all problems. This means informed selection of an AD system and combinations can be a problem-specific variable that can greatly impact performance. In the Julia programming language, the major AD systems target the same input and thus in theory can compose. Hitherto, switching between AD packages in the Julia Language required end-users to familiarize themselves with the user-facing API of the respective packages. Furthermore, implementing a new, usable AD package required AD package developers to write boilerplate code to define convenience API functions for end-users. As a response to these issues, we present AbstractDifferentiation.jl for the automatized generation of an extensive, unified, user-facing API for any AD package. By splitting the complexity between AD users and AD developers, AD package developers only need to implement one or two primitive definitions to support various utilities for AD users like Jacobians, Hessians and lazy product operators from native primitives such as pullbacks or pushforwards, thus removing tedious -- but so far inevitable -- boilerplate code, and enabling the easy switching and composing between AD implementations for end-users.
https://arxiv.org/abs/2109.12449
Frank Schäfer, Mohamed Tarek, Lyndon White (Frames Catherine White), Chris Rackauckas
https://www.invenia.ca/wp-content/uploads/Screen-Shot-2022-06-14-at-8.14.05-PM.png
Assessing the Cost of Network Simplifications in Long-Term Hydrothermal Dispatch Planning Models
The sustainable utilization of hydro energy relies on accurate estimates of the opportunity cost of the water. This value is calculated through long-term hydrothermal dispatch problems (LTHDP), and the recent literature has raised awareness about the consequences of modeling simplifications in these problems. The inaccurate representation of Kirchhoff's voltage law under the premise of a DC power flow is an example. Under a non-linear AC model, however, the LTHDP becomes intractable, and the literature lacks an accurate evaluation method of different modeling alternatives. In this paper, we extend the state-of-the-art cost-assessment framework of network approximations for LTHDP and bring relevant and practical new insights. First, we increase the quality of the assessment by using an AC power flow to simulate and compare the performance of five policies based on different network approximations. Second, we find that the tightest network relaxation (based on semidefinite programming) is not the one exhibiting the best performance. Results show that the DC power flow with quadratic losses approximation exhibits the lowest expected cost and inconsistency gaps. Finally, its computational burden is lower than that exhibited by the semidefinite relaxation, whereas market distortions are significantly reduced in comparison to previously published benchmarks based on DC power flow.
https://ieeexplore.ieee.org/document/9521833
Andrew W. Rosemberg; Alexandre Street; Joaquim Dias Garcia; Davi M. Valladão; Thuener Silva; Oscar Dowson
https://www.invenia.ca/wp-content/uploads/How-Tight-Can-PAC-Bayes-be-in-the-Small-Data-Regime_.png
How Tight Can PAC-Bayes be in the Small Data Regime?
In this paper, we investigate the question: Given a small number of datapoints, for example N = 30, how tight can PAC-Bayes and test set bounds be made? For such small datasets, test set bounds adversely affect generalisation performance by withholding data from the training procedure. In this setting, PAC-Bayes bounds are especially attractive, due to their ability to use all the data to simultaneously learn a posterior and bound its generalisation risk. We focus on the case of i.i.d. data with a bounded loss and consider the generic PAC-Bayes theorem of Germain et al. While their theorem is known to recover many existing PAC-Bayes bounds, it is unclear what the tightest bound derivable from their framework is. For a fixed learning algorithm and dataset, we show that the tightest possible bound coincides with a bound considered by Catoni; and, in the more natural case of distributions over datasets, we establish a lower bound on the best bound achievable in expectation. Interestingly, this lower bound recovers the Chernoff test set bound if the posterior is equal to the prior. Moreover, to illustrate how tight these bounds can be, we study synthetic one-dimensional classification tasks in which it is feasible to meta-learn both the prior and the form of the bound to numerically optimise for the tightest bounds possible. We find that in this simple, controlled scenario, PAC-Bayes bounds are competitive with comparable, commonly used Chernoff test set bounds. However, the sharpest test set bounds still lead to better guarantees on the generalisation error than the PAC-Bayes bounds we consider.
https://arxiv.org/abs/2106.03542
Andrew Y. K. Foong, Wessel P. Bruinsma, David R. Burt, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/The-Gaussian-Neural-Process.png
The Gaussian Neural Process
Neural Processes (NPs; Garnelo et al., 2018a,b) are a rich class of models for meta-learning that map data sets directly to predictive stochastic processes. We provide a rigorous analysis of the standard maximum-likelihood objective used to train conditional NPs. Moreover, we propose a new member to the Neural Process family called the Gaussian Neural Process (GNP), which models predictive correlations, incorporates translation equivariance, provides universal approximation guarantees, and demonstrates encouraging performance.
https://arxiv.org/abs/2101.03606
Wessel P. Bruinsma, James Requeima, Andrew Y. K. Foong, Jonathan Gordon, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/Deep-learning-architectures-for-inference-of-AC-OPF-solutions.png
Deep learning architectures for inference of AC-OPF solutions
Machine learning assisted optimal power flow (OPF) aims to reduce the computational complexity of these non-linear and non-convex constrained optimization problems by consigning expensive (online) optimization to offline training. The majority of work in this area typically employs fully connected neural networks (FCNN). However, recently convolutional (CNN) and graph (GNN) neural networks have also been investigated, in effort to exploit topological information within the power grid. Although promising results have been obtained, there lacks a systematic comparison between these architectures throughout literature. Accordingly, we introduce a concise framework for generalizing methods for machine learning assisted OPF and assess the performance of a variety of FCNN, CNN and GNN models for two fundamental approaches in this domain: regression (predicting optimal generator set-points) and classification (predicting the active set of constraints). For several synthetic power grids with interconnected utilities, we show that locality properties between feature and target variables are scarce and subsequently demonstrate marginal utility of applying CNN and GNN architectures compared to FCNN for a fixed grid topology. However, with variable topology (for instance, modeling transmission line contingency), GNN models are able to straightforwardly take the change of topological information into account and outperform both FCNN and CNN models.
Posterhttps://arxiv.org/abs/2011.03352
Thomas Falconer, Letif Mones
https://www.invenia.ca/wp-content/uploads/WEmbSim-A-Simple-yet-Effective-Metric-for-Image-Captioning.png
WEmbSim: A Simple yet Effective Metric for Image Captioning
(DSTG Best Contribution to Science Award)
The area of automatic image caption evaluation is still undergoing intensive research to address the needs of generating captions which can meet adequacy and fluency requirements. Based on our past attempts at developing highly sophisticated learning-based metrics, we have discovered that a simple cosine similarity measure using the Mean of Word Embeddings (MOWE) of captions can actually achieve a surprisingly high performance on unsupervised caption evaluation. This inspires our proposed work on an effective metric WEmbSim, which beats complex measures such as SPICE, CIDEr and WMD at system-level correlation with human judgments. Moreover, it also achieves the best accuracy at matching human consensus scores for caption pairs, against commonly used unsupervised methods. Therefore, we believe that WEmbSim sets a new baseline for any complex metric to be justified.
http://www.dicta2020.org/wp-content/uploads/2020/09/1_CameraReady.pdf
Naeha Sharif, Lyndon White (Frames Catherine White), Mohammed Bennamoun, Wei Liu, Syed Afaq Ali Shah
https://www.invenia.ca/wp-content/uploads/Meta-Learning-Stationary-Stochastic-Process-Prediction-with-Convolutional-Neural-Processes.png
Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes
Stationary stochastic processes (SPs) are a key component of many probabilistic models, such as those for off-the-grid spatio-temporal data. They enable the statistical symmetry of underlying physical phenomena to be leveraged, thereby aiding generalization. Prediction in such models can be viewed as a translation equivariant map from observed data sets to predictive SPs, emphasizing the intimate relationship between stationarity and equivariance. Building on this, we propose the Convolutional Neural Process (ConvNP), which endows Neural Processes (NPs) with translation equivariance and extends convolutional conditional NPs to allow for dependencies in the predictive distribution. The latter enables ConvNPs to be deployed in settings which require coherent samples, such as Thompson sampling or conditional image completion. Moreover, we propose a new maximum-likelihood objective to replace the standard ELBO objective in NPs, which conceptually simplifies the framework and empirically improves performance. We demonstrate the strong performance and generalization capabilities of ConvNPs on 1D regression, image completion, and various tasks with real-world spatio-temporal data.
https://arxiv.org/abs/2007.01332
Andrew Y. K. Foong, Wessel P. Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/WordTokenizers.jl-Basic-tools-for-tokenizing-natural-language-in-Julia.png
WordTokenizers.jl: Basic tools for tokenizing natural language in Julia
WordTokenizers.jl is a tool to help users of the Julia programming language (Bezanson, Edelman, Karpinski, & Shah, 2014) work with natural language. In natural language processing (NLP) tokenization refers to breaking a text up into parts – the tokens. Generally, tokenization refers to breaking a sentence up into words and other tokens such as punctuation. Complementary to word tokenization is sentence segmentation or sentence splitting (occasionally also called sentence tokenization), where a document is broken up into sentences, which can then be tokenized into words. Tokenization and sentence segmentation are some of the most fundamental operations to be performed before applying most NLP or information retrieval algorithms.
WordTokenizers.jl is currently being used by packages like TextAnalysis.jl, Transformers.jl and CorpusLoaders.jl for tokenizing text.
https://joss.theoj.org/papers/10.21105/joss.01956
Ayush Kaushal , Lyndon White (Frames Catherine White), Mike Innes , Rohit Kumar
https://www.invenia.ca/wp-content/uploads/Scalable-Exact-Inference-in-Multi-Output-Gaussian-Processes.png
Scalable Exact Inference in Multi-Output Gaussian Processes
Multi-output Gaussian processes (MOGPs) leverage the flexibility and interpretability of GPs while capturing structure across outputs, which is desirable, for example, in spatio-temporal modelling. The key problem with MOGPs is their computational scaling O(n3p3 ), which is cubic in the number of both inputs n (e.g., time points or locations) and outputs p. For this reason, a popular class of MOGPs assumes that the data live around a low-dimensional linear subspace, reducing the complexity to O(n3m3 ). However, this cost is still cubic in the dimensionality of the subspace m, which is still prohibitively expensive for many applications. We propose the use of a sufficient statistic of the data to accelerate inference and learning in MOGPs with orthogonal bases. The method achieves linear scaling in m in practice, allowing these models to scale to large m without sacrificing significant expressivity or requiring approximation. This advance opens up a wide range of real-world tasks and can be combined with existing GP approximations in a plug-and-play way. We demonstrate the efficacy of the method on various synthetic and real-world data sets.
http://proceedings.mlr.press/v119/bruinsma20a.html
Wessel Bruinsma, Eric Perim, William Tebbutt, Scott Hosking, Arno Solin, Richard Turner
https://www.invenia.ca/wp-content/uploads/Square-Reduction-of-the-Optimal-Power-Flow-Problem-through-Meta-Optimization-1.png
Reduction of the Optimal Power Flow Problem through Meta-Optimization
We introduce a method for solving Optimal Power Flow (OPF) using meta-optimization, which can substantially reduce solution times. A pre-trained classifier that predicts the binding constraints of the system is used to generate an initial reduced OPF problem, defined by removing the predicted non-binding constraints. Through an iterative procedure, this initial set of constraints is then ex- tended by those constraints that are violated but not represented in the reduced OPF, guaranteeing an optimal solution of the original OPF problem with the full set of constraints. The classifier is trained using a meta-loss objective, defined by the computational cost of the series of reduced OPF problems.
Posterhttps://www.climatechange.ai/papers/neurips2019/16/paper.pdf
Alex Robson, Mahdi Jamei, Cozmin Ududec, Letif Mones
https://www.invenia.ca/wp-content/uploads/Square-GP-ALPS-Automatic-Latent-Process-Selection-for-Multi-Output-Gaussian-Process-Models.png
GP-ALPS: Automatic Latent Process Selection for Multi-Output Gaussian Process Models
A simple and widely adopted approach to extend Gaussian processes (GPs) to multiple outputs is to model each output as a linear combination of a collection of shared, unobserved latent GPs. An issue with this approach is choosing the number of latent processes and their kernels. These choices are typically done manually, which can be time consuming and prone to human biases. We propose Gaussian Process Automatic Latent Process Selection (GP-ALPS), which automatically chooses the latent processes by turning off those that do not meaningfully contribute to explaining the data. We develop a variational inference scheme, assess the quality of the variational posterior by comparing it against the gold standard MCMC, and demonstrate the suitability of GP-ALPS in a set of preliminary experiments.
AABI_Posterhttps://arxiv.org/abs/1911.01929
Pavel Berkovich, Eric Perim, Wessel Bruinsma
https://www.invenia.ca/wp-content/uploads/Square-Learning-an-Optimally-Reduced-Formulation-of-OPF-through-Meta-optimization.png
Learning an Optimally Reduced Formulation of OPF through Meta-optimization
With increasing share of renewables in power generation mix, system operators would need to run Optimal Power Flow (OPF) problems closer to real-time to better manage uncertainty. Given that OPF is an expensive optimization problem to solve, shifting computational effort away from real-time to offline training by machine learning techniques has become an intense research area. In this paper, we introduce a method for solving OPF problems, which can substantially reduce solve times of the two-step hybrid techniques that comprise of a neural network with a subsequent OPF step guaranteeing optimal solutions. A neural network that predicts the binding status of constraints of the system is used to generate an initial reduced OPF problem, defined by removing the predicted non-binding constraints. This reduced model is then extended in an iterative manner until guaranteeing an optimal solution to the full OPF problem. The classifier is trained using a meta-loss objective, defined by the total computational cost of solving the reduced OPF problems constructed during the iterative procedure. Using a wide range of DC- and AC-OPF problems, we demonstrate that optimizing this meta-loss objective results in a classifier that significantly outperforms conventional loss functions used to train neural network classifiers. We also provide an extensive analysis of the investigated grids as well as an empirical limit of performance of machine learning techniques providing optimal OPF solutions.
https://arxiv.org/abs/1911.06784
Alex Robson, Mahdi Jamei, Cozmin Ududec, Letif Mones
https://www.invenia.ca/wp-content/uploads/Convolutional-Conditional-Neural-Processes.png
Convolutional Conditional Neural Processes
We introduce the Convolutional Conditional Neural Process (ConvCNP), a new member of the Neural Process family that models translation equivariance in the data. Translation equivariance is an important inductive bias for many learning problems including time series modelling, spatial data, and images. The model embeds data sets into an infinite-dimensional function space as opposed to a finite-dimensional vector space. To formalize this notion, we extend the theory of neural representations of sets to include functional representations, and demonstrate that any translation-equivariant embedding can be represented using a convolutional deep set. We evaluate ConvCNPs in several settings, demonstrating that they achieve state-of-the-art performance compared to existing NPs. We demonstrate that building in translation equivariance enables zero-shot generalization to challenging, out-of-domain tasks.
https://arxiv.org/abs/1910.13556
Gordon, J., Bruinsma W. P., Foong, A. Y. K., Requeima, J., Dubois Y., Turner R. E.
https://www.invenia.ca/wp-content/uploads/Square-Meta-Optimization-of-Optimal-Power-Flow.png
Meta-Optimization of Optimal Power Flow
The planning and operation of electricity grids is carried out by solving various forms of con- strained optimization problems. With the increasing variability of system conditions due to the integration of renewable and other distributed energy resources, such optimization problems are growing in complexity and need to be repeated daily, often limited to a 5 minute solve-time. To address this, we propose a meta-optimizer that is used to initialize interior-point solvers. This can significantly reduce the number of iterations to converge to optimality.
Posterhttps://www.climatechange.ai/papers/icml2019/42/paper.pdf
Mahdi Jamei, Letif Mones, Alex Robson, Lyndon White (Frames Catherine White), James Requeima, Cozmin Ududec
https://www.invenia.ca/wp-content/uploads/Square-Memristive-networks-From-graph-theory-to-statistical-physics.png
Memristive networks: From graph theory to statistical physics
This paper is an introduction to a very specific toy model of memristive networks, for which an exact differential equation for the internal memory which contains the Kirchhoff laws is known. In particular, we highlight how the circuit topology enters the dynamics via an analysis of directed graph. We try to highlight in particular the connection between the asymptotic states of memristors and the Ising model, and the relation to the dynamics and statics of disordered systems.
http://iopscience.iop.org/article/10.1209/0295-5075/125/10001/pdf
A. Zegarac and F. Caravelli
https://www.invenia.ca/wp-content/uploads/Square-Business-models-design-space-for-electricity-storage-systems-Case-study-of-the-Netherlands-1.png
Business models design space for electricity storage systems: Case study of the Netherlands
Because of weather uncertainty and dynamics, power generation from some renewable energy technologies is variable. Electricity storage is recognized as a solution to better integrate variable renewable generation into the electricity system. Despite considerable growth in the research on the electricity storage, implementation of electricity storage systems (ESS) is globally negligible because of technical, institutional, and business model challenges. We use literature review and data analysis to provide a conceptual framework and a design space for ESS business models in the case of Dutch electricity sector by taking technological, institutional, and business model considerations into account. We provide a map of single-application business models for ESS in the Netherlands which can be used as a basis for making ESS application portfolios and evaluating ESS business models in other parts of the world as well. Furthermore, this research can be used to inform models that explore the evolution of ESS.
https://www.sciencedirect.com/science/article/pii/S2352152X18301609?viaihub
S.A.R. Mir Mohammadi Kooshknow, C.B. Davis
https://www.invenia.ca/wp-content/uploads/Square-The-Gaussian-Process-Autoregressive-Regression-Model-GPAR-1.png
The Gaussian Process Autoregressive Regression Model (GPAR)
Multi-output regression models must exploit dependencies between outputs to maximise predictive performance. The application of Gaussian processes (GPs) to this setting typically yields models that are computationally demanding and have limited representational power. We present the Gaussian Process Autoregressive Regression (GPAR) model, a scalable multi-output GP model that is able to capture nonlinear, possibly input-varying, dependencies between outputs in a simple and tractable way: the product rule is used to decompose the joint distribution over the outputs into a set of conditionals, each of which is modelled by a standard GP. GPAR’s efficacy is demonstrated on a variety of synthetic and real-world problems, outperforming existing GP models and achieving state-of-the-art performance on the tasks with existing benchmarks.
https://www.invenia.ca/wp-content/uploads/The-Gaussian-Process-Autoregressive-Regression-Model-GPAR.pdf
James Requeima, Wessel Bruinsma, Will Tebbutt, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/Square-Learning-Causally-Generated-Stationary-Time-Series.png
Learning Causally-Generated Stationary Time Series
We present the Causal Gaussian Process Convolution Model (CGPCM), a doubly nonparametric model for causal, spectrally complex dynamical phenomena. The CGPCM is a generative model in which white noise is passed through a causal, nonparametric-window moving-average filter, a construction that we show to be equivalent to a Gaussian process with a nonparametric kernel that is biased towards causally-generated signals. We develop enhanced variational inference and learning schemes for the CGPCM and its previous acausal variant, the GPCM (Tobar et al., 2015b), that significantly improve statistical accuracy. These modelling and inferential contributions are demonstrated on a range of synthetic and real-world signals.
https://www.invenia.ca/wp-content/uploads/Learning-Causally-Generated-Stationary-Time-Series.pdf
Wessel Bruinsma, Richard E. Turner
https://www.invenia.ca/wp-content/uploads/Square-Correlations-and-Clustering-in-Wholesale-Electricity-Markets-thumbnail.jpeg
Correlations and Clustering in Wholesale Electricity Markets
We study the structure of locational marginal prices in day-ahead and real-time wholesale electricity markets. In particular, we consider the case of two North American markets and show that the price correlations contain information on the locational structure of the grid. We study various clustering methods and introduce a type of correlation function based on event synchronization for spiky time series, and another based on string correlations of location names provided by the markets. This allows us to reconstruct aspects of the locational structure of the grid.
https://www.invenia.ca/wp-content/uploads/Correlations-and-Clustering-in-Wholesale-Electricity-Markets.pdf
Tianyu Cui, Francesco Caravelli, Cozmin Ududec
https://www.invenia.ca/wp-content/uploads/The-mise-01.png
The mise en scène of memristive networks: effective memory, dynamics and learning
We discuss the properties of the dynamics of purely memristive circuits. In particular, we show that the amount of memory in a memristive circuit is constrained by the conservation laws of the circuit, and that the dynamics preserves the symmetry by means of a projection on this subspace. We obtain these results both for current and voltage controlled linear memristors. Moreover, we discuss the symmetries of the dynamics which are due to the circuit cohomology, and study the weak and strong non-linear regimes.
https://arxiv.org/abs/1611.02104
Francesco Caravelli
https://www.invenia.ca/wp-content/uploads/Square-The-complex-dynamics-of-memristive-circuits-analytical-results-and-universal-slow-relaxation.png
The complex dynamics of memristive circuits: analytical results and universal slow relaxation
Networks with memristive elements (resistors with memory) are being explored for a variety of applications ranging from unconventional computing to models of the brain. However, analytical results that highlight the role of the graph connectivity on the memory dynamics are still a few, thus limiting our understanding of these important dynamical systems. In this paper, we derive an exact matrix equation of motion that takes into account all the network constraints of a purely memristive circuit, and we employ it to derive analytical results regarding its relaxation properties. We are able to describe the memory evolution in terms of orthogonal projection operators onto the subspace of fundamental loop space of the underlying circuit. This orthogonal projection explicitly reveals the coupling between the spatial and temporal sectors of the memristive circuits and compactly describes the circuit topology. For the case of disordered graphs, we are able to explain the emergence of a power law relaxation as a superposition of exponential relaxation times with a broad range of scales using random matrices. This power law is also {\it universal}, namely independent of the topology of the underlying graph but dependent only on the density of loops. In the case of circuits subject to alternating voltage instead, we are able to obtain an approximate solution of the dynamics, which is tested against a specific network topology. These result suggest a much richer dynamics of memristive networks than previously considered.
https://arxiv.org/abs/1608.08651
Francesco Caravelli, Fabio Lorenzo Traversa, Massimiliano Di Ventra
https://www.invenia.ca/wp-content/uploads/Square-Trajectories-entropy-in-dynamical-graphs-with-memory.png
Trajectories entropy in dynamical graphs with memory
In this paper we investigate the application of non-local graph entropy to evolving and dynamical graphs. The measure is based upon the notion of Markov diffusion on a graph, and relies on the entropy applied to trajectories originating at a specific node. In particular, we study the model of reinforcement-decay graph dynamics, which leads to scale free graphs. We find that the node entropy characterizes the structure of the network in the two parameter phase-space describing the dynamical evolution of the weighted graph. We then apply an adapted version of the entropy measure to purely memristive circuits. We provide evidence that meanwhile in the case of DC voltage the entropy based on the forward probability is enough to characterize the graph properties, in the case of AC voltage generators one needs to consider both forward and backward based transition probabilities. We provide also evidence that the entropy highlights the self-organizing properties of memristive circuits, which re-organizes itself to satisfy the symmetries of the underlying graph.
http://arxiv.org/abs/1511.07135
Francesco Caravelli
https://www.invenia.ca/wp-content/uploads/2016/08/conformity-driven.png
Conformity Driven Agents Support Ordered Phases in the Spatial Public Goods Game
We investigate the spatial Public Goods Game in the presence of conformity-driven agents on a bi-dimensional lattice with periodic boundary conditions. The present setting usually considers fitness-driven agents, i.e., agents that tend to imitate the strategy of their fittest neighbors. Here, fitness is a general property usually adopted to quantify the extent to which individuals are able to succeed, or at least to survive, in a competitive environment. However, when social systems are considered, the evolution of a population might be affected also by social behaviors as conformity, stubbornness, altruism, and selfishness. Although the term evolution can assume different meanings depending on the considered domain, here it corresponds to the set of processes that lead a system towards an equilibrium or a steady-state. In doing so, we use two types of strategy update rules: fitness-driven and conformity-driven. We map fitness to the agents' payoff so that richer agents are those most imitated by fitness-driven agents, while conformity-driven agents tend to imitate the strategy assumed by the majority of their neighbors. Numerical simulations aim to identify critical phenomena, on varying the amount of the relative density of conformity-driven agents in the population, and to study the nature of related equilibria. Remarkably, we find that conformity fosters ordered phases and may also lead to bistable behaviors.
http://arxiv.org/abs/1602.01808
Marco Alberto Javarone, Alberto Antonioni, Francesco Caravelli
https://www.invenia.ca/wp-content/uploads/Square-Optimal-growth-trajectories-with-finite-carrying-capacity.png
Optimal growth trajectories with finite carrying capacity
We investigate the spatial Public Goods Game in the presence of conformity-driven agents on a bi-dimensional lattice with periodic boundary conditions. The present setting usually considers fitness-driven agents, i.e., agents that tend to imitate the strategy of their fittest neighbors. Here, fitness is a general property usually adopted to quantify the extent to which individuals are able to succeed, or at least to survive, in a competitive environment. However, when social systems are considered, the evolution of a population might be affected also by social behaviors as conformity, stubbornness, altruism, and selfishness. Although the term evolution can assume different meanings depending on the considered domain, here it corresponds to the set of processes that lead a system towards an equilibrium or a steady-state. In doing so, we use two types of strategy update rules: fitness-driven and conformity-driven. We map fitness to the agents' payoff so that richer agents are those most imitated by fitness-driven agents, while conformity-driven agents tend to imitate the strategy assumed by the majority of their neighbors. Numerical simulations aim to identify critical phenomena, on varying the amount of the relative density of conformity-driven agents in the population, and to study the nature of related equilibria. Remarkably, we find that conformity fosters ordered phases and may also lead to bistable behaviors.
http://arxiv.org/abs/1602.01808
Marco Alberto Javarone, Alberto Antonioni, Francesco Caravelli
https://www.invenia.ca/wp-content/uploads/Square-Neurogenesis-Paradoxically-Decreases-Both-Pattern-Separation-and-Memory-Interference-1.jpeg
Neurogenesis Paradoxically Decreases Both Pattern Separation and Memory Interference
The hippocampus has been the focus of memory research for decades. While the functional role of this structure is not fully understood, it is widely recognized as being vital for rapid yet accurate encoding and retrieval of associative memories. Since the discovery of adult hippocampal neurogenesis in the dentate gyrus by Altman and Das in the 1960's, many theories and models have been put forward to explain the functional role it plays in learning and memory. These models postulate different ways in which new neurons are introduced into the dentate gyrus and their functional importance for learning and memory. Few if any previous models have incorporated the unique properties of young adult-born dentate granule cells and the developmental trajectory. In this paper, we propose a novel computational model of the dentate gyrus that incorporates the developmental trajectory of the adult-born dentate granule cells, including changes in synaptic plasticity, connectivity, excitability and lateral inhibition, using a modified version of the Restricted Boltzmann machine. Our results show superior performance on memory reconstruction tasks for both recent and distally learned items, when the unique characteristics of young dentate granule cells are taken into account. Even though the hyperexcitability of the young neurons generates more overlapping neural codes, reducing pattern separation, the unique properties of the young neurons nonetheless contribute to reducing retroactive and proactive interference, at both short and long time scales. The sparse connectivity is particularly important for generating distinct memory traces for highly overlapping patterns that are learned within the same context.
http://journal.frontiersin.org/article/10.3389/fnsys.2015.00136/full
Becker, Finnegan
https://www.invenia.ca/wp-content/uploads/Square-Correlation-Structure-of-Spiky-Financial-Data-The-Case-of-Congestion-in-Day-Ahead-Energy-Markets.png
Correlation Structure of Spiky Financial Data: The Case of Congestion in Day-Ahead Energy Markets
I study the correlation structure and argue that these should be ltered. I propose the use of di erent correlation measures other than Pearson, in particular a modi cation of Event Synchronization adapted to negative values or a ltered correlation matrix.
https://www.invenia.ca/wp-content/uploads/Internal2_Caravelli.pdf
Caravelli
https://www.invenia.ca/wp-content/uploads/Square-On-Moments-of-the-Integrated-Exponential-Brownian-Motion.png
On Moments of the Integrated Exponential Brownian Motion
We present new exact expressions for a class of moments for the geometric Brownian motion, in terms of determinants, obtained using a recurrence relation and combinatorial arguments for the case of a Ito’s Wiener process. We then apply the obtained exact formulas to computing averages of the solution of the logistic stochastic differential equation via a series expansion, and compare the results to the solution obtained via Monte Carlo.
http://arxiv.org/abs/1509.05980
Caravelli, Mansur, Severini, Sindoni
https://www.invenia.ca/wp-content/uploads/Square-Bounds-on-Transient-Instability-For-Complex-Ecosystems-1.png
Bounds on Transient Instability For Complex Ecosystems
Stability is a desirable property of complex ecosystems. If a community of interacting species is at a stable equilibrium point then it is able to withstand small perturbations without any adverse effect. In ecology, the Jacobian matrix evalufated at an equilibrium point is known as the community matrix, which represents the population dynamics of interacting species. The system’s asymptotic short- and long-term behaviour can be determined from eigenvalues derived from the community matrix. Here we use results from the theory of pseudospectra to describe intermediate, transient dynamics. We show that the transition from stable to unstable dynamics includes a region of transient instability, where the effect of a small perturbation is amplified before ultimately decaying. The shift from stability to transient instability depends on the magnitude of a perturbation, and we show how to determine lower and upper bounds to the maximum amplitude of perturbations. Of five different types of community matrix, we find that amplification is least severe with predatorprey interactions. This analysis is relevant to other systems whose dynamics can be expressed in terms of the Jacobian matrix. Through understanding transient instability, we can learn under what conditions multiple perturbations—multiple external shocks—will irrecoverably break stability.
http://arxiv.org/pdf/1506.06971.pdf
Caravelli, Staniczenko
https://www.invenia.ca/wp-content/uploads/Square-Multi-scaling-of-wholesale-electricity-prices-1.png
Multi-scaling of wholesale electricity prices
We empirically analyze the most volatile component of the electricity price time series from two North-American wholesale electricity markets. We show that these time series exhibit fluctuations which are not described by a Brownian Motion, as they show multi-scaling, high Hurst exponents and sharp price movements. We use the generalized Hurst exponent (GHE, H(q)) to show that although these time-series have strong cyclical components, the fluctuations exhibit persistent behaviour, i.e., H(q)>0.5. We investigate the effectiveness of the GHE as a predictive tool in a simple linear forecasting model, and study the forecast error as a function of H(q), with q=1 and q=2. Our results suggest that the GHE can be used as prediction tool for these time series when the Hurst exponent is dynamically evaluated on rolling time windows of size ≈50−100 hours. These results are also compared to the case in which the cyclical components have been subtracted from the time series, showing the importance of cyclicality in the prediction power of the Hurst exponent.
http://arxiv.org/abs/1507.06219
Ashtari, Aste, Caravelli, Di Matteo, Requeima, Ududec
https://www.invenia.ca/wp-content/uploads/Square-Scale-free-networks.png
Scale-free networks as an epiphenomenon of memory
Many realistic networks are scale-free, with small characteristic path lengths, high clustering, and power law in their degree distribution. They can be obtained by dynamical networks in which a preferential attachment process takes place. However, this mechanism is nonlocal, in the sense that it requires knowledge of the whole graph in order for the graph to be updated. Instead, if preferential attachment and realistic networks occur in physical systems, these features need to emerge from a local model. In this paper, we propose a local model and show that a possible ingredient (which is often underrated) for obtaining scale-free networks with local rules is memory. Such a model can be realised in solid-state circuits, using non-linear passive elements with memory such as memristors, and thus can be tested experimentally.
http://arxiv.org/pdf/1312.2289.pdf
Caravelli, Di Ventra, Hamma
