Generalized Physics-Informed Learning through Language-Wide Differentiable Programming (Video)
March 31 2020 in Differential Equations, Mathematics, Science, Scientific ML | Tags: physics-informed machine learning, pinn, scientific machine learning, scientific ml, sciml | Author: Christopher Rackauckas
Chris Rackauckas (MIT), “Generalized Physics-Informed Learning through Language-Wide Differentiable Programming”
Scientific computing is increasingly incorporating the advancements in machine learning to allow for data-driven physics-informed modeling approaches. However, re-targeting existing scientific computing workloads to machine learning frameworks is both costly and limiting, as scientific simulations tend to use the full feature set of a general purpose programming language. In this manuscript we develop an infrastructure for incorporating deep learning into existing scientific computing code through Differentiable Programming (∂P). We describe a ∂P system that is able to take gradients of full Julia programs, making Automatic Differentiation a first class language feature and compatibility with deep learning pervasive. Our system utilizes the one-language nature of Julia package development to augment the existing package ecosystem with deep learning, supporting almost all language … READ MORE
Scientific Machine Learning: Interpretable Neural Networks That Accurately Extrapolate From Small Data
January 14 2020 in Differential Equations, Julia, Mathematics, Science, Scientific ML | Tags: neural ode, physics-informed, sciml, small data, universal differential equations | Author: Christopher Rackauckas
The fundamental problems of classical machine learning are:
- Machine learning models require big data to train
- Machine learning models cannot extrapolate out of the their training data well
- Machine learning models are not interpretable
However, in our recent paper, we have shown that this does not have to be the case. In Universal Differential Equations for Scientific Machine Learning, we start by showing the following figure:
Indeed, it shows that by only seeing the tiny first part of the time series, we can automatically learn the equations in such a manner that it predicts the time series will be cyclic in the future, in a way … READ MORE
Recent advancements in differential equation solver software
October 16 2019 in Differential Equations, Julia, Mathematics, Scientific ML, Uncategorized | Tags: | Author: Christopher Rackauckas
This was a talk given at the Modelica Jubilee Symposium – Future Directions of System Modeling and Simulation.
Recent Advancements in Differential Equation Solver Software
Since the time of the ancient Fortran methods like dop853 and DASSL were created, many advancements in numerical analysis, computational methods, and hardware have accelerated computing. However, many applications of differential equations still rely on the same older software, possibly to their own detriment. In this talk we will describe the recent advancements being made in differential equation solver software, focusing on the Julia-based DifferentialEquations.jl ecosystem. We will show how high order Rosenbrock and IMEX methods have been proven advantageous over traditional BDF implementations in certain problem domains, and the types of issues that give rise to general performance characteristics between the methods. Extensions of these … READ MORE
A Collection of Jacobian Sparsity Acceleration Tools for Julia
October 6 2019 in Differential Equations, Julia, Programming, Scientific ML | Tags: | Author: Christopher Rackauckas
Over the summer there have been a whole suite of sparsity acceleration tools for Julia. These are encoded in the packages:
The toolchain is showcased in the following blog post by Pankaj Mishra, the student who build a lot of the Jacobian coloring and decompression framework. Langwen Huang setup the fast paths for structured matrices (tridiagonal, banded, and block-banded matrices) and also integrated these tools with DifferentialEquations.jl. Shashi Gowda then setup a mechanism for automatically detecting the sparsity of Julia programs (!!!).
A tutorial using this workflow together is described in the SparseDiffTools.jl README. In summary, to use the tools you have the following flow:
- Find your sparsity pattern, Jacobian structure (i.e. Jacobian type), or automatically detect it with SparsityDetection.jl.
- Call `matrix_colors(A)` from SparseDiffTools.jl to get the `colorvec` for A. This is the vector that the … READ MORE
The Essential Tools of Scientific Machine Learning (Scientific ML)
August 20 2019 in Differential Equations, Julia, Mathematics, Programming, Scientific ML | Tags: ai, differential equations, natural language processing, scientific machine learning, scientific ml, sciml | Author: Christopher Rackauckas
Scientific machine learning is a burgeoning discipline which blends scientific computing and machine learning. Traditionally, scientific computing focuses on large-scale mechanistic models, usually differential equations, that are derived from scientific laws that simplified and explained phenomena. On the other hand, machine learning focuses on developing non-mechanistic data-driven models which require minimal knowledge and prior assumptions. The two sides have their pros and cons: differential equation models are great at extrapolating, the terms are explainable, and they can be fit with small data and few parameters. Machine learning models on the other hand require “big data” and lots of parameters but are not biased by the scientists ability to correctly identify valid laws and assumptions.
However, the recent trend has been to merge the two disciplines, allowing explainable models that are data-driven, require less data than traditional machine learning, and utilize the … READ MORE
Neural Jump SDEs (Jump Diffusions) and Neural PDEs
June 5 2019 in Differential Equations, Julia, Mathematics, Stochastics | Tags: CUDA, differentiable programming, DifferentialEquations.jl, gpu, julia, stochastic differential equations | Author: Christopher Rackauckas
This is just an exploration of some new neural models I decided to jot down for safe keeping. DiffEqFlux.jl gives you the differentiable programming tools to allow you to use any DifferentialEquations.jl problem type (DEProblem) mixed with neural networks. We demonstrated this before, not just with neural ordinary differential equations, but also with things like neural stochastic differential equations and neural delay differential equations.
At the time we made DiffEqFlux, we were the “first to the gate” for many of these differential equations types and left it as an open question for people to find a use for these tools. And judging by the Arxiv papers that went out days after NeurIPS submissions were due, it looks like people now have justified some machine learning use cases for them. There were two separate papers on neural … READ MORE
Solving Partial Differential Equations with Julia
August 8 2018 in Differential Equations, HPC, Julia, Mathematics, Programming, Uncategorized | Tags: | Author: Christopher Rackauckas
Here is a talk from JuliaCon 2018 where I describe how to use the tooling across the Julia ecosystem to solve partial differential equations (PDEs), and how the different areas of the ecosystem are evolving to give top-notch PDE solver support.
DifferentialEquations.jl’s Confederated Modular API
July 18 2018 in Differential Equations, Julia, Mathematics, Programming | Tags: | Author: Christopher Rackauckas
I wrote a manuscript describing DifferentialEquations.jl’s confederated modular API and its effect on the local scientific computing ecosystem. It’s now on Arxiv until we can find the right venue for it.
Simulation and Control of Biological Stochasticity
June 5 2018 in Biology, Differential Equations, Mathematics, Stochastics | Tags: | Author: Christopher Rackauckas
Yesterday I defended my PhD thesis entitled “Simulation and Control of Biological Stochasticity”. Here’s the recording:
Algorithm efficiency comes from problem information
January 4 2018 in Differential Equations, Julia, Mathematics, Programming | Tags: | Author: Christopher Rackauckas
This is a high level post about algorithms (especially mathematical, scientific, and data analysis algorithms) which I hope can help people who are not researchers or numerical software developers better understand how to choose and evaluate algorithms. When scientists and programmers think about efficiency of algorithms, they tend to think about high level ideas like temporary arrays, choice of language, and parallelism. Or they tend to think about low level ideas like pointer indirection, cache efficiency, and SIMD vectorization. However, while these features matter, most decent implementations of the same algorithm tend to get quite close in efficiency to each other (probably <3x if everyone is using a performant enough programming language or constructs). If you're sitting down thinking "what is the next algorithm that I am going to invent that will be truly a step beyond what existed before?", ... READ MORE