Machine learning with hard constraints: Neural Differential-Algebraic Equations (DAEs) as a general formalism
June 3 2025 in Differential Equations, Julia, Mathematics, Programming, Science, Scientific ML, Uncategorized | Tags: adjoint methods, differential-algebraic equations, julia, modelingtoolkit, neural dae, numerical solvers | Author: Christopher Rackauckas
We recently released a new manuscript Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints where we showed a way to develop neural networks where any arbitrary constraint function can be directly imposed throughout the evolution equation to near floating point accuracy. However, in true academic form it focuses directly on getting to the point about the architecture, but here I want to elaborate about the mathematical structures that surround the object, particularly the differential-algebraic equation (DAE), how its various formulations lead to the various architectures (such as stabilized neural ODEs), and elaborate on the other related architectures which haven’t had a paper yet but how you’d do it (and in what circumstances they would make sense).
READ MOREHow chaotic is chaos? How some AI for Science / SciML papers are overstating accuracy claims
May 26 2025 in Differential Equations, Julia, Mathematics, Programming, Science, Scientific ML | Tags: | Author: Christopher Rackauckas
Just how chaotic are chaotic systems? Many of you may have heard of “the butterfly effect” but don’t quite know the mathematics behind such systems. What I want to demonstrate is the “sensitive dependence to initial conditions” property of chaotic systems and just how sensitive these systems are. The reason this has come up is that I have seen some AI papers claiming to be able to predict the timeseries of a chaotic system (many more can be found online too, just highlighting a few random ones). What I want to bring to the forefront is an examination of what is really being claimed: just how hard is it to actually forecast a chaotic system? And if they aren’t doing that, what have they done instead?
Quick Understanding of Chaos: Sensitive Dependence and the Shadowing Lemma
First of … READ MORE
Open Source Component-Based Modeling with ModelingToolkit
May 5 2025 in Differential Equations, Julia, Mathematics, Programming | Tags: | Author: Christopher Rackauckas
Component-based modeling systems such as Simulink and Dymola allow for building scientific models in a way that can be composed. For example, Bob can build a model of an engine, and Alice can build a model of a drive shaft, and you can then connect the two models and have a model of a car. These kinds of tools are used all throughout industrial modeling and simulation in order to allow for “separation of concerns”, allowing experts to engineer their domain and compose the final digital twins with reusable scientific modules. But what about open source? In this talk we will introduce ModelingToolkit, an open source component-based modeling framework that allows for composing pre-built models and scales to large high-fidelity digital twins.
PyData is … READ MORE
The Numerical Analysis of Differentiable Simulation: Automatic Differentiation Can Be Incorrect
April 20 2025 in Differential Equations, Julia, Mathematics, Scientific ML | Tags: | Author: Christopher Rackauckas
The Numerical Analysis of Differentiable Simulation: How Automatic Differentiation of Physics Can Give Incorrect Derivatives
Scientific machine learning (SciML) relies heavily on automatic differentiation (AD), the process of constructing gradients which include machine learning integrated into mechanistic models for the purpose of gradient-based optimization. While these differentiable programming approaches pitch an idea of “simply put the simulator into a loss function and use AD”, it turns out there are a lot more subtle details to consider in practice. In this talk we will dive into the numerical analysis of differentiable simulation and ask the question: how numerically stable and robust is AD? We will use examples from the Python-based Jax (diffrax) and PyTorch (torchdiffeq) libraries in order to demonstrate how canonical … READ MORE
JuliaSim: Building a Product which improves Open Source Sustainability
January 26 2025 in Differential Equations, HPC, Julia, Scientific ML | Tags: | Author: Christopher Rackauckas
How do you build products that support open source communities? In this non-technical talk with OpenTeams I discuss how the MIT Julia Lab, PumasAI, and JuliaHub have all been essential pillars of the julialang opensource community in its goal to achieve sustainable open science. If you’ve ever been curious about what the difference is between the Julia Lab and JuliaHub is, the evolution of these groups, and what kinds of different contributions they make to the open source community, in this talk I go through as many details as I could!
Differences Between Methods for Solving Stiff ODEs
April 6 2024 in Differential Equations, Mathematics | Tags: | Author: Christopher Rackauckas
I found these notes from August 2018 and thought they might be useful so I am posting them verbatim.
A stiff ordinary differential equation is a difficult problem to integrator. However, many of the ODE solver suites offer quite a few different choices for this kind of problem. DifferentialEquations.jl offers almost 200 different choices for example. In this article we will dig into what the differences between these integrators really is so that way you can more easily find which one will be most efficient for your problem.
Quick Overview (tl;dr)
- A BDF, Rosenbrock, ESDIRK method are standard
- For small equations, Rosenbrock methods have performance advantages
- For very stiff systems, Rosenbrock and Rosenbrock-W methods do not require convergence of Newton’s method and thus can take larger steps, being more efficient
- BDF integrators are only L-stable (and A-stable) to order 2, so if the problem is … READ MORE
Direct Automatic Differentiation of (Differential Equation) Solvers vs Analytical Adjoints: Which is Better?
October 11 2022 in Differential Equations, Julia, Mathematics, Science, Scientific ML | Tags: automatic differentiation, differentiable programming, sciml | Author: Christopher Rackauckas
Automatic differentiation of a “solver” is a subject with many details for doing it in the most effective form. For this reason, there are a lot of talks and courses that go into lots of depth on the topic. I recently gave a talk on some of the latest stuff in differentiable simulation with the American Statistical Association, and have some detailed notes on such adjoint derivations as part of the 18.337 Parallel Computing and Scientific Machine Learning graduate course at MIT. And there are entire organizations like my SciML Open Source Software Organization which work day-in and day-out on the development of new differentiable solvers.
I’ll give a brief summary of all my materials here below.
Continuous vs Discrete Differentiation of Solvers
AD of a solver can be done in essentially two different ways: either directly performing automatic … READ MORE
Composing Modeling and Simulation with Julia (2021 Modelica Conference)
April 8 2022 in Differential Equations, Julia, Science, Scientific ML | Tags: | Author: Christopher Rackauckas
In this paper we introduce JuliaSim, a high-performance programming environment designed to blend traditional modeling and simulation with machine learning. JuliaSim can build accelerated surrogates from component-based models, such as those conforming to the FMI standard, using continuous-time echo state networks (CTESN). The foundation of this environment, ModelingToolkit.jl, is an acausal modeling language which can compose the trained surrogates as components within its staged compilation process. As a complementary factor we present the JuliaSim model library, a standard library with differential-algebraic equations and pre-trained surrogates, which can be composed using the modeling system for design, optimization, and control. We demonstrate the effectiveness of the surrogate-accelerated modeling and simulation approach on HVAC dynamics by showing that the CTESN surrogates accurately capture the dynamics of a HVAC … READ MORE
Learning Epidemic Models That Extrapolate, AI4Pandemics
July 25 2021 in Differential Equations, Julia, Mathematics, Science, Scientific ML | Tags: epidemics, scientific machine learning, sciml | Author: Christopher Rackauckas
I think this talk was pretty good so I wanted to link it here!
Title: Learning Epidemic Models That Extrapolate
Speaker Chris Rackauckas, https://chrisrackauckas.com/
Abstract:
Modern techniques of machine learning are uncanny in their ability to automatically learn predictive models directly from data. However, they do not tend to work beyond their original training dataset. Mechanistic models utilize characteristics of the problem to ensure accurate qualitative extrapolation but can lack in predictive power. How can we build techniques which integrate the best of both approaches? In this talk we will discuss the body of work around universal differential equations, a technique which mixes traditional differential equation modeling with machine learning for accurate extrapolation from small data. We will showcase how incorporating different variations of the technique, such … READ MORE
ModelingToolkit, Modelica, and Modia: The Composable Modeling Future in Julia
April 19 2021 in Differential Equations, Julia, Mathematics, Programming, Science, Scientific ML | Tags: acausal, dae, language, modelica, modeling, modelingtoolkit, modia | Author: Christopher Rackauckas
Let me take a bit of time here to write out a complete canonical answer to ModelingToolkit and how it relates to Modia and Modelica. This question comes up a lot: why does ModelingToolkit exist instead of building on tooling for Modelica compilers? I’ll start out by saying I am a huge fan of Martin and Hilding’s work and we work very closely with them on the direction of Julia-based tooling for modeling and simulation. ModelingToolkit, being a new system, has some flexibility in the design space it explores, and while we are following a different foundational philosophy, we have many of the same goals.
Composable Abstractions for Model Transformations
Everything in the SciML organization is built around a principle of confederated modular development: let other packages influence the capabilities of your own. This is highlighted in a … READ MORE