The tl;dr: Julia is built around types. Software architectures in Julia are built around good use of the type system. This makes it easy to build generic code which works over a large range of types and gets good performance. The result is highperformance code that has many features. In fact, with generic typing, your code may have more features than you know of! The ... READ MORE
The post TypeDispatch Design: Post ObjectOriented Programming for Julia appeared first on Stochastic Lifestyle.
]]>The tl;dr: Julia is built around types. Software architectures in Julia are built around good use of the type system. This makes it easy to build generic code which works over a large range of types and gets good performance. The result is highperformance code that has many features. In fact, with generic typing, your code may have more features than you know of! The purpose of this tutorial is to introduce the multiple dispatch designs that allow this to happen.
Now let's discuss the main components of this design!
If it quacks like a duck, it might as well be a duck. This is the idea of defining an object by the way that it acts. This idea is central to typebased designs: abstract types are defined by how they act. For example, a `Number` is some type that can do things like +,,*, and /. In this category we have things like Float64 and Int32. An AbstractFloat is some floating point number, and so it should have a dispatch of eps(T) that gives its machine epsilon. An AbstractArray is a type that can be indexed like `A[i]`. An AbstractArray may be mutable, meaning it can be "set": A[i]=v.
These abstract types then have actions which abstract from their underlying implmentation. A.*B does elementwise multiplication, and in many cases it does not matter what kind of array this is done on. The default is Array which is a contiguous array on the CPU, but this action is common amongst AbstractArray types. If a user has a DistributedArray (DArray), then A.*B will work on multiple nodes of a cluster. If the user uses a `GPUArray`, then A.*B will be performed on the GPU. Thus, if you don't restrict the usage of your algorithm to Array, then your algorithm actually "just works" as many different algorithms.
This is all well and good, but this would not be worthwhile if it were not performant. Thankfully, Julia has an answer to this. Every function autospecializes on the types which it is given. Thus if you look at something like:
my_square(x) = x^2
then we see that this function will be efficient for the types that we give it. Looking at the generated code:
@code_llvm my_square(1) define i64 @julia_my_square_72669(i64) #0 { top: %1 = mul i64 %0, %0 ret i64 %1 }
@code_llvm my_square(1.0) define double @julia_my_square_72684(double) #0 { top: %1 = fmul double %0, %0 ret double %1 }
See that the function which is generated by the compiler is different in each case. The first specifically is an integer multiplication x*x of the input x. The other is a floating point multiplication x*x of the input x. But this means that it does not matter what kind of Number we put in here: this function will work as long as * is defined, and it will be efficient by Julia's multiple dispatch design.
Thus we don't need to restrict the types we allow in functions in order to get performance. That means that
my_restricted_square(x::Int) = x^2
is no more efficient than the version above, and actually generates the same exact compiled code:
@code_llvm my_restricted_square(1) define i64 @julia_my_restricted_square_72686(i64) #0 { top: %1 = mul i64 %0, %0 ret i64 %1 }
Thus we can write generic and efficient code by leaving our functions unrestricted. This is the practice of ducktyping functions. We just let them work on any input types. If the type has the correct actions, the function will "just work". If it does not have the correct actions, for our example above say * is undefined, then a MethodError saying the action is not defined will be thrown.
We can be slightly more conservative by restricting to abstract types. For example:
my_number_restricted_square(x::Number) = x^2
will allow any Number. There are things which can square which aren't Numbers for which this will now throw an error (a matrix is a simple example). But, this can let us clearly define the interface for our package/script/code. Using these assertions, we can then dispatch differently for different type classes. For example:
my_number_restricted_square(x::AbstractArray) = (println(x);x.^2)
Now, my_number_restricted_square calculates x^2 on a Number, and for an array it will print the array and calculate x^2 elementwise. Thus we are controlling behavior with broad strokes using classes of types and their associated actions.
This idea of control leads to type hierarchies. In objectoriented programming languages, you sort objects by their implementation. Fields, the pieces of data that an object holds, are what is inherited.
There is an inherent limitation to that kind of thinking when looking to achieve good performance. In many cases, you don't need as much data to do an action. A good example of this is the range type, for example 1:10.
a = 1:10
This type is an abstract array:
typeof(a) <: AbstractArray #true
It has actions like an Array
fieldnames(a)
# Output
2element Array{Symbol,1}:
:start
:stop
It is an immutable type which just holds the start and stop values. This means that its indexing, A[i], is just a function. What's nice about this is that means that no array is ever created. Creating large arrays can be a costly action:
@time collect(1:10000000)
0.038615 seconds (308 allocations: 76.312 MB, 45.16% gc time)
But creating an immutable type of two numbers is essentially free, no matter what those two numbers are:
@time 1:10000000
0.000001 seconds (5 allocations: 192 bytes)
The array takes memory to store its values while this type is , using a constant 192 bytes (if the start and stop are Int64). Yet, in cases where we just want to index values, they act exactly the same.
Another nice example is the UniformScaling operator, which acts like an identity matrix without forming an identity matrix.
println(I[10,10]) # prints 1
println(I[10,2]) # prints 0
This can calculate expressions like Ab*I without ever forming the matrix (eye(n)) which would take memory.
This means that a lot of efficiency can be gained by generalizing our algorithms to allow for generic typing and organization around actions. This means that, while in an objectoriented programming language you group by implementation details, in typeddispatch programming you group by actions. Number is an abstract type for "things which act like numbers, i.e. do things like *", while AbstractArray is for "things which index and sometimes set".
This is the key idea to keep in mind when building type hierarchies: things which subtype are inheriting behavior. You should setup your abstract types to mean the existence or nonexistence of some behavior. For example:
abstract AbstractPerson
abstract AbstractStudent <: AbstractPerson
abstract AbstractTeacher <: AbstractPerson
type Person <: AbstractPerson
name::String
end
type Student <: AbstractStudent
name::String
grade::Int
hobby::String
end
type MusicStudent <: AbstractStudent
grade::Int
end
type Teacher <: AbstractTeacher
name::String
grade::Int
end
This can be interpreted as follows. At the top we have AbstractPerson. Our interface here is "a Person is someone who has a name which can be gotten by get_name".
get_name(x::AbstractPerson) = x.name
Thus codes which are written for an AbstractPerson can "know" (by our informal declaration of the interface) that get_name will "just work" for its subtypes. However, notice that MusicStudent doesn't have a name field. This is because MusicStudents just want to be named whatever the trendiest band is, so we can just replace the usage of the field by the action:
get_name(x::MusicStudent) = "Justin Bieber"
In this way, we can use get_name to get the name, and how it was implemented (whether it's pulling something that had to be stored from memory, or if it's something magically known in advance) does not matter. We can keep refining this: an AbstractStudent has a get_hobby, but a MusicStudent's hobby is always Music, so there's not reason to store that data in the type and instead just have its actions implicitly "know" this. In nontrivial examples (like the range and UniformScaling above), this distinction by action and abstraction away from the actual implementation of the types allows for full optimization of generic codes.
Small Functions and Constant Propagation
The next question to ask is, does storing information in functions and actions affect performance? The answer is yes, and in favor of the function approach! To see this, let's see what happens when we use these functions. To make it simpler, let's use a boolean function. Teachers are old and don't like music, while students do like music. But generally people like music. This means that:
likes_music(x::AbstractTeacher) = false
likes_music(x::AbstractStudent) = true
likes_music(x::AbstractPerson) = true
Now how many records would these people buy at a record store? If they don't like music, they will buy zero records. If they like music, then they will pick up a random number between 1 and 10. If they are a student, they will then double that (impulsive Millennials!).
function number_of_records(x::AbstractPerson)
if !likes_music(x)
return 0
end
num_records = rand(10)
if typeof(x) <: AbstractStudent
return 2num_records
else
return num_records
end
end
Let's check the code that is created:
x = Teacher("Randy",11)
println(number_of_records(x))
@code_llvm number_of_records(x)
on v0.6, we get:
on v0.6, we get:
; Function Attrs: uwtable
define i64 @julia_number_of_records_63848(i8** dereferenceable(16)) #0 !dbg !5 {
top:
ret i64 0
}
Notice that the entire function compiled away, and the resulting compiled code is "return 0"! Then for a music student:
x = MusicStudent(10)
@code_typed number_of_records(x)
# Output
CodeInfo(:(begin
NewvarNode(:(num_records))
goto 4 # line 30:
4: # line 32:
@@0@@(Expr(:foreigncall, :(:jl_alloc_array_1d), Array{Float64,1}, svec(Any, Int64), Array{Float64,1}, 0, 10, 0))
# meta: pop location
# meta: pop location
# meta: pop location
# meta: pop location
@@1@@(Expr(:invoke, MethodInstance for rand!(::MersenneTwister, ::Array{Float64,1}, ::Int64, ::Type{Base.Random.CloseOpen}), :(Base.Random.rand!), :(Base.Random.GLOBAL_RNG), SSAValue(1), :((Base.arraylen)(SSAValue(1))::Int64), :(Base.Random.CloseOpen))) # line 33:
(MusicStudent <: Main.AbstractStudent)::Bool # line 34:
return $(Expr(:invoke, MethodInstance for *(::Int64, ::Array{Float64,1}), :(Main.*), 2, :(num_records))) # line 36:
end))=>Array{Float64,1}
we get a multiplication by 2, while for a regular person,
x = Person("Miguel")
@code_typed number_of_records(x)
# Output
CodeInfo(:(begin
NewvarNode(:(num_records))
goto 4 # line 30:
4: # line 32:
@@2@@(Expr(:foreigncall, :(:jl_alloc_array_1d), Array{Float64,1}, svec(Any, Int64), Array{Float64,1}, 0, 10, 0))
# meta: pop location
# meta: pop location
# meta: pop location
# meta: pop location
@@3@@(Expr(:invoke, MethodInstance for rand!(::MersenneTwister, ::Array{Float64,1}, ::Int64, ::Type{Base.Random.CloseOpen}), :(Base.Random.rand!), :(Base.Random.GLOBAL_RNG), SSAValue(1), :((Base.arraylen)(SSAValue(1))::Int64), :(Base.Random.CloseOpen))) # line 33:
(Person <: Main.AbstractStudent)::Bool
goto 22 # line 34:
22: # line 36:
return num_records
end))=>Array{Float64,1}
we do not get a multiplication by 2. This is all in the compiledcode, so this means that in one case the *2 simply doesn't exist at runtime, not even a check for whether to do it.
The key thing to see from the typed code is that the "branches" (the if statements) all compiled away. Since types are known at compile time (remember, functions specialize on types), the dispatch of likes_music is known at compiletime. But this means, since the result is directly inferred from the dispatch, the boolean value true/false is known at compile time. This means that the compiler can directly infer the answer to all of these checks, and will use this information to skip them at runtime.
This is the distinction between compiletime information and runtime information. At compiletime, what is known is:
 The types of the inputs
 Any types which can be inferred from the input types (via typestability)
 The function dispatches that will be internally called (from types which have been inferred)
Note that what cannot be inferred by the compiler is the information in fields. Information in fields is strictly runtime information. This is easy to see since there is no way for the compiler to know that person's name was "Miguel": that's ephemeral and part of the type instance we just created.
Thus by putting our information into our functions and dispatches, we are actually giving the compiler more information to perform more optimizations. Therefore using this "actionbased design", we are actually giving the compiler leeway to perform many extra optimizations on our code as long as we define our interfaces by the actions that are used. Of course, at the "very bottom" our algorithms have to use the fields of the types, but the full interface can then be built up using a simple set of functions which in many cases with replace runtime data with constants.
Traits and THTT
What we just saw is a "trait". Traits are compiletime designations about types which are distinct from their abstract hierarchy. likes_music is a trait which designates which people like music, and it could in cases not be defined using the abstract types. For example, we can, using dispatch, create a WeirdStudent which does not like music, and that will still be compiletime information which is fully optimized. This means that these small functions which have constant return values allow for compiletime inheritance of behavior, and these traits don't have to be tied to abstract types (all of our examples were on AbstractPerson, but we could've said a GPUArray likes music if we felt like it!). Traits are multipleinheritance for type systems.
Traits can be more refined than just true/false. This can be done by having the return be a type itself. For example, we can create music genre types:
abstract MusicGenres
abstract RockGenre <: MusicGenres
immutable ClassicRock <: RockGenre end
immutable AltRock <: RockGenre end
immutable Classical <: MusicGenres end
These "simple types" are known as singleton types. This means that we can have traits like:
favorite_genre(x::AbstractPerson) = ClassicRock()
favorite_genre(x::MusicStudent) = Classical()
favorite_genre(x::AbstractTeacher) = AltRock()
This gives us all of the tools we need to compile the most efficient code, and structure our code around types/actions/dispatch to get high performance out. The last thing we need is syntactic sugar. Since traits are compiletime information, the compiler could in theory dispatch on them. While this is currently not part of Julia, it's scheduled to be part of a future version of Julia (2.0?). The design for this (since Julia is written in Julia!) is known as the Tim Holy Trait Trick (THTT), named after its inventor. It's described in detail on this page. But in the end, macros can make this easier. A package which implements traitdispatch is SimpleTraits.jl, which allows you to dispatch on a trait IsNice like:
@traitfn ft(x::::IsNice) = "Very nice!"
@traitfn ft(x::::(!IsNice)) = "Not so nice!"
Composition vs Inheritance
The last remark that is needed is a discussion of composition vs inheritance. While the previous discussions have all explained why "information not in fields" makes structural relations compiletime information and increases the efficiency. However, there are cases where we want to share runtime structure. Thus the great debate of composition vs inheritance, comes up.
Composition vs inheritance isn't a Julia issue, it's a long debate in objectoriented programming. The idea is that, inheritance is inherently (puninteded) inflexible. It forces an "is a" relation: A inherits from B means A is a B, and adds a few things. It copies behavior from something defined elsewere. This is a recipe for havoc. Here's a few links which discuss this in more detail:
https://en.wikipedia.org/wiki/Composition_over_inheritance
https://www.thoughtworks.com/insights/blog/compositionvsinheritancehowchoose
So if possible, give composition a try. Say you have MyType, and it has some function f defined on it. This means that you can extend MyType by making it a field in another type:
type MyType2
mt::MyType
... # Other stuff
end
f(mt2::MyType2) = f(mt2.mt)
The pro here is that it's explicit: you've made the choice for each extension. The con is that this can require some extra code, though this can be automated by metaprogramming.
What if you really really really want inheritance of fields? There are solutions via metaprogramming. One simple solution is the @def macro.
macro def(name, definition)
return quote
macro $name()
esc($(Expr(:quote, definition)))
end
end
end
This macro is very simple. What it does is compiletime copy/paste. For example:
@def give_it_a_name begin
a = 2
println(a)
end
defines a macro @give_it_a_name that will paste in those two lines of code wherever it is used. As another example, the reused fields of Optim.jl's solvers could be put into an @def:
@def add_generic_fields begin
method_string::String
n::Int64
x::Array{T}
f_x::T
f_calls::Int64
g_calls::Int64
h_calls::Int64
end
and those fields can be copied around with
type LBFGSState{T}
@add_generic_fields
x_previous::Array{T}
g::Array{T}
g_previous::Array{T}
rho::Array{T}
# ... more fields ...
end
Because @def works at compiletime, there is no cost associated with this. Similar metaprogramming can be used to build an "inheritance feature" for Julia. One package which does this is ConcreteAbstractions.jl which allows you to add fields to abstract types and make the child types inherit the fields:
# The abstract type
@base type AbstractFoo{T}
a
b::Int
c::T
d::Vector{T}
end
# Inheritance
@extend type Foo <: AbstractFoo
e::T
end
where the @extend macro generates the typedefinition:
type Foo{T} <: AbstractFoo
a
b::Int
c::T
d::Vector{T}
e::T
end
But it's just a package? Well, that's the beauty of Julia. Most of Julia is written in Julia, and Julia code is first class and performant (here, this is all at compiletime, so again runtime is not affected at all). Honestly, if something ever gets added to Julia's Base library for this, it will likely look very similar, and the only real difference to the user will be that the compiler will directly recognize the keywords, meaning you would use base and extend instead of @base and @extend. So if you have something that really really really needs inheritance, go for it: there's no downsides to using a package + macro for this. But you should really try other means to reduce the runtime information and build a more performant and more Julian architecture first.
Conclusion
Programming for type systems has a different architecture than objectoriented systems. Instead of being oriented around the objects and their fields, typedispatch systems are oriented around the actions of types. Using traits, multiple inheritance behavior can be given. Using this structure, the compiler can have maximal information, and use this to optimize the code. But also, this directly generalizes the vast majority of the code to not be "implementationdependent", allowing for ducktyped code to be fully performant, with all of the details handled by dispatch/traits/abstract types. The end result is flexible, generic, and high performance code.
The post TypeDispatch Design: Post ObjectOriented Programming for Julia appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/typedispatchdesignpostobjectorientedprogrammingjulia/feed/
0
668

Video Blog: Developing and Editing Julia Packages
http://www.stochasticlifestyle.com/videoblogdevelopingeditingjuliapackages/
http://www.stochasticlifestyle.com/videoblogdevelopingeditingjuliapackages/#respond
Tue, 16 May 2017 20:37:57 +0000
http://www.stochasticlifestyle.com/?p=660
Google Summer of Code is starting up, so I thought it would be a good time to share my workflow for developing my own Julia packages, as well as my workflow for contributing to other Julia packages. This does not assume familiarity with commandline Git, and instead shows you how to use a GUI (GitKraken) to make branches and PRs, as well as reviewing and merging code. You can think of it as an update to my old blog post on package development in Julia. However, this is not only updated but also improved since I am now able to walk through the "noncode" parts of package developing (such as setting up AppVeyor and code coverage).
Enjoy! (I quite like this video blog format: it was a lot less work)
The post Video Blog: Developing and Editing Julia Packages appeared first on Stochastic Lifestyle.
]]>
Google Summer of Code is starting up, so I thought it would be a good time to share my workflow for developing my own Julia packages, as well as my workflow for contributing to other Julia packages. This does not assume familiarity with commandline Git, and instead shows you how to use a GUI (GitKraken) to make branches and PRs, as well as reviewing and merging code. You can think of it as an update to my old blog post on package development in Julia. However, this is not only updated but also improved since I am now able to walk through the "noncode" parts of package developing (such as setting up AppVeyor and code coverage).
Enjoy! (I quite like this video blog format: it was a lot less work)
The post Video Blog: Developing and Editing Julia Packages appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/videoblogdevelopingeditingjuliapackages/feed/
0
660

DifferentialEquations.jl 2.0: State of the Ecosystem
http://www.stochasticlifestyle.com/differentialequationsjl20stateecosystem/
http://www.stochasticlifestyle.com/differentialequationsjl20stateecosystem/#comments
Mon, 08 May 2017 12:13:30 +0000
http://www.stochasticlifestyle.com/?p=613
In this blog post I want to summarize what we have accomplished with DifferentialEquations' 2.0 release and detail where we are going next. I want to put the design changes and development work into a larger context so that way everyone can better understand what has been achieved, and better understand how we are planning to tackle our next challenges.
If you find this project interesting and would like to support our work, please star our Github repository. Thanks!
Now let's get started.
DifferentialEquations.jl 1.0: The Core
Before we start talking about 2.0, let's understand first what 1.0 was all about. DifferentialEquations.jl 1.0 was about answering a single question: how can we put the wide array of differential equations into one simple and efficient interface. The result of this was the common interface explained in the first blog post. Essentially, we created ... READ MORE
The post DifferentialEquations.jl 2.0: State of the Ecosystem appeared first on Stochastic Lifestyle.
]]>
In this blog post I want to summarize what we have accomplished with DifferentialEquations' 2.0 release and detail where we are going next. I want to put the design changes and development work into a larger context so that way everyone can better understand what has been achieved, and better understand how we are planning to tackle our next challenges.
If you find this project interesting and would like to support our work, please star our Github repository. Thanks!
Now let's get started.
DifferentialEquations.jl 1.0: The Core
Before we start talking about 2.0, let's understand first what 1.0 was all about. DifferentialEquations.jl 1.0 was about answering a single question: how can we put the wide array of differential equations into one simple and efficient interface. The result of this was the common interface explained in the first blog post. Essentially, we created one interface that could:
 Specify a differential equation problem
 Solve a differential equation problem
 Analyze a differential equation problem
The problem types, solve command, and solution interface were all introduced here as part of the unification of differential equations. Here, most of the work was on developing the core. DifferentialEquations.jl 1.0 was about having the core methods for solving ordinary differential equations, stochastic differential equations, and differential algebraic equations. There were some nice benchmarks to show that our core native solvers were on the right track, even besting wellknown Fortran methods in terms of efficiency, but the key of 1.0 was the establishment of this high level unified interface and the core libraries for solving the problems.
DifferentialEquations.jl 2.0: Extended Capabilities
DifferentialEquations.jl 2.0 asked a unique question for differential equations libraries. Namely, "how flexible can a differential equations solver be?" This was motivated by an offputting remark where someone noted that standard differential equations solvers were limited in their usefulness because many of the higher level analyses that people need to do cannot be done with a standard differential equations solver.
So okay, then we won't be a standard differential equations solver. But what do we need to do to make all of this possible? I gathered a list of things which were considered impossible to do with "blackbox" differential equations solvers. People want to model continuous equations for protein concentrations inside of each cell, but allow the number of cells (and thus the number of differential equations) to change stochastically over time. People want to model multiscale phenomena, and have discontinuities. Some "differential equations" may only be discontinuous changes of discrete values (like in Gillespie models). People want to solve equations with colored noise, and reuse the same noise process in other calculations. People want to solve the same ODE efficiently hundreds of times, and estimate parameters. People want to quantify the uncertainty and the sensitivity of their model. People want their solutions conserve properties like energy.
People want to make simulations of reality moreso than solve equations.
And this became the goal for DifferentialEquations.jl 2.0. But the sights were actually set a little higher. The underlying question was:
How do you design a differential equations suite such that it can have this "simulation engine" functionality, but also such that adding new methods automatically makes the method compatible with all of these features?
That is DifferentialEquations.jl 2.0. the previous DifferentialEquations.jl ecosystem blog post details the strategies we were going to employ to achieve this goal, but let me take a little bit of time to explain the solution that eventually resulted.
The Integrator Interface
The core of the solution is the integrator interface. Instead of just having an interface on the highlevel solve command, the integrator interface is the interface on the core type. Everything inside of the OrdinaryDiffEq.jl, StochasticDiffEq.jl, DelayDiffEq.jl packages (will be referred to as the *DiffEq solvers) is actually just a function on the integrator type. This means that anything that the solver can do, you can do by simply having access to the integrator type. Then, everything can be unified by documenting this interface.
This is a powerful idea. It makes development easy, since the devdocs just explain what is done internally to the integrator. Adding new differential equations algorithms is now simply adding a new perform_step dispatch. But this isn't just useful for development, this is useful for users too. Using the integrator, you can step one at a time if you wanted, and do anything you want between steps. Resizing the differential equation is now just a function on the integrator type since this type holds all of the cache variables. Adding discontinuities is just changing integrator.u.
But the key that makes this all work is Julia. In my dark past, I wrote some algorithms which used R's S3 objects, and I used objects in numerical Python codes. Needless to say, these got in the way of performance. However, the process of implementing the integrator type was a full refactor from straight loops to the type format. The result was no performance loss (actually, there was a very slight performance gain!). The abstraction that I wanted to use did not have a performance tradeoff because Julia's type system optimized its usage. I find that fact incredible.
But back to the main story, the event handling framework was rebuilt in order to take full advantage of the integrator interface, allowing the user to directly affect the integrator. This means that doubling the size of your differential equation the moment some value hits 1 is now a possibility. It also means you can cause your integration to terminate when "all of the bunnies" die. But this became useful enough that you might not want to just use it for traditional event handling (aka cause some effect when some function hits zero, which we call the ContinuousCallback), but you may just want to apply some affect after steps. The DiscreteCallback allows one to check a boolean function for true/false, and if true apply some function to the integrator. For example, we can use this to always apply a projection to a manifold at the end of each step, effectively preserving the order of the integration while also conserving model properties like energy or angular momentum.
The integrator interface and thus its usage in callbacks then became a way that users could add arbitrary functionality. It's useful enough that a DiscreteProblem (an ODE problem with no ODE!) is now a thing. All that is done is the discrete problem walks through the ODE solver without solving a differential equation, just hitting callbacks.
But entirely new sets of equations could be added through callbacks. For example, discrete stochastic equations (or Gillespie simulations) are models where rate equations determine the time to the next discontinuity or "jump". The JumpProblem types simply add callbacks to a differential (or discrete) equation that perform these jumps at specific rates. This effectively turns the "blank ODE solver" into an equation which can solve these models of discrete proteins stochastically changing their levels over time. In addition, since it's built directly onto the differential equations solvers, mixing these types of models is an instant side effect. These models which mix jumps and differential equations, such as jump diffusions, were an immediate consequence of this design.
The design of the integrator interface meant that dynamicness of the differential equation (changing the size, the solver options, or any other property in the middle of solving) was successfully implemented, and handling of equations with discontinuities directly followed. This turned a lot of "not differential equations" into "models and simulations which can be handled by the same DifferentialEquations.jl interface".
Generic algorithms over abstract types
However, the next big problem was being able to represent a wider array of models. "Models and simulations which do weird nondifferential equation things over time" are handled by the integrator interface, but "weird things which aren't just a system of equations which do weird nondifferential equation things over time" were still out of reach.
The solution here is abstract typing. The *DiffEq solvers accept two basic formats. Let's stick to ODEs for the explanation. For ODEs, there is the outofplace format
du = f(t,u)
where the derivative/change is returned by the function, and there is the inplace format
f(t,u,du)
where the function modifies the object du which stores the derivative/change. Both of these formats were generalized to the extreme. In the end, the requirements for a type to work in the outofplace format can be described as the ability to do basic arithmetic (+,,/,*), and you add the requirement of having a linear index (or simply having a broadcast! function defined) in order to satisfy the inplace format. If the method is using adaptivity, the user can pass an appropriate norm function to be used for calculating the norm of the error estimate.
This means that wild things work in the ODE solvers. I have already demonstrated arbitrary precision numbers, and unitchecked arithmetic.
But now there's even crazier. Now different parts of your equation can have different units using the ArrayPartition. You can store and update discrete values along with your differential equation using the DEDataArray type. Just the other day I showed this can be used to solve problems where the variable is actually a symbolic mathematical expression. We are in the late stages of getting a type which represents a spectral discretization of a function compatible with the *DiffEq solvers.
But what about those "purely scientific nondifferential equations" applications? A multiscale model of an embryo which has tissues, each with different populations of cells, and modeling the proteins in each cell? That's just a standard application of the AbstractMultiScaleArray.
Thus using the abstract typing, even simulations which don't look like systems of equations can now be directly handled by DifferentialEquations.jl. But not only that, since this is done simply via Julia's generic programming, this compatibility is true for any of the new methods which are added (one caveat: if they use an external library like ForwardDiff.jl, their compatibility is limited by the compatibility of that external library).
Refinement of the problem types
The last two big ideas made it possible for a very large set of problems to be written down as a "differential equation on an array" in a much expanded sense of the term. However, there was another design problem to solve: not every algorithm could be implemented with "the information" we had! What I mean by "information", I mean the information we could get from the user. The ODEProblem type specified an ODE as
but some algorithms do special things. For example, for the ODE
the LawsonEuler algorithm for solving the differential equation is
This method exploits the fact that it knows that the first part of the equation is for some matrix, and uses it directly to improve the stability of the algorithm. However, if all we know is , we could never implement this algorithm. This would violate our goal of "full flexibility at full performance" if this algorithm was the most efficient for the problem!
The solution is to have a more refined set of problem types. I discussed this a bit at the end of the previous blog post that we could define things like splitting problems. The solution is quite general, where
can be defined using the SplitODEProblem (M being a mass matrix). Then specific methods can request specific forms, like here the linearnonlinear ODE. Together, the ODE solver can implement this algorithm for the ODE, and that implementation, being part of a *DiffEq solver, will have interpolations, the integrator interface, event handling, abstract type compatibility, etc. all for free. Check out the other "refined problem types": these are capable of covering wild things like staggered grid PDE methods and symplectic integrators.
In addition to specifying the same equations in new ways, we created avenues for common analyses of differential equations which are not related to simulating them over time. For example, one common problem is to try to find steady states, or points where the differential equation satisfies . This can now easily be done by defining a SteadyStateProblem from an ODE, and then using the steady state solver. This new library will also lead to the implementation of accelerated methods for finding steady states, and the development of new accelerated methods. The steady state behavior can now also be analyzed using the bifurcation analysis tools provided by the wrapper to PyDSTool.
Lastly, the problem types themselves have become much more expressive. In addition to solving the standard ODE, one can specify mass matrices in any appropriate DE type, to instead solve the equation
where is some linear operator (similarly in DDEs and SDEs). While the vast majority of solvers are not able to use right now, this infrastructure is there for algorithms to support it. In addition, one can now specify the noise process used in random and stochastic equations, allowing the user to solve problems with colored noise. Using the optional fields, a user can define nondiagonal noise problems, and specify sparse noise problems using sparse matrices.
As of now, only some very basic methods using all of this infrastructure have been made for the most extreme examples for testing purposes, but these show that the infrastructure works and this is ready for implementing new methods.
Common solve extensions
Okay, so once we can specify efficient methods for weird models which evolve over time in weird ways, we can simulate and get what solutions look like. Great! We have a tool that can be used to get solutions! But... that's only the beginning of most analyses!
Most of the time, we are simulating solutions to learn more about the model. If we are modeling chemical reactions, what is the reaction rate that makes the model match the data? How sensitive is our climate model to our choice of the albedo coefficient?
To back out information about the model, we rely on analysis algorithms like parameter estimation and sensitivity analysis. However, the common solve interface acts as the perfect level for implementing these algorithms because they can be done problem and algorithm agnostic. I discuss this in more detail in a previous blog post, but the general idea is that most of these algorithms can be written with a term which is the solution of a differential equation. Thus we can write the analysis algorithms at a very high level and allow the user to pass in the arguments for a solve command use that to generate the . The result is an implementation of the analysis algorithm which works with any of the problems and methods which use the common interface. Again, chaining all of the work together to get one more complete product. You can see this in full force by looking at the parameter estimation docs.
Modeling Tools
In many cases one is solving differential equations not for their own sake, but to solve scientific questions. To this end, we created a framework for modeling packages which make this easier. The financial models make it easy to specify common financial equations, and the biological models make it easy to specify chemical reaction networks. This functionality all works on the common solver / integrator interface, meaning that models specified in these forms can be used with the full stack and analysis tools. Also, I would like to highlight BioEnergeticFoodWebs.jl as a great modeling package for bioenergetic food web models.
Over time, we hope to continue to grow these modeling tools. The financial tools I hope to link with Julia Computing's JuliaFin tools (Miletus.jl) in order to make it easy to efficiently solve the SDE and PDE models which result from their financial DSL. In addition, DiffEqPhysics.jl is planned to make it easy to specify the equations of motion just by giving a Hamiltonian or Lagrangian, or by giving the the particles + masses and automatically developing a differential equation. I hope that we can also tackle domains like mechanical systems and pharmacokinetics/pharmacodynamics to continually expand what is easily able to be solved using this infrastructure.
DifferentialEquations 2.0 Conclusion
In the end, DifferentialEquations 2.0 was about finding the right infrastructure such that pretty much anything CAN be specified and solved efficiently. While there were some bumps along the road (that caused breaking API changes), I believe we came up with a very good solution. The result is a foundation which feeds back onto itself, allowing projects like parameter estimation of multiscale models which change size due to events to be standard uses of the ODE solver.
And one of the key things to note is that this follows by design. None of the algorithms were specifically written to make this work. The design of the *DiffEq packages gives interpolation, event handling, compatibility with analysis tools, etc. for free for any algorithm that is implemented in it. One contributor, @ranocha, came to chat in the chatroom and on a few hours later had implemented 3 strong stability preserving RungeKutta methods (methods which are efficient for hyperbolic PDEs) in the *DiffEq solvers. All of this extra compatibility followed for free, making it a simple exercise. And that leads me to DifferentialEquations 3.0.
DifferentialEquations 3.0: Stiff solvers, parallel solvers, PDEs, and improved analysis tools
1.0 was about building the core. 2.0 was about making sure that the core packages were built in a way that could be compatible with a wide array of problems, algorithms, and analysis tools. However, in many cases, only the simplest of each type of algorithm was implemented since this was more about building out the capabilities than it was to have completeness in each aspect. But now that we have expanded our capabilities, we need to fill in the details. These details are efficient algorithms in the common problem domains.
Stiff solvers
Let's start by talking about stiff solvers. As of right now, we have the all of the standard solvers (CVODE, LSODA, radau) wrapped in the packages Sundials.jl, LSODA.jl, and ODEInterface.jl respectively. These can all be used in the DifferentialEquations.jl common interface, meaning that it's mostly abstracted away from the user that these aren't actually Julia codes. However, these lower level implementations will never be able to reach the full flexibility of the native Julia solvers simply because they are restricted in the types they use and don't fully expose their internals. This is fine, since our benchmarks against the standard RungeKutta implementations (dopri5, dop853) showed that the native Julia solvers, being more modern implementations, can actually have performance gains over these older methods. But, we need to get our own implementations of these high order stiff solvers.
Currently there exists the Rosenbrock23 method. This method is similar to the MATLAB ode23s method (it is the Order 2/3 ShampineRosenbrock method). This method is A and L stable, meaning it's great for stiff equations. This was thus used for testing event handling, parameter estimation, etc.'s capabilities and restrictions with the coming set of stiff solvers. However, where it lacks is order. As an order 2 method, this method is only efficient at higher error tolerances, and thus for "standard tolerances" it tends not to be competitive with the other methods mentioned before. That is why one of our main goals in DiffEq 3.0 will be the creation of higher order methods for stiff equations.
The main obstacle here will be the creation of a library for making the differentiation easier. There are lots of details involved here. Since a function defined using the macros of ParameterizedFunctions can symbolically differentiate the users function, in some cases a precomputed function for the inverted or factorized Jacobian can be used to make a stiff method explicit. In other cases, we need autodifferentiation, and in some we need to use numerical differentiation. This is all governed by a system of traits setup behind the scenes, and thus proper logic for determining and using Jacobians can immensely speed up our calculations.
The Rosenbrock23 method did some of this adhocly within its own method, but it was determined that the method would be greatly simplified if there was some library that could handle this. In fact, if there was a library to handle this, then the Rosenbrock23 code for defining steps would be as simple as defining steps for explicit RK methods. The same would be true for implicit RK methods like radau. Thus we will be doing that: building a library which handles all of the differentiation logic. The development of this library, DiffEqDiffTools.jl, is @miguelraz 's Google Summer of Code project. Thus with the completion of this project (hopefully summer?), efficient and fully compatible high order Rosenbrock methods and implicit RK methods will easily follow. Also included will be additive RungeKutta methods (IMEX RK methods) for SplitODEProblems. Since these methods double as native Julia DAE solvers and this code will make the development of stiff solvers for SDEs, this will be a major win to the ecosystem on many fronts.
Stiffness Detection and Switching
In many cases, the user may not know if a problem is stiff. In many cases, especially in stochastic equations, the problem may be switching between being stiff and nonstiff. In these cases, we want to change the method of integration as we go along. The general setup for implementing switching methods has already been implemented by the CompositeAlgorithm. However, current usage of the CompositeAlgorithm requires that the user define the switching behavior. This makes it quite difficult to use.
Instead, we will be building methods which make use of this infrastructure. Stiffness detection estimates can be added to the existing methods (in a very efficient manner), and could be toggled on/off. Then standard switching strategies can be introduced such that the user can just give two algorithms, a stiff and a nonstiff solvers, and basic switching can then occur. What is deemed as the most optimal strategies can then be implemented as standard algorithm choices. Then at the very top, these methods can be added as defaults for solve(prob), making the fully automated solver efficiently handle difficult problems. This will be quite a unique feature and is borderline a new research project. I hope to see some really cool results.
Parallelintime ODE/BVP solvers
While traditional methods (RungeKutta, multistep) all step one time point at a time, in many cases we want to use parallelism to speed up our problem. It's not hard to buy an expensive GPU, and a lot of us already have one for optimization, so why not use it?
Well, parallelism for solving differential equations is very difficult. Let's take a look at some quick examples. In the Euler method, the discretization calculates the next time step from the previous time step using the equation
In code, this is the update step
u .= uprev .+ dt.*f(t,uprev)
I threw in the .'s to show this is broadcasted over some arrays, i.e. for systems of equations is a vector. And that's it, that's what the inner loop is. The most you can parallelize are the loops internal to the broadcasts. This means that for very large problems, you can parallelize this method efficiently (this form is called parallelism within the method). Also, if your input vector was a GPUArray, this will broadcast using CUDA or OpenCL. However, if your problem is not a sufficiently large vector, this parallelism will not be very efficient.
Similarly for implicit equations, we need to repeatedly solve where is the Jacobian matrix. This linear solve will only parallelize well if the Jacobian matrix is sufficiently large. But many small differential equations problems can still be very difficult. For example, this about solving a very stiff ODE with a few hundred variables. Instead, the issue is that we are stepping serially over time, and we need to use completely different algorithms which parallelize over time.
One of these approaches is a collocation method. Collocation methods build a very large nonlinear equation which describes a numerical method over all time points at once, and simultaneously solves for all of the time points using a nonlinear solver. Internally, a nonlinear solver is just a linear solver, , with a very large . Thus, if the user passes in a custom linear solver method, say one using PETSc.jl or CUSOLVER, this is parallelized efficiently over many nodes of an HPC or over a GPU. In fact, these methods are the standard methods for Boundary Value Problems (BVPs). The development of these methods is the topic of @YingboMa's Google Summer of Code project. While written for BVPs, these same methods can then solve IVPs with a small modification (including stochastic differential equations).
By specifying an appropriate preconditioner with the linear solver, these can be some of the most efficient parallel methods. When no good preconditioner is found, these methods can be less efficient. One may wonder then if there's exists a different approach, one which may sacrifice some "theoretical top performance" in order to be better in the "low user input" case (purely automatic). There is! Another approach to solving the parallelism over time issue is to use a neural network. This is the topic of @akaysh's Google Summer of Code project. Essentially, you can define a cost function which is the difference between the numerical derivative and at each time point. This then gives an optimization problem: find the at each time point such that the difference between the numerical and the desired derivatives is minimized. Then you solve that cost function minimization using a neural network. The neat thing here is that neural nets are very efficient to parallelize over GPUs, meaning that even for somewhat small problems we can get out very good parallelism. These neural nets can use very advanced methods from packages like Knet.jl to optimize efficiently and parallel with very little required user input (no preconditioner to set). There really isn't another standard differential equations solver package which has a method like this, so it's hard to guess how efficient it will be in advance. But given the properties of this setup, I suspect this should be a very good "automatic" method for mediumsized (100's of variables) differential equations.
The really great thing about these parallelintime methods is that they are inherently implicit, meaning that they can be used even on extremely stiff equations. There are also simple extensions to make these solver SDEs and DDEs. So add this to the bank of efficient methods for stiff diffeqs!
Improved methods for stochastic differential equations
As part of 3.0, the hope is to release brand new methods for stochastic differential equations. These methods will be high order and highly stable, some explicit and some implicit, and will have adaptive timestepping. This is all of the details that I am giving until these methods are published, but I do want to tease that many types of SDEs will become much more efficient to solve.
Improved methods for jump equations
For jump equations, in order to show that everything is complete and can work, we have only implemented the Gillespie method. However, we hope to add many different forms of tauleaping and Poisson(/Binomial) RungeKutta methods for these discrete stochastic equations. Our roadmap is here and it seems there may be a great deal of participation to complete this task. Additionally, we plan on having a specialized DSL for defining chemical reaction networks and automatically turn them into jump problems or ODE/SDE systems.
Geometric and symplectic integrators
In DifferentialEquations.jl 2.0, the ability to partition ODEs for dynamical problems (or directly specify a second order ODE problem) was added. However, only a symplectic Euler method has been added to solve this equations so far. This was used to make sure the *DiffEq solvers were compatible with this infrastructure, and showed that event handling, resizing, parameter estimation, etc. all works together on these new problem types. But, obviously we need more algorithms. Velocity varlet and higher order Nystrom methods are asking to be implemented. This isn't difficult for the reasons described above, and will be a very nice boost to DifferentialEquations.jl 3.0.
(Stochastic/Delay) Partial differential equations
Oh boy, here's a big one. Everyone since the dawn of time has wanted me to focus on building a method that makes solving the PDE that they're interested in dead simple to do. We have a plan for how to get there. The design is key: instead of onebyone implementing numerical methods for each PDE, we needed a way to pool the efforts together and make implementations on one PDE matter for other PDEs.
Let's take a look at how we can do this for efficient methods for reactiondiffusion equations. In this case, we want to solve the equation
The first step is always to discretize this over space. Each of the different spatial discretization methods (finite difference, finite volume, finite element, spectral), end up with some equation
where now is a vector of points in space (or discretization of some basis). At this point, a specialized numerical method for stepping this equation efficiently in the time can be written. For example, if diffusion is fast and is stiff, one really efficient method is the implicit integrating factor method. This would solve the equation by updating time points like:
where we have to solve this implicit equation each time step. The nice thing is that the implicit equation decouples in space, and so we actually only need to solve a bunch of very small implicit equations.
How can we do this in a way that is not "specific to the heat equation"? There were two steps here, the first is discretizing in space, the second is writing an efficient method specific to the PDE. The second part we already have an answer for: this numerical method can be written as one of the methods for linear/nonlinear SplitODEProblems that we defined before. Thus if we just write a SplitODEProblem algorithm that does this form of updating, it can be applied to any ODE (and any PDE discretization) which splits out a linear part. Again, because it's now using the ODE solver, all of the extra capabilities (event handling, integrator interface, parameter estimation tools, interpolations, etc.) all come for free as well. The development of ODE/SDE/DDE solvers for handling this split, like implicit integrating factor methods and exponential RungeKutta methods, is part of DifferentialEquations.jl 3.0's push for efficient (S/D)PDE solvers.
So with that together, we just need to solve the discretization problem. First let's talk about finite difference. For the Heat Equation with a fixed gridsize , many people know what the secondorder discretization matrix is in advance. However, what if you have variable grid sizes, and want different order discretizations of different derivatives (say a third derivative)? In this case the Fornberg algorithm can be used to construct this . And instead of making this an actual matrix, because this is sparse we can make this very efficient by creating a "matrixfree type" where acts like the appropriate matrix multiplication, but in reality no matrix is ever created. This can save a lot of memory and make the multiplication a lot more efficient by ignoring the zeros. In addition, because of the reduced memory requirement, we easily distribute this operator to the GPU or across the cluster, and make the function utilize this parallelism.
The development of these efficient linear operators is the goal of @shivin9's Google Summer of Code project. The goal is to get a function where the user can simply specify the order of the derivative and the order of the discretization, and it will spit out this highly efficient to be used in the discretization, turning any PDE into a system of ODEs. In addition, other operators which show up in finite difference discretizations, such as the upwind scheme, can be encapsulated in such an . Thus this project would make turning these types of PDEs into efficient ODE discretizations much easier!
The other very popular form of spatial discretization is the finite element method. For this form, you chose some basis function over space and discretize the basis function. The definition of this basis function's discretization then derives what the discretization of the derivative operators should be. However, there is a vast array of different choices for basis and the discretization. If we wanted to create a toolbox which would implement all of what's possible, we wouldn't get anything else done. Thus we will instead, at least for now, piggyback off of the efforts of FEniCS. FEniCS is a toolbox for the finite element element method. Using PyCall, we can build an interface to FEniCS that makes it easy to receive the appropriate linear operator (usually sparse matrix) that arises from the desired discretization. This, the development of a FEniCS.jl, is the topic of @ysimillides's Google Summer of Code. The goal is to make this integration seamless, so that way getting ODEs out for finite element discretizations is a painless process, once again reducing the problem to solving ODEs.
The last form of spatial discretization is spectral discretizations. These can be handled very well using the Julia library ApproxFun.jl. All that is required is to make it possible to step in time the equations which can be defined using the ApproxFun types. This is the goal of DiffEqApproxFun.jl. We already have large portions of this working, and for fixed basis lengths the ApproxFunProblems can actually be solved using any ODE solver (not just native Julia solvers, but also methods from Sundials and ODEInterface work!). This will get touched up soon and will be another type of PDE discretization which will be efficient and readily available.
Improved Analysis Tools
What was described above is how we are planning to solve very common difficult problems with high efficiency and simplify the problems for the user, all without losing functionality. However, the tools at the very top of the stack, the analysis tools, can also become much more efficient as well. This is the other focus of DifferentialEquations.jl 3.0.
Local sensitivity analysis is nice because it not only tells you how sensitive your model is to the choice of parameters, but it gives this information at every time point. However, in many cases this is overkill. Also, this makes the problem much more computationally difficult. If we wanted to classify parameter space, like to answer the question "where is the model least sensitive to parameters?", we would have to solve this equation many times. When this is the question we wish to answer, global sensitivity analysis methods are much more efficient. We plan on adding methods like the Morris method in order for sensitives to be globally classified.
In addition, we really need better parameter estimation functionality. What we have is very good: you can build an objective function for your parameter estimation problem to then use Optim.jl, BlackBoxOptim.jl or any MathProgBase/JuMP solver (example: NLopt.jl) to optimize the parameters. This is great, but it's very basic. In many cases, more advanced methods are necessary in order to get convergence. Using likelihood functions instead of directly solving the nonlinear regression can often times be more efficient. Also, in many cases statistical methods (the twostage method) can be used to approximately optimize parameters without solving the differential equations repeatedly, a huge win for performance. Additionally, Bayesian methods will not only give optimal parameters, but distributions for the parameters which the user can use to quantify how certain they are about estimation results. The development of these methods is the focus of @Ayushiitkgp's Google Summer of Code project.
DifferentialEquations.jl 3.0 Conclusion
2.0 was about building infrastructure. 3.0 is about filling out that infrastructure and giving you the most efficient methods in each of the common problem domains.
DifferentialEquations.jl 4.0 and beyond
I think 2.0 puts us in a really great position. We have a lot, and the infrastructure allows us to be able to keep expanding and adding more and more algorithms to handle different problem types more efficiently. But there are some things which are not slated in the 3.0 docket. One thing that keeps getting pushed back is the automatic promotion of problem types. For example, if you specified a SplitODEProblem and you want to use an algorithm which wants an ODEProblem (say, a standard RungeKutta algorithm like Tsit5), it's conceivable that this conversion can be handled automatically. Also, it's conceivable that since you can directly convert an ODEProblem into a SteadyStateProblem, that the steady state solvers should work directly on an ODEProblem as well. However, with development time always being a constraint, I am planning on spending more time developing new efficient methods for these new domain rather than the automatic conversions. However, if someone else is interested in tackling this problem, this could probably be addressed much sooner!
Additionally, there is one large set of algorithms which have not been addressed in the 3.0 plan: multistep methods. I have been holding off on these for a few reasons. For one, we have wrappers to Sundials, DASKR, and LSODA which supply wellknown and highly efficient multistep methods. However, these wrappers, having the internals not implemented in Julia, are limited in their functionality. They will never be able to support arbitrary Julia types and will be constrained to equations on contiguous arrays of Float64s. Additionally, the interaction with Julia's GC makes it extremely difficult to implement the integrator interface and thus event handling (custom allocators would be needed). Also, given what we have seen with other wrappers, we know we can actually improve the efficiency of these methods.
But lastly, and something I think is important, these methods are actually only efficient in a small but important problem domain. When the ODE is not "expensive enough", implicit RungeKutta and Rosenbrock methods are more efficient. When it's a discretization of a PDE and there exists a linear operator, exponential RungeKutta and implicit integrating factor methods are more efficient. Also, if there are lots of events or other dynamic behavior, multistep methods have to "restart". This is an expensive process, and so in most cases using a onestep method (any of the previously mentioned methods) is more efficient. This means that multistep methods like Adams and BDF (/NDF) methods are really only the most efficient when you're talking about a large spatial discretization of a PDE which doesn't have a linear operator that you can take advantage of and events are nonexistent/rare. Don't get me wrong, this is still a very important case! But, given the large amount of wrappers which handle this quite well already, I am not planning on tackling these until the other parts are completed. Expect the *DiffEq infrastructure to support multistep methods in the future (actually, there's already quite a bit of support in there, just the adaptive order and adaptive time step needs to be made possible), but don't count on it in DifferentialEquations 3.0.
Also not part of 3.0 but still of importance is stochastic delay differential equations. The development of a library for handling these equations can follow in the same manner as DelayDiffEq.jl, but likely won't make it into 3.0 as there are more pressing (more commonly used) topics to address first. In addition, methods for delay equations with nonconstant lags (and neutral delay equations) also will likely have to wait for 4.0.
In the planning stages right now is a new domainspecific language for the specification of differential equations. The current DSL, the @ode_def macro in ParameterizedFunctions.jl does great for the problem that it can handle (ODEs and diagonal SDEs). However, there are many good reasons to want to expand the capabilities here. For example, equations defined by this DSL can be symbolically differentiated, leading to extremely efficient code even for stiff problems. In addition, one could theoretically "split the ODE function" to automatically turn the problem in a SplitODEProblem with a stiff and nonstiff part suitable for IMEX solvers. If PDEs also can be written in the same syntax, then the PDEs can be automatically discretized and solved using the tools from 2.0. Additionally, one can think about automatically reducing the index of DAEs, and specifying DDEs.
This all sounds amazing, but it will need a new DSL infrastructure. After a discussion to find out what kind of syntax would be necessary, it seems that a major overhaul of the @ode_def macro would be needed in order to support all of this. The next step will be to provide a new library, DiffEqDSL.jl, for providing this enhanced DSL. As described in the previously linked roadmap discussion, this DSL will likely take a form closer in style to JuMP, and the specs seem to be compatible at reaching the above mentioned goals. Importantly, this will be developed as a new DSL and thus the current @ode_def macro will be unchanged. This is a large development which will most likely not be in 3.0, but please feel free to contribute to the roadmap discussion which is now being continued at the new repository.
Conclusion
DifferentialEquations.jl 1.0 was about establishing a core that could unify differential equations. 2.0 was about developing the infrastructure to tackle a very vast domain of scientific simulations which are not easily or efficiently written as differential equations. 3.0 will be be about producing efficient methods for the most common sets of problems which haven't adequately addressed yet. This will put the ecosystem in a very good state and hopefully make it a valuable tool for many people. After this, 4.0+ will keep adding algorithms, expand the problem domain some more, and provide a new DSL.
The post DifferentialEquations.jl 2.0: State of the Ecosystem appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/differentialequationsjl20stateecosystem/feed/
3
613

Some Fun With Julia Types: Symbolic Expressions in the ODE Solver
http://www.stochasticlifestyle.com/funjuliatypessymbolicexpressionsodesolver/
http://www.stochasticlifestyle.com/funjuliatypessymbolicexpressionsodesolver/#comments
Thu, 04 May 2017 05:57:04 +0000
http://www.stochasticlifestyle.com/?p=573
In Julia, you can naturally write generic algorithms which work on any type which has specific "actions". For example, an "AbstractArray" is a type which has a specific set of functions implemented. This means that in any genericallywritten algorithm that wants an array, you can give it an AbstractArray and it will "just work". This kind of abstraction makes it easy to write a simple algorithm and then use that same exact code for other purposes. For example, distributed computing can be done by just passing in a DistributedArray, and the algorithm can be accomplished on the GPU by using a GPUArrays. Because Julia's functions will autospecialize on the types you give it, Julia automatically makes efficient versions specifically for the types you pass in which, at compiletime, strips away the costs of the abstraction.
This means ... READ MORE
The post Some Fun With Julia Types: Symbolic Expressions in the ODE Solver appeared first on Stochastic Lifestyle.
]]>
In Julia, you can naturally write generic algorithms which work on any type which has specific "actions". For example, an "AbstractArray" is a type which has a specific set of functions implemented. This means that in any genericallywritten algorithm that wants an array, you can give it an AbstractArray and it will "just work". This kind of abstraction makes it easy to write a simple algorithm and then use that same exact code for other purposes. For example, distributed computing can be done by just passing in a DistributedArray, and the algorithm can be accomplished on the GPU by using a GPUArrays. Because Julia's functions will autospecialize on the types you give it, Julia automatically makes efficient versions specifically for the types you pass in which, at compiletime, strips away the costs of the abstraction.
This means that one way to come up with new efficient algorithms in Julia is to simply pass a new type to an existing algorithm. Does this kind of stuff really work? Well, I wanted to show that you can push this to absurd places by showing one of my latest experiments.
Note
The ODE solver part of this post won't work on release until some tags go through (I'd say at least in a week?). The fix that I had to do was make the "internal instability checks", which normally default to checking if the number is NaN, instead always be false for numbers which aren't AbstractFloats (meaning, no matter what don't halt integration due to instability). Do you understand why that would be a problem when trying to use a symbolic expression? This is a good question to keep in mind while reading.
SymEngine.jl
Let me first give a little bit of an introduction to SymEngine.jl. SymEngine is a rewrite of SymPy into C++ for increased efficiency, and SymEngine.jl is a wrapper for Julia. The only part of the package I will use is its symbolic expression type, the SymEngine.Basic.
In many cases, this is all you need out of the package. This is because dispatch and generic algorithms tend to handle everything else you need. For example, what if you want the symbolic expression for the inverse of a matrix? Just make a matrix of expressions, and call the Julia inverse function:
y1,y2,y3,y4 = @vars y1 y2 y3 y4
A = [y1 y1+y2
y3+y2 y4]
println(inv(A))
#SymEngine.Basic[(1 + y2*y3/(y1*(y4  y2*y3/y1)))/y1 y2/(y1*(y4  y2*y3/y1)); y3/(y1*(y4  y2*y3/y1)) (y4  y2*y3/y1)^(1)]
Notice that the only thing that's directly used here from SymEngine is the declaration of the variables for the symbolic expressions (the first line). The second line just creates a Julia matrix of these expressions. The last line prints the inv function from Julia's Base library applied to this matrix of expressions.
Julia's inverse function is generically written, and thus it just needs to have that the elements satisfy some properties of numbers. For example, they need to be able to add, subtract, multiply, and divide. Since our "number" type can do these operations, the inverse function "just works". But also, by Julia's magic, a separate function is created and compiled just for doing this, which makes this into a highly efficient function. So easy + efficient: the true promise of Julia is satisfied.
Going Deep: The ODE Solver
Now let's push this. Let's say we had a problem where we wanted to find out which initial condition is required in order to get out a specific value. One way to calculate this is to use Boundary Value Problem (BVP) solvers, but let's say we will want to do this at a bunch of different timepoints.
How about using a numerical solver for ODEs, except instead of using numbers, let's use symbolic expressions for our initial conditions so that way we can calculate approximate functions for the timepoints. Sounds fun and potentially useful, so let's give it a try.
The ODE solvers for Julia are in the package DifferentialEquations.jl. Let's solve the linear ODE:
with an initial condition which is a symbolic variable. Following the tutorial, let's swap out the numbers for symbolic expressions. To do this, we simply make the problem type and solve it:
using DifferentialEquations, SymEngine
y0 = symbols(:y0)
u0 = y0
f = (t,y) > 2y
prob = ODEProblem(f,u0,(0.0,1.0))
sol = solve(prob,RK4(),dt=1/10)
println(sol)
# SymEngine.Basic[y0,1.2214*y0,1.49181796*y0,1.822106456344*y0,2.22552082577856*y0,2.71825113660594*y0,3.32007193825049*y0,4.05513586537915*y0,4.95294294597409*y0,6.04952451421275*y0,7.38888924165946*y0,7.38888924165946*y0]
The solution is an array of symbolic expressions for what the RK4 method (the order 4 RungeKutta method) gives at each timepoint, starting at 0.0 and stepping 0.1 units at a time up to 1.0. We can then use the SymEngine function lambdify to turn the solution at the end into a function, and check it. For our reference, I will resolve the differential equation at a much higher accuracy. We can test this against the true solution, which for the linear ODE we know this is:
sol = solve(prob,RK4(),dt=1/1000)
end_solution = lambdify(sol[end])
println(end_solution(2)) # 14.778112197857341
println(2exp(2)) # 14.7781121978613
println(end_solution(3)) # 22.167168296786013
println(3exp(2)) # 22.16716829679195
We have successfully computed a really high accuracy approximation to our solution which is a function of the initial condition! We can even do this for systems of ODEs. For example, let's get a function which approximates a solution to the LotkaVolterra equation:
using SymEngine, DifferentialEquations
# Make our initial condition be symbolic expressions from SymEngine
x0,y0 = @vars x0 y0
u0 = [x0;y0]
f = function (t,y,dy)
dy[1] = 1.5y[1]  y[1]*y[2]
dy[2] = 3y[2] + y[1]*y[2]
end
prob = ODEProblem(f,u0,(0.0,1.0))
sol = solve(prob,RK4(),dt=1/2)
The result is a stupidly large expression because it grows exponentially and SymEngine doesn't have a simplification function yet (it's still pretty new!), but hey this is super cool anyways.
Going Past The Edge: What Happens With Incompatibility?
There are some caveats here. You do need to work through MethodErrors. For example, if you wanted to use the more efficient version of ode45/dopri5 (Tsit5), you'll get an error:
sol = solve(prob,Tsit5())
MethodError: no method matching abs2(::SymEngine.Basic)
Closest candidates are:
abs2(!Matched::Bool) at bool.jl:38
abs2(!Matched::ForwardDiff.Dual{N,T<:Real}) at C:\Users\Chris\.julia\v0.5\ForwardDiff\src\dual.jl:325
abs2(!Matched::Real) at number.jl:41
...
What this is saying is that there is no abs2 function for symbolic expressions. The reason why this is used is because the adaptive algorithm uses normed errors in order to find out how to change the stepsize. However, in order to get a normed error, we'd actually have to know the initial condition... so this just isn't going to work.
Conclusion
This is kind of a silly example just for fun, but there's a more general statement here. In Julia, new algorithms can just be passing new types to prewritten functionality. This vastly decreases the amount of work that you actually have to do, without a loss to efficiency! You can add this to your Julia bag of tricks.
The post Some Fun With Julia Types: Symbolic Expressions in the ODE Solver appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/funjuliatypessymbolicexpressionsodesolver/feed/
2
573

Building a Web App in Julia: DifferentialEquations.jl Online
http://www.stochasticlifestyle.com/buildingwebappjuliadifferentialequationsjlonline/
http://www.stochasticlifestyle.com/buildingwebappjuliadifferentialequationsjlonline/#comments
Tue, 17 Jan 2017 21:35:15 +0000
http://www.stochasticlifestyle.com/?p=535
Web apps are all the rage because of accessibility. However, there are usually problems with trying to modify some existing software to be a web app: specifically user interface and performance. In order for a web application to perform well, it must be able to have a clean user interface with a "smart" backend which can automatically accommodate to the user's needs, but also give back a nice looking response in under a second. This has typically limited the types of applications which can become responsive web applications.
Scientific software is one domain where user interface has been obstructive at best, and always in need of better performance. However, the programming language Julia has allowed us to build both an easy to use and highly performant ecosystem of numerical differential equation solvers over the last 8 months. Thus we had ... READ MORE
The post Building a Web App in Julia: DifferentialEquations.jl Online appeared first on Stochastic Lifestyle.
]]>
Web apps are all the rage because of accessibility. However, there are usually problems with trying to modify some existing software to be a web app: specifically user interface and performance. In order for a web application to perform well, it must be able to have a clean user interface with a "smart" backend which can automatically accommodate to the user's needs, but also give back a nice looking response in under a second. This has typically limited the types of applications which can become responsive web applications.
Scientific software is one domain where user interface has been obstructive at best, and always in need of better performance. However, the programming language Julia has allowed us to build both an easy to use and highly performant ecosystem of numerical differential equation solvers over the last 8 months. Thus we had to ask the next question: can Julia be used as a viable backend for scientific web applications?
The answer is a definitive yes! Today I am announcing DifferentialEquations.jl Online, a web interface for DifferentialEquations.jl, a library of numerical methods for differential equations written in Julia. Using this web application, you can easily solve biological models, draw interactive 3D phase diagrams, and easily see what happens when you add stochasticity (randomness) to a model. While we restrict the user to a light computational load (maxes out at 1000 iterations), this serves as a good educational tool, a good tool for easily exploring models, and as a gateway to DifferentialEquations.jl.
If you like this work and would like to support it, please star the DifferentialEquations.jl repository. In this blog post I would like to share how we built this app and the lack of difficulties that we encountered.
Getting The Prototype Together
I took the idea over to Julia's discourse forum where many people throw around some ideas for how to get this working. Alex Mellnik is the star of the show who developed a working prototype in what must have been a few hours using JuliaWebAPI.jl along with an AngularJS frontend (though he shortly after changed the backend to Mux.jl). The resulting code was short and simple enough that I was able to dive right into it.
Next we had to look around for how to host it. While we were at first were looking at using AWS Lambda, we eventually settled on using Heroku and deploying via Docker containers. This setup allows us to build an install of Julia the way we like and then just ship it over to the server. Major kudos to Alex for figuring out how to set this up. You can see how we develop and deploy by looking at our Github repository for the frontend and the repository for the backend.
Optimizing the Web Application
There are a few things to note about Julia and how we had to make the app. Specifically, Julia is a really interesting language because it allows the use of justintime (JIT) compilation in order to speedup functions. It does this by specializing functions to the arguments which they are given, and caching that function call for further uses. DifferentialEquations.jl is built around optimizing the numerical solvers for the most difficult differential equations you can throw at it. However, this means that it specializes and recompiles a new version of the solver functions for each ODE/SDE/etc. that you give it. In most cases, this is the right thing to do and on beefier computers has a startup cost of around 0.4 seconds. This form of hyperspecialization allows our solvers to routinely beat even the classic FORTRAN codes in performance benchmarks. However, for our web application we are restricting users to only 1000 iterations (and thus easy problems), so instead of brute performance we needed a fast response time. This kind of hyperspecialization caused some response time issues since the web server was quite slow at compiling (around 1 second).
But Julia is very flexible, and making it so that way it wouldn't fully specialize and recompile was as easy as setting up a few new types. While the standard problem types that are generated for the differential equations are strongly typed in the function fields, we made a special set of types for handling these "quick problems" which only specify "f" as a Function (which is an abstract and not a concrete type, since each function is its own type). Those 44 lines were pretty much the only changes required to have Julia automatically optimize for quick problems and response time instead of long problems. The rest of the logic all calls the exact same functions as the normal types because it's all handled via dispatches on the abstract type hierarchy. The result? Less than 100 lines of code were required to get average response times come out to be around 0.2 seconds (with plotting via Plots.jl taking around half of that time), which is more than responsive enough for a computationally heavy web application!
Conclusion
And that's the main conclusion I wanted to share. Not only was it quick and easy to make this web application in Julia, but the resulting product is quick to respond and easy to use. Since it plugs into DifferentialEquations.jl, there really is not much special code required for the web server other than a few type definitions. However, those type definitions were all that was required to make Julia pretty much automatically optimize the resulting code for a completely different application. The result is both easy to maintain and easy to extend. I couldn't be happier with this experience.
The post Building a Web App in Julia: DifferentialEquations.jl Online appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/buildingwebappjuliadifferentialequationsjlonline/feed/
4
535

6 Months of DifferentialEquations.jl: Where We Are and Where We Are Going
http://www.stochasticlifestyle.com/6monthsdifferentialequationsjlgoing/
http://www.stochasticlifestyle.com/6monthsdifferentialequationsjlgoing/#respond
Thu, 15 Dec 2016 06:41:26 +0000
http://www.stochasticlifestyle.com/?p=483
So around 6 months ago, DifferentialEquations.jl was first registered. It was at first made to be a library which can solve "some" types of differential equations, and that "some" didn't even include ordinary differential equations. The focus was mostly fast algorithms for stochastic differential equations and partial differential equations.
Needless to say, Julia makes you too productive. Ambitions grew. By the first release announcement, much had already changed. Not only were there ordinary differential equation solvers, there were many. But the key difference was a change in focus. Instead of just looking to give a productionquality library of fast methods, a major goal of DifferentialEquations.jl became to unify the various existing packages of Julia to give one userfriendly interface.
Since that release announcement, we have made enormous progress. At this point, I believe we have both the most expansive and flexible ... READ MORE
The post 6 Months of DifferentialEquations.jl: Where We Are and Where We Are Going appeared first on Stochastic Lifestyle.
]]>
So around 6 months ago, DifferentialEquations.jl was first registered. It was at first made to be a library which can solve "some" types of differential equations, and that "some" didn't even include ordinary differential equations. The focus was mostly fast algorithms for stochastic differential equations and partial differential equations.
Needless to say, Julia makes you too productive. Ambitions grew. By the first release announcement, much had already changed. Not only were there ordinary differential equation solvers, there were many. But the key difference was a change in focus. Instead of just looking to give a productionquality library of fast methods, a major goal of DifferentialEquations.jl became to unify the various existing packages of Julia to give one userfriendly interface.
Since that release announcement, we have made enormous progress. At this point, I believe we have both the most expansive and flexible differential equation library to date. I would like to take this time to explain the overarching design, where we are, and what you can expect to see in the near future.
(Note before we get started: if you like what you see, please star the DifferentialEquations.jl repository. I hope to use this to show interest in the package so that one day we can secure funding and make this my main project. Thank you for your support!)
JuliaDiffEq Structure
If you take a look at the source for DifferentialEquations.jl, you will notice that almost all of the code has gone. What has happened?
The core idea behind this change is explained in another blog post on modular interfaces for scientific computing in Julia. The key idea is that we built an interface which is "packageindependent", meaning that the same lines of code can call solvers from different packages. There are many advantages to this which will come up later in the section which talks about the "addon packages", but one organizational advantage is that it lets us split up the repositories as needed. The core differential equation solvers from DifferentialEquations.jl reside in OrdinaryDiffEq.jl, StochasticDiffEq.jl, etc. (you can see more at our webpage). Packages like Sundials.jl, ODEInterface.jl, ODE.jl, etc. all have bindings to this same interface, making them all work similarly.
One interesting thing about this setup is that you are no longer forced to contribute to these packages in order to contribute to the ecosystem. If you are a developer or a researcher in the field, you can develop your own package with your own license which has a common interface binding, and you will get the advantages of the common interface without any problems. This may be necessary for some researchers, and so we encourage you to join in and contribute as you please.
The Common Interface
Let me then take some time to show you what this common interface looks like. To really get a sense, I would recommend checking out the tutorials in the documentation and the extra Jupyter notebook tutorials in DiffEqTutorials.jl. The idea is that solving differential equations always has 3 steps:
 Defining a problem.
 Solving the problem.
 Analyzing the solution.
Defining a problem
What we created was a generic typestructure for which dispatching handles the details. For defining an ODE problem, one specifies the function for the ODE, the initial condition, and the timespan that the problem is to be solved on. The ODE
with an initial condition and and timespan is then written as:
prob = ODEProblem(f,u0,(t0,tf))
There are many different problem types for different types of differential equations. Currently we have types (and solvers) for ordinary differential equations, stochastic differential equations, differential algebraic equations, and some partial differential equations. Later in the post I will explain how this is growing.
Solving the problem
To solve the problem, the common solve command is:
sol = solve(prob,alg;kwargs...)
where alg is a typeinstance for the algorithm. It is by dispatch on alg that the package is chosen. For example, we can call the 14thOrder Feagin Method from OrdinaryDiffEq.jl via
sol = solve(prob,Feagin14();kwargs...)
We can call the BDF method from Sundials.jl via
sol = solve(prob,CVODE_BDF();kwargs...)
Due to this structure (and the productivity of Julia), we have a ridiculous amount of methods which are available as is seen in the documentation. Later I will show that we do not only have many options, but these options tend to be very fast, often benchmarking as faster than classic FORTRAN codes. Thus one can choose the right method for the problem, and efficient solve it.
Notice I put in the trailing "kwargs...". There are many keyword arguments that one is able to pass to this solve command. The "Common Solver Options" are documented at this page. Currently, all of these options are supported by the OrdinaryDiffEq.jl methods, while there is general support for large parts of this for the other methods. This support will increase overtime, and I hope to include a table which shows what is supported where.
Analyzing the solution
Once you have this solution type, what does it do? The details are explained in this page of the manual, but I would like to highlight some important features.
First of all, the solution acts as an array. For the solution at the ith timestep, you just treat it as an array:
sol[i]
You can also get the ith timepoint out:
sol.t[i]
Additionally, the solution lets you grab individual components. For example, the jth component at the ith timepoint is found by:
sol[i,j]
These overloads are necessary since the underlying data structure can actually be a more complicated vector (some examples explained later), but this lets you treat it in a simple manner.
Also, by default many solvers have the option "dense=true". What this means is that the solution has a dense (continuous) output, which is overloaded on to the solver. This looks like:
sol(t)
which gives the solution at time t. This continuous version of the solution can be turned off using "dense=false" (to get better performance), but in many cases it's very nice to have!
Not only that, but there are some standard analysis functions available on the solution type as well. I encourage you to walk through the tutorial and see for yourself. Included are things like plot recipes for easy plotting with Plots.jl:
plot(sol)
Now let me describe what is available with this interface.
EcosystemWide Development Tools and Benchmarks
Since all of the solutions act the same, it's easy to create tools which build off of them. One fundamental set of tools are those included in DiffEqDevTools.jl. DiffEqDevTools.jl includes a bunch of functions for things like convergence testing and benchmarking. This not only means that all of the methods have convergence tests associated with them to ensure accuracy and correctness, but also that we have ecosystemwide benchmarks to know the performance of different methods! These benchmarks can be found at DiffEqBenchmarks.jl and will be referenced throughout this post.
Very Efficient Nonstiff ODE Solvers
The benchmarks show that the OrdinaryDiffEq.jl methods achieve phenomenal performance. While in many cases other libraries resort to the classic dopri5 and dop853 methods due to Hairer, our ecosystem has these methods available via the ODEInterface.jl glue package ODEInterfaceDiffEq.jl and so these can be directly called from the common interface. From the benchmarks on nonstiff problems you can see that the OrdinaryDiffEq.jl methods are much more efficient than these classic codes when one is looking for the highest performance. This is even the case for DP5() and DP8() which have the same exact timestepping behavior as dopri5() and dop853() respectively, showing that these implementations are top notch, if not the best available.
These are the benchmarks on the implementations of the DormandPrince 4/5 methods. Also included is a newer method, the Tsitorous 4/5 method, which is now the default nonstiff method in DifferentialEquations.jl since our research has shown that it is more efficient than the classical methods (on most standard problems).
A Wide Array of Stiff ODE Solvers
There is also a wide array of stiff ODE solvers which are available. BDF methods can be found from Sundials.jl, Radau methods can be found from ODEInterface.jl, and a welloptimized 2ndOrder Rosenbrock method can be found in OrdinaryDiffEq.jl. One goal in the near future will be to implement higher order Rosenbrock methods in this fashion, since it will be necessary to get better performance, as shown in the benchmarks. However, the Sundials and ODEInterface methods, being that they use FORTRAN interop, are restricted to equations on Float64, while OrdinaryDiffEq.jl's methods support many more types. This allows one to choose the best method for the job.
Wrappers for many classic libraries
Many of the classic libraries people are familiar with are available from the common interface, including:
 CVODE
 LSODA
 The Hairer methods
and differential algebraic equation methods including
 IDA (Sundials)
 DASKR
Native Julia Differential Algebraic Equation Methods
DASSL.jl is available on the common interface and provides a method to solve differential algebraic equations using a variabletimestep BDF method. This allows one to support some Juliabased types like arbitraryprecision numbers which are not possible with the wrapped libraries.
Extensive Support for Juliabased Types in OrdinaryDiffEq.jl
Speaking of support for types, what is supported? From testing we know that the following work with OrdinaryDiffEq.jl:
 Arbitrary precision arithmetic via BigFloats, ArbFloats, DecFP
 Numbers with units from Unitful.jl
 Ndimensional arrays
 Complex numbers (the nonstiff solvers)
 "Very arbitrary arrays"
Your numbers can be ArbFloats of 200bit precision in 3dimensional tensors with units (i.e. "these numbers are in Newtons"), and the solver will ensure that the dimensional constraints are satisfied, and at every timestep give you a 3dimensional tensor with 200bit ArbFloats. The types are declared to match the initial conditions: if you start with u0 having BigFloats, you will be guaranteed to have BigFloat solutions. Also, the types for time are determined by the types for the times in the solution interval (t0,tf). Therefore can have the types for time be different than the types for the solution (say, turn off adaptive timestepping and do fixed timestepping with rational numbers or integers).
Also, by "very arbitrary arrays" I mean, any type which has a linear index can be used. One example which recently came up in this thread involves solving a hybriddynamical system which has some continuous variables and some discrete variables. You can make a type which has a linear index over the continuous variables and simply throw this into the ODE solver and it will know what to do (and use callbacks for discrete updates). All of the details like adaptive timestepping will simply "just work".
Thus, I encourage you to see how these methods can work for you. I myself have been developing MultiScaleModels.jl to build multiscale hybrid differential equations and solve them using the methods available in DifferentialEquations.jl. This shows that heuristic for classic problems which you "cannot use a package for" no longer apply: Julia's dispatch system allows DifferentialEquations.jl to handle these problems, meaning that there is no need for you to have to ever reinvent the wheel!
Event Handling and Callbacks in OrdinaryDiffEq.jl
OrdinaryDiffEq.jl already has extensive support for callback functions and event handling. The documentation page describes a lot of what you can do with it. There are many things you can do with this, not just bouncing a ball, but you can also use events to dynamically change the size of the ODE (as demonstrated by the cell population example).
Specification of Extra Functions for Better Performance
If this were any other library, the header would have been "Pass Jacobians for Better Performance", but DifferentialEquations.jl's universe goes far beyond that. We named this set of functionality Performance Overloads. An explicit function for a Jacobian is one type of performance overload, but you can pass the solvers many other things. For example, take a look at:
f(Val{:invW},t,u,Î³,iW) # Call the explicit inverse RosenbrockW function (M  Î³J)^(1)
This seems like an odd definition: it is the analytical function for the equation for some mass matrix built into the function. The reason why this is so necessary is because Rosenbrock methods have to solve this every step. What this allows the developers to do is write a method which goes like:
if has_invW(f)
# Use the function provided by the user
else
# Get the Jacobian
# Build the W
# Solve the linear system
end
Therefore, whereas other libraries would have to use a linear solver to solve the implicit equation at each step, DifferentialEquations.jl allows developers to write this to use the precomputed inverses and thus get an explicit method for stiff equations! Since the linear solves are the most expensive operation, this can lead to huge speedups in systems where the analytical solution can be computed. But is there a way to get these automatically?
Parameterized Functions and Function Definition Macros
ParameterizedFunctions.jl is a library which solves many problems at one. One question many people have is, how do you provide the model parameters to an ODE solver? While the standard method of "use a closure" is able to work, there are many higherorder analyses which require the ability to explicitly handle parameters. Thus we wanted a way to define functions with explicit parameters.
The way that this is done is via call overloading. The syntax looks like this. We can define the LotkaVolterra equations with explicit parameters and via:
type LotkaVolterra <: Function
a::Float64
b::Float64
end
f = LotkaVolterra(0.0,0.0)
(p::LotkaVolterra)(t,u,du) = begin
du[1] = p.a * u[1]  p.b * u[1]*u[2]
du[2] = 3 * u[2] + u[1]*u[2]
end
Now f is a function where f.a and f.b are the parameters in the function. This type of function can then be seamlessly used in the DifferentialEquations.jl solvers, including those which use interop like Sundials.jl.
This is very general syntax which can handle any general function. However, the next question was, is there a way to do this better for the problems that people commonly encounter? Enter the library ParameterizedFunctions.jl. As described in the manual, this library (which is part of DifferentialEquations.jl) includes a macro @ode_def which allows you to define the LotkaVolterra equation
as follows:
f = @ode_def LotkaVolterraExample begin
dx = a*x  b*x*y
dy = c*y + d*x*y
end a=>1.5 b=>1.0 c=>3.0 d=1.0
Notice that at the bottom you pass in the parameters. => keeps the parameters explicit, while = passes them in as a constant. Flip back and forth to see that it matches the LaTeX so that way it's super easy to debug and maintain.
But this macro isn't just about ease of use: it's also about performance! What happens silently within this macro is that symbolic calculations occur via SymEngine.jl. The Performance Overloads functions are thus silently symbolically computed, allowing the solvers to then get maximal performance. This gives you an easy way to define a stiff equation of 100 variables in a very intuitive way, yet get a faster solution than you would find with other libraries.
Sensitivity Analysis
Once we have explicit parameters, we can generically implement algorithms which use them. For example, DiffEqSensitivity.jl transforms a ParameterizedFunction into the senstivity equations which are then solved using any ODE solver, outputting both the ODE's solution and the parameter sensitivities at each timestep. This is described in the manual in more detail. The result is the ability to use whichever differential equation method in the common interface matches your problem to solve the extended ODE and output how sensitive the solution is to the parameters. This is the glory of the common interface: tools can be added to every ODE solver all at once!
In the future we hope to increase the functionality of this library to include functions for computing global and adjoint sensitivities via methods like the Morris method. However, the current setup shows how easy this is to do, and we just need someone to find the time to actually do it!
Parameter Estimation
Not only can we identify parameter sensitivities, we can also estimate model parameters from data. The design of this is described in more detail is explained in the manual. It is contained in the package DiffEqParamEstim.jl. Essentially, you can define a function using the @ode_def macro, and then pair it with any ODE solver and (almost) any optimization method, and together use that to find the optimal parameters.
In the near future, I would like to increase the support of DiffEqParamEstim.jl to include machine learning methods from JuliaML using its Learn.jl. Stay tuned!
Adaptive Timestepping for Stochastic Differential Equations
Adaptive timestepping is something you will not find in other stochastic differential equation libraries. The reason is because it's quite hard to do correctly and, when finally implemented, can have so much overhead that it does not actually speedup the runtime for most problems.
To counteract this, I developed a new form of adaptive timestepping for stochastic differential equations which focused on both correctness and an algorithm design which allows for fast computations. The result was a timestepping algorithm which is really fast! This paper has been accepted to Discrete and Continuous Dynamical Systems Series B, and where we show that the correctness of the algorithm and its efficiency. We actually had to simplify the test problem so that way we could time the speedup over fixed timestep algorithms, since otherwise they weren't able to complete in a reasonable time without numerical instability! When simplified, the speedup over all of the tested fixed timestep methods was >12x, and this speedup only increases as the problem gets harder (again, we chose the simplified version only because testing the fixed timestep methods on harder versions wasn't even computationally viable!).
These methods, Rejection Sampling with Memory (RSwM), are available in DifferentialEquations.jl as part of StochasticDiffEq.jl. It should help speed up your SDE calculations immensely. For more information, see the publication "Adaptive Methods for Stochastic Differential Equations via Natural Embeddings and Rejection Sampling with Memory".
Easy Multinode Parallelism For Monte Carlo Problems
Also included as part of the stochastic differential equation suite are methods for parallel solving of Monte Carlo problems. The function is called as follows:
monte_carlo_simulation(prob,alg;numMonte=N,kwargs...)
where the extra keyword arguments are passed to the solver, and N is the number of solutions to obtain. If you've setup Julia on a multinode job on a cluster, this will parallelize to use every core.
In the near future, I hope to expand this to include a Monte Carlo simulation function for random initial conditions, and allow using this on more problems like ODEs and DAEs.
Smart Defaults
DifferentialEquations.jl also includes a new cool feature for smartly choosing defaults. To use this, you don't really have to do anything. If you've defined a problem, say an ODEProblem, you can just call:
sol = solve(prob;kwargs...)
without passing the algorithm and DifferentialEquations.jl will choose an algorithm for you. Also included is an `alg_hints` parameter with allows you to help the solver choose the right algorithm. So lets say you want to solve a stiff stochastic differential equations, but you do not know much about the algorithms. You can do something like:
sol = solve(prob,alg_hints=[:stiff])
and this will choose a good algorithm for your problem and solve it. This reduces userburden to only having to know properties of the problem, while allowing us to proliferate the solution methods. More information is found in the Common Solver Options manual page.
Progress Monitoring
Another interesting feature is progress monitoring. OrdinaryDiffEq.jl includes the ability to use Juno's progressbar. This is done via the keyword arguments like:
sol = solve(prob,progress=true,
progress_steps=100)
You can also set a progress message, for which the default is:
ODE_DEFAULT_PROG_MESSAGE(dt,t,u) = "dt="*string(dt)*"\nt="*string(t)*"\nmax u="*string(maximum(abs.(u)))
When you scroll over the progressbar, it will show you how close it is to the final timepoint and use linear extrapolation to estimate the amount of time left to solve the problem.
When you scroll over the top progressbar, it will display the message. Here, it tells us the current dt, t, and the maximum value of u (the independent variable) to give a sanity check that it's all working.
The keyword argument progress_steps lets you control how often the progress bar updates, so here we choose to do it every 100 steps. This means you can do some very intense sanity checks inside of the progress message, but reduce the number of times that it's called so that way it doesn't affect the runtime.
All in all, having this built into the interface should make handling long and difficult problems much easier, I problem that I had when using previous packages.
(Stochastic) Partial Differential Equations
There is rudimentary support for solving some stochastic partial differential equations which includes semilinear Poisson and Heat equations. This is able to be done on a large set of domains using a finite element method as provided by FiniteElementDiffEq.jl. I will say that this library is in need of an update for better efficiency, but it shows how we are expanding into the domain of adding easytodefine PDE problems, which then create the correct ODEProblem/DAEProblem discretization and which then gets solved using the ODE methods.
Modular Usage
While all of this creates the DifferentialEquations.jl package, the JuliaDiffEq ecosystem is completely modular. If you want to build a library which uses only OrdinaryDiffEq.jl's methods, you can directly use those solvers without requiring the rest of DifferentialEquations.jl
An Update Blog
Since things are still changing fast, the website for JuliaDiffEq contains a news section which will describe updates to packages in the ecosystem as they occur. To be notified of updates, please subscribe to the RSS feed.
Coming Soon
Let me take a few moments to describe some works in progress. Many of these are past planning stages and have partial implementations. I find some of these very exciting.
Solver Method Customization
The most common reason to not use a differential equation solver library is because you need more customization. However, as described in this blog post, we have developed a design which solves this problem. The advantages are huge. Soon you will be able to choose the linear and nonlinear solvers which are employed in the differential equation solving methods. For linear solvers, you will be able to use any method which solves a linear map. This includes direct solvers from Base, iterative solvers from IterativeSolvers.jl, parallel solvers from PETSc.jl, GPU methods from CUSOLVER.jl: it will be possible to even use your own linear solver if you wish. The same will be true for nonlinear solvers. Thus you can choose the internal methods which match the problem to get the most efficiency out.
Specialized Problem Types and Promotions
One interesting setup that we have designed is a hierarchy of types. This is best explained by example. One type of ODE which shows up are "ImplicitExplicit ODEs", written as:
where is a "stiff" function and is a "nonstiff" function. These types of ODEs with a natural splitting commonly show up in discretizations of partial differential equations. Soon we will allow one to define an IMEXProblem(f,g,u0,tspan) for this type of ODE. Specialized methods such as the ARKODE methods from Sundials will then be able to utilize this form to gain speed advantages.
However, let's say you just wanted to use a standard RungeKutta method to solve this problem? What we will automatically do via promotion is make
and then behind the scenes the RungeKutta method will solve the ODE
Not only that, but we can go further and define
to get the equation
which is a differential algebraic equation solver. This autopromotion means that any method will be able to solve any problem type which is lower than it.
The advantages are twofold. For one, it allows developers to write a code to the highest problem available, and automatically have it work on other problem types. For example, the classic Backwards Differentiation Function methods (BDF) which are seen in things like MATLAB's ode15s are normally written to solve ODEs, but actually can solve DAEs. In fact, DASSL.jl is an implementation of this algorithm. When this promotion structure is completed, DASSL's BDF method will be a native BDF method not just for solving DAEs, but also ODEs, and there is no specific development required on the part of DASSL.jl. And because Julia's closures compile to fast functions, all of this will happen with little to no overhead.
In addition to improving developer productivity, it allows developers to specialize methods to problems. The splitting methods for implicitexplicit problems can be tremendously more performant since it reduces the complexity of the implicit part of the equation. However, with our setup we go even further. One common case that shows up in partial differential equations is that one of these equations is linear. For example, in a discretization of the semilinear Heat Equation, we arise at an ODE
where is a matrix which is the discretization of the LaPlacian. What our ecosystem will allow is for the user to specify that the first function is a linear function by wrapping it in a LinearMap type from LinearMaps.jl. Then the solvers can use this information like:
if is_linear(f)
# Perform linear solve
else
# Perform nonlinear solve
end
This way, the solvers will be able to achieve even more performance by specializing directly to the problem at hand. In fact, it will allow methods require this type of linearity like Exponential RungeKutta methods to be able to be developed for the ecosystem and be very efficient methods when applicable.
In the end, users can just define their ODE by whatever problem type makes sense, and promotion magic will make tons of methods available, and typechecking within the solvers will allow them to specialize directly to every detail of the ODE for as much speed as possible. With DifferentialEquations.jl also choosing smart default methods to solve the problem, the userburden is decreased and very specialized methods can be used to get maximal efficiency. This is a win for everybody!
Ability to Solve Fun Types from ApproxFun.jl
ApproxFun.jl provides an easy way to do spectral approximations of functions. In many cases, these spectral approximations are very fast and are used to decompose PDEs in space. When paired with timestepping methods, this gives an easy way to solve a vast number of PDEs with really good efficiency.
The link between these two packages is currently found in SpectralTimeStepping.jl. Currently you can fix the basis size for the discretization and use that to solve the PDE with any ODE method in the common interface. However, we are looking to push further. Since OrdinaryDiffEq.jl can handle many different Juliadefined types, we are looking to make it support solving the ApproxFun.jl Fun type directly, which would allow the ODE solver to adapt the size of the spectral basis during the computation. This would tremendously speedup the methods and make it as fast as if you were to specifically design a spectral method to a PDE. We are really close to getting this!
New Methods for Stochastic Differential Equations
I can't tell you too much about this because these methods will be kept secret until publication, but there are some very computationallyefficient methods for nonstiff and semistiff equations which have already been implemented and are being thoroughly tested. Go tell the people doing the peer review to hurry up the process if you want these quicker!
Improved Plot Recipes
There is already an issue open for improving the plot recipes. Essentially what will come out of this will be the ability to automatically draw phase plots and other diagrams from the plot command. This should make using DifferentialEquations.jl even easier than before.
Uncertainty Quantification
One major development in scientific computing has been the development of methods for uncertainty quantification. This allows you to quantify the amount of error that comes from a numerical method. There is already a design for how to use the ODE methods to implement a popular uncertainty quantification algorithm, which would allow you to see a probability distribution for the numerical solution to show the uncertainty in the numerical values. Like the sensitivity analysis and parameter estimation, this can be written in a solverindependent manner so that way it works with any solver on the common interface (which supports callbacks). Coming soon!
Optimal Control
We have in the works for optimal control problem types which will automatically dispatch to PDE solvers and optimization methods. This is a bit off in the distance, but is currently being planned.
Geometric and Symplectic Integrators
A newcomer to the Juliasphere is GeometricIntegrators.jl. We are currently in the works for attaching this package to the common interface so that way it will be easily accessible. Then, Partitioned ODE and DAE problems will be introduced (with a promotion structure) which will allow users to take advantage of geometric integrators for their physical problems.
Bifurcation Analysis
Soon you will be able to take your ParameterizedFunction and directly generate bifurcation plots from it. This is done by a wrapper to the PyDSTool library via PyDSTool.jl, and a linker from this wrapper to the JuliaDiffEq ecosystem via DiffEqBifurcate.jl. The toolchain already works, but... PyCall has some nasty segfaults. When these segfaults are fixed in PyCall.jl, this functionality will be documented and released.
Models Packages
This is the last coming soon, but definitely not the least. There are already a few "models packages" in existence. What these packages do is provide functionality which makes it easy to define very specialized differential equations which can be solved with the ODE/SDE/DAE solvers. For example, FinancialModels.jl makes it easy to define common equations like Heston stochastic volatility models, which will then convert into the appropriate stochastic differential equation or PDE for use in solver methods. MultiScaleModels.jl allows one to specify a model on multiple scales: a model of proteins, cells, and tissues, all interacting dynamically with discrete and continuous changes, mixing in stochasticity. Also planned is PhysicalModels.jl which will allow you to define ODEs and DAEs just by declaring the Hamiltonian or Legrangian functions. Together, these packages should help show how the functionality of DifferentialEquations.jl reaches far beyond what previous differential equation packages have allowed, and make it easy for users to write very complex simulations (all of course without the loss of performance!).
Conclusion
I hope this has made you excited to use DifferentialEquaitons.jl, and excited to see what comes in the future. To support this development, please star the DifferentialEquations.jl repository. I hope to use these measures of adoption to one day secure funding. In the meantime, if you want to help out, join in on the issues in JuliaDiffEq, or come chat in the JuliaDiffEq Gitter chatroom. We're always looking for more hands! And to those who have already contributed: thank you as this would not have been possible without each and every one of you.
The post 6 Months of DifferentialEquations.jl: Where We Are and Where We Are Going appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/6monthsdifferentialequationsjlgoing/feed/
0
483

Modular Algorithms for Scientific Computing in Julia
http://www.stochasticlifestyle.com/modularalgorithmsscientificcomputingjulia/
http://www.stochasticlifestyle.com/modularalgorithmsscientificcomputingjulia/#comments
Wed, 07 Dec 2016 16:55:56 +0000
http://www.stochasticlifestyle.com/?p=390
When most people think of the Julia programming language, they usually think about its speed. Sure, Julia is fast, but what I emphasized in a previous blog post is that what makes Julia tick is not justintime compilation, rather it's specialization. In that post, I talked a lot about how multiple dispatch allows for Julia to automatically specialize functions on the types of the arguments. That microspecialization is what allows Julia to compile functions which are as fast as those from C/Fortran.
However, there is a form of "macro specialization" that Julia's design philosophy and multiple dispatch allows one to build libraries for. It allows you to design an algorithm in a very generic form, essentially writing your full package with inputs saying "insert scientific computing package here", allowing users to specialize the entire overarching algorithm to the specific problem. ... READ MORE
The post Modular Algorithms for Scientific Computing in Julia appeared first on Stochastic Lifestyle.
]]>
When most people think of the Julia programming language, they usually think about its speed. Sure, Julia is fast, but what I emphasized in a previous blog post is that what makes Julia tick is not justintime compilation, rather it's specialization. In that post, I talked a lot about how multiple dispatch allows for Julia to automatically specialize functions on the types of the arguments. That microspecialization is what allows Julia to compile functions which are as fast as those from C/Fortran.
However, there is a form of "macro specialization" that Julia's design philosophy and multiple dispatch allows one to build libraries for. It allows you to design an algorithm in a very generic form, essentially writing your full package with inputs saying "insert scientific computing package here", allowing users to specialize the entire overarching algorithm to the specific problem. JuliaDiffEq, JuliaML, JuliaOpt, and JuliaPlots have all be making use of this style, and it has been leading to some really nice results. What I would like to do is introduce how to program in this style, and the advantages that occur simply by using this design. I want to show that not only is this style very natural, but it solves many extendability problems that libraries in other languages did not have a good answer for. The end result is a unique Julia design which produces both more flexible and more performant code.
A Case Study: Parameter Estimation in Differential Equations
I think the easiest way to introduce what's going on and why it's so powerful is to give an example. Let's look at parameter estimation in differential equations. The setup is as follows. You have a differential equation which explains how evolves over time with being the coefficients of the model. For example, could be the reproduction rates of different species, and this ODE describes how the populations of the ecosystem (rabbits, wolves, etc.) evolve over time (with being a vector at each time which is indicative of the amount of each species). So if we just have rabbits and wolves, would mean there's twice as many rabbits as there are wolves at time , and this could be with reproduction rates births per time unit for each species.
Assume that we have some measurements which are real measurements for the number of rabbits and wolves. What we would like to do is come up with a good number for the reproduction rates such that our model's outputs match reality. The way that one does this is you set up a optimization problem. The easiest problem is to simply say you want to minimize the (Euclidian) difference between the model's output and the data. In mathematical notation, this is written as:
or more simply, find the such that the loss function is minimized (the norm notation just means sum of the squares, so this is the sum of squared differences between the model and reality. This is the same loss measure which is used in regressions). Intuitive, right?
The Standard Approach to Developing an Algorithms for Estimation
That's math. To use math, we need to pull back on the abstraction a little bit. In reality, is not a function at all times, we have discrete times where we measured data points. And also, we don't know ! We know the differential equation, but most differential equations cannot be solved analytically. So what we must do to solve this is:
1. Numerically solve the differential equation.
2. Use the numerical solution in some optimization algorithm.
At this point, the simple mathematical problem no longer sounds so simple: we have to develop a software for solving differential equations, and software for doing optimization!
If you check out how packages normally will do this, you will see that they tend to reinvent the wheel. There are issues with this approach. For one, you probably won't develop the fastest most complex numerical differential equation algorithms or optimization algorithms because this might not be what you do! So you'll opt for simpler implementations of these parts, and then the package won't get optimal performance, and you'll have more to do/maintain.
Another approach is to use packages. For example, you can say "I will call _____ for solving the differential equations, and ________ for solving the optimization problem". However, this has some issues. First of all, many times in scientific computing, in order to compute the solution more efficiently, the algorithms may have to be tailored to the problem. This is why there's so much research in new multigrid methods, new differential equations methods, etc! So the only way to make this truly useful is to make it expansive enough such that users can pick from many choices. This sounds like a pain.
But what if I told you there's a third way, where "just about any package" can be used? This is the modular programming approach in Julia.
Modular Programming through Common Interfaces
The key idea here is that multiple dispatch allows many different packages to implement the same interface, and so if you write code to that interface, you will automatically be "packageindependent". Let's take a look at how DiffEqParamEstim.jl does it (note that this package is still under development so there are a few rough edges, but it still illustrates the idea beautifully).
In JuliaDiffEq, there is something called the "Common Interface". This is documented in the DifferentialEquations.jl documentation. What it looks like is this: to solve an ODE with initial condition over a time interval , you make the problem type:
prob = ODEProblem(f,y0,tspan)
and then you call solve with some algorithm. For example, if we want to use the Tsit5 algorithm from OrdinaryDiffEq.jl, we do:
sol = solve(prob,Tsit5())
and we can pass a bunch more common arguments. More details for using this can be found in the tutorial. But the key idea is that you can call the solvers from Sundials.jl, a different package, using
sol = solve(prob,CVODE_BDF())
Therefore, this "solve(prob,alg)" is syntax which is "packageindependent" (the restrictions will be explained in a later section). What we can then do is write our algorithm at a very generic level with "put ODE solver here", and the user could then use any ODE solver package which implements the common interface.
Here's what this looks like. We want to take in an ODEProblem like defined above (assume it's the problem which defines our rabbit/wolves example), the timepoints for which we have data at, the array of data values, and the common interface algorithm to solve the ODE. The resulting code is very simple:
function build_optim_objective(prob::AbstractODEProblem,t,data,alg;loss_func = L2DistLoss,kwargs...)
f = prob.f
cost_function = function (p)
for i in eachindex(f.params)
setfield!(f,f.params[i],p[i])
end
sol = solve(prob,alg;kwargs...)
y = vecvec_to_mat(sol(t))
norm(value(loss_func(),vec(y),vec(data)))
end
end
In takes in what I said and spits out a function which:
1. Sets the model to have the parameters
2. Solves the differential equation with the chosen algorithm
3. Computes the loss function using LossFunctions.jl
We can then use this as described in the DifferentialEquations.jl manual with any ODE solver on the common interface by plugging it into Optim.jl or whatever optimization package you want, and can change the loss function to any that's provided by LossFunctions.jl. Now any package which follows these interfaces can be directly used as a substitute without users having to reinvent this function (you can even use your own ODE solver).
(Note this algorithm still needs to be improved. For example, if an ODE solver errors, which likely happens if the parameters enter a bad range, it should still give a (high) loss value. Also, this implementation assumes the solver implements the "dense" continuous output, but this can be eased to something else. Still, this algorithm in its current state is a nice simple example!)
How does the common interface work?
I believe I've convinced you by now that this common interface is kind of cool. One question you might have is, "I am not really a programmer but I have a special method for my special differential equation. My special method probably doesn't exist in a package. Can I use this for parameter estimation?". The answer is yes! Let me explain how.
All of these organizations (JuliaDiffEq, JuliaML, JuliaOpt, etc.) have some kind of "Base" library which has the functions which are extended. In JuliaDiffEq, there's DiffEqBase.jl. DiffEqBase defines the method "solve". Each other the component packages (OrdinaryDiffEq.jl, Sundials.jl, ODE.jl, ODEInterface.jl, and recently LSODA.jl) import DiffEqBase:
import DiffEqBase: solve
and add a new dispatch to "solve". For example, in Sundials.jl, it defines (and exports) the algorithm types (for example, CVODE_BDF) all as subtypes of SundialsODEAlgorithm, like:
# Abstract Types
abstract SundialsODEAlgorithm{Method,LinearSolver} <: AbstractODEAlgorithm
# ODE Algorithms
immutable CVODE_BDF{Method,LinearSolver} <: SundialsODEAlgorithm{Method,LinearSolver}
# Extra details not necessary!
end
and then it defines a dispatch to solve:
function solve{uType,tType,isinplace,F,Method,LinearSolver}(
prob::AbstractODEProblem{uType,tType,isinplace,F},
alg::SundialsODEAlgorithm{Method,LinearSolver},args...;kwargs...)
...
where I left out the details on the extra arguments. What this means is that now
sol = solve(prob,CVODE_BDF())
which points to the method defined here in the Sundials.jl package. Since Julia is cool and compiles all of the specializations based off of the given types, this will compile to have no overhead, giving us the benefit of the common API with essentially no cost! All that Sundials.jl has to do is make sure that its resulting Solution type acts according to the common interface's specifications and then this package can seamlessly be used in any place where an ODE solver call is used.
So yes, if you import DiffEqBase and put this interface on your ODE solver, parameter estimation from DiffEqParamEstim.jl will "just work". Sensitivity analysis from DiffEqSensitivity.jl will "just work". If the callbacks section of the API is implemented, the coming (still in progress) uncertainty quantification from DiffEqUncertainty.jl will "just work". Any code written to the common interface doesn't actually care what package was used to solve the differential equation, it just wants a solution type to come back and be used the way the interface dictates!
Summary of what just happened!
This means that if domains of scientific computing in Julia conform to common interfaces, not only do users need to learn only one API, but that API can be used to write modular code where users can stick in any package/method/implementation they like, without even affecting the performance.
Where do we go from here?
Okay, so I've probably convinced you that these common interfaces are extremely powerful. What is their current state? Well, let's look the workhorse algorithms of scientific computing:
1. Optimization
2. Numerical Differential Equations
3. Machine Learning
4. Plotting
5. Linear Solving (Ax=b)
6. Nonlinear Solving (F(x)=0, find x)
Almost every problem in scientific computing seems to boil down to repeated application of these algorithms, so if one could write all of these in a modular fashion on common APIs like above, then the level of control users would have over algorithms from packages would be unprecedented: no other language will have ever been this flexible and achieve this high of performance!
One use case is as follows: say your research is in numerical linear algebra, and you've developed some new multigrid method with helps speed up Rosenbrock ODE solvers. If the ODE solver took in an argument which lets you specify by a linear solver interface how to solve the linear equation Ax=b, you'd be able to implement your linear solver inside of the Rosenbrock solver from OrdinaryDiffEq.jl by passing it in like Rosenbrock23(linear_solver=MyLinearSolver()). Even better, since this ODE solver is on the common interface, this algorithm could then be used in the parameter estimation scheme described above. This means that you can simply make a linear solver, and instantly be able to test how it helps speed up parameter estimation for ODEs with only ~10 lines of code, without sacrificing performance! Lastly, since the same code is used for everywhere except the choice of the linear solver, this is trivial to benchmark. You can just lineup and test every possibility with almost no work put in:
algs = [Rosenbrock23(linear_solver=MyLinearSolver());
Rosenbrock23(linear_solver=qrfact());
Rosenbrock23(linear_solver=lufact());
...
]
for alg in algs
sol = solve(prob,alg)
@time sol = solve(prob,alg)
end
and tada you're benchmarking every linear solver on the same ODE using the Rosenbrock23 method (this is what DiffEqBenchmarks.jl does to test all of the ODE solvers).
Are we there yet? No, but we're getting there. Let's look organization by organization.
JuliaOpt
JuliaOpt provides a common interface for optimization via MathProgBase.jl. This interface is very mature. However, Optim.jl doesn't seem to conform to it, so there may be some updates happening in the near future. However, this is already usable, and as you can see from JuMP's documentation, you can already call many different solver methods (including commercial solvers) with the same codes
JuliaDiffEq
JuliaDiffEq is rather new, but it has rapidly developed and now it offers a common API which has been shown in this blog post and is documented at DiffEqDocs.jl with developer documentation at DiffEqDevDocs.jl.
DifferentialEquations.jl no longer directly contains any solver packages. Instead, its ODE solvers were moved to OrdinaryDiffEq.jl, its SDE (stochastic differential equation) solvers were moved to StochasticDiffEq.jl, etc. (I think you get the naming scheme!), with the parts for the common API moved to DiffEqBase.jl. DifferentialEquations.jl is now a metapackage which pulls the common interface packages into one convenient metapackage, and the components can thus be used directly if users/developers please. In addition, this means that any one could add their own methods to solve an ODEProblem, DAEProblem, etc. by importing DiffEqBase and implementing a solve method. The number of packages which have implemented this interface is rapidly increasing, even to packages outside of JuliaDiffEq. Since I have the reins here, expect this common interface to be well supported.
JuliaML
JuliaML is still under heavy development (think of it currently as a "developer preview"), but its structure is created in this same common API / modular format. You can expect Learn.jl to be a metapackage like DifferentialEquations.jl in the near future, and the ecosystem to have a common interface where you can mix and match all of the details for machine learning. It will be neat!
JuliaPlots
Plots.jl (technically not in JuliaPlots, but ignore that) implements a common interface for plotting using many different plotting packages as "backends". Through its recipes system, owners of other packages can extend Plots.jl so that way it has "nice interactions". For example, the solutions from DiffEqBase.jl have plot recipes so that way the command
plot(sol)
plots the solution, and using Plots.jl's backend controls, this works with a wide variety of packages like GR, PyPlot, Plotly, etc. This is really welldesigned and already very mature.
Linear Solving
We are close to having a common API for linear solving. The reason is because Base uses the type system to dispatch on \. For example, to solve Ax=b by LUfactorization, one uses the command:
K = lufact(A)
x = K\b
and to solve using QRfactorization, one instead would use
K = qrfact(A)
x = K\b
Thus what one could do is take in a function for a factorization type and use that:
function test_solve(linear_solve)
K = linear_solve(A)
x = K\b
end
Then the user can pass in whatever method they think is best, or implement their own linear_solve(A) which makes a type and has a dispatch on \ to give the solution via their method.
Fortunately, this even works with PETSc.jl:
K = KSP(A, ksp_type="gmres", ksp_rtol=1e6)
x = K\b
so in test_solve, one could make the user do:
test_solve((A)>KSP(A, ksp_type="gmres", ksp_rtol=1e6))
(since linear_solve is supposed to be a function of the matrix, so use an anonymous function to make a closure have all of the right arguments). However, this setup is not currently compatible with IterativeSolvers.jl. I am on a mission to make it happen, and have opened up issues and a discussion on Discourse. Feel free to chime in with your ideas. I am not convinced this anonymous function / closure API is nice to use.
Nonlinear Solving
There is no common interface in Julia for nonlinear solving. You're back to the old way of doing it right now. Currently I know of 3 good packages for solving nonlinear equations F(x)=0:
1. NLsolve.jl
2. Roots.jl
3. Sundials.jl (KINSOL)
NLsolve.jl and Roots.jl are native Julia methods and support things like higher precision numbers, while Sundials.jl only supports Float64 and tends to be faster (it's been being optimized for ages). In this state, you have to choose which package to use and stick with it. If this had a common interface, then users could pass in the method for nonlinear solving and it could "just work" like how the differential equations stack works. I opened up a discussion on Discourse to try and get this going!
Conclusion
These common interfaces which we are seeing develop in many different places in the Julia package ecosystem are not just nice for users, but they are also incredibly powerful for algorithm developers and researchers. I have shown that this insane amount of flexibility allows one to choose the optimal method for every mathematical detail of the problem, without performance costs. I hope we soon see the scientific computing stack all written in this manner. The end result would be that, in order to test how your research affects some large realworld problem, you simply plug your piece into the modular structure and let it run.
That is the power of multiple dispatch.
The post Modular Algorithms for Scientific Computing in Julia appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/modularalgorithmsscientificcomputingjulia/feed/
3
390

7 Julia Gotchas and How to Handle Them
http://www.stochasticlifestyle.com/7juliagotchashandle/
http://www.stochasticlifestyle.com/7juliagotchashandle/#comments
Tue, 04 Oct 2016 00:50:56 +0000
http://www.stochasticlifestyle.com/?p=362
Let me start by saying Julia is a great language. I love the language, it is what I find to be the most powerful and intuitive language that I have ever used. It's undoubtedly my favorite language. That said, there are some "gotchas", tricky little things you need to know about. Every language has them, and one of the first things you have to do in order to master a language is to find out what they are and how to avoid them. The point of this blog post is to help accelerate this process for you by exposing some of the most common "gotchas" offering alternative programming practices.
Julia is a good language for understanding what's going on because there's no magic. The Julia developers like to have clearly defined rules for how things act. This means that all behavior ... READ MORE
The post 7 Julia Gotchas and How to Handle Them appeared first on Stochastic Lifestyle.
]]>
Let me start by saying Julia is a great language. I love the language, it is what I find to be the most powerful and intuitive language that I have ever used. It's undoubtedly my favorite language. That said, there are some "gotchas", tricky little things you need to know about. Every language has them, and one of the first things you have to do in order to master a language is to find out what they are and how to avoid them. The point of this blog post is to help accelerate this process for you by exposing some of the most common "gotchas" offering alternative programming practices.
Julia is a good language for understanding what's going on because there's no magic. The Julia developers like to have clearly defined rules for how things act. This means that all behavior can be explained. However, this might mean that you need to think about what's going on to understand why something is happening. That's why I'm not just going to lay out some common issues, but I am also going to explain why they occur. You will see that there are some very similar patterns, and once you catch onto the patterns, you will not fall for any of these anymore. Because of this, there's a slightly higher learning curve for Julia over the simpler languages like MATLAB/R/Python. However, once you get the hang of this, you will fully be able to use the conciseness of Julia while obtaining the performance of C/Fortran. Let's dig in.
Gotcha #1: The REPL (terminal) is the Global Scope
For anyone who is familiar with the Julia community, you know that I have to start here. This is by far the most common problem reported by new users of Julia. Someone will go "I heard Julia is fast!", open up the REPL, quickly code up some algorithm they know well, and execute that script. After it's executed they look at the time and go "wait a second, why is this as slow as Python?"
Because this is such an important issue and pervasive, let's take some extra time delving into why this happens so we understand how to avoid it.
Small Interlude into Why Julia is Fast
To understand what just happened, you have to understand that Julia is about not just code compilation, but also typespecialization (i.e. compiling code which is specific to the given types). Let me repeat: Julia is not fast because the code is compiled using a JIT compiler, rather it is fast because typespecific code is compiled and ran.
If you want the full story, checkout some of the notes I've written for an upcoming workshop. I am going to summarize the necessary parts which are required to understand why this is such a big deal.
Typespecificity is given by Julia's core design principle: multiple dispatch. When you write the code:
function f(a,b)
return 2a+b
end
you may have written only one "function", but you have written a very large amount of "methods". In Julia parlance, a function is an abstraction and what is actually called is a method. If you call f(2.0,3.0), then Julia will run a compiled code which takes in two floating point numbers and returns the value 2a+b. If you call f(2,3), then Julia will run a different compiled code which takes in two integers and returns the value 2a+b. The function f is an abstraction or a shorthand for the multitude of different methods which have the same form, and this design of using the symbol "f" to call all of these different methods is called multiple dispatch. And this goes all the way down: the + operator is actually a function which will call methods depending on the types it sees.
Julia actually gets its speed is because this compiled code knows its types, and so the compiled code that f(2.0,3.0) calls is exactly the compiled code that you would get by defining the same C/Fortran function which takes in floating point numbers. You can check this with the @code_native macro to see the compiled assembly:
@code_native f(2.0,3.0)
# This prints out the following:
pushq %rbp
movq %rsp, %rbp
Source line: 2
vaddsd %xmm0, %xmm0, %xmm0
vaddsd %xmm1, %xmm0, %xmm0
popq %rbp
retq
nop
This is the same compiled assembly you would expect from the C/Fortran function, and it is different than the assembly code for integers:
@code_native f(2,3)
pushq %rbp
movq %rsp, %rbp
Source line: 2
leaq (%rdx,%rcx,2), %rax
popq %rbp
retq
nopw (%rax,%rax)
The Main Point: The REPL/Global Scope Does Not Allow Type Specificity
This brings us to the main point: The REPL / Global Scope is slow because it does not allow type specification. First of all, notice that the REPL is the global scope because Julia allows nested scoping for functions. For example, if we define
function outer()
a = 5
function inner()
return 2a
end
b = inner()
return 3a+b
end
you will see that this code works. This is because Julia allows you to grab the "a" from the outer function into the inner function. If you apply this idea recursively, then you understand the highest scope is the scope which is directly the REPL (which is the global scope of a module Main). But now let's think about how a function will compile in this situation. Let's do the same case as before, but using the globals:
a=2.0; a=3.0
function linearcombo()
return 2a+b
end
ans = linearcombo()
a = 2; b = 3
ans2= linearcombo()
Question: What types should the compiler assume "a" and "b" are? Notice that in this example we changed the types and still called the same function. In order for this compiled C function to not segfault, it needs to be able to deal with whatever types we throw at it: floats, ints, arrays, weird userdefined types, etc. In Julia parlance, this means that the variables have to be "boxed", and the types are checked with every use. What do you think that compiled code looks like?
pushq %rbp
movq %rsp, %rbp
pushq %r15
pushq %r14
pushq %r12
pushq %rsi
pushq %rdi
pushq %rbx
subq $96, %rsp
movl $2147565792, %edi # imm = 0x800140E0
movabsq $jl_get_ptls_states, %rax
callq *%rax
movq %rax, %rsi
leaq 72(%rbp), %r14
movq $0, 88(%rbp)
vxorps %xmm0, %xmm0, %xmm0
vmovups %xmm0, 72(%rbp)
movq $0, 56(%rbp)
movq $10, 104(%rbp)
movq (%rsi), %rax
movq %rax, 96(%rbp)
leaq 104(%rbp), %rax
movq %rax, (%rsi)
Source line: 3
movq pcre2_default_compile_context_8(%rdi), %rax
movq %rax, 56(%rbp)
movl $2154391480, %eax # imm = 0x806967B8
vmovq %rax, %xmm0
vpslldq $8, %xmm0, %xmm0 # xmm0 = zero,zero,zero,zero,zero,zero,zero,zero,xmm0[0,1,2,3,4,5,6,7]
vmovdqu %xmm0, 80(%rbp)
movq %rdi, 64(%rbp)
movabsq $jl_apply_generic, %r15
movl $3, %edx
movq %r14, %rcx
callq *%r15
movq %rax, %rbx
movq %rbx, 88(%rbp)
movabsq $586874896, %r12 # imm = 0x22FB0010
movq (%r12), %rax
testq %rax, %rax
jne L198
leaq 98096(%rdi), %rcx
movabsq $jl_get_binding_or_error, %rax
movl $122868360, %edx # imm = 0x752D288
callq *%rax
movq %rax, (%r12)
L198:
movq 8(%rax), %rax
testq %rax, %rax
je L263
movq %rax, 80(%rbp)
addq $5498232, %rdi # imm = 0x53E578
movq %rdi, 72(%rbp)
movq %rbx, 64(%rbp)
movq %rax, 56(%rbp)
movl $3, %edx
movq %r14, %rcx
callq *%r15
movq 96(%rbp), %rcx
movq %rcx, (%rsi)
addq $96, %rsp
popq %rbx
popq %rdi
popq %rsi
popq %r12
popq %r14
popq %r15
popq %rbp
retq
L263:
movabsq $jl_undefined_var_error, %rax
movl $122868360, %ecx # imm = 0x752D288
callq *%rax
ud2
nopw (%rax,%rax)
For dynamic languages without typespecialization, this bloated code with all of the extra instructions is as good as you can get, which is why Julia slows down to their speed.
To understand why this is a big deal, notice that every single piece of code that you write in Julia is compiled. So let's say you write a loop in your script:
a = 1
for i = 1:100
a += a + f(a)
end
The compiler has to compile that loop, but since it cannot guarantee the types do not change, it conservatively gives that nasty long code, leading to slow execution.
How to Avoid the Issue
There are a few ways to avoid this issue. The simplest way is to always wrap your scripts in functions. For example, with the previous code we can do:
function geta(a)
# can also just define a=1 here
for i = 1:100
a += a + f(a)
end
return a
end
a = geta(1)
This will give you the same output, but since the compiler is able to specialize on the type of a, it will give the performant compiled code that you want. Another thing you can do is define your variables as constants.
const b = 5
By doing this, you are telling the compiler that the variable will not change, and thus it will be able to specialize all of the code which uses it on the type that it currently is. There's a small quirk that Julia actually allows you to change the value of a constant, but not the type. Thus you can use "const" as a way to tell the compiler that you won't be changing the type and speed up your codes. However, note that there are some small quirks that come up since you guaranteed to the compiler the value won't change. For example:
const a = 5
f() = a
println(f()) # Prints 5
a = 6
println(f()) # Prints 5
this does not work as expected because the compiler, realizing that it knows the answer to "f()=a" (since a is a constant), simply replaced the function call with the answer, giving different behavior than if a was not a constant.
This is all just one big way of saying: Don't write your scripts directly in the REPL, always wrap them in a function.
Let's hit one related point as well.
Gotcha #2: TypeInstabilities
So I just made a huge point about how specializing code for the given types is crucial. Let me ask a quick question, what happens when your types can change?
If you guessed "well, you can't really specialize the compiled code in that case either", then you are correct. This kind of problem is known as a typeinstability. These can show up in many different ways, but one common example is that you initialize a value in a way that is easy, but not necessarily that type that it should be. For example, let's look at:
function g()
x=1
for i = 1:10
x = x/2
end
return x
end
Notice that "1/2" is a floating point number in Julia. Therefore it we started with "x=1", it will change types from an integer to a floating point number, and thus the function has to compile the inner loop as though it can be either type. If we instead had the function:
function h()
x=1.0
for i = 1:10
x = x/2
end
return x
end
then the whole function can optimally compile knowing x will stay a floating point number (this ability for the compiler to judge types is known as type inference). We can check the compiled code to see the difference:
pushq %rbp
movq %rsp, %rbp
pushq %r15
pushq %r14
pushq %r13
pushq %r12
pushq %rsi
pushq %rdi
pushq %rbx
subq $136, %rsp
movl $2147565728, %ebx # imm = 0x800140A0
movabsq $jl_get_ptls_states, %rax
callq *%rax
movq %rax, 152(%rbp)
vxorps %xmm0, %xmm0, %xmm0
vmovups %xmm0, 80(%rbp)
movq $0, 64(%rbp)
vxorps %ymm0, %ymm0, %ymm0
vmovups %ymm0, 128(%rbp)
movq $0, 96(%rbp)
movq $18, 144(%rbp)
movq (%rax), %rcx
movq %rcx, 136(%rbp)
leaq 144(%rbp), %rcx
movq %rcx, (%rax)
movq $0, 88(%rbp)
Source line: 4
movq %rbx, 104(%rbp)
movl $10, %edi
leaq 477872(%rbx), %r13
leaq 10039728(%rbx), %r15
leaq 8958904(%rbx), %r14
leaq 64(%rbx), %r12
leaq 10126032(%rbx), %rax
movq %rax, 160(%rbp)
nopw (%rax,%rax)
L176:
movq %rbx, 128(%rbp)
movq 8(%rbx), %rax
andq $16, %rax
movq %r15, %rcx
cmpq %r13, %rax
je L272
movq %rbx, 96(%rbp)
movq 160(%rbp), %rcx
cmpq $2147419568, %rax # imm = 0x7FFF05B0
je L272
movq %rbx, 72(%rbp)
movq %r14, 80(%rbp)
movq %r12, 64(%rbp)
movl $3, %edx
leaq 80(%rbp), %rcx
movabsq $jl_apply_generic, %rax
vzeroupper
callq *%rax
movq %rax, 88(%rbp)
jmp L317
nopw %cs:(%rax,%rax)
L272:
movq %rcx, 120(%rbp)
movq %rbx, 72(%rbp)
movq %r14, 80(%rbp)
movq %r12, 64(%rbp)
movl $3, %r8d
leaq 80(%rbp), %rdx
movabsq $jl_invoke, %rax
vzeroupper
callq *%rax
movq %rax, 112(%rbp)
L317:
movq (%rax), %rsi
movl $1488, %edx # imm = 0x5D0
movl $16, %r8d
movq 152(%rbp), %rcx
movabsq $jl_gc_pool_alloc, %rax
callq *%rax
movq %rax, %rbx
movq %r13, 8(%rbx)
movq %rsi, (%rbx)
movq %rbx, 104(%rbp)
Source line: 3
addq $1, %rdi
jne L176
Source line: 6
movq 136(%rbp), %rax
movq 152(%rbp), %rcx
movq %rax, (%rcx)
movq %rbx, %rax
addq $136, %rsp
popq %rbx
popq %rdi
popq %rsi
popq %r12
popq %r13
popq %r14
popq %r15
popq %rbp
retq
nop
Verses:
pushq %rbp
movq %rsp, %rbp
movabsq $567811336, %rax # imm = 0x21D81D08
Source line: 6
vmovsd (%rax), %xmm0 # xmm0 = mem[0],zero
popq %rbp
retq
nopw %cs:(%rax,%rax)
Notice how many fewer computational steps are required to compute the same value!
How to Find and Deal with TypeInstabilities
At this point you might ask, "well, why not just use C so you don't have to try and find these instabilities?" The answer is:
 They are easy to find
 They can be useful
 You can handle necessary instabilities with function barriers
How to Find TypeInstabilities
Julia gives you the macro @code_warntype to show you where type instabilities are. For example, if we use this on the "g" function we created:
@code_warntype g()
Variables:
#self#::#g
x::ANY
#temp#@_3::Int64
i::Int64
#temp#@_5::Core.MethodInstance
#temp#@_6::Float64
Body:
begin
x::ANY = 1 # line 3:
SSAValue(2) = (Base.select_value)((Base.sle_int)(1,10)::Bool,10,(Base.box)(Int64,(Base.sub_int)(1,1)))::Int64
#temp#@_3::Int64 = 1
5:
unless (Base.box)(Base.Bool,(Base.not_int)((#temp#@_3::Int64 === (Base.box)(Int64,(Base.add_int)(SSAValue(2),1)))::Bool)) goto 30
SSAValue(3) = #temp#@_3::Int64
SSAValue(4) = (Base.box)(Int64,(Base.add_int)(#temp#@_3::Int64,1))
i::Int64 = SSAValue(3)
#temp#@_3::Int64 = SSAValue(4) # line 4:
unless (Core.isa)(x::UNION{FLOAT64,INT64},Float64)::ANY goto 15
#temp#@_5::Core.MethodInstance = MethodInstance for /(::Float64, ::Int64)
goto 24
15:
unless (Core.isa)(x::UNION{FLOAT64,INT64},Int64)::ANY goto 19
#temp#@_5::Core.MethodInstance = MethodInstance for /(::Int64, ::Int64)
goto 24
19:
goto 21
21:
#temp#@_6::Float64 = (x::UNION{FLOAT64,INT64} / 2)::Float64
goto 26
24:
#temp#@_6::Float64 = $(Expr(:invoke, :(#temp#@_5), :(Main./), :(x::Union{Float64,Int64}), 2))
26:
x::ANY = #temp#@_6::Float64
28:
goto 5
30: # line 6:
return x::UNION{FLOAT64,INT64}
end::UNION{FLOAT64,INT64}
Notice that it tells us at the top that the type of x is "ANY". It will capitalize any type which is not inferred as a "strict type", i.e. it is an abstract type which needs to be boxed/checked at each step. We see that at the end we return x as a "UNION{FLOAT64,INT64}", which is another nonstrict type. This tells us that the type of x changed, causing the difficulty. If we instead look at the @code_warntype for h, we get all strict types:
@code_warntype h()
Variables:
#self#::#h
x::Float64
#temp#::Int64
i::Int64
Body:
begin
x::Float64 = 1.0 # line 3:
SSAValue(2) = (Base.select_value)((Base.sle_int)(1,10)::Bool,10,(Base.box)(Int64,(Base.sub_int)(1,1)))::Int64
#temp#::Int64 = 1
5:
unless (Base.box)(Base.Bool,(Base.not_int)((#temp#::Int64 === (Base.box)(Int64,(Base.add_int)(SSAValue(2),1)))::Bool)) goto 15
SSAValue(3) = #temp#::Int64
SSAValue(4) = (Base.box)(Int64,(Base.add_int)(#temp#::Int64,1))
i::Int64 = SSAValue(3)
#temp#::Int64 = SSAValue(4) # line 4:
x::Float64 = (Base.box)(Base.Float64,(Base.div_float)(x::Float64,(Base.box)(Float64,(Base.sitofp)(Float64,2))))
13:
goto 5
15: # line 6:
return x::Float64
end::Float64
Indicating that this function is type stable and will compile to essentially optimal C code. Thus typeinstabilities are not hard to find. What's harder is to find the right design.
Why Allow TypeInstabilities?
This is an age old question which has lead to dynamicallytyped languages dominating the scripting language playing field. The idea is that, in many cases you want to make a tradeoff between performance and robustness. For example, you may want to read a table from a webpage which has numbers all mixed together with integers and floating point numbers. In Julia, you can write your function such that if they were all integers, it will compile well, and if they were all floating point numbers, it will also compile well. And if they're mixed? It will still work. That's the flexibility/convenience we know and love from a language like Python/R. But Julia will explicitly tell you (via @code_warntype) when you are making this performance tradeoff.
How to Handle TypeInstabilities
There are a few ways to handle typeinstabilities. First of all, if you like something like C/Fortran where your types are declared and can't change (thus ensuring typestability), you can do that in Julia. You can declare your types in a function with the following syntax:
local a::Int64 = 5
This makes "a" an 64bit integer, and if future code tries to change it, an error will be thrown (or a proper conversion will be done. But since the conversion will not automatically round, it will most likely throw errors). Sprinkle these around your code and you will get type stability the C/Fortran way.
A less heavy handed way to handle this is with typeassertions. This is where you put the same syntax on the other side of the equals sign. For example:
a = (b/c)::Float64
This says "calculate b/c, and make sure that the output is a Float64. If it's not, try to do an autoconversion. If it can't easily convert, throw an error". Putting these around will help you make sure you know the types which are involved.
However, there are cases where type instabilities are necessary. For example, let's say you want to have a robust code, but the user gives you something crazy like:
arr = Vector{Union{Int64,Float64},2}(4)
arr[1]=4
arr[2]=2.0
arr[3]=3.2
arr[4]=1
which is a 4x4 array of both integers and floating point numbers. The actual element type for the array is "Union{Int64,Float64}" which we saw before was a nonstrict type which can lead to issues. The compiler only knows that each value can be either an integer or a floating point number, but not which element is which type. This means that naively performing arithmetic on this array, like:
function foo{T,N}(array::Array{T,N})
for i in eachindex(array)
val = array[i]
# do algorithm X on val
end
end
will be slow since the operations will be boxed.
However, we can use multipledispatch to run the codes in a typespecialized manner. This is known as using function barriers. For example:
function inner_foo{T<:Number}(val::T)
# Do algorithm X on val
end
function foo2{T,N}(array::Array{T,N})
for i in eachindex(array)
inner_foo(array[i])
end
end
Notice that because of multipledispatch, calling inner_foo either calls a method specifically compiled for floating point numbers, or a method specifically compiled for integers. In this manner, you can put a long calculation inside of inner_foo and still have it perform well do to the strict typing that the function barrier gives you.
Thus I hope you see that Julia offers a good mixture between the performance of strict typing and the convenience of dynamic typing. A good Julia programmer gets to have both at their disposal in order to maximize performance and/or productivity when necessary.
Gotcha #3: Eval Runs at the Global Scope
One last typing issue: eval. Remember this: eval runs at the global scope.
One of the greatest strengths of Julia is its metaprogramming capabilities. This allows you to effortlessly write code which generates code, effectively reducing the amount of code you have to write and maintain. Macro is a function which runs at compile time and (usually) spits out code. For example:
macro defa()
:(a=5)
end
will replace any instance of "@defa" with the code "a=5" (":(a=5)" is the quoted expression for "a=5". Julia code is all expressions, and thus metaprogramming is about building Julia expressions). You can use this to build any complex Julia program you wish, and put it in a function as a type of really clever shorthand.
However, sometimes you may need to directly evaluate the generated code. Julia gives you the "eval" function or the "@eval" macro for doing so. In general, you should try to avoid eval, but there are some codes where it's necessary, like my new library for transferring data between different processes for parallel programming. However, note that if you do use it:
@eval :(a=5)
then this will evaluate at the global scope (the REPL). Thus all of the associated problems will occur. However, the fix is the same as the fixes for globals / type instabilities. For example:
function testeval()
@eval :(a=5)
return 2a+5
end
will not give a good compiled code since "a" was essentially declared at the REPL. But we can use the tools from before to fix this. For example, we can bring the global in and assert a type to it:
function testeval()
@eval :(a=5)
b = a::Int64
return 2b+5
end
Here "b" is a local variable, and the compiler can infer that its type won't change and thus we have typestability and are living in good performance land. So dealing with eval isn't difficult, you just have to remember it works at the REPL.
That's the last of the gotcha's related to typeinstability. You can see that there's a very common thread for why it occurs and how to handle them.
Gotcha #4: How Expressions Break Up
This is one that got me for awhile at first. In Julia, there are many cases where expressions will continue if they are not finished. For this reason linecontinuation operators are not necessary: Julia will just read until the expression is finished.
Easy rule, right? Just make sure you remember how functions finish. For example:
a = 2 + 3 + 4 + 5 + 6 + 7
+8 + 9 + 10+ 11+ 12+ 13
looks like it will evaluate to 90, but instead it gives 27. Why? Because "a = 2 + 3 + 4 + 5 + 6 + 7" is a complete expression, so it will make "a=27" and then skip over the nonsense "+8 + 9 + 10+ 11+ 12+ 13". To continue the line, we instead needed to make sure the expression wasn't complete:
a = 2 + 3 + 4 + 5 + 6 + 7 +
8 + 9 + 10+ 11+ 12+ 13
This will make a=90 as we wanted. This might trip you up the first time, but then you'll get used to it.
The more difficult issue dealing with array definitions. For example:
x = rand(2,2)
a = [cos(2*pi.*x[:,1]).*cos(2*pi.*x[:,2])./(4*pi) sin(2.*x[:,1]).*sin(2.*x[:,2])./(4)]
b = [cos(2*pi.*x[:,1]).*cos(2*pi.*x[:,2])./(4*pi)  sin(2.*x[:,1]).*sin(2.*x[:,2])./(4)]
at glance you might think a and b are the same, but they are not! The first will give you a (2,2) matrix, while the second is a (1dimensional) vector of size 2. To see what the issue is, here's a simpler version:
a = [1 2]
b = [1  2]
In the first case there are two numbers: "1" and "2". In the second there is an expression: "12" (which is evaluated to give the array [1]). This is because of the special syntax for array definitions. It's usually really lovely to write:
a = [1 2 3 4
2 3 1 4]
and get the 2x4 matrix that you'd expect. However, this is the tradeoff that occurs. However, this issue is also easy to avoid: instead of concatenating using a space (i.e. in a whitespacesensitive manner), instead use the "hcat" function:
a = hcat(cos(2*pi.*x[:,1]).*cos(2*pi.*x[:,2])./(4*pi),sin(2.*x[:,1]).*sin(2.*x[:,2])./(4))
Problem solved!
Gotcha #5: Views, Copy, and Deepcopy
One way in which Julia gets good performance is by working with "views". An "Array" is actually a "view" to the contiguous blot of memory which is used to store the values. The "value" of the array is its pointer to the memory location (and its type information). This gives (and useful) interesting behavior. For example, if we run the following code:
a = [3;4;5]
b = a
b[1] = 1
then at the end we will have that "a" is the array "[1;4;5]", i.e. changing "b" changes "a". The reason is "b=a" set the value of "b" to the value of "a". Since the value of an array is its pointer to the memory location, what "b" actually gets is not a new array, rather it gets the pointer to the same memory location (which is why changing "b" changes "a").
This is very useful because it also allows you to keep the same array in many different forms. For example, we can have both a matrix and the vector form of the matrix using:
a = rand(2,2) # Makes a random 2x2 matrix
b = vec(a) # Makes a view to the 2x2 matrix which is a 1dimensional array
Now "b" is a vector, but changing "b" still changes "a", where "b" is indexed by reading down the columns. Notice that this whole time, no arrays have been copied, and therefore these operations have been excessively cheap (meaning, there's no reason to avoid them in performance sensitive code).
Now some details. Notice that the syntax for slicing an array will create a copy when on the righthand side. For example:
c = a[1:2,1]
will create a new array, and point "c" to that new array (thus changing "c" won't change "a"). This can be necessary behavior, however note that copying arrays is an expensive operation that should be avoided whenever possible. Thus we would instead create more complicated views using:
d = @view a[1:2,1]
e = view(a,1:2,1)
Both "d" and "e" are the same thing, and changing either "d" or "e" will change "a" because both will not copy the array, just make a new variable which is a Vector that only points to the first column of "a". (Another function which creates views is "reshape" which lets you reshape an array.)
If this syntax is on the lefthand side, then it's a view. For example:
a[1:2,1] = [1;2]
will change "a" because, on the lefthand side, "a[1:2,1]" is the same as "view(a,1:2,1)" which points to the same memory as "a".
What if we need to make copies? Then we can use the copy function:
b = copy(a)
Now since "b" is a copy of "a" and not a view, changing "b" will not change "a". If we had already defined "a", there's a handy inplace copy "copy!(b,a)" which will essentially loop through and write the values of "a" to the locations of "a" (but this requires that "b" is already defined and is the right size).
But now let's make a slightly more complicated array. For example, let's make a "Vector{Vector}":
a = Vector{Vector{Float64}}(2)
a[1] = [1;2;3]
a[2] = [4;5;6]
Each element of "a" is a vector. What happens when we copy a?
b = copy(a)
b[1][1] = 10
Notice that this will change a[1][1] to 10 as well! Why did this happen? What happened is we used "copy" to copy the values of "a". But the values of "a" were arrays, so we copied the pointers to memory locations over to "b", so "b" actually points to the same arrays. To fix this, we instead use "deepcopy":
b = deepcopy(a)
This recursively calls copy in such a manner that we avoid this issue. Again, the rules of Julia are very simple and there's no magic, but sometimes you need to pay closer attention.
Gotcha #6: Temporary Allocations, Vectorization, and InPlace Functions
In MATLAB/Python/R, you're told to use vectorization. In Julia you might have heard that "devectorized code is better". I wrote about this part before so I will refer back to my previous post which explains why vectorized codes give "temporary allocations" (i.e. they make middleman arrays which aren't needed, and as noted before, array allocations are expensive and slow down your code!).
For this reason, you will want to fuse your vectorized operations and write them inplace in order to avoid allocations. What do I mean by inplace? An inplace function is one that updates a value instead of returning a value. If you're going to continually operate on an array, this will allow you to keep using the same array, instead of creating new arrays each iteration. For example, if you wrote:
function f()
x = [1;5;6]
for i = 1:10
x = x + inner(x)
end
return x
end
function inner(x)
return 2x
end
then each time inner is called, it will create a new array to return "2x" in. Clearly we don't need to keep making new arrays. So instead we could have a cache array "y" which will hold the output like so:
function f()
x = [1;5;6]
y = Vector{Int64}(3)
for i = 1:10
inner(y,x)
for i in 1:3
x[i] = x[i] + y[i]
end
copy!(y,x)
end
return x
end
function inner!(y,x)
for i=1:3
y[i] = 2*x[i]
end
nothing
end
Let's dig into what's happening here. "inner!(y,x)" doesn't return anything, but it changes "y". Since "y" is an array, the value of "y" is the pointer to the actual array, and since in the function those values were changed, "inner!(y,x)" will have "silently" changed the values of "y". Functions which do this are called inplace. They are usually denoted with a "!", and usually change the first argument (this is just by convention). So there is no array allocation when "inner!(y,x)" is called.
In the same way, "copy!(y,x)" is an inplace function which writes the values of "x" to "y", updating it. As you can see, this means that every operation only changes the values of the arrays. Only two arrays are ever created: the initial array for "x" and the initial array for "y". The first function created a new array every since time "x + inner(x)" was called, and thus 11 arrays were created in the first function. Since array allocations are expensive, the second function will run faster than the first function.
It's nice that we can get fast, but the syntax bloated a little when we had to write out the loops. That's where loopfusion comes in. In Julia v0.5, you can now use the "." symbol to vectorize any function (also known as broadcasting because it is actually calling the "broadcast" function). While it's cool that "f.(x)" is the same thing as applying "f" to each value of "x", what's cooler is that the loops fuse. If you just applied "f" to "x" and made a new array, then "x=x+f.(x)" would have a copy. However, what we can instead do is designate everything as array functions:
x .= x .+ f.(x)
The ".=" will do elementwise equals, so this will essentially turn be the code
for i = 1:length(x)
x[i] = x[i] + f(x[i])
end
which is the allocationfree loop we wanted! Thus another way to write our function
would've been:
function f()
x = [1;5;6]
for i = 1:10
x .= x .+ inner.(x)
end
return x
end
function inner(x)
return 2x
end
Therefore we still get the concise vectorized syntax of MATLAB/R/Python, but this version doesn't create temporary arrays and thus will be faster. This is how you can use "scripting language syntax" but still get C/Fortranlike speeds. If you don't watch for temporaries, they will bite away at your performance (same in the other languages, it's just that using vectorized codes is faster than not using vectorized codes in the other languages. In Julia, we have the luxury of something faster being available).
**** Note: Some operators do not fuse in v0.5. For example, ".*" won't fuse yet. This is still a work in progress but should be all together by v0.6 ****
Gotcha #7: Not Building the System Image for your Hardware
This is actually something I fell prey to for a very long time. I was following all of these rules thinking I was a Julia champ, and then one day I realized that not every compiler optimization was actually happening. What was going on?
It turns out that the prebuilt binaries that you get via the downloads off the Julia site are toneddown in their capabilities in order to be usable on a wider variety of machines. This includes the binaries you get from Linux when you do "aptget install" or "yum install". Thus, unless you built Julia from source, your Julia is likely not as fast as it could be.
Luckily there's an easy fix provided by Mustafa Mohamad (@musm). Just run the following code in Julia:
include(joinpath(dirname(JULIA_HOME),"share","julia","build_sysimg.jl")); build_sysimg(force=true)
If you're on Windows, you may need to run this code first:
Pkg.add("WinRPM");
WinRPM.install("gcc", yes=true)
WinRPM.install("winpthreadsdevel", yes=true)
And on any system, you may need to have administrator privileges. This will take a little bit but when it's done, your install will be tuned to your system, giving you all of the optimizations available.
Conclusion: Learn the Rules, Understand Them, Then Profit
To reiterate one last time: Julia doesn't have compiler magic, just simple rules. Learn the rules well and all of this will be second nature. I hope this helps you as you learn Julia. The first time you encounter a gotcha like this, it can be a little hard to reason it out. But once you understand it, it won't hurt again. Once you really understand these rules, your code will compile down to essentially C/Fortran, while being written in a concise highlevel scripting language. Put this together with broadcast fusing and metaprogramming, and you get an insane amount of performance for the amount of code you're writing!
Here's a question for you: what Julia gotchas did I miss? Leave a comment explaining a gotcha and how to handle it. Also, just for fun, what are your favorite gotchas from other languages? [Mine has to be the fact that, in Javascript inside of a function, "var x=3" makes "x" local, while "x=3" makes "x" global. Automatic globals inside of functions? That gave some insane bugs that makes me not want to use Javascript ever again!]
The post 7 Julia Gotchas and How to Handle Them appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/7juliagotchashandle/feed/
8
362

Introducing DifferentialEquations.jl
http://www.stochasticlifestyle.com/introducingdifferentialequationsjl/
http://www.stochasticlifestyle.com/introducingdifferentialequationsjl/#comments
Mon, 01 Aug 2016 14:24:10 +0000
http://www.stochasticlifestyle.com/?p=322
Edit: This post is very old. See this post for more uptodate information.
Differential equations are ubiquitous throughout mathematics and the sciences. In fact, I myself have studied various forms of differential equations stemming from fields including biology, chemistry, economics, and climatology. What was interesting is that, although many different people are using differential equations for many different things, pretty much everyone wants the same thing: to quickly solve differential equations in their various forms, and make some pretty plots to describe what happened.
The goal of DifferentialEquations.jl is to do exactly that: to make it easy solve differential equations with the latest and greatest algorithms, and put out a pretty plot. The core idea behind DifferentialEquations.jl is that, while it is easy to describe a differential equation, they have such diverse behavior that experts have spent over a century compiling ... READ MORE
The post Introducing DifferentialEquations.jl appeared first on Stochastic Lifestyle.
]]>
Edit: This post is very old. See this post for more uptodate information.
Differential equations are ubiquitous throughout mathematics and the sciences. In fact, I myself have studied various forms of differential equations stemming from fields including biology, chemistry, economics, and climatology. What was interesting is that, although many different people are using differential equations for many different things, pretty much everyone wants the same thing: to quickly solve differential equations in their various forms, and make some pretty plots to describe what happened.
The goal of DifferentialEquations.jl is to do exactly that: to make it easy solve differential equations with the latest and greatest algorithms, and put out a pretty plot. The core idea behind DifferentialEquations.jl is that, while it is easy to describe a differential equation, they have such diverse behavior that experts have spent over a century compiling different ways to think about and handle differential equations. Most users will want to just brush past all of the talk about which algorithms simply ask: "I have this kind of differential equation. What does the solution look like?"
DifferentialEquations.jl's User Interface
To answer that question, the user should just have to say what their problem is, tell the computer to solve it, and then tell the computer to plot it. In DifferentialEquations.jl, we use exactly those terms. Let's look at an Ordinary Differential Equation (ODE): the linear ODE . It is described as the function
To use DifferentialEquations.jl, you first have to tell the computer what kind of problem you have, and what your data is for the problem. Recall the general ordinary differential equation is of the form
and initial condition , so in this case, we have an ODE with data and . DifferentialEquations.jl is designed as a software for a highlevel language, Julia. There are many reasons for this choice, but the one large reason is its type system and multiple dispatch. For our example, we tell the machine what type of problem we have by building a DEProblem type. The code looks like this:
using DifferentialEquations
alpha = 0.5 #Setting alpha to 1/2
f(y,t) = alpha*y
u0 = 1.5
timespan = (0.0,1.0) # Solve from time = 0 to time = 1
prob = ODEProblem(f,u0,timespan)
where prob contains everything about our problem. You can then tell the computer to solve it and give you a plot by, well, solve and plot:
sol = solve(prob) # Solves the ODE
plot(sol) # Plots the solution using Plots.jl
And that's the key idea: the user should simply have to tell the program what the problem is, and the program should handle the details. That doesn't mean that the user won't have access to to all of the details. For example, we can control the solver and plotters in more detail, using something like
sol = solve(prob,RK4()) # Unrolled and optimzed RK4
plot(sol,lw=3) # All of the Plots.jl attributes are available
However, in many cases a user may approach the problem for which they don't necessarily know anything about the algorithms involved in approximating the problem, and so obfuscating the API with these names is simply confusing. One place where this occurs is solving stochastic differential equations (SDEs). These have been recently growing in popularity in many of the sciences (especially systems biology) due to their added realism and their necessity when modeling rare and random phenomena. In DifferentialEquations.jl, you can get started by simply knowing that an SDE problem is defined by the functions and in the form
with initial condition , and so the steps for defining and solving the linear SDE is
g(u,t) = 0.3u
prob = SDEProblem(f,g,u0,timespan)
sol = solve(prob)
plot(sol)
If you wish to dig into the manual, you will see that the default solver that is used is a RosslerSRI type of method and will (soon) have adaptivity which is complex enough to be a numerical analysis and scientific computing research project. And you can dig into the manual to find out how to switch to different solvers, but the key idea is that you don't have to. Everything is simply about defining a problem, and asking for solutions and plots.
And that's it. For more about the API, take a look at the documentation or the tutorial IJulia notebooks. What I want to discuss is why I believe this is the right choice, where we are, and where we can go with it.
What exactly does that give us?
Julia was created to solve the manylanguage problem in scientific computing. Before people would have to write out the inner loops as C/Fortran, and bind it to a scripting language that was never designed with performance in mind. Julia has done extremely well as solving this problem via multipledispatch. Multiple dispatch is not just about ease of use, but it is also the core of what allows Julia to be fast . From a quote I am stealing from IRC: "Julia: come for the fast loops, stay for the typesystem".
In my view, the manylanguage problem always had an uglier cousin: the manyAPI problem. Every package has its own way of interacting with the user, and it becomes a burden to remember how all of them work. However, in Julia there seems to be a beautiful emergence of packages which solve the manyAPI problem via Julia's multipledispatch and metaprogramming functionalities. Take for example Plots.jl. There are many different plotting packages in Julia. However, through Plots.jl, you can plot onto any "backend" (other plotting package) using just one API. You can mix and match plotting in PyPlot (matplotlib), GR, Plotly, and unicode. It's all the same commands. Another example of this is JuMP. Its core idea is solver independence: you take your optimization problem, define the model in JuMP's lingo, and then plug into many different solvers all by flipping a switch.
DifferentialEquations.jl is extending this idea to the realm of differential equations. By using the keyword `alg=:ode45`, the solver can call functions from ODE.jl. And changing it to `alg=:dopri5`, DifferentialEquations.jl will solve your ODE problem using the coveted dopri5 Fortran software. The complexity of learning and understanding many different APIs is no longer a requirement for trying different algorithms.
But why "Differential Equations"? Isn't that broad?
Sure, there are packages for solving various types of differential equations, all specializing in one little part. But when I was beginning my PhD, quickly found that these packages were missing something. The different types of differential equations that we encounter are not different but many times embody the same problem: a PDE when discretized is a system of ODEs, the probability distribution of evolving SDEs is a PDE (a form of the Heat Equation), and all of the tools that they use to get good performance are the same. Indeed, many contemporary research questions can be boiled down to addressing the question: what happens if we change the type of differential equation? What happens if we add noise to our ODEs which describe population dispersal? What happens if we add to our model that RNA production is reading a delayed signal? Could we make this highdimensional PDE computationally feasible via a Monte Carlo experiment combined with FeynmanKac's theorem?
Yet, our differential equations libraries are separate. Our PDEs are kept separate from our SDEs, while our delay equations hang out in their own world. Mixing and matching solvers requires learning complex APIs, usually large C/Fortran libraries with opaque function names. That is what DifferentialEquations.jl is looking to solve. I am building DifferentialEquations.jl as a hub for differential equations, the general sense of the term.
If you have defined an SDE problem, then via the Forward Kolmorogov equation there is a PDE associated to the SDE. In many cases like the BlackScholes model, both the SDE and the PDE are canonical ways of looking at the same problem. The solver should translate between them, and the solver should handle both types of differential equations. With one API and the functionality for these contained within the same package, no longer are they separate entities to handle computationally.
Where are we currently?
DifferentialEquations.jl is still very young. Indeed, the project only started a few months ago, and during that time period I was taking 6 courses. However, the package already has a strong core, including
 Most of the standard ODE, SDE, and PDE (Heat and Poisson) solvers.
 Plot recipes for all the basic types.
 Tests for convergence of every algorithm.
 Extensive documentation and tutorials.
In fact, there are already a lot of features which are unique to DifferentialEquations.jl:
 Implementations of Feagin's Order 10, 12, and 14 RungeKutta methods.
 Compatibility with Juliadefined number types. This has been tested to work with Bigs, DecFP, and ArbFloats, and is actively being tested with ArbReals and DoubleDouble.
 Wrappers to ODE.jl and ODEInterface, giving you instant access to tons of different solver methods just by changing the `alg` keyword.
 Stateoftheart stochastic differential equation solvers. As noted before, implemented are results from recent papers, and many other algorithms are waiting on a private branch until papers are published.
 Finite element solvers for some common stochastic PDEs, including the ReactionDiffusion equation used to describe Turing Morphogenesis.
 An algorithm design and testing suite.
You may have been thinking, "but I am a numerical analyst. How could this program help me?". DifferentialEquations.jl has a bunch of functionalities for quickly designing and testing algorithms. All of the DEProblems allow for one to give them the analytical solution, and the solvers will then automatically calculate the errors. Thus by using some simple macros, one can define new algorithms in just a few lines of code, test the convergence, benchmark times, and have the algorithm available as an `alg` option in no time (note: all of the ODE solvers were written in one morning!). Thus it is easy to define the loop, and the rest of the functionality will come by default. It's both a great way to test algorithms, and share algorithms. Contributing will both help you and DifferentialEquations.jl!.
Where are we going?
I have big plans for DifferentialEquations.jl. For example:
 I will be rolling out an efficiency testing suite so that one could just specify the algorithms you'd like to solve a problem, and along what convergence axis (i.e. choose a few s, or along changing tolerances), and it will output comparisons of the computational efficiencies and make some plots. It will be similar in functionality to the ConvergenceSimulation suite.
 Finite difference methods for Heat and Poisson equation. These are long overdue for the research I do.
 Changing the tableaus in ODEs and SDEs to StaticArrays so they are stack allocated. This has already been tested and works well on v0.5.
 Higherorder methods for parabolic SPDEs (a research project with promising results!).
 Blazing fast adaptivity for SDEs. (Once the paper I have submitted for it is published, it will be available. It's already implemented!)
 Highstability high order methods for SDEs (another research project).
 Parallel methods. I have already implemented parallel (Xeon Phi) solvers and described them in previous blog posts. They simply need to be integrated into DifferentialEquations.jl. I would like to have native GPU solvers as well.
 Delay and algebraic differential equations.
 Wrapping more popular solvers. I'd like to add Sundials, LSODE, and PetsC to the list.
 A web interface via Escher.jl to define DEProblems and get the solution plots. I am looking to have this hosted as an XSEDE Gateway.
If you'd like to influence where this project is going, please file an issue on the Github repository. I am always open for suggestions.
I hope this gives you a good idea on what my plans are for DifferentialEquations.jl. Check out the documentation and give the package a whirl!
The post Introducing DifferentialEquations.jl appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/introducingdifferentialequationsjl/feed/
7
322

Using Julia's Type System For Hidden Performance Gains
http://www.stochasticlifestyle.com/usingjuliastypesystemhiddenperformancechunkedarraysgrowablearraysellipsisnotation/
http://www.stochasticlifestyle.com/usingjuliastypesystemhiddenperformancechunkedarraysgrowablearraysellipsisnotation/#comments
Tue, 07 Jun 2016 20:54:08 +0000
http://www.stochasticlifestyle.com/?p=308
What I want to share today is how you can use Julia's type system to hide performance gains in your code. What I mean is this: in many cases you may find out that the optimal way to do some calculation is not a "clean" solution. What do you do? What I want to do is show how you can define special arrays which are wrappers such that these special "speedups" are performed in the background, while having not having to keep all of that muck in your main algorithms. This is easiest to show by example.
The examples I will be building towards are useful for solving ODEs and SDEs. Indeed, these tricks have all been implemented as part of DifferentialEquations.jl and so these examples come from a real use case! They really highlight a main feature of Julia: ... READ MORE
The post Using Julia's Type System For Hidden Performance Gains appeared first on Stochastic Lifestyle.
]]>
What I want to share today is how you can use Julia's type system to hide performance gains in your code. What I mean is this: in many cases you may find out that the optimal way to do some calculation is not a "clean" solution. What do you do? What I want to do is show how you can define special arrays which are wrappers such that these special "speedups" are performed in the background, while having not having to keep all of that muck in your main algorithms. This is easiest to show by example.
The examples I will be building towards are useful for solving ODEs and SDEs. Indeed, these tricks have all been implemented as part of DifferentialEquations.jl and so these examples come from a real use case! They really highlight a main feature of Julia: since Julia code is fast (as long as you have type stability!), you don't need to worry about writing code outside of Julia, and you can take advantage of higherlevel features (with caution). In Python, normally one would only get speed benefits by going down to C, and so utilizing these complex objects would not get speed benefits over simply using numpy arrays in the most vectorized fashion. The same holds for R. In MATLAB... it would be really tough to implement most of this in MATLAB!
Taking Random Numbers in Chunks: ChunkedArrays
Let's say we need to take random numbers in a loop like the following:
for i = 1:N
dW = randn(size(u))
#Do some things and add dW to u
end
While this is the "intuitive" code to write, it's not necessarily the best. While there have been some improvements made since early Julia, in principle it's just slower to make 1000 random numbers via randn() than to use randn(1000). This is because of internal speedups due to caching, SIMD, etc. and you can find mentions of this fact all over the web especially when people are talking about fast random number generators like from the VSL library.
So okay, what we really want to do is the following. Every "bufferSize" steps, create a new random number dW which is of size size(u)*bufferSize, and go through using the buffer until it is all used up, and then grab another buffer.
for i = 1:N
if i%bufferSize == 0
dW = randn(size(u),bufferSize)
end
#Do some things and add dW[..,i] to u
end
But wait? What if we don't always use one random number? Sometimes the algorithm may need to use more than one! So you can make an integer k which tracks the current state in the buffer, and then at each point where it can be incremented, you add the conditional to grab a new buffer, etc. Also, what if you want to have the buffer generated in parallel? As you can see, code complexity explosion, just to go a little faster?
This is where ChunkedArrays come in. What I did is defined an array which essentially does the chunking/buffering in the background, so that way the code in the algorithm could be clean. A ChunkedArray is a wrapper over an array, and then used the next command to hide all of this complexity. Thus, to generate random numbers in chunks to get this speed improvement, you can use code like this:
rands = ChunkedArray(u)
for i = 1:N
if i%bufferSize == 0
dW = next(rands)
end
#Do some things and add dW[..,i] to u
end
Any time another random number is needed, you just call next. It internally stores an array and the state of the buffer, and the next function automatically check / replenishes the buffer, and can launch another process to do this in parallel if the user wants. Thus we get the optimal solution without sacrificing cleanliness. I chopped off about 10% of a runtime in EulerMaruyama code in DifferentialEquations.jl by switching to ChunkedArrays, and haven't thought about doing a benchmark since.
Safe Vectors of Arrays and Conversion: GrowableArrays
First let's look at the performance difference between Vectors of Arrays and higherdimensional contiguous arrays when using them in a loop. Julia's arrays can take in a parametric type which makes the array hold arrays, this makes the array essentially an array of pointers. The issue here is that this adds an extra cost every time the array is dereferenced. However, for highdimensional arrays, the : way of referencing has to generate a slice each time. Which way is more performant?
function test1()
u = Array{Int}(4,4,3)
u[:,:,1] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
u[:,:,2] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
u[:,:,3] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
j = 1
for i = 1:100000
j += sum(u[:,:,1] + u[:,:,2] + 3u[:,:,3] + u[:,:,i%3+1]  u[:,:,(i%j)%2+1])
end
end
function test2()
u = Vector{Matrix{Int}}(3)
u[1] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
u[2] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
u[3] = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
j = 1
for i = 1:100000
j += sum(u[1] + u[2] + 3u[3] + u[i%3+1]  u[(i%j)%2+1])
end
end
function test3()
u = Array{Int}(4,4,4)
u[1,:,:] = reshape([1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1],(1,4,4))
u[2,:,:] = reshape([1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1],(1,4,4))
u[3,:,:] = reshape([1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1],(1,4,4))
j = 1
for i = 1:100000
j += sum(u[1,:,:] + u[2,:,:] + 3u[3,:,:] + u[i%3+1,:,:]  u[(i%j)%2+1,:,:])
end
end
#Precompile
test1()
test2()
test3()
t1 = @elapsed for i=1:10 test1() end
t2 = @elapsed for i=1:10 test2() end
t3 = @elapsed for i=1:10 test3() end
println("Test results: t1=$t1, t2=$t2, t3=$t3")
#Test results: t1=1.239379946, t2=0.576053075, t3=1.533462129
So using Vectors of Arrays is fast for dereferecing.
Now think about adding to an array. If you have a Vector of pointers and need to resize the array, it's much easier to resize and copy over some pointers then it is to copy over all of the arrays. So, if you're going to grow an array in a loop, the Vector of Arrays is the fastest implementation! Here's a quick benchmark from GrowableArrays.jl:
using GrowableArrays, EllipsisNotation
using Base.Test
tic()
const NUM_RUNS = 100
const PROBLEM_SIZE = 1000
function test1()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = u
for i = 1:PROBLEM_SIZE
uFull = hcat(uFull,u)
end
uFull
end
function test2()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = u
for i = 1:PROBLEM_SIZE
uFull = vcat(uFull,u)
end
uFull
end
function test3()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = Vector{Int}(0)
sizehint!(uFull,PROBLEM_SIZE*16)
append!(uFull,vec(u))
for i = 1:PROBLEM_SIZE
append!(uFull,vec(u))
end
reshape(uFull,4,4,PROBLEM_SIZE+1)
uFull
end
function test4()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = Vector{Array{Int}}(0)
push!(uFull,copy(u))
for i = 1:PROBLEM_SIZE
push!(uFull,copy(u))
end
uFull
end
function test5()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = Vector{Array{Int,2}}(0)
push!(uFull,copy(u))
for i = 1:PROBLEM_SIZE
push!(uFull,copy(u))
end
uFull
end
function test6()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = Vector{typeof(u)}(0)
push!(uFull,u)
for i = 1:PROBLEM_SIZE
push!(uFull,copy(u))
end
uFull
end
function test7()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = GrowableArray(u)
for i = 1:PROBLEM_SIZE
push!(uFull,u)
end
uFull
end
function test8()
u = [1 2 3 4
1 3 3 4
1 5 6 3
5 2 3 1]
uFull = GrowableArray(u)
sizehint!(uFull,PROBLEM_SIZE)
for i = 1:PROBLEM_SIZE
push!(uFull,u)
end
uFull
end
println("Run Benchmarks")
println("PreCompile")
#Compile Test Functions
test1()
test2()
test3()
test4()
test5()
test6()
test7()
test8()
println("Running Benchmarks")
t1 = @elapsed for i=1:NUM_RUNS test1() end
t2 = @elapsed for i=1:NUM_RUNS test2() end
t3 = @elapsed for i=1:NUM_RUNS test3() end
t4 = @elapsed for i=1:NUM_RUNS test4() end
t5 = @elapsed for i=1:NUM_RUNS test5() end
t6 = @elapsed for i=1:NUM_RUNS test6() end
t7 = @elapsed for i=1:NUM_RUNS test7() end
t8 = @elapsed for i=1:NUM_RUNS test8() end
println("Benchmark results: $t1 $t2 $t3 $t4 $t5 $t6 $t7 $t8")
#Benchmark results: 1.923640854 2.131108443 0.012493308 0.00866045 0.005246504 0.00532613 0.00773568 0.00819909
As you can see in test7 and test8, I created a "GrowableArray" which is an array which acts like a Vector of Arrays. However, it has an added functionality that if you copy(G), then what you get is the contiguous array. Therefore in the loop you can grow the array the quickest way as a storage machine, but after the loop (say to plot the array), but at any time you can copy it to a contiguous array which is more suited for interop with C and other goodies.
It also hides a few painful things. Notice that in the code we pushed a copy of u (copy(u)). This is because when u is an array, it's only the reference to the array, so if we simply push!(uFull,u), every element of uFull is actually the same item! This benchmark won't catch this issue, but try changing u and you will see that every element of uFull changes if you don't use copy. This can be a nasty bug, so instead we build copy() into the push!() command for the GrowableArray. This gives another issue. Since copying a GrowableArray changes it, you need to make sure push! doesn't copy on arguments of GrowableArrays (to create GrowableArrays of GrowableArrays). However, this is easily managed via dispatch.
Helping Yourself and the Julia Community
Simple projects like these lead to reusable solutions to improve performance while allowing for ease of use. I have just detailed some projects I have personally done (and have more to do!), but there are others that should be pointed out. I am fond of projects like VML.jl which speedup standard functions, and DoubleDouble.jl which implements efficient quadprecision numbers that you can then use in place of other number types.
I think Julia will succeed not by the "killer packages" that are built in Julia, but by a rich type ecosystem that will make everyone want to build their "killer package" in Julia.
The post Using Julia's Type System For Hidden Performance Gains appeared first on Stochastic Lifestyle.
]]>
http://www.stochasticlifestyle.com/usingjuliastypesystemhiddenperformancechunkedarraysgrowablearraysellipsisnotation/feed/
2
308