When you see something like:

collection.iter()
    .map(...)
    .filter(...)
    .take(...)
    .collect();

Why Iterators Are Useful

• Abstraction: They hide the underlying container’s implementation. You don’t need to know if it’s an array, a linked list, or some more specialized structure.

• Safety / Encapsulation: With an iterator, you typically avoid directly indexing into a collection’s memory. This reduces off-by-one errors, out-of-bounds errors, or concurrency issues.

• Composability: Iterators can be transformed, filtered, and combined to express complex operations in a clean, functional style.

Iterators in Rust

Rust’s iterator system is powerful and versatile, offering developers a robust toolset for managing collections efficiently. It simplifies the code and reduces the potential for errors. This not only makes the code more readable but also aligns with functional programming paradigms, making it easier for developers familiar with languages like C# or Java to adapt.

1. They Are Lazy

   • In Rust, when you call methods like .map(), .filter(), or .take() on an iterator, it does not immediately create a new collection. Instead, these methods return a new iterator that, when advanced, processes items on demand.

   • This means you can build up chained iterator calls without incurring overhead until you actually consume the iterator (e.g., in a for loop, or when calling a terminal method like .collect()).

2. Three Main “Iterator” Traits

   • Iterator: The trait for iterators that produce items by value. Typically you implement this when you want to define how to produce a sequence of items.

   • IntoIterator: For types (like Vec, slices, and so on) that can be converted into an iterator.

   • DoubleEndedIterator: An extension that allows “reverse” iteration (calling .next_back() in addition to .next() from the front).

3. Ownership, Borrowing, and Lifetimes

   • Rust’s type system enforces rules to ensure safety. For iterators, this means that when you iterate over a collection, either you move (take ownership of) the collection or you borrow from it in a well-defined way.

   • iter() borrows each element by reference.

   • into_iter() consumes (takes ownership of) the elements, so the iterator yields the items by value.

   • iter_mut() borrows each element mutably, allowing you to modify items in place.

4. Common Methods

   • map(f): Apply a function f to each item.

   • filter(pred): Keep only items where pred(item) is true.

   • fold(init, f): Accumulate items into a single value, starting from init, applying f in a fold/reduction manner.

   • collect(): Consume an iterator and gather the items into a collection (like a Vec, HashSet, etc.).

   • take(n), skip(n), enumerate(), etc.: Additional combinators for slicing, offset, or enumerating.

5. Consuming vs. Adapting Iterators
   In Rust’s library documentation, you might see a distinction between “consuming” vs. “adapting” iterators:

   • Consuming: These methods take ownership of the iterator to produce a final value or another data structure (e.g., collect(), fold()). Once consumed, the iterator cannot be used again.

   • Adapting: These methods produce a new iterator (e.g., map, filter). These can be chained before any final consumption.

Examples

Below is some "cookbook"-style collection of common and practical iterator patterns in Rust. This will help you become more comfortable and familiar with iterators, allowing you to later explore them in greater depth.

1. Iterate and Collect into a New Vector

A basic example: transform and collect.

fn example_collect() {
    let numbers = vec![1, 2, 3, 4, 5];
    // Double each number and collect into a new Vec<i32>
    let doubled: Vec<i32> = numbers.iter().map(|x| x * 2).collect();
    println!("{:?}", doubled); // Output: [2, 4, 6, 8, 10]
}

iter() borrows each element (so x is &i32). map(|x| x * 2) applies a function to each element. |x| x * 2 is closure. collect() finalizes the lazy chain, producing the Vec<i32>.

2. Filter Items Based on a Condition

Use filter to keep only items meeting a criterion.

fn example_filter() {
    let numbers = vec![10, 15, 20, 25, 30];
    // Keep only multiples of 10
    let multiples_of_ten: Vec<i32> = numbers
        .into_iter()        // Now x is i32, not &i32
        .filter(|x| x % 10 == 0)
        .collect();

    println!("{:?}", multiples_of_ten); // Output: [10, 20, 30]
}

into_iter() consumes numbers and yields i32 (by value). filter(...) returns a new iterator keeping only those items where the predicate is true.

3. Filter and Map in One Step: filter_map

Sometimes you want to transform items and discard certain items altogether. filter_map combines filter and map in one step, working with Option.

let inputs = vec!["42", "93", "hello", "128", "xyz"];
// Try parsing each string as an integer.
// If successful, keep the integer; otherwise, skip.
let parsed: Vec<i32> = inputs
        .iter()
        .filter_map(|s| s.parse::<i32>().ok())
        .collect();
// parsed: [42, 93, 128]

s.parse::<i32>().ok() returns Some(i32) if parse succeeded, or None otherwise. filter_map only yields Some items.

4. Combine Multiple Iterators: chain

Use chain to create a single iterator that yields items from two or more sources sequentially.

let nums1 = vec![1, 2, 3];
let nums2 = vec![4, 5, 6];
let chained: Vec<i32> = nums1
        .into_iter()
        .chain(nums2.into_iter())
        .collect();
// chained: [1, 2, 3, 4, 5, 6]

chain takes two iterators and iterates over the first fully, then the second. In this example, nums1 and nums2 are both consumed (moved) by into_iter().

5. Enumerate Items with enumerate

Sometimes you need both the index and the value.

let animals = vec!["cat", "dog", "bird"];
for (index, animal) in animals.iter().enumerate() {
    println!("{}: {}", index, animal);
}
// Output:
// 0: cat
// 1: dog
// 2: bird

enumerate() yields (index, item) pairs. The index starts at 0 by default.

6. Find the First Matching Item: find

The find method returns the first item that satisfies a condition (as an Option).

let numbers = vec![1, 3, 5, 7, 9];
// Find the first number that is divisible by 3
if let Some(found) = numbers.iter().find(|&&x| x % 3 == 0) {
    println!("Found: {}", found); // Output: Found: 3
} else {
    println!("No match found");
}

find stops searching after the first match. Returns None if no match is found.

7. Take While a Condition is True: take_while

Use take_while to yield items only as long as a predicate holds. (Note: This is available via Iterator::take_while in stable Rust 1.52+.)

    let numbers = vec![2, 4, 6, 8, 10, 1, 12];
    // Take only even numbers until we reach the first odd
    let evens_up_to_odd: Vec<i32> = numbers
        .into_iter()
        .take_while(|x| x % 2 == 0)
        .collect();

    println!("{:?}", evens_up_to_odd); // Output: [2, 4, 6, 8, 10]

Once take_while sees a value that doesn’t satisfy the predicate, it stops altogether.

8. Accumulating/Reducing: fold

Use fold to accumulate values into a single result. This is often used for sums, products, or more advanced aggregations.

fn example_fold() {
    let numbers = vec![1, 2, 3, 4, 5];

    let sum = numbers.iter().fold(0, |acc, &x| acc + x);
    println!("Sum is {}", sum); // Output: 15

    let product = numbers.iter().fold(1, |acc, &x| acc * x);
    println!("Product is {}", product); // Output: 120
}

The first argument to fold is the initial accumulator value. The closure receives the accumulator and the next item.

9. Partition Items into Two Groups: partition

partition splits items into two collections based on a condition, returning (Vec<T>, Vec<T>).

fn example_partition() {
    let numbers = vec![1, 2, 3, 4, 5, 6];

    let (even, odd): (Vec<i32>, Vec<i32>) = numbers
        .into_iter()
        .partition(|x| x % 2 == 0);

    println!("Even: {:?}", even); // Even: [2, 4, 6]
    println!("Odd: {:?}", odd);   // Odd: [1, 3, 5]
}

The first Vec collects items where the predicate is true, and the second collects items where it is false.

10. Zip Two Iterators Together: zip

zip pairs items from two iterators into (item1, item2) tuples.

fn example_zip() {
    let letters = vec!['a', 'b', 'c'];
    let numbers = vec![1, 2, 3, 4];

    // Zip together into pairs
    let zipped: Vec<(char, i32)> = letters.into_iter().zip(numbers).collect();
    println!("{:?}", zipped); // Output: [('a', 1), ('b', 2), ('c', 3)]
}

The iteration stops when the shortest iterator ends. In this example, letters has 3 items, numbers has 4, so only 3 pairs are created.

11. Flatten an Iterator of Iterators: flatten

If you have nested iterators (e.g., a Vec<Vec<T>>), you can flatten it into a single sequence.

fn example_flatten() {
    let nested = vec![vec![1, 2], vec![3, 4, 5], vec![6]];
    let flattened: Vec<i32> = nested.into_iter().flatten().collect();

    println!("{:?}", flattened); // Output: [1, 2, 3, 4, 5, 6]
}

flatten automatically iterates through each sub-iterator/item. You can also use flat_map if you need to transform and flatten in one step.

12. Skipping Elements or Taking a Specific Count: skip / take

Use skip to ignore a certain number of items, take to limit iteration to a certain count.

fn example_skip_take() {
    let numbers = vec![10, 20, 30, 40, 50, 60];

    // Skip the first 2, then take the next 3
    let slice: Vec<i32> = numbers
        .iter()
        .skip(2)
        .take(3)
        .copied()  // because we had .iter() -> &i32
        .collect();

    println!("{:?}", slice); // Output: [30, 40, 50]
}

skip(n) ignores the first n items. take(n) yields only the next n items, then stops. .copied() turns &i32 into i32; you could also use .cloned() or just leave them as references if that’s acceptable.

IntoIterator Trait

IntoIterator is a trait that defines how a type can be converted into an iterator. It's one of the fundamental traits in Rust's collections and iteration system.

pub trait IntoIterator {
    type Item;
    type IntoIter: Iterator<Item = Self::Item>;

    fn into_iter(self) -> Self::IntoIter;
}

This trait has three key components:

• Item: The type of item that the iterator will produce

• IntoIter: The specific iterator type that will be returned

• into_iter(): The method that consumes the collection and returns an iterator.

How IntoIterator Is Used

For loops: When you write a for loop in Rust, the compiler automatically calls into_iter() on the collection you're iterating over:

for element in collection {
    // This is actually doing: for element in collection.into_iter() { ... }
}

When you write a for loop or manually call collection.into_iter(), Rust decides which implementation to call based on whether you’re iterating over the collection by value (collection.into_iter()), by reference ((&collection).into_iter()), or by mutable reference ((&mut collection).into_iter()).

Converting collections to iterators:

You can explicitly call into_iter() to convert a collection into an iterator:

let vec = vec![1, 2, 3];
let iter = vec.into_iter(); // Consumes vec, returns an iterator

Different Implementations

Most collections in Rust implement IntoIterator in multiple ways:

For the collection itself (by value): Consumes the collection, returning an iterator that takes ownership of the elements:

impl<T> IntoIterator for Vec<T> {
    type Item = T;
    type IntoIter = std::vec::IntoIter<T>;

    fn into_iter(self) -> Self::IntoIter { /* ... */ }
}

For references to the collection (&): Creates an iterator that borrows the elements:

impl<'a, T> IntoIterator for &'a Vec<T> {
    type Item = &'a T;
    type IntoIter = std::slice::Iter<'a, T>;

    fn into_iter(self) -> Self::IntoIter { /* ... */ }
}

For mutable references (&mut): Creates an iterator that mutably borrows the elements:

impl<'a, T> IntoIterator for &'a mut Vec<T> {
    type Item = &'a mut T;
    type IntoIter = std::slice::IterMut<'a, T>;

    fn into_iter(self) -> Self::IntoIter { /* ... */ }
}

std::slice::Iter<'a, T> , std::slice::IterMut<'a, T> and std::vec::IntoIter<T> are different iterator types you're seeing in Rust. Each one corresponds to a different way of iterating over a collection, based on ownership and mutability.

Understanding into_iter() and Ownership

The into_iter() method can handle different ownership patterns, making Rust's iteration system super flexible. Let's discuss how it works with values, references, and mutable references, and how iter() fits into this system.

How into_iter() works with different ownership types

The into_iter() method behaves differently depending on whether you call it on:

1. Value (T): Consumes the collection, taking ownership

2. Reference (&T): Borrows the collection immutably

3. Mutable reference (&mut T): Borrows the collection mutably

This is achieved through separate implementations of the IntoIterator trait for each case.

// For Vec<T> as an example:
// Takes ownership (consumes the Vec)
impl<T> IntoIterator for Vec<T> {
    type Item = T;  // Iterator yields owned values
    // ...
}
// Borrows immutably
impl<'a, T> IntoIterator for &'a Vec<T> {
    type Item = &'a T;  // Iterator yields references
    // ...
}
// Borrows mutably
impl<'a, T> IntoIterator for &'a mut Vec<T> {
    type Item = &'a mut T;  // Iterator yields mutable references
    // ...
}

Examples:

let v = vec![1, 2, 3];

// Takes ownership - consumes v
let iter1 = v.into_iter();  // Iterator yields values (T)
// v is no longer usable here

let v = vec![1, 2, 3];

// Borrows immutably
let iter2 = (&v).into_iter();  // Iterator yields references (&T)
// v is still usable here

// Borrows mutably
let mut v = vec![1, 2, 3];
let iter3 = (&mut v).into_iter();  // Iterator yields mutable references (&mut T)
// v is still usable after iter3 is dropped

// Standard collection methods
let v = vec![1, 2, 3];
let iter_a = v.iter();          // Always yields &T
let iter_b = v.iter_mut();      // Always yields &mut T
let iter_c = v.into_iter();     // Consumes v, yields T

The flexibility of into_iter() is why for loops in Rust work with all three ownership models. The compiler automatically chooses the appropriate implementation based on how you use the collection in the loop.

Iterator Trait

The Iterator trait provides a way to process sequences of items one at a time, and it serves as the foundation for many functional programming patterns. Iterator and IntoIterator are two related but different traits that help in working with collections and iterators.

Iterator is used when you already have an iterator and want to iterate over it. The next() method is used to fetch elements one by one.

Example:

let numbers = vec![1, 2, 3];
let mut iter = numbers.iter(); // Create an iterator over `numbers`

println!("{:?}", iter.next()); // Some(1)
println!("{:?}", iter.next()); // Some(2)
println!("{:?}", iter.next()); // Some(3)
println!("{:?}", iter.next()); // None (end of iteration)

numbers.iter() creates an iterator and calling next() moves through the elements.

Basic Structure

trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;
    // Many default methods omitted...
}

The trait has only one required method to implement: next(). Everything else is built on top of this foundation.

Core Components

1. Associated Type Item: Defines what type of elements the iterator will produce.

2. next() Method: The heart of the iterator pattern.

   • Returns Some(item) if there's another item in the sequence

   • Returns None when iteration is complete

   • Takes &mut self because advancing the iterator changes its internal state

Implementing Iterator

Here's a simple example of implementing the Iterator trait:

struct Fibonacci {
    curr: u32,
    next: u32,
}

impl Iterator for Fibonacci {
    type Item = u32;

    fn next(&mut self) -> Option<Self::Item> {
        let current = self.curr;
        self.curr = self.next;
        self.next = current + self.next;
        Some(current)
    }
}

// Create an iterator that generates the Fibonacci sequence
fn fibonacci() -> Fibonacci {
    Fibonacci { curr: 0, next: 1 }
}

let fib = fibonacci();
let first_10: Vec<u32> = fib.take(10).collect();

Relationship to IntoIterator

Iterator defines how to iterate through a sequence. IntoIterator defines how to create an iterator from a value. These two work hand in hand to make a full iteration system. The IntoIterator::into_iter() method creates an Iterator, and when you use a for loop, it automatically uses IntoIterator behind the scenes.

This design allows for flexibility and powerful abstractions across Rust's standard library and ecosystem. The combination of these traits enables many of Rust's most elegant patterns for working with collections and sequences.

The iter() Method vs. into_iter()

The iter() method is not syntactic sugar for &collection.into_iter(), but they're closely related.

iter() is a method implemented specifically for collections in the standard library, while into_iter() comes from the IntoIterator trait. They serve similar purposes but work differently:

let v = vec![1, 2, 3];

// These two are equivalent:
let iter1 = v.iter();
let iter2 = (&v).into_iter();
// Both produce iterators over &T (references)

Important Distinctions:

• iter() is a method found directly on collection types like Vec and HashMap, while into_iter() comes from the IntoIterator trait.

• iter() always borrows the collection and never consumes it, whereas into_iter() might consume the collection, depending on whether it's called on a value or a reference.

• iter() always gives you an iterator over references, while into_iter() behaves differently based on how you use it. And iter_mut() always provides an iterator over mutable references.

Full Example

Below is a detailed example of a custom Portfolio struct, which contains a Vec<String>. This example includes three distinct implementations of the IntoIterator trait, each designed to allow different types of iteration over the Portfolio:

1. Consuming iteration: This type of iteration occurs by value, meaning it takes ownership of the elements. It yields owned String values, effectively consuming the Portfolio as it iterates through its elements. This is useful when you need to take ownership of the data for further processing or transformation.

2. Shared iteration: This iteration happens by shared reference, which means it does not take ownership of the elements. Instead, it yields &String, allowing you to read the data without modifying it. This is ideal for scenarios where you need to access the data without altering the original Portfolio.

3. Mutable iteration: This iteration is performed by mutable reference, yielding &mut String. It lets you change the elements of the Portfolio as you go through them. This is especially useful when you need to update or transform the data right where it is.

For each of these iteration types, we'll show how they work with a custom iterator type: PortfolioIntoIter, PortfolioIter, and PortfolioIterMut. Each of these custom iterator types implements the Iterator trait, giving you the tools you need to go through the Portfolio in the way you want. This setup not only highlights the flexibility of Rust's iteration patterns but also shows how you can create custom iterators to fit your specific needs.

#[derive(Debug)]
struct Portfolio {
    instruments: Vec<String>,
}

// An iterator for consuming `Portfolio` (yields owned `String`)
struct PortfolioIntoIter {
    inner: std::vec::IntoIter<String>,
}

// An iterator for `&Portfolio` (yields `&String`)
struct PortfolioIter<'a> {
    inner: std::slice::Iter<'a, String>,
}

// An iterator for `&mut Portfolio` (yields `&mut String`)
struct PortfolioIterMut<'a> {
    inner: std::slice::IterMut<'a, String>,
}

// Constructor for convenience
impl Portfolio {
    fn new(instruments: Vec<String>) -> Self {
        Portfolio { instruments }
    }
}

Now, let's delve into implementing the three distinct variants of the IntoIterator trait for our custom iterator types.

// 1) By value (Portfolio -> owned iteration)
impl IntoIterator for Portfolio {
    type Item = String;
    type IntoIter = PortfolioIntoIter;

    fn into_iter(self) -> Self::IntoIter {
        PortfolioIntoIter {
            // Move out the vector’s own iterator
            inner: self.instruments.into_iter(),
        }
    }
}

// 2) By shared reference (&Portfolio -> shared iteration)
impl<'a> IntoIterator for &'a Portfolio {
    type Item = &'a String;
    type IntoIter = PortfolioIter<'a>;

    fn into_iter(self) -> Self::IntoIter {
        PortfolioIter {
            // Borrow the underlying vector’s iterator
            inner: self.instruments.iter(),
        }
    }
}

// 3) By mutable reference (&mut Portfolio -> mutable iteration)
impl<'a> IntoIterator for &'a mut Portfolio {
    type Item = &'a mut String;
    type IntoIter = PortfolioIterMut<'a>;

    fn into_iter(self) -> Self::IntoIter {
        PortfolioIterMut {
            // Borrow the underlying vector’s mutable iterator
            inner: self.instruments.iter_mut(),
        }
    }
}

Now, let's proceed to implement the Iterator trait for each of our custom iterator types. This step is crucial because it defines how each iterator will behave when traversing through the elements of the Portfolio. By implementing the Iterator trait, we specifically implement next() for different iterators to advance the iterator and yield the value.

impl Iterator for PortfolioIntoIter {
    type Item = String;

    fn next(&mut self) -> Option<Self::Item> {
        self.inner.next()
    }
}

impl<'a> Iterator for PortfolioIter<'a> {
    type Item = &'a String;

    fn next(&mut self) -> Option<Self::Item> {
        self.inner.next()
    }
}

impl<'a> Iterator for PortfolioIterMut<'a> {
    type Item = &'a mut String;

    fn next(&mut self) -> Option<Self::Item> {
        self.inner.next()
    }
}

Example usage in main() where we create a Portfolio instance and iterate over its elements using the different iterator implementations we have defined above.

fn main() {
    // 1) By value: move the Portfolio and get owned Strings
    let portfolio = Portfolio::new(vec!["AAPL".into(), "GOOG".into(), "MSFT".into()]);
    println!("Iterate by value:");
    for instrument in portfolio.into_iter() {
        // instrument is String (owned)
        println!("{}", instrument);
    }
    // `portfolio` is consumed here—can't use it again.

    // 2) By shared reference: get &String
    let portfolio_ref = Portfolio::new(vec!["TSLA".into(), "AMZN".into(), "META".into()]);
    println!("\nIterate by shared reference:");
    for instrument in &portfolio_ref {
        // instrument is &String
        println!("{}", instrument);
    }
    // `portfolio_ref` is still usable (not consumed).

    // 3) By mutable reference: get &mut String
    let mut portfolio_mut = Portfolio::new(vec!["NFLX".into(), "DIS".into()]);
    println!("\nIterate by mutable reference:");
    for instrument in &mut portfolio_mut {
        // instrument is &mut String
        instrument.push_str(" (edited)");
    }
    println!("{:?}", portfolio_mut);
}

Output

With this setup, the same .into_iter() method name can produce three different iterator types depending on the context of how you call it (owned, immutable reference, or mutable reference)—exactly like Vec<T> does in the standard library.

Iterators Without Boilerplate: successors

Instead of writing a custom iterator, you can define your sequence generation logic inline with just a closure using successors. successors is a function in the Rust standard library (std::iter) that creates an iterator from an initial value and a “successor function.” Formally:

pub fn successors<T, F>(first: Option<T>, succ: F) -> Successors<T, F>
where
    F: FnMut(&T) -> Option<T>,

first is an Option<T> – your starting point (may or may not exist) and succ is a closure or function that, given a reference to the most recently yielded value, returns the next value as an Option<T>.

The iterator produced will:

• Yield the first value if it’s Some(...).

• Then repeatedly call succ on the last yielded value to get the next.

• Stop once succ returns None.

This means you can generate a sequence on-the-fly without building your own struct and implementing Iterator manually.

If you like chaining and pipelining, successors fits right in. You can follow it with methods like .map(...), .filter(...), etc., and then .take(...) or .collect() as needed.

Here’s a simple example that starts from Some(1) and then keeps adding 1 until we decide to stop at a certain condition.

use std::iter::successors;
fn main() {
    let mut iter = successors(Some(1), |&prev| {
        let next = prev + 1;
        if next <= 5 {
            Some(next)
        } else {
            None
        }
    });

    // Collect into a vector
    let values: Vec<i32> = iter.collect();
    println!("{:?}", values); // [1, 2, 3, 4, 5]
}

We pass Some(1) as the first value, so 1 is yielded immediately when iteration begins. For each subsequent item, we apply the closure |&prev| { ... }. The iterator yields 1, 2, 3, 4, 5, then stops.

We can implement our previous example Fibonacci sequence using std::iter::successors as well.

Here’s the equivalent version of Fibonacci iterator using successor.

use std::iter;
fn fibonacci() -> impl Iterator<Item = u32> {
    iter::successors(Some((0, 1)), |&(curr, next)| {
        Some((next, curr + next))
    })
    .map(|(curr, _)| curr)
}

fn main() {
    let first_10: Vec<u32> = fibonacci().take(10).collect();
    println!("{:?}", first_10); // [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
}

Overall Easier Setup: successors(Some(start), |&prev| Some(next)) is all you need to define a sequence and it is more functional-style code.

Conclusion

Iterators are a fundamental concept in programming that provide a powerful and flexible way to traverse and manipulate collections. By abstracting the underlying data structure, iterators offer a safe and efficient means to perform operations like mapping, filtering, and reducing. In Rust, the iterator system is particularly robust, supporting lazy evaluation, ownership, borrowing, and a wide range of combinators that enhance code readability and maintainability. Understanding the different traits and methods associated with iterators, such as Iterator and IntoIterator, allows you to write more expressive and efficient code.

Capturing variables from its enclosing scope means you don’t have to manually pass those variables as parameters; the compiler implicitly makes them accessible inside the closure.

let x = 10;
// `double_x` is a closure that captures `x` from the environment
let double_x = || x * 2;

Closure Syntax

Parameters in pipes (|)

let closure = |param1, param2| {
    // body
};

Optional curly braces if the body is a single expression:

let closure = |x| x + 1;

Often, you don't have to specify types for the parameters, but if you'd like to be clear, you certainly can.

let add = |a, b| a + b;
let add = |a: i32, b: i32| -> i32 { a + b };

Capturing Variables

Unlike standard functions, closures have a special talent for capturing variables from the scope where they are created. Rust closures can capture:

1. By reference (&T)

2. By mutable reference (&mut T)

3. By value / move (T)

How Rust Decides the Capture Method

Rust uses capture inference to figure out the best way to capture each variable. It decides this based on how the closure uses the variable:

1. If the closure just needs to read the variable, it will capture it by immutable reference.

2. If the closure needs to change the variable, it will capture it by mutable reference.

3. If the closure needs to take full ownership of the variable (like when it moves it or keeps it somewhere that lasts longer than the current scope), it will capture it by value (ownership).

fn main() {
    let mut s = String::from("hello");
    let print_s = || println!("{}", s); // 1) Capture by reference

    print_s(); // Prints "hello"
    // This closure modifies `s`, so it captures `s` by &mut.
    let mut append_exclamation = || {    // 2) Capture by mutable reference
        s.push('!');
    };

    append_exclamation();
    println!("{}", s); // "hello!"

    // 3) Capture by value (move) Using move explicitly forces ownership capture.
    let take_ownership = move || {
        println!("Moved: {}", s);
    }; 
    take_ownership();
}

Lets try to print s after running take_ownerhship closure

println!("{}", s); 

error[E0382]: borrow of moved value: `s`
  --> src/main.rs:20:20
   |
2  |     let mut s = String::from("hello");
   |         ----- move occurs because `s` has type `String`, which does not implement the `Copy` trait
...
15 |     let take_ownership = move || {
   |                          ------- value moved into closure here
16 |         // We won't be able to use `s` after this closure is created if it truly takes it by value.
17 |         println!("Moved: {}", s);
   |                               - variable moved due to use in closure
...
20 |     println!("{}", s); // Error: `s` was moved
   |                    ^ value borrowed here after move

The move Keyword

By default, the compiler uses the least restrictive capture mode it can. But if you want to change that, you can use move. This makes sure all captures in that closure take ownership, using move semantics.

Closure Traits: Fn, FnMut, and FnOnce

Closures can implement one or more of these function traits, based on how they capture their environment:

Fn: A closure that only needs an immutable reference to its environment.

FnMut: A closure that only needs a mutable reference to its environment.

FnOnce: A closure that can be called at least once. It may consume (move) variables it captures, so it can only be called once if it moves.

Conceptually, you can think of them as:

• FnOnce: uses self

• FnMut: uses &mut self

• Fn: uses &self

Let do a bit of deep dive on these traits.

Fn trait

As described in Rust documentation, Fn is the trait for function-like types that can be called repeatedly without needing to change anything in their environment. Put simply, if a piece of code can be called multiple times and doesn’t need to mutate (change) any variables it has captured, it usually implements Fn. This trait (Fn) should not be confused with function pointers (fn).

Fn vs. fn (Function Pointers)

• fn (lowercase) refers to a function pointer type. For example, fn(i32) -> i32 is a pointer to a function that takes an i32 and returns an i32.

• Fn (uppercase) is a trait, which can be implemented by any callable type, such as:

  • Closures (anonymous functions that capture variables),

  • Function pointers (fn(...)) themselves,

  • or even your own custom types that implement the Fn trait.

To illustrate, consider a function that takes an integer as input and returns its square. In Rust, a function pointer has the type fn(…) -> …. For example, here is a simple function that squares an integer:

fn square(x: i32) -> i32 {
    x * x
}

We can create a function that accepts a function pointer of this type. apply_square takes a parameter f: fn(i32) -> i32. This means the caller needs to pass a pointer to a function with that exact signature (no captures, no closures). We can pass square (a regular function) to it, and it works perfectly.

// Accepts a function pointer "fn(i32) -> i32".
fn apply_square(f: fn(i32) -> i32, value: i32) -> i32 {
    f(value)
}

fn main() {
    let result = apply_square(square, 4);
    println!("Result using function pointer: {}", result); 
}

If we try to pass a closure instead of a function pointer, you may expect a compilation error or warning.

let closure = |x: i32| x * x;
let result = apply_square(closure, 4);

Wait, it worked perfectly as well and we know Closures in Rust are not the same type as fn(…) -> ….

In this case, a closure does not capture anything from its environment (i.e., it doesn’t use any local variables from the enclosing scope) is effectively equivalent to a plain function. Rust can coerce (convert) that closure into a function pointer. This is a special-case automatic conversion allowed by the compiler.

Lets capture some variable from its environment.

let x = 4;
let closure = || x * x;
let result = apply_square(closure, x);

error[E0308]: mismatched types
  --> src/main.rs:24:40
   |
22 |     let closure = || x * x;
   |                   -- the found closure
23 |     
24 |     let result = apply_square(closure, x);
   |                  --------------------- ^^^^^^^ expected fn pointer, found closure
   |                  |
   |                  arguments to this function are incorrect
   |
   = note: expected fn pointer `fn(i32) -> i32`
                 found closure `{closure@src/main.rs:22:19: 22:21}`
note: closures can only be coerced to `fn` types if they do not capture any variables
  --> src/main.rs:22:22
   |
22 |     let closure = || x * x;
   |                      ^ `x` captured here
note: function defined here
  --> src/main.rs:7:4

To allow a capturing closure to pass, you need to use a trait bound like Fn

fn apply_square_fn<F>(f: F) -> i32
where
    F: Fn() -> i32,
{
    f()
}
let x = 4;
let closure = || x * x;
let result = apply_square_fn(closure);

The main thing to remember is that fn(i32) -> i32 is a function pointer type. It's only for real functions or some non-capturing closures that can be turned into a fn pointer. If you set your parameter as a function pointer type, you can't pass a capturing closure. It does not capture any state.

FnMut Trait

A closure that only needs a mutable reference to its environment. It can be called repeatedly, but it might mutate captured variables. FnMut is implemented automatically by closures which take mutable references to captured variables, as well as all types that implement Fn. Any FnMut is automatically also an FnOnce, because if you can call something multiple times, you can at least call it once.

In the below example, the closure changes count. Therefore, it needs a mutable borrow of count. That means it implements at least FnMut.

let mut count = 0;
// This closure captures `count` by mutable reference,
let mut increment = || { count += 1  }; // compiler infers `increment: FnMut() -> ()`.

increment(); //"count: 1"
increment(); //"count: 2"
println!("Final count: {}", count); // 2

Using FnMut as a Trait Bound

Often, you’ll write generic functions that accept any closure (or function) with certain capabilities. If your function needs to call a closure multiple times and allow it to mutate captured state, you should use an FnMut bound.

// It requires that `f` can be called multiple times AND can mutate state.
fn call_multiple_times<F>(mut f: F, times: usize)
where
    F: FnMut(),
{
    for _ in 0..times {
        f();
    }
}

fn main() {
    let mut counter = 0;

    let mut increment = || {
        counter += 1;
        println!("counter: {}", counter);
    };
    // The closure implements `FnMut`, so it's valid here.
    call_multiple_times(increment, 3);
    // Output:
    // counter: 1
    // counter: 2
    // counter: 3

    println!("Final counter: {}", counter); // 3
}

FnMut is the trait for callables that may mutate captured state and can be called multiple times.

FnOnce Trait

A closure that can be called at least once. It may consume (move) variables it captures, so it can only be called once if it moves something out of the environment. Every closure implements FnOnce because, at the bare minimum, you can call it once.

In simple terms, if a closure takes ownership of data from its environment, it's at least FnOnce. It could also be FnMut or Fn, based on whether it changes or just reads that data.

Every closure implements at least one of these traits, and sometimes more, depending on how it handles variables.

Essentially:

FnOnce <= FnMut <= Fn

(FnOnce is the “weakest” requirement; Fn is the “strongest”)

Manually Implementing Fn

In most real-world Rust code, you do not manually implement Fn, because closures and function pointers automatically implement it. However, if you want a custom struct to behave like a function, you need to implement Fn, FnMut, and FnOnce by hand.

Important: This is only possible with nightly Rust, using special features. You can manually implement Fn, FnMut, or FnOnce by enabling unboxed_closures and fn_traits, using the extern "rust-call" syntax.

Below is a nightly-only example. It shows a struct MultiplyBy that holds a multiplier. We implement all three traits so that it can be called like a function.

#![feature(fn_traits)]
#![feature(unboxed_closures)]
use std::ops::{Fn, FnMut, FnOnce};

struct MultiplyBy(i32); // A struct that holds a multiplier.

//1) Implement FnOnce for MultiplyBy. Notice the signature uses `call_once(self, (arg,): (i32,))`.
impl FnOnce<(i32,)> for MultiplyBy {
    type Output = i32;

    extern "rust-call" fn call_once(self, (arg,): (i32,)) -> Self::Output {
        self.0 * arg
    }
}
//2)Implement FnMut for MultiplyBy. This time the method is `call_mut(&mut self, (arg,): (i32,))`.
impl FnMut<(i32,)> for MultiplyBy {
    extern "rust-call" fn call_mut(&mut self, (arg,): (i32,)) -> Self::Output {
        self.0 * arg
    }
}
// 3) Finally, implement Fn for MultiplyB. Now we use `&self`.
impl Fn<(i32,)> for MultiplyBy {
    extern "rust-call" fn call(&self, (arg,): (i32,)) -> Self::Output {
        self.0 * arg
    }
}
fn main() {
    let multiply_by_10 = MultiplyBy(10);
    // Because we've implemented Fn, we can call our struct like a function:
    println!("{}", multiply_by_10(5));  // 50
    println!("{}", multiply_by_10(12)); // 120
}

Why implement all three?
In Rust, Fn is a supertrait of FnMut, which is a supertrait of FnOnce. So you need to implement them in ascending order if you want a type to fully behave like a closure that can be called multiple times without mutation.

• FnOnce says “I can be called once.”

• FnMut says “I can be called multiple times, mutably.”

• Fn says “I can be called multiple times, immutably.”

Under the hood: Anonymous Struct Holding Captures

When you write a closure, Rust will create a hidden struct (let’s call it the closure object) that:

1. Has fields for each variable it captures.

2. Implements one or more of the FnOnce, FnMut, and/or Fn traits.

3. Implements a “call” method (i.e. call, call_mut, or call_once) that executes the closure body, referencing or consuming those fields as needed.

Compiler generates this struct with name something like closure@ID.

fn main() {
    let mut count = 0;
    let mut increment = || {
        count += 1;
        println!("count: {}", count);
    };

    increment();
    increment();
}

Conceptually, Rust might generate a struct like:

struct Closure_CountMut<'a> {
    count_ref: &'a mut i32,
}

// This closure implements FnMut, because it mutates its captures.
impl<'a> FnMut(()) for Closure_CountMut<'a> {
    extern "rust-call" fn call_mut(&mut self, _args: ()) {
        *self.count_ref += 1;
        println!("count: {}", *self.count_ref);
    }
}

fn main() {
    let mut count = 0;
    // The closure is constructed
    let mut increment = Closure_CountMut {
        count_ref: &mut count,
    };

    // When we call it, it calls `call_mut`
    increment.call_mut(());
    increment.call_mut(());
}

You can say that closure is syntactic sugar for an anonymous struct that holds the captured variables.

This is all done automatically by the compiler, but understanding this model helps clarify why closures have their particular capturing rules and trait bounds.

Summary

In summary, I would say if you need a small piece of logic that depends on local variables, but you don’t want to define a whole new named function. That’s exactly where closures come in. A closure is essentially an anonymous function that automatically captures variables from the scope in which it’s defined. Sometimes it merely borrows these variables (immutably or mutably), and other times it takes full ownership—Rust decides which strategy to use based on how you interact with those variables.

Once created, the closure acts like a callable object, implementing one or more of the function traits—Fn, FnMut, or FnOnce—depending on whether it only reads, mutates, or consumes its captured data. In cases where a closure doesn’t capture anything at all, the compiler can even coerce it into a plain function pointer.

Under the hood, Rust constructs an anonymous struct to hold whatever variables the closure captures, then it implements the relevant trait methods for that struct. Most of the time, these traits are derived automatically by the compiler.

Ultimately, this allows you to write concise, powerful inline functions that interact seamlessly with their surrounding environment—without ever losing track of Rust’s safety and performance guarantees. When it comes to performance, the use of closures in Rust is truly remarkable. Closures are designed to be efficient and fast, allowing you to write code that executes quickly without sacrificing safety or functionality. The Rust compiler is highly optimized for closures.

Immutability in Rust

In Rust, immutability is a core feature and is enforced by the language's ownership and borrowing system. Variables are immutable by default. Once a value is assigned to a variable, it cannot be changed unless explicitly declared as mutable.

let x = 5; // x is immutable
// x = 6; // This would cause a compile-time error
let mut y = 10; // y is mutable
y = 20; // This is allowed

Rust's immutability ensures thread safety and prevents data races. Immutable data can be shared across threads without the risk of concurrent modification. Rust's borrow checker enforces rules around mutable and immutable references to ensure memory safety.

Rust’s borrowing system ensures memory safety:

• Any number of immutable references (&T) to a resource is allowed simultaneously.

• You can have either a single mutable reference (&mut T) or any number of immutable references, but not both at the same time.

If you have a struct instance that is immutable, then its fields are also immutable—you can't change them.

struct Point {
    x: i32,
    y: i32,
}
let p = Point { x: 1, y: 2 }; // p is immutable
// p.x = 3; // This would cause a compile-time error

Immutability in C++

C++ does not enforce immutability by default, but it provides tools to achieve it. Variables are mutable unless explicitly marked as const. The const keyword is used to declare immutable variables or pointers.

const int x = 5; // x is immutable
// x = 6; // This would cause a compile-time error

In C++, you can mark member functions as const to indicate that they do not modify the object's state.

class Point {
public:
    int getX() const { return x; } // This function does not modify the object
private:
    int x;
};

Immutability in Java

Java provides immutability through the final keyword and immutable classes. The final keyword is used to declare immutable variables, methods, or classes.

final int x = 5; // x is immutable
// x = 6; // This would cause a compile-time error

In Java, immutable classes are created by making all fields final and not providing setters.

public final class Point {
    private final int x;
    private final int y;
    public Point(int x, int y) {
        this.x = x;
        this.y = y;
    }
    public int getX() { return x; }
    public int getY() { return y; }
}

Embrace immutability where you can, and you’ll find your code more predictable, maintainable, and (in many cases) faster or easier to optimize.

The building blocks of OOPs are attributes, methods, classes and object. The principles of OOPs are encapsulation, abstraction, inheritance and polymorphism. We will discuss these blocks and principles.

Rust isn't "object-oriented" in the usual inheritance-based way, but it offers tools—structs, impl blocks, and traits—that let you use many of the same design patterns: encapsulation, abstraction, and polymorphism.

In fact, Rust's approach is very close to object oriented way of thinking, can help you create more maintainable and strong systems.

Encapsulation

Encapsulation is a core principle of object-oriented programming (OOP). It refers to bundling data (fields/attributes) and behavior (methods) together. In Rust, this is done by defining structs (similar to classes in other programming languages). Beyond just packaging data, encapsulation also involves hiding an object's internal state and implementation details, while providing a clear and accessible way to interact with it. In many traditional OOP languages like Java, C#, and C++, encapsulation is achieved using private fields (attributes) and public methods (getters, setters, business logic) to control access to these fields.

Structs

In classical OO languages:

• A class is a blueprint that defines the data (fields or attributes) and behaviors (methods).

• An object is an instance of a class.

In Rust, structs can be used to group related data (similar to fields in a class). Along with impl blocks, which define associated functions and methods, struct + impl can serve a similar role to classes.

Attributes are variables that represent the state of the object. In the below example future contract attributes represent its state. In other words attributes (or fields) refer to the data contained within an object.

Example: A Basic Future Contract Struct

A futures contract has attributes such as:

• The underlying asset (e.g., a commodity, stock index)

• The quantity or number of units

• The maturity date

• The entry price or the price at which the contract was agreed

Here’s an example of how we can represent these attributes in Rust:

#[derive(Debug)]
struct FutureContract {
    underlying: String,
    quantity: u64,
    maturity_date: String,
    entry_price: f64,
}

This FutureContract struct holds all the relevant data for our futures contract. It’s analogous to defining a class with four attributes in an OO language.

We can create instances of a struct (akin to objects) by calling the struct’s name with the field values.

fn main() {
    let future = FutureContract {
        underlying: String::from("Crude Oil"),
        quantity: 100,
        maturity_date: String::from("2025-12-31"),
        entry_price: 60.50,
    };

    println!("Created a future: {:?}", future);
}

Created a future: FutureContract { underlying: "Crude Oil", quantity: 100, maturity_date: "2025-12-31", entry_price: 60.5 }

Here, future is an instance of FutureContract, just as an object would be an instance of a class in a classical OO paradigm.

Rust ensures that objects (instances of structs) are in a valid state after they are created. In an object-oriented way of thinking, an object must have a state; it cannot exist without a valid state. For example, if you create a point with x-axis and y-axis for a 2D coordinate system, a point object must have valid coordinates. You cannot create a point where you know the x-coordinate but the y-coordinate is unknown. Rust provides this by default. In C# you can achieve this using the required modifier.

#[derive(Debug)]
struct point{
    x:i64 ,
    y: i64
}
fn main() {
    let a = point{x:1};
    println!("Created a point: {:?}", a);
}

error[E0063]: missing field `y` in initializer of `point`
 --> src/main.rs:8:13
  |
8 |     let a = point{x:1};
  |             ^^^^^ missing `y`

If we try to create a point without y coordinate we get compile time error.

Methods and Associated Functions

In many OO languages, methods are defined within the class. Rust does not store methods within structs but uses impl blocks (implementation blocks) to define methods associated with a given struct.

Let’s add a constructor-like function (commonly named new in Rust). Rust does not have constructors in the classical sense. Instead, Rust uses associated functions (often called “factory” or “constructor-like” functions, typically named new) within an impl block to initialize structs.

Another method to calculate the contract’s notional value (quantity times entry price) and a method to settle the contract.

impl FutureContract {
    /// Associated function (like a constructor)
    fn new(underlying: &str, quantity: u64, maturity_date: &str, entry_price: f64) -> Self {
        FutureContract {
            underlying: String::from(underlying),
            quantity,
            maturity_date: String::from(maturity_date),
            entry_price,
        }
    } 
    /// Calculate the notional value of the future
    fn calculate_notional(&self) -> f64 {
        self.quantity as f64 * self.entry_price
    }
    /// Some method representing contract settlement
    fn settle(&self, settlement_price: f64) -> f64 {
        // Gain or loss = (Settlement price - Entry price) * Quantity
        let pnl = (settlement_price - self.entry_price) * self.quantity as f64;
        pnl
    }
}

To define the function within the context of FutureContract, we start an impl (implementation) block for FutureContract. Everything within this impl block will be associated with the FutureContract type.

All functions defined within an impl block are called associated functions because they’re associated with the type named after the impl. The associated functions that take self as parameters are called methods because they need instance of the struct. Associated functions that aren’t methods (don’t need self in parameters) are often used for constructors that will return a new instance of the struct. These are often called new, but new isn’t a special name and isn’t built into the language. The new function in our struct doesn’t take self and act as a constructor.

The Self keywords in the return type and in the body of the function are aliases for the type that appears after the impl keyword, which in this case is FutureContract. To call this associated function, we use the :: syntax with the struct name.

fn main() {
    let oil_future = FutureContract::new("Crude Oil", 100, "2025-12-31", 60.50);
    let notional = oil_future.calculate_notional();
    println!("Notional value: ${:.2}", notional);
    let pnl = oil_future.settle(65.00);
    if pnl > 0.0 {
        println!("Profit: ${:.2}", pnl);
    } else {
        println!("Loss: ${:.2}", pnl.abs());
    }
}

Self, self and &self in Methods

Self (capital “S”) refers to the type within an impl block. For instance, in our impl FutureContract { ... } block, Self is an alias for FutureContract. Typically used in associated functions (which do not take a self parameter) to construct or return an instance of the same type, for example, the new function in impl FutureContract { ... }. Here, Self is simply FutureContract. If you rename the struct, you won’t need to update the type references in the methods—Self will remain valid as it automatically refers to the type in the current impl.

Additionally, you can also see Self used as a return type in methods that consume self and return another struct of the same type, or partial transformations:

impl FutureContract {
    fn into_option(self) -> Option<Self> {
        // Example of returning an Option of Self.
        Some(self)
    }
}

Here, Option<Self> could also be written as Option<FutureContract>.

Rust requires us to explicitly declare how we want to receive the instance: (self), by immutable reference (&self), or by mutable reference (&mut self).

&self — Immutable Reference

When a method signature takes &self, it borrows the struct immutably. This means the method can read the fields of the struct but cannot modify them. calculate_notional borrows self immutably, we can call it multiple times on the same instance, even simultaneously, as long as we don’t try to have mutable reference.

self — Ownership Transfer

A method that takes self literally consumes the struct instance. After calling this method, the caller loses ownership unless something is returned.

&mut self — Mutable Reference

A method with &mut self borrows the instance mutably, allowing the method to modify the struct’s fields.

impl FutureContract {
    // A method that mutates the struct
    fn update_price(&mut self, new_price: f64) {
        self.entry_price = new_price;
    }
}

Only one mutable reference to a given object can exist at a time, enforcing Rust’s borrow-checker rules.

The self is similar to self in Python, in both Python and Rust, “self” is the way you reference the current instance of a type. Both are explicit in the sense that they appear in the method signature. Rust requires you to be explicit about the ownership and borrowing (self, &self, &mut self). In Python, you can read and write to attributes through self without any explicit ownership or mutability rules. Rust enforces these constraints via the type system.

Associated Functions vs. Static Methods in Other Languages

In many object-oriented languages (e.g., Java, C#), you can mark a method as static, meaning it does not need an instance of the class to be called. This approach goes against the principles of OOP and can lead to less maintainable code. This might be why there is no specific static keyword for methods in Rust.

As we discussed earlier, associated functions that do not have a self parameter are called directly from the type, like this: TypeName::function_name(), instead of from an instance of the type. These associated functions are the closest equivalent to a "static method."

In JAVA/C#, you write public static void foo() { ... }, and call MyClass.foo(). In C++, You might have static void foo(); inside a class, and call MyClass ::foo();. Python uses @staticmethod decorator. In Rust, again, an associated function with no self is your “static method.”

You write:

impl MyStruct {
    fn foo() {
        // ...
    }
}

and call MyStruct::foo().

No static field inside a struct

Using the static keyword on a class field means that all instances share that field, which isn't really in line with OOP principles and can lead to issues. Many object-oriented languages have moved away from OOP principles to include this feature. In Java/C#, the static keyword is used on a class field and is usually accessed with ClassName.staticField. C++ follows the same idea. Python doesn't have a static keyword but uses class attributes that all instances share. Luckily, Rust doesn't directly support a "static field inside a struct." It avoids this behavior and sticks more closely to OOP principles in this respect.

pub keywords

Encapsulation is also a practice of restricting direct access to some of the object's components. In Rust, this is done using pub (public) and private visibility modifiers.

Overloading

When you have two or more methods with the same name but different parameters, that's called overloading. Rust does not support function or method overloading in the same way that languages like C++, Java, or C# do. In those languages, you can define multiple functions with the same name but different parameter lists or types (e.g., foo(int x) vs. foo(double x) vs. foo(int x, int y)).

Rust, however, disallows having two or more functions in the same scope.

#[derive(Debug)]
struct point{
    x:i64 ,
    y:i64
}
impl point{
    fn my_point(){
        println!("my_point function with no parameter");
    }
    fn my_point(_x:i64){
        println!("my_point function with int parameter");
    }
}
fn main() {
    let a = point{x:1,y:2};
    println!("Created a point: {:?}", a);
    point::my_point();
}

Rust does not allow these two my_point methods to coexist, even though they have different signatures, because it doesn’t support method overloading by parameter signatures.

error[E0592]: duplicate definitions with name `my_point`
  --> src/main.rs:10:5
   |
7  |     fn my_point(){
   |     ------------- other definition for `my_point`
...
10 |     fn my_point(_x:i64){
   |     ^^^^^^^^^^^^^^^^^^^ duplicate definitions for `my_point`

Some would argue that overloadimg makes code base more readable and manageable, but Rust don’t believe in this. Overloading can obscure which function is actually getting called, especially if conversions are implicit. Rust reduce confusion and make sure you have cleaner code. You can use some pattern to achieve this.

Polymorphism

Polymorphism is a big idea in programming languages. It basically means "many forms." Depending on the situation, your functions can take on different forms and behave differently. For example, function overloading is one kind of polymorphism. Generics are another type, where a function or struct is written in a way that it can work with different types while still keeping everything type-safe, like this: fn do_something<T: SomeTrait>(x: T) { ... }.

In the context of object-oriented concepts, we're particularly interested in subtype polymorphism (often called inheritance-based polymorphism or inclusion polymorphism), where an object (or reference) of a subtype can be used anywhere its supertype is expected.

In classical OOP languages (e.g., Java, C#), subtype polymorphism typically relies on inheritance: a derived class “is a kind of” the base class and can be used in any context expecting the base. Rust does not support inheritance in that traditional sense, but it provides a powerful mechanism—trait objects—which enable runtime polymorphism (often called “dynamic dispatch”).

Rust uses traits to define shared behaviors or capabilities. In Rust, when we talk about subtype polymorphism, it means that if different concrete types implement the same trait, you can keep them in the same data structure or pass them to the same function by just referring to that trait.

Traits

Traits are fundamental feature in Rust that provide a way for shared behavior, abstraction and polymorphism. You can think of traits as interfaces in other programming languages. This is how The Rust Programming Language (“the Rust book”) defines traits:

A trait tells the Rust compiler about functionality a particular type has and can share with other types. We can use traits to define shared behavior in an abstract way. We can use trait bounds to specify that a generic can be any type that has certain behavior.

This is specially important as Rust is not an object-oriented language, in a traditional sense. Struct doesn't support inheritance.

Define a Trait

Let's say we have a variety of financial instruments from different asset classes like equity, equity options, and interest rate derivatives. We need a few generic methods on all the contracts so that we can analyze the portfolio of trades. We define a trait named Contract with one method named npv. The may look like a virtual method in C++.

    pub trait Contract {
        fn npv(&self) -> f64;
    }

In this example, Contract is a trait that requires an implementing type to define the npv method. The npv method takes a reference to self and returns a f64 value.

Implement a Trait

Any type that implements the Contract trait must provide an implementation of the npv method. Here is an example of a struct named EquityOption that implements the Contract trait. impl keyword is used to implement traits for a struct. I believe the impl keyword is inspired by the implements keyword in Java.

    pub struct EquityOption {
        pub strike: f64,
        pub expiry: f64,
        pub is_call: bool,
    }
    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

In this example, EquityOption is a struct that implements the Contract trait. If you're familiar with C++, it's like class EquityOption: public Contract. Now, let's create another struct called InterestRateSwap that also implements the Contract trait.

    pub struct InterestRateSwap {
        pub notional: f64,
        pub fixed_rate: f64,
        pub floating_rate: f64,
    }
    impl Contract for InterestRateSwap {
        fn npv(&self) -> f64 {
            println!("Calculating npv for InterestRateSwap");
            0.0
        }
    }

We have totally different asset classes, EquityOption and InterestRateSwap, but they both implement the Contract trait. We can create a portfolio of equity options and interest rate swaps and calculate the net present value of the portfolio.

    fn calculate_portfolio_npv(contracts: Vec<&dyn Contract>) -> f64 {
        let mut npv = 0.0;
        for contract in contracts {
            npv += contract.npv();
        }
        npv
    }

Here, dyn Contract is a trait object. This means we can pass a vector of any type that implements the Contract trait. This is an example of dynamic dispatch. We'll talk more about this when we dive into dyn in detail. Now, let's take a look at another implementation of the same function using static dispatch. Traits can also be used to set constraints on generic types, known as trait bounds. Here's an example of a generic function that calculates the net present value of a portfolio of contracts.

    fn calculate_portfolio_npv<T: Contract>(contracts: Vec<T>) -> f64 {
        let mut npv = 0.0;
        for contract in contracts {
            npv += contract.npv();
        }
        npv
    }

In this example, T: Contract is a trait bound that specifies that T must implement the Contract trait. This is called static dispatch. The compiler generates a separate version of calculate_portfolio_npv for each type T used.

Static vs Dynamic Dispatch

Rust support both static and dynamic dispatch.

• Static dispatch is resolved at compile time. The compiler knows the exact type at compile time and method calls are resolved during compilation.

• Dynamic dispatch is resolved at runtime. The compiler doesn't know the exact type at compile time and method calls are resolved via virtual table (vtable), similar to virtual function in C++.

Rust's traits are closely tied to its ownership and borrowing model, and it clearly distinguishes between static and dynamic dispatch.

Inheritance

Actually no inheritance, no super or “abstract class” or no deep class hierarchies. You can’t call a parent class method from a child; Rust’s approach is about interfaces and implementations. Inheritance in rust is only for traits. You can define a trait that inherits from another trait. Here is an example of a trait named Option that inherits from the Contract trait.

pub trait Option: Contract {
        fn delta(&self) -> f64;
    }

In this example, Option is a trait that inherits from the Contract trait and requires an implementing type to define the delta method. If we have a struct named EquityOption that implements the Option trait, it must provide implementations of both the npv and delta methods by implementing respective traits.

    pub struct EquityOption {
        pub strike: f64,
        pub expiry: f64,
        pub is_call: bool,
    }

    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

    impl Option for EquityOption {
        fn delta(&self) -> f64 {
            println!("Calculating delta for EquityOption");
            0.0
        }
    }

If you implement the Option traits then you must implement the Contract trait as well, otherwise you have error like this:

    error[E0277]: the trait bound `EquityOption: Contract` is not unfulfilled

This is another example of trait bounds in Rust. It forces the implementing type to implement all the required traits.

Default Implementations of Traits

You can provide default implementations for trait methods.

    pub trait Contract {
        fn npv(&self) -> f64 {
            println!("Calculating npv for Contract");
            0.0
        }
    }

In this example, Contract is a trait that provides a default implementation for the npv method. In this case, you don't have to provide the implementation of the npv method in the implementing type.

impl Contract for EquityOption {}

Multiple Trait Implementations

You can have only one implementation for each type.

    pub trait Contract {
        fn npv(&self) -> f64;
        fn expiry(&self) -> f64;
    }
    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

    impl Contract for EquityOption {
        fn expiry(&self) -> f64 {
            println!("Calculating expiry for EquityOption in days");
            0.0
        }
    }

This will give you an error like this:

    error[E0119]: conflicting implementations of trait `Contract` for type `EquityOption`

This prevents the ambiguity in the code. You can have multiple traits but only one implementation for each type.

Abstraction

Abstraction is a key idea in Object-Oriented Programming (OOP) that lets you hide complex implementation details and show only the essential features or functions to the user. In Rust, you achieve abstraction using traits, modules, and encapsulation.

Traits define a set of methods that a type must implement, allowing you to abstract over behavior without specifying the implementation.

Modules (Organizing and Abstracting Code)

Rust’s module system allows you to organize code into logical units and control visibility. By default, items in a module are private, and you can expose only what’s necessary using pub.

A trait tells the Rust compiler about functionality a particular type has and can share with other types. We can use traits to define shared behavior in an abstract way. We can use trait bounds to specify that a generic can be any type that has certain behavior.

This is specially important as Rust is not an object-oriented language, in a traditional sense. Struct doesn't support inheritance. You can not define an abstract base struct and inherit to make a derived struct. To achieve the polymorphism in Rust, you need a pointer (reference) that can hold references of any type that have shared behavior. This is where traits come in.

Define a Trait

Let's say we have a variety of financial instruments from different asset classes like equity, equity options, and interest rate derivatives. We need a few generic methods on all the contracts so that we can analyze the portfolio of trades. We define a trait named Contract with one method named npv. The may look like a virtual method in C++.

    pub trait Contract {
        fn npv(&self) -> f64;
    }

In this example, Contract is a trait that requires an implementing type to define the npv method. The npv method takes a reference to self and returns a f64 value.

Implement a Trait

Any type that implements the Contract trait must provide an implementation of the npv method. Here is an example of a struct named EquityOption that implements the Contract trait. impl keyword is used to implement traits for a struct. I think the impl keyword comes from implements keyword in Java.

    pub struct EquityOption {
        pub strike: f64,
        pub expiry: f64,
        pub is_call: bool,
    }

    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

In this example, EquityOption is a struct that implements the Contract trait. If you are familiar with C++ then it is similar to class EquityOption: public Contract. Let's create one more struct named InterestRateSwap that implements the Contract trait.

    pub struct InterestRateSwap {
        pub notional: f64,
        pub fixed_rate: f64,
        pub floating_rate: f64,
    }

    impl Contract for InterestRateSwap {
        fn npv(&self) -> f64 {
            println!("Calculating npv for InterestRateSwap");
            0.0
        }
    }

We have totally different asset classes, EquityOption and InterestRateSwap, but they both implement the Contract trait. We can create a portfolio of equity options and interest rate swaps and calculate the net present value of the portfolio.

Polymorphism with Traits

    fn calculate_portfolio_npv(contracts: Vec<&dyn Contract>) -> f64 {
        let mut npv = 0.0;
        for contract in contracts {
            npv += contract.npv();
        }
        npv
    }

Here, dyn Contract is a trade object. We can pass a vector of any type that implements the Contract trait. This is example of dynamic dispatch. We will discuss more shortly when discuss dyn in details. Let's understand see the another implementation of the same function with static dispatch. Traits can also be used to impose constraints on generic types, known as trait bounds. Here is an example of a generic function that calculates the net present value of a portfolio of contracts.

    fn calculate_portfolio_npv<T: Contract>(contracts: Vec<T>) -> f64 {
        let mut npv = 0.0;
        for contract in contracts {
            npv += contract.npv();
        }
        npv
    }

In this example, T: Contract is a trait bound that specifies that T must implement the Contract trait. This is called static dispatch. The compiler generates a separate version of calculate_portfolio_npv for each type T used.

Static vs Dynamic Dispatch

Rust support both static and dynamic dispatch.

• Static dispatch is resolved at compile time. The compiler knows the exact type at compile time and method calls are resolved during compilation.
• Dynamci dispatch is resolved at runtime. The compiler doesn't know the exact type at compile time and method calls are resolved via virtual table (vtable), similar to virtual function in C++.

Rust's traits are closed tied to its ownsership and borrowing model, and it distinguishes between static and dynamic dispatch explicitly.

Inheritance in Rust

Inheritance in rust is onlt for traits. You can define a trait that inherits from another trait. Here is an example of a trait named Option that inherits from the Contract trait.

    pub trait Option: Contract {
        fn delta(&self) -> f64;
    }

In this example, Option is a trait that inherits from the Contract trait and requires an implementing type to define the delta method. If we have a struct named EquityOption that implements the Option trait, it must provide implementations of both the npv and delta methods by implementing respective traits.

    pub struct EquityOption {
        pub strike: f64,
        pub expiry: f64,
        pub is_call: bool,
    }

    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

    impl Option for EquityOption {
        fn delta(&self) -> f64 {
            println!("Calculating delta for EquityOption");
            0.0
        }
    }

If you implement the Option traits then you must implement the Contract trait as well, otherwise you have error like this:

    error[E0277]: the trait bound `EquityOption: Contract` is not unfulfilled

This is another example of trait bounds in Rust. It forces the implementing type to implement all the required traits.

Default Implementations

You can provide default implementations for trait methods.

    pub trait Contract {
        fn npv(&self) -> f64 {
            println!("Calculating npv for Contract");
            0.0
        }
    }

In this example, Contract is a trait that provides a default implementation for the npv method. In this case, you don't have to provide the implementation of the npv method in the implementing type.

impl Contract for EquityOption {}

Multiple Trait Implementations

You can have only one implementation for each type.

    pub trait Contract {
        fn npv(&self) -> f64;
        fn expiry(&self) -> f64;
    }
    impl Contract for EquityOption {
        fn npv(&self) -> f64 {
            println!("Calculating npv for EquityOption");
            0.0
        }
    }

    impl Contract for EquityOption {
        fn expiry(&self) -> f64 {
            println!("Calculating expiry for EquityOption in days");
            0.0
        }
    }

This will give you an error like this:

    error[E0119]: conflicting implementations of trait `Contract` for type `EquityOption`

This prevents the ambiguity in the code. You can have multiple traits but only one implementation for each type.

Using dyn with Traits

The dyn keyword is used to indicate a trait object, making it explict when dynamic dispatch is used. A trait object is a pointer to some type that implements a trait, but exact type is not known at compile time. Here is an example of a function that takes a trait object as an argument.

    fn calculate_npv(contract: &dyn Contract) -> f64 {
        contract.npv()
    }
    fn main() {
        let equity_option = EquityOption {
            strike: 100.0,
            expiry: 1.0,
            is_call: true,
        };
        let interest_rate_swap = InterestRateSwap {
            notional: 1000000.0,
            fixed_rate: 0.05,
            floating_rate: 0.03,
        };
        println!("Equity Option NPV: {}", calculate_npv(&equity_option));
        println!("Interest Rate Swap NPV: {}", calculate_npv(&interest_rate_swap));
    }

In this example, calculate_npv takes a reference to a trait object that implements the Contract trait.

Using dyn with Box

You can also use the Box type to create a trait object.

    fn main() {
        let equity_option = EquityOption {
            strike: 100.0,
            expiry: 1.0,
            is_call: true,
        };
        let interest_rate_swap = InterestRateSwap {
            notional: 1000000.0,
            fixed_rate: 0.05,
            floating_rate: 0.03,
        };
        let contract: Box<dyn Contract> = Box::new(equity_option);
        println!("Equity Option NPV: {}", contract.npv());
        let contract = Box::new(interest_rate_swap) as Box<dyn Contract>;
        println!("Interest Rate Swap NPV: {}", contract.npv());
    }

By boxing and casting it to Box, you create a trait object on the heap.

Traits with Generics

Blanket Implementations

Rust allows you to provide a blanket implementation for all types that satisfy a certain trait bound. Here is an example of a blanket implementation. We define two more trait DiscountFactorProvider and CashFlow.

    pub trait DiscountFactorProvider {
    fn discount_factor(&self, date: NaiveDate) -> f64;
    }
    pub trait CashFlow {
        fn amount(&self) -> f64;
        fn date(&self) -> NaiveDate;
    }

We can implement Contract with CashFlow trait

    impl<T> Contract for T
    where
        T: CashFlow + DiscountFactorProvider,
    {
        fn npv(&self) -> f64 {
            let df = self.discount_factor(self.date());
            let npv = self.amount() * df;
            println!("Calculating NPV using CashFlow and DiscountFactorProvider traits.");
            npv
        }
    }

Any type implementing both CashFlow and DiscountFactorProvider will have npv implementation. Let's consider another example: Suppose you work on equity asset class and other team worked on fixed income asset class. Now your teams wants to do the cross asset combine Bonds with Equity. Bond doesn't have the npv method instead it has price method and different traits.

pub trait Pricable {
    fn price(&self) -> f64;
}
use chrono::NaiveDate;
pub struct Bond {
    pub face_value: f64,
    pub coupon_rate: f64,
    pub maturity: NaiveDate,
}
impl Pricable for Bond {
    fn price(&self) -> f64 {
        // Simplified pricing logic
        self.face_value * (1.0 + self.coupon_rate)
    }
}

Since Bond implements Pricable, we can implement Contract for Bond by using the blanket implementation.

impl<T> Contract for T
where
    T: Pricable,
{
    fn npv(&self) -> f64 {
        println!("Calculating NPV using Pricable trait.");
        self.price()
    }
}

The npv method uses the price method from Pricable.

Using Traits with Generics Types

You can use traits with generic types to provide more flexibility.

pub trait EventHandler<E> {
    fn handle_event(&mut self, event: E);
}

EventHandler handles events of different types, useful in event-driven trading systems. E is a generic type that represents the event type. Here is an example of a struct named EventProcessor that implements the EventHandler trait.

pub struct EventProcessor {
    pub name: String,
}
impl<E> EventHandler<E> for EventProcessor {
    fn handle_event(&mut self, event: E) {
        println!("{} handling event", self.name);
    }
}

Summary

In summary, traits enables ad-hoc polymorphism in Rust, allowing functions to operate on any type, as long as it satisfies certain constraints. Understanding how to define and use traits with generics is essential for leveraging Rust's strengths in abstraction and performance. Generics allow your traits to work with a variety of data types and structures, enabling you to model complex behaviors in a flexible and efficient way.

std::vector<int> vec = {10, 20, 5, 30, 15};
auto max_iter = std::max_element(vec.begin(), vec.end());
if (max_iter != vec.end()) {
    std::cout << "The maximum value is " << *max_iter << "\n";
} else {
    std::cout << "The vector is empty.\n";
}

Exercise 2: Remove the last element from the vector

To retrieve and remove the last element of a std::vector in C++, you can use a combination of the back() and pop_back() member functions. Alternatively using Iterators, you can use end and erase although this approach is less efficient for removing than pop_back();

int lastElement = vec.back();  // Get the last element 15
vec.pop_back(); // // Remove 15
auto it = vec.end() - 1;
int lastElement = *it;
myVector.erase(it);

“Design” as a word is both an activity (verb) and a product (noun). As a verb it’s the process of thinking through and creating blueprints for classes, architectures, or systems. Design as a Noun is the artifact that results from the design activity. It might be a blueprint for a class, a system’s architectural diagram, or an entire ecosystem of interacting components.

When we talk about “design,” we’re not just envisioning static documents or diagrams; we’re imagining a dynamic ecosystem, much like a pond where water flows through pipes, where fish of different species interact, and where various environmental factors maintain a stable balance. A well-designed software system similarly has multiple components—classes, modules, services, or patterns—that function together harmoniously, each constrained by and supportive of the larger architecture. Similarly software design imposes conditions on components, dictating how they can or cannot interact.

In object-oriented software design, design patterns are like ecological rules or best practices that address common problems and encourage stable interactions:

• Creational Patterns: Provide ways to instantiate objects

• Structural Patterns: Arrange classes and objects

• Behavioral Patterns: Govern how objects communicate and respond

While design patterns solve localized design issues, software architecture design looks at the macro-level:

• Layered Architectures: Separate responsibilities into layers

• Service-Oriented or Microservices: Break the system into independent yet interacting services

• Event-Driven Architectures: Focus on asynchronous communication and flows

Why Emphasize Design?

1. Maintainability: A thoughtful design is easier to extend and adapt.

2. Scalability: Well-chosen architectural decisions can accommodate growing demands without fundamental redesign.

3. Reusability: Components, once designed, can be repurposed in future projects or domains.

4. Predictability: Like a stable ecosystem, a well-architected system exhibits predictable behaviors, simplifying testing and debugging.

The main focus of this blog is design patterns using object oriented concepts.

What is Algorithmic Trading?

Algorithmic trading models are automated trading strategies that use predefined rules, instructions and quantitative techniques to execute trades in financial markets. These are highly quantitative process used by large financial institutions. Now when technology touches every part of our lives, it's no surprise that algorithmic trading, once only used by big firms, is now available to individual traders and investors. This democratization has created many opportunities, but it also brings challenges and risks. This section will introduce you to algorithmic trading.

As executing large buy or sell order will push the price higher or lower that is also a cost for the firm. This cost associated with this is market impact cost. The algorithmics that aim to minimize the market impact cost along with other execution cost such as bid-ask spread are called order execution algorithms.

The algorithmic trading / systematic trading firms called buy-side develop algorithms / models aim to capitalize on market inefficiencies, trends or patterns to generate trading signals by analyzing vast amounts of data. There are also market making firms (called sell-side) that use trading strategy to foster market liquidity by continuously quoting both bid and ask prices for a specific security, effectively creating a two-way market. By actively participating in the market, market markers profit from the bid-ask spread and rebates.

At the heart of an algorithmic trading system are several essential components, including Order Management Systems (OMS), Execution Management Systems (EMS), backtesting frameworks and many more. Before exploring the design patterns, let's take a moment to discuss these components briefly to gain a better understanding of the design pattern examples. In addition to the algorithmic trading components, we will also cover essential terms related to the trading system. In this blog, as we implement various design pattern, we will learn Rust core concepts, data structures and algorithmic trading concepts.

Order management systems (OMS)

Order management systems (OMS) play a crucial role in monitoring and executing trades for clients, such as portfolio managers and traders. For instance, when a portfolio manager decides to purchase 5,000 shares of NVDA at a limit price of $130 per share, they enter this order into the OMS. The OMS checks whether the order violates fund's concentration limit and if it does then alerts the PM. If all okay then analyzes current market conditions and routes the order to an exchange where the likelihood of execution at the desired price is highest. After execution, OMS sends trade details to the back-office system.

It's impressive how OMS can manage such high volumes of orders from various sources, control the flow, validate them, and ensure that trades are completed both efficiently and accurately.

Order Execution Algorithms

Arrival Price (Implementation Shortfall)

The arrival price will attempt to achieve, over the course of the order, the bid/ask midpoint price at the time the order is submitted. The arrival algo keeps the hidden orders that will impact a high percentage of the average daily volume.

Accumulate / Distribute

This algo achieve the best price for a large volume order without being noticed in the market, and can be set up for high frequency trading by slicing the order into smaller randomly sized order increments that are released at random time intervals within a used defined time period.

TWAP (Time Weighted Average Price)

TWAP algorithms are designed to execute trades based on a rate proposed to time. It aims to achieve the time weighted average price calculated from the time of submitted order to the time it completes.

VWAP (Volume Weighted Average Price)

VWAP algo seeks to achieve the volume weighted average price, calculated from the time of submitted order. VWAP schedule is proportional to the historical intraday volume profile. This algo trade more at some time frame when the stock traded more historically at same time frame.

Percentage of Volume

The POV is an order execution trading strategy that allows trades to execute the desired percentage of the total market volume called participation rate over a specified period. It’s an adaptive strategy designed to adjust the order size dynamically based on prevailing or forecasted market volume in real-time.

The SOLID Design Principles

Let's review the SOLID design principles since we might mention them while discussing design patterns. It's helpful to remember these principles when designing maintainable and scalable object-oriented systems.

1. Single Responsibility Principle (SRP): A class should have only one reason to exist, meaning it should have only one job or responsibility. You should not overload your objects with two many responsibility, just create a new class for it.

2. Open/Closed Principle (OCP): Software entities (classes, modules, functions, etc.) should be open for extension but closed for modification. This means you should be able to add new functionality without changing existing code.

3. Liskov Substitution Principle (LSP): Objects of a superclass should be replaceable with objects of a subclass without affecting the correctness of the program. This ensures that a subclass can stand in for its superclass.

4. Interface Segregation Principle (ISP): Clients should not be forced to depend on interfaces they do not use. This means creating smaller, more specific interfaces rather than a large, general-purpose one.

5. Dependency Inversion Principle (DIP): High-level modules should not depend on low-level modules. Both should depend on abstractions. Additionally, abstractions should not depend on details. Details should depend on abstractions. This principle helps in reducing the coupling between different parts of a system.

🔍Strategy Pattern

The Strategy design pattern is a behavioral design pattern that enables selecting an algorithm's behavior at runtime. This pattern is based on composition. Lets see the definition from the “Head First Design Pattern”:

The strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Strategy lets algorithm vary independently from clients that use it.

It is naturally useful in trading systems where algorithms need to switch between different execution methods. In this example, we'll implement a simplified trading system in Rust that executes orders using different strategies like TWAP (Time-Weighted Average Price), VWAP (Volume-Weighted Average Price), and POV (Percentage of Volume). These are your execution strategies. You can choose one of the strategies based on certain factors or constraints.

Structure

1. The Strategy interface is common to all concrete strategies. It declares a method the context uses to execute a strategy. We define a common interface for all execution strategies, the method execute_order that all execution strategies must implement.

trait ExecutionStrategy {
    fn execute_order(&self, order_id: u32, quantity: u32);
}

2. Concrete Strategies implement different variations of an algorithm the context uses. In this example, we write implementations of TWAP, VWAP, and POV strategies.

    struct TwapStrategy;
   
    impl ExecutionStrategy for TwapStrategy {
        fn execute_order(&self, order_id: u32, quantity: u32) {
            println!("Executing order {} using TWAP for {} units.", order_id, quantity);
            // Implement TWAP logic here
        }
    }
    struct VwapStrategy;
   
    impl ExecutionStrategy for VwapStrategy {
        fn execute_order(&self, order_id: u32, quantity: u32) {
            println!("Executing order {} using VWAP for {} units.", order_id, quantity);
            // Implement VWAP logic here
        }
    }
    struct PovStrategy {
        participation_rate: f64,
    }
   
    impl ExecutionStrategy for PovStrategy {
        fn execute_order(&self, order_id: u32, quantity: u32) {
            println!(
                "Executing order {} using POV at {}% participation for {} units.",
                order_id, self.participation_rate * 100.0, quantity
            );
            // Implement POV logic here
        }
    }
   

3. The Context maintains a reference to one of the concrete strategies and communicates with this object only via the strategy interface. Here we write the OrderExecutor that holds a reference to a strategy and uses it to execute orders. The context calls the execution method on the linked strategy object each time it needs to run the algorithm. The context doesn’t know what type of strategy it works with or how the algorithm is executed.

    struct OrderExecutor {
        strategy: Box<dyn ExecutionStrategy>,
    }
   
    impl OrderExecutor {
        fn new(strategy: Box<dyn ExecutionStrategy>) -> Self {
            OrderExecutor { strategy }
        }
        fn set_strategy(&mut self, strategy: Box<dyn ExecutionStrategy>) {
            self.strategy = strategy;
        }
        fn execute(&self, order_id: u32, quantity: u32) {
            self.strategy.execute_order(order_id, quantity);
        }
    }
   

4. The Client creates a specific strategy object and passes it to the context. The context exposes a setter which lets clients replace the strategy associated with the context at runtime.

    fn main() {
        let order_id = 101;
        let quantity = 1000;
   
        // Using TWAP Strategy
        let twap_strategy = Box::new(TwapStrategy);
        let mut executor = OrderExecutor::new(twap_strategy);
        executor.execute(order_id, quantity);
   
        // Switching to VWAP Strategy
        let vwap_strategy = Box::new(VwapStrategy);
        executor.set_strategy(vwap_strategy);
        executor.execute(order_id+1, quantity);
   
        // Switching to POV Strategy
        let pov_strategy = Box::new(PovStrategy {
            participation_rate: 0.1,
        });
        executor.set_strategy(pov_strategy);
        executor.execute(order_id+2, quantity);
    }
   

Strategy pattern Class Diagram

👀Observer Pattern

The Observer design pattern is a behavioral design pattern that enables an object, known as the Subject, to maintain a list of its dependents, called Observers, and automatically notify them of any state changes, typically by calling one of their methods. This pattern is particularly useful in Event-Driven Systems, when you need to notify multiple objects about events without tightly coupling them. The definition from the “Head First Design Pattern”:

The observer pattern defines a one to many dependency between objects so that when one object changes state, all of its dependents are notified and updated automatically.

In the context of algorithmic trading , the Observer pattern can be applied to scenarios such as:

• Market Data Feeds: Notifying trading strategies when new market data arrives.

• Order Execution Updates: Informing interested parties when an order is executed or its status changes.

• Price Alerts: Triggering alerts when certain price thresholds are crossed.

• 

Structure

• Subject (Observable or Publisher): The Subject issues events of interest to other objects. These events occur when the subject (publisher) changes its state or executes some behaviors. Subjects maintain a subscription infrastructure, list of observers and provides methods to attach, detach, and notify them.

• Observer (Subscriber): The Observer interface declares the notification interface. In most cases, it consists of a single update method.

• Concrete Subject: Implements the subject interface and holds the state of interest.

• Concrete Observer: Concrete Observer perform some actions in response to notifications issued by the subject. All of these classes must implement the same interface so the subject isn’t coupled to concrete classes.

Usually, subscribers need some contextual information to handle the update correctly. For this reason, publishers often pass some context data as arguments of the notification method. The publisher can pass itself as an argument, letting subscriber fetch any required data directly.

Implementation Example

We will implement a simplified version of trading system using Rust where we have:

• Market Data Feed (Subject): Provides live price updates for various financial instruments.

• Trading Strategies (Observers): React to market data updates to make trading decisions.

Let's take a look at the UML diagram of this example and code implementation.

Observer Trait

The Observer trait declares the update method, which observers must implement.

use std::rc::Rc;
use std::cell::RefCell;

trait Observer {
    fn update(&self, instrument_id: &str, price: f64);
}

Implement Concrete Observers (Trading Strategies)

struct MomentumStrategy {
    name: String,
    threshold: f64,
}

impl Observer for MomentumStrategy {
    fn update(&self, instrument_id: &str, price: f64) {
        if price > self.threshold {
            println!(
                "{}: [{}] Price crossed above threshold! Price: {}",
                self.name, instrument_id, price
            );
            // Implement buy logic
        } else {
            println!(
                "{}: [{}] Price below threshold. Price: {}",
                self.name, instrument_id, price
            );
            // Implement hold or sell logic
        }
    }
}

struct MeanReversionStrategy {
    name: String,
    average_price: RefCell<f64>,
}

impl Observer for MeanReversionStrategy {
    fn update(&self, instrument_id: &str, price: f64) {
        let mut avg = self.average_price.borrow_mut();
        *avg = (*avg * 0.9) + (price * 0.1); // Update moving average
        if price < *avg {
            println!(
                "{}: [{}] Price below average! Price: {}, Average: {:.2}",
                self.name, instrument_id, price, *avg
            );
            // Implement buy logic
        } else {
            println!(
                "{}: [{}] Price above average. Price: {}, Average: {:.2}",
                self.name, instrument_id, price, *avg
            );
            // Implement sell logic
        }
    }
}

Define the Subject Trait

trait Subject {
    fn attach(&mut self, observer: Rc<dyn Observer>);
    fn detach(&mut self, observer: &Rc<dyn Observer>);
    fn notify(&self);
}

Implement the Concrete Subject (Market Data Feed)

struct MarketDataFeed {
    instrument_id: String,
    observers: RefCell<Vec<Rc<dyn Observer>>>,
    price: RefCell<f64>,
}

impl MarketDataFeed {
    fn new(instrument_id: &str) -> Self {
        MarketDataFeed {
            instrument_id: instrument_id.to_string(),
            observers: RefCell::new(Vec::new()),
            price: RefCell::new(0.0),
        }
    }
    fn set_price(&self, new_price: f64) {
        *self.price.borrow_mut() = new_price;
        self.notify();
    }
}
impl Subject for MarketDataFeed {
    fn attach(&mut self, observer: Rc<dyn Observer>) {
        self.observers.borrow_mut().push(observer);
    }
    fn detach(&mut self, observer: &Rc<dyn Observer>) {
        let mut observers = self.observers.borrow_mut();
        if let Some(pos) = observers.iter().position(|x| Rc::ptr_eq(x, observer)) {
            observers.remove(pos);
        }
    }
    fn notify(&self) {
        let price = *self.price.borrow();
        let instrument_id = &self.instrument_id;
        for observer in self.observers.borrow().iter() {
            observer.update(instrument_id, price);
        }
    }
}

Client Code

fn main() {
    // Create market data feed for AAPL
    let mut market_data_feed = MarketDataFeed::new("AAPL");

    // Create observers
    let momentum_strategy: Rc<dyn Observer> = Rc::new(MomentumStrategy {
        name: String::from("MomentumStrategy"),
        threshold: 150.0,
    });

    let mean_reversion_strategy: Rc<dyn Observer> = Rc::new(MeanReversionStrategy {
        name: String::from("MeanReversionStrategy"),
        average_price: RefCell::new(145.0),
    });

    // Attach observers
    market_data_feed.attach(momentum_strategy.clone());
    market_data_feed.attach(mean_reversion_strategy.clone());

    // Simulate market data updates
    let price_updates = vec![148.0, 151.0, 149.5, 152.5, 147.0];

    for price in price_updates {
        println!("\nMarketDataFeed [{}]: New price is {}", "AAPL", price);
        market_data_feed.set_price(price);
    }

    // Detach momentum strategy
    market_data_feed.detach(&momentum_strategy);

    // More updates
    let more_price_updates = vec![153.0, 146.5];

    for price in more_price_updates {
        println!("\nMarketDataFeed [{}]: New price is {}", "AAPL", price);
        market_data_feed.set_price(price);
    }
}

Notes on Rust Implementation

• Reference Counting (Rc): Used to allow multiple ownership of observers by the subject.

• Interior Mutability (RefCell): Allows us to mutate data even when it is wrapped in an immutable reference, which is necessary when observers need to update their internal state upon receiving updates.

• Trait Objects (dyn Trait): Trait objects like dyn Observer allow for dynamic dispatch in Rust. They allow the subject to hold a collection of different types that implement the same trait.

In Rust, comparing trait objects (dyn Observer) directly is not straightforward because trait objects do not implement PartialEq by default. We use Rc::ptr_eq to compare the pointers of the Rc smart pointers, which checks if they point to the same allocation.

In the line, if we don’t explicitly say the type Rc<dyn Observer> then momentum_strategy will be of type Rc<MomentumStrategy> and it will be fine for attach method however not work with detach method.

let momentum_strategy: Rc<dyn Observer> = Rc::new(MomentumStrategy{...});

In Rust, even though MomentumStrategy implements the Observer trait, Rc<MomentumStrategy> and Rc<dyn Observer> are different types and are not directly interchangeable.

💡Note: The Rc::ptr_eq function requires that both Rc pointers have the same type parameter. By ensuring that both Rc pointers are of type Rc<dyn Observer>, we satisfy this requirement.

Rust can automatically coerce a reference to a concrete type into a reference to a trait object if the type implements the trait. This is what happens when you assign Rc::new(MomentumStrategy { /* fields */ }) to a variable of type Rc<dyn Observer>.

Decorator Pattern

The Decorator Pattern is a structural design pattern that allows behavior to be added to individual objects dynamically without affecting the behavior of other objects from the same class. The definition from the “Head First Design Pattern”:

The decorator pattern attaches additional responsibilities to an object dynamically. Decorators provides a flexible alternative to subclassing for extending functionality.

Structure

• Component Interface: Defines the interface for objects that can have responsibilities added to them dynamically.

• Concrete Component: The original object to which additional responsibilities are added.

• Decorator: Abstract class that implements the component interface and contains a reference to a component object.

• Concrete Decorators: Extend the functionality of the component by overriding methods and adding additional behaviors.

UML Diagram

Example: Enhancing Order Execution with Decorators

We have an OrderExecutor component responsible for executing trades. We want to add additional behaviors:

• LoggingDecorator: Logs the details of each order execution.

• ValidationDecorator: Validates orders before execution.

Component Trait

#[derive(Debug)]
struct Order {
    symbol: String,
    quantity: i32,
    price: f64,
}
trait OrderExecutor {
    fn execute_order(&self, order: &Order) -> Result<(), String>;
}

Concrete Component

struct BasicOrderExecutor;

impl OrderExecutor for BasicOrderExecutor {
    fn execute_order(&self, order: &Order) -> Result<(), String> {
        // Simulate order execution logic
        println!("Executing order: {:?}", order);
        Ok(())
    }
}

Concrete Decorators

struct LoggingDecorator<T: OrderExecutor> {
    execu
tor: T,
}

impl<T: OrderExecutor> LoggingDecorator<T> {
    fn new(executor: T) -> Self {
        LoggingDecorator { executor }
    }
}

impl<T: OrderExecutor> OrderExecutor for LoggingDecorator<T> {
    fn execute_order(&self, order: &Order) -> Result<(), String> {
        println!("LoggingDecorator: Order received: {:?}", order);
        let result = self.executor.execute_order(order);
        println!("LoggingDecorator: Order execution result: {:?}", result);
        result
    }
}
struct ValidationDecorator<T: OrderExecutor> {
    executor: T,
}

impl<T: OrderExecutor> ValidationDecorator<T> {
    fn new(executor: T) -> Self {
        ValidationDecorator { executor }
    }
}

impl<T: OrderExecutor> OrderExecutor for ValidationDecorator<T> {
    fn execute_order(&self, order: &Order) -> Result<(), String> {
        if self.validate(order) {
            println!("Validated Order: {:?}", order);
            self.executor.execute_order(order)
        } else {
            Err(String::from("Validation failed"))
        }
    }
}

impl<T: OrderExecutor> ValidationDecorator<T> {
    fn validate(&self, order: &Order) -> bool {
        // Implement validation logic
        order.quantity > 0 && order.price > 0.0
    }
}

Client Code

fn main() {
    let order = Order {
        symbol: String::from("AAPL"),
        quantity: 100,
        price: 150.0,
    };

    // Basic executor
    let basic_executor = BasicOrderExecutor;

    // Decorate with validation
    let validated_executor = ValidationDecorator::new(basic_executor);

    // Further decorate with logging
    let logged_executor = LoggingDecorator::new(validated_executor);

    // Execute the order
    let result = logged_executor.execute_order(&order);
    match result {
        Ok(_) => println!("Order executed successfully."),
        Err(e) => println!("Order execution failed: {}", e),
    }
}

Notes on Rust Implementation

In this example, I used generics allowing static dispatch. By the way, in Rust, traits are not types themselves; they are a collection of methods that types can implement. You cannot instantiate a trait directly or store it as a field without using a pointer or a generic parameter. The following will result in compilation error because OrderExecutor is a trait.

struct ValidationDecorator {
    executor: OrderExecutor,
}

In Rust, all types must have a known size at compile time unless they are behind a pointer (like &, Box, or Rc) or used as a generic type parameter with trait bounds. By default, all generic type parameters and struct fields have an implicit Sized bound. This means that the compiler needs to know the size of the type at compile time.

The choice between generics and trait objects depends on your specific needs for performance and flexibility.

Instead of generics, you can use Trait Objects as follows:

struct ValidationDecorator {
    executor: Box<dyn OrderExecutor>,
}

This uses dynamic dispatch, which introduces a slight runtime overhead due to the use of a vtable.

🏭Factory Method Pattern

Factory Method is a creational design pattern that provides an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created. This definition is by the refactoring.guru. The “Head First Design Pattern” more or less says the same:

The Factory Method Patterns defines an interface for creating an object, but lets subclasses decide which class to instantiate. Factory method lets a class defer instantiation to subclasses.

Structure

• Product Interface: Defines the interface of objects the factory method creates.

• Concrete Products: Various implementations of the product interface.

• Creator (Factory): Declares the factory method that returns new product objects.

• Concrete Creators: Override the factory method to change the resulting product's type.

Example: Order Creation Factory

Let's consider an example where we need to create different types of orders based on the current state of our trading strategy.

Order Types

• Market Order: Executes immediately at the current market price.

• Limit Order: Executes at a specified price or better.

• Stop Order: Becomes a market order when a specified price is reached.

Factory Method UML Diagram

This is the class diagram structure we are implementing.

Product Interface

This interface is common to all the objects (in our case order types) that can be produced by the creator and its subclasses.

trait Order {
    fn place(&self);
}

Concrete Products

Concrete Products are different implementations of the product interface. In our case different implementation of order types.

struct MarketOrder {
    symbol: String,
    quantity: u32,
}

impl Order for MarketOrder {
    fn place(&self) {
        println!("Placing Market Order: Buy {} units of {} at market price.",
            self.quantity, self.symbol
        );
        // Implement order placement logic here
    }
}
struct LimitOrder {
    symbol: String,
    quantity: u32,
    limit_price: f64,
}

impl Order for LimitOrder {
    fn place(&self) {
        println!("Placing Limit Order: Buy {} units of {} at ${}.",
            self.quantity, self.symbol, self.limit_price
        );
        // Implement order placement logic here
    }
}
struct StopOrder {
    symbol: String,
    quantity: u32,
    stop_price: f64,
}

impl Order for StopOrder {
    fn place(&self) {
        println!(
            "Placing Stop Order: Buy {} units of {} when price reaches ${}.",
            self.quantity, self.symbol, self.stop_price
        );
        // Implement order placement logic here
    }
}

Creator (Order Factory)

It returns a different type of order.

enum OrderType {
    Market,
    Limit(f64), // Limit price
    Stop(f64),  // Stop price
}
struct OrderFactory;

impl OrderFactory {
    fn create_order(order_type: OrderType, symbol: String, quantity: u32) -> Box<dyn Order> {
        match order_type {
            OrderType::Market => Box::new(MarketOrder { symbol, quantity }),
            OrderType::Limit(limit_price) => Box::new(LimitOrder {
                symbol,
                quantity,
                limit_price,
            }),
            OrderType::Stop(stop_price) => Box::new(StopOrder {
                symbol,
                quantity,
                stop_price,
            }),
        }
    }
}

Client Code

fn main() {
    let symbol = String::from("AAPL");
    let quantity = 100;
    // Decide which order type to use
    let order_type = OrderType::Limit(149.0);
    // Create the order using the factory
    let order = OrderFactory::create_order(order_type, symbol.clone(), quantity);
    // Place the order
    order.place();
}

Singleton Pattern

Singleton is a creational design pattern that lets you ensure that a class has only one instance, while providing a global access point to this instance. Overall Singleton has almost the same pros and cons as global variables and generally a bad idea in context of concurrency.

In Rust, implementing singletons are pretty tricky due to its ownership model and emphasis on safe concurrency. It must solve these two problem:

• Ensure a Class Has Only One Instance: Prevents multiple instances, guaranteeing that all clients use the same instance.

• Provide a Global Point of Access: Offers a way to access the single instance from anywhere in the application.

Rust discourages global mutable state to prevent data races and ensure thread safety. And any global state must be safe to access from multiple threads. All these makes implementing singletons non-trivial. We can achieve this using different approach like using Mutex, static lifetime, OnceCell crate.

While a mutable singleton can be convenient, global mutable state can lead to code that's difficult to maintain and test. You always have option to pass the instance as a parameter to the parts of your code that needs it. As we are discussing singletons, lets implement using OnceCell.

OnceCell<T>

When you need to initialize data at runtime, possibly in a lazy manner, and ensure it is set only once. The core API looks roughly like

impl OnceCell<T> {
    fn new() -> OnceCell<T> { ... }
    fn set(&self, value: T) -> Result<(), T> { ... }
    fn get(&self) -> Option<&T> { ... }
}

OnceCell has two varient std::cell::OnceCell<T> for single-threaded scenarios and std::sync::OnceCell<T> for thread-safe scenarios, can be shared between threads.

https://github.com/matklad/once_cell

However, since OnceCell provides immutable access to the initialized value, we need a way to mutate the instance after it's been set because singleton patterns allows mutable instance. To achieve a mutable singleton, we need combine OnceCell with interior mutability patterns. This involves wrapping the struct in types that allow mutation through immutable references, such as RefCell for single-threaded applications or Mutex/RwLock for multi-threaded applications.

Example: Trading System ConfigManager

Config

The Config struct holds various configuration settings. It derives Deserialize to allow loading from a JSON file.

use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct Config {
    pub api_key: String,
    pub db_connection_string: String,
    pub trading_parameters: TradingParameters,
}

#[derive(Debug, Deserialize)]
pub struct TradingParameters {
    pub max_positions: usize,
    pub risk_tolerance: f64,
    // Add other parameters as needed
}

ConfigManager Singleton

OnceCell ensures that the ConfigManager is initialized only once in a thread-safe manner. instance method provides global access to the singleton instance. The new method is private to prevent direct instantiation.

use once_cell::sync::OnceCell;
use std::fs;

pub struct ConfigManager {
    config: Config,
}

impl ConfigManager {
    fn new() -> Self {
        let config_data = fs::read_to_string("config.json")
            .expect("Failed to read configuration file");
        let config: Config = serde_json::from_str(&config_data)
            .expect("Failed to parse configuration file");
        ConfigManager { config }
    }

    pub fn instance() -> &'static ConfigManager {
        static INSTANCE: OnceCell<ConfigManager> = OnceCell::new();
        INSTANCE.get_or_init(|| ConfigManager::new())
    }
    pub fn get_config(&self) -> &Config {
        &self.config
    }
}

Use ConfigManager

Now we can’t just use this TWAP that depends on a ConfigManager in some other context, without carrying over the ConfigManager to the other context, e.g. Unit Tests.

pub trait OrderExecution {
    fn execute(&self);
}
pub struct TWAP;

impl OrderExecution for TWAP {
    fn execute(&self) {
        let config_manager = ConfigManager::instance();
        let max_positions = config_manager.get_config().trading_parameters.max_positions;
        let risk_tolerance = config_manager.get_config().trading_parameters.risk_tolerance;
        // Implement execution logic using the configurations
    }
}
fn main() {
    let strategy = TWAP;
    strategy.execute();

    // Access the ConfigManager directly elsewhere
    let api_key = ConfigManager::instance().get_config().api_key.clone();
    println!("Using API Key: {}", api_key);
}

In this example, we can't change the ConfigManager instance. We've basically set up an immutable singleton (or a single Flyweight object).

To implement a mutable singleton ConfigManager struct using OnceCell, you need to wrap the ConfigManager in an Interior Mutability Type like using RefCell<ConfigManager> for single-threaded applications or using using Mutex<ConfigManager> or RwLock<ConfigManager> for multi-threaded applications. For example:

static CONFIG: OnceCell<RefCell<ConfigManager>> = OnceCell::new();
//Multi-threaded
static CONFIG: OnceCell<Mutex<ConfigManager>> = OnceCell::new();

Notes on Rust

What is Mutex<T>

Mutex stands for mutual exclusion, which means it is used to protect shared data by allowing only one thread to access a resource or critical section at a time e.g. only exclusive borrows. Once a thread acquires a mutex, all other threads attempting to acquire the same mutex are blocked until the first thread releases the mutex.

The Rust standard library provides std::sync::Mutex<T>. It is generic over a type T, which is the type of the data the mutex is protecting. Its lock() method returns a special type called a MutexGuard. This guard represents the guarantee that we have locked the mutex. You unlock the mutex simply by dropping the guard.

Rust’s Mutex holds the data it’s protecting. In C++, on the other hand, std::mutex doesn’t hold the data it’s protecting and doesn’t even know what it’s protecting.

What is Arc<Mutex<T>>

Here, we want to share ownership of the data (Mutex<T>) between multiple threads. Arc provides shared ownership with thread-safe reference counting. Arc<Mutex<T>> allows multiple threads to own the same data and mutate it safely.

Command Pattern

Command is a behavioral design pattern that turns a request into a stand-alone object that contains all information about the request. This transformation lets you pass requests as a method arguments, delay or queue a request’s execution, and support undoable operations. This definition is by the refactoring.guru. The “Head First Design Pattern” says:

The command pattern encapsulates a request as an object, thereby letting you parameterize other objects with different requests, queue or log requests and support undoable operations.

Let consider an example where we have a Portfolio that keeps track of stock positions and cash balance. We support actions like Add, Remove, or Reduce positions, as well as Deposit and Withdraw funds to our portfolio. Lets make it more interesting and take a step forward and consider multiple treading scenario as one tread could be buying SPY, another selling MSFT and another thread depositing funds. We need to share Portfolio with different threads and it handles different commends.

Structure

• Command interface: The Command interface usually methods for executing the command. We can defines execute and rollback methods with thread safety in mind.

• Receiver: The receiver class contains the business logic. Most commands only handle the details of how a request is passed to the receiver, while the receiver itself does the actual work. In our case the Portfolio is a receiver that stores positions and balance, protected by Mutex for safe concurrent access.

• Concrete Commands: Implement the Command trait for specific actions. A concrete command isn’t supposed to perform the work on its own, but rather to pass to the receiver. In our case we will implement add position and add funds commends.

• Invoker: The Invoker class is responsible for initiating requests. In our example we define a Broker that executes commands and maintains a history.

• Client: The application that creates commands and interacts with the broker.

Command Trait

use std::error::Error;
trait Command: Send + 'static {
    fn execute(&mut self) -> Result<(), Box<dyn Error>>;
    fn rollback(&mut self) -> Result<(), Box<dyn Error>>;
}

Send Trait ensures commands can be transferred across threads. The Box<dyn Error> is for flexible error types.

Receiver: Portfolio

use std::collections::HashMap;
use std::sync::{Arc, Mutex};
#[derive(Clone)]
struct Portfolio {
    positions: Arc<Mutex<HashMap<String, i32>>>, // symbol -> quantity
    balance: Arc<Mutex<f64>>,                    // cash balance
}
impl Portfolio {
    fn new() -> Self {
        Portfolio {
            positions: Arc::new(Mutex::new(HashMap::new())),
            balance: Arc::new(Mutex::new(0.0)),
        }
    }
    fn add_position(&self, symbol: &str, quantity: i32) -> Result<(), String> {
        let mut positions = self.positions.lock().unwrap();
        let entry = positions.entry(symbol.to_string()).or_insert(0);
        *entry += quantity;
        println!("Added {} shares of {}", quantity, symbol);
        Ok(())
    }
    fn reduce_position(...){...} //Similar
    fn deposit(&self, amount: f64) -> Result<(), String> {
        let mut balance = self.balance.lock().unwrap();
        *balance += amount;
        println!("Deposited ${}", amount);
        Ok(())
    }
    fn withdraw(...){...} //Similar to deposit
    fn get_balance(&self) -> f64 {
        let balance = self.balance.lock().unwrap();
        *balance
    }

    fn get_positions(&self) -> HashMap<String, i32> {
        let positions = self.positions.lock().unwrap();
        positions.clone()
    }
}

We used Arc<Mutex<HashMap<String, i32>>> to allow shared mutable access across threads for position.

Concrete Commands

We implement two concreate commands AddPositionCommand and WithdrawCommand. Both command interacts with the Portfolio in a thread-safe manner and implements rollback functionality.

struct AddPositionCommand {
    receiver: Arc<Portfolio>,
    symbol: String,
    quantity: i32,
}
impl Command for AddPositionCommand {
    fn execute(&mut self) -> Result<(), Box<dyn Error>> {
        self.receiver.add_position(&self.symbol, self.quantity)?;
        Ok(())
    }
    fn rollback(&mut self) -> Result<(), Box<dyn Error>> {
        self.receiver.reduce_position(&self.symbol, self.quantity)?;
        Ok(())
    }
}
struct WithdrawCommand {
    receiver: Arc<Portfolio>,
    amount: f64,
}
impl Command for WithdrawCommand {
    fn execute(&mut self) -> Result<(), Box<dyn Error>> {
        self.receiver.withdraw(self.amount)?;
        Ok(())
    }
    fn rollback(&mut self) -> Result<(), Box<dyn Error>> {
        self.receiver.deposit(self.amount)?;
        Ok(())
    }
}

Invoker: Broker

struct Broker {
    history: Vec<Box<dyn Command>>,
}
impl Broker {
    fn new() -> Self {
        Broker {
         history: Vec::new(),
        }
    }
    fn execute_command(&mut self, mut command: Box<dyn Command>) -> Result<(), Box<dyn Error>> {
        command.execute()?;
        self.history.push(command);
        Ok(())
    }
    fn rollback_last(&mut self) -> Result<(), Box<dyn Error>> {
        if let Some(mut command) = self.history.pop() {
            command.rollback()?;
            Ok(())
        } else {
            Err("No commands to rollback.".into())
        }
    }
}

Client Code

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let portfolio = Arc::new(Portfolio::new());
    let broker = Arc::new(Mutex::new(Broker::new()));

    let portfolio_clone1 = Arc::clone(&portfolio);
    let broker_clone1 = Arc::clone(&broker);
    // Add position
    let add_position_command = AddPositionCommand {
        receiver: portfolio.clone(),
        symbol: "AAPL".to_string(),
        quantity: 50,
    };
    let deposit_handle = thread::spawn(move || {
        let deposit_command = DepositCommand {
            receiver: portfolio_clone1,
            amount: 10000.0,
        };
        let mut broker1 = broker_clone1.lock().unwrap();
        broker1.execute_command(Box::new(deposit_command)).unwrap();
        drop(broker1);
    });
    let broker_clone2 = Arc::clone(&broker);
    {
        let mut broker2 = broker_clone2.lock().unwrap();
        broker2.execute_command(Box::new(add_position_command))?;
    }

    deposit_handle.join().unwrap();
    // Print current portfolio state
    println!("Current balance: ${}", portfolio.get_balance());
    println!("Current positions: {:?}", portfolio.get_positions());
    // Rollback last command (Withdraw)
    {
        let mut broker3 = broker.lock().unwrap();
        broker3.rollback_last()?;
    }
    println!("\nAfter rollback of last command:");
    println!("Current balance: ${}", portfolio.get_balance());
    println!("Current positions: {:?}", portfolio.get_positions());
    Ok(())
}

Here, I used Arc<Mutex<Broker>> to let multiple threads share access to the broker, ensuring the execute_command method is synchronized. While this isn't the perfect example since locking Broker might slow things down, it gives you a taste of multithreading and the command pattern.

As Broker might be doing tons of other operation we should at least minimize locking and lock only history to prevent contention and improve performance.

So, let's take out the Mutex around the Broker and redesign the broker as:

struct Broker {
    history: Arc<Mutex<Vec<Box<dyn Command>>>>,
}

Now, the history is an Arc<Mutex<Vec<Box<dyn Command>>>>, which lets multiple parts of the program access the Broker at the same time while keeping the history synchronized.

The ring buffer pattern

The ring buffer pattern is not listed as a design pattern in the GoF book. It is more of a data structure or Producer-Consumer pattern than a design pattern. However, it is relevant in trading systems and fundamental for other pattern to understand. Let's dive in. Challenges Ahead!

The ring buffer as name suggest offers a fixed-size buffer that replaces old data when it fills up, making it perfect for processing data streams without interruption. It operates as a circular queue data structure, maintaining a first-in, first-out (FIFO) behavior, guided by two indices: one for reading and another for writing. So far so good.

🔗What Is a Ring Buffer?

A ring buffer is a fixed-size data structure that uses a single, contiguous block of memory in a circular fashion. When the write pointer reaches the end of the buffer, it wraps around to the beginning, overwriting the oldest data. Since it's a fixed-size buffer, it doesn't require dynamic memory allocation, and reading/writing operations are generally constant time, O(1), making it highly desirable for low latency.

It has two pointers:

• Head (Write Pointer): Indicates where the next data element will be written.

• Tail (Read Pointer): Indicates where the next data element will be read.

  

Define the Ring Buffer Struct

We have learned the multithreading and Mutex in previous section. We can assume there will be two thread one for consumer and one for producer. As in the previous section we can use Mutex to lock head and tail and we should be good. However in this example we will do Lock-Free Ring Buffer Implementation for a Single Producer and Single Consumer. In general you almost never need to do the lock-free implementation (warning: stay away from Atomic), and your life is much better to just use locks. But we will do anyway, otherwise, how would we learn the Atomic and memory ordering. This is going to be mind blowing.

Lets define the RingBuffer struct.

pub struct RingBuffer<T> {
    buffer: Vec<UnsafeCell<MaybeUninit<T>>>,
    capacity: usize,
    head: AtomicUsize, // Index for the producer
    tail: AtomicUsize, // Index for the consumer
}

In this example, we're diving into the unsafe part, which means we're turning off some of the compiler's safety features. First, let's revisit some Rust concepts together.

What is Cell<T>?

Cell<T> is a type in Rust's standard library that provides interior mutability for values that implement the Copy trait. It allows you to get and set the value inside the Cell<T> even when you have only an immutable reference to it. This is particularly useful when you need to mutate data within a struct that is otherwise immutable.

There are scenarios where you need to mutate data even though you have only an immutable reference to it. This pattern is known as interior mutability. Cell<T> enables interior mutability by encapsulating the value and providing methods to access and modify it without requiring a mutable reference to the Cell<T> itself. Internally, it uses UnsafeCell<T> to bypass Rust's usual borrowing rules safely.

use std::cell::Cell;
fn main() {
    let cell = Cell::new(10);
    // Even though `cell` is immutable, we can get and set its value.
    println!("Initial value: {}", cell.get());
    cell.set(20);
    println!("Updated value: {}", cell.get());
}

What is UnsafeCell<T>?

UnsafeCell<T> is a type that wraps some T and indicates unsafe interior operations on the wrapped type. Types like Cell<T> and RefCell<T> in Rust's standard library are built upon UnsafeCell<T>. The UnsafeCell API .get() gives you a raw pointer *mut T to its contents. It is up to you to use that raw pointer correctly.

Let's create our own cell!

use std::cell::UnsafeCell;
struct MyCell<T> {
    value: UnsafeCell<T>,
}
impl<T> MyCell<T> {
    fn new(value: T) -> MyCell<T> {
        MyCell {
            value: UnsafeCell::new(value),
        }
    }
    fn get(&self) -> &T {
        unsafe { &*self.value.get() }
    }
    fn set(&self, new_value: T) {
        unsafe {
            *self.value.get() = new_value;
        }
    }
}
fn main() {
    let cell = MyCell::new(42);
    println!("Cell value: {}", cell.get());
    cell.set(24);
    println!("Cell value: {}", cell.get());
}

You have immutable cell with value 42 and and you can set to 24 without any problem.

Before we move to implementation, lets visit one more concepts Send and Sync.

impl<T: Send> Sync for StructName {}

In Rust, the Send and Sync traits are fundamental to the language's concurrency guarantees. They determine how types can be safely shared or transferred across threads when you see a line like

impl<T: Send> Sync for StructName<T> {}

It means that we're implementing the Sync trait for the type StructName<T>, where T is any type that implements Send. This implementation allows references to StructName<T> to be shared safely across multiple threads, provided that T is Send. A type T is Send if it is safe to transfer ownership of values of type T to another thread. A type T is Sync if it is safe to share references (&T) to T between multiple threads.

Here we are asserting to the compiler that StructName<T> is Sync whenever T is Send and allow us to share the references to StructName<T> safely shared between threads.

struct StructName<T> {
    data: T,
}
impl<T: Send> Sync for StructName<T> {}

When we compile this code, we see the error that says “An unsafe trait was implemented without an unsafe implementation.”

error[E0200]: the trait Sync requires an unsafe impl declaration
   |
27 | impl<T: Send> Sync for StructName<T> {}
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: the trait `Sync` enforces invariants that the compiler can't check.
 Review the trait documentation and make sure this implementation upholds those invariants before adding the `unsafe` keyword
help: add `unsafe` to this trait implementation
   |
27 | unsafe impl<T: Send> Sync for StructName<T> {}
   | ++++++

As compiler cannot check its safety, we need to add ‘unsafe’ keyword. The unsafe keyword indicates that you are taking responsibility for ensuring that your implementation upholds the required safety guarantees.

What is MaybeUninit<T>?

A type that represents uninitialized memory. MaybeUninit<T> serves to enable unsafe code to deal with uninitialized data. It is a signal to the compiler indicating that the data here might not be initialized. Refer to below link for more details and examples.

https://doc.rust-lang.org/std/mem/union.MaybeUninit.html

Overall UnsafeCell and MaybeUninit provide safer abstractions over raw pointers.

assume_init_read(&self) -> T and assume_init_ref(&self) -> T

assume_init_read reads the value from the MaybeUninit<T> container. The resulting T is subject to the usual drop handling. This function creates a bitwise copy of the contents, regardless whether the contained type implements the Copy trait or not.

assume_init_ref gets a shared reference to the contained value. This can be useful when we want to access a MaybeUninit that has been initialized but don’t have ownership of the MaybeUninit (preventing the use of .assume_init()).

Ring Buffer Implementation

unsafe impl<T: Send> Sync for RingBuffer<T> {}
impl<T> RingBuffer<T> {
    pub fn new(capacity: usize) -> Self {
        let mut buffer = Vec::with_capacity(capacity);
        for _ in 0..capacity {
            buffer.push(UnsafeCell::new(MaybeUninit::uninit()));
        }
        RingBuffer {
            buffer,
            capacity,
            head: AtomicUsize::new(0),
            tail: AtomicUsize::new(0),
        }
    }

    pub fn push(&self, item: T) -> Result<(), T> {
        let head = self.head.load(Ordering::Acquire);
        let next_head = (head + 1) % self.capacity;
        // Check if buffer is full
        if next_head == self.tail.load(Ordering::Acquire) {
            // Buffer is full
            return Err(item);
        }
        unsafe {
            let cell = self.buffer[head].get();
            (*cell).write(item);
        }
        self.head.store(next_head, Ordering::Release);
        Ok(())
    }

    pub fn pop(&self) -> Option<T> {
        let tail = self.tail.load(Ordering::Relaxed);

        if tail == self.head.load(Ordering::Acquire) {
            // Buffer is empty
            return None;
        }
        let item = unsafe {
            let cell = self.buffer[tail].get();
            (*cell).assume_init_read()
        };
        let next_tail = (tail + 1) % self.capacity;
        self.tail.store(next_tail, Ordering::Release);
        Some(item)
    }
}
impl<T> Drop for RingBuffer<T> {
    fn drop(&mut self) {
        // Clean up any initialized elements
        let head = self.head.load(Ordering::Relaxed);
        let tail = self.tail.load(Ordering::Relaxed);

        let mut idx = tail;
        while idx != head {
            unsafe {
                let cell = self.buffer[idx].get();
                (*cell).assume_init_drop();
            }
            idx = (idx + 1) % self.capacity;
        }
    }
}

Let's go through this step by step. There's a lot to understand. We've already talked about MaybeUninit<T> and its methods, which are quite low-level. But what do Ordering::Acquire or Ordering::Release mean when we read and update head and tail? Memory ordering is the next concept we need to discuss.

Memory Ordering

In simple terms, memory ordering refers to the sequence in which read and write operations on memory happen in a concurrent or multi-threaded environment. This order can vary because of optimizations like instruction reordering, caching, and the architecture's memory model.

Compilers and CPUs re-order your instructions to optimize the performance. We use AtomicUsize and its method e.g. store, load, swap, compare_and_swap, etc. These atomic operation is an indivisible operation that appears instantaneous to other threads. It ensures that no other thread can observe the operation in a partially completed state. When using atomic operations, we need to specify the memory ordering, which controls how operations are ordered with respect to other memory operations.

Common C++ Memory Orderings are:

• Relaxed (memory_order_relaxed): No ordering constraints; only atomicity is guaranteed.

• Acquire (memory_order_acquire): Prevents memory reordering of subsequent reads/writes.

• Release (memory_order_release): Prevents memory reordering of previous reads/writes.

• Acquire-Release: Combines acquire and release semantics.

• Sequentially Consistent (memory_order_seq_cst): Provides a total ordering; all threads agree on the order of operations.

Different CPU architectures have different memory models:

• x86/x64 (Intel/AMD): Generally strong memory ordering; most reads and writes are not reordered.

• ARM, PowerPC: Allow more reordering; require explicit memory barriers to enforce ordering.

Memory Ordering in Rust

Rust uses the same memory model as C++ for atomic operations. Memory Ordering is specified using Ordering enum. Ordering Variants in Rust

• Ordering::Relaxed

• Ordering::Acquire

• Ordering::Release

• Ordering::AcqRel

• Ordering::SeqCst

This is a complicated topic, so for now, we'll just follow a few simple rules. We'll skip Ordering::Relaxed because it is quite weak. We can discuss it further later.

We apply Ordering::Release when writing to an atomic variable that will be read by other threads. It guarantees that all memory writes (before the release store) in the current thread become visible to other threads that perform an acquire load on the same atomic variable. In a way it says "I'm releasing my changes so other threads can see them."

We apply Ordering::Acquire when reading from an atomic variable that was written by other threads. It guarantees that all memory reads and writes (after the acquire load) in the current thread see the effects of the writes that happened before the release store in another thread. In a way it says "I'm acquiring the latest changes made by other threads."

Together, they ensure that memory operations are properly ordered and visible across threads, enabling safe and correct concurrent programming.

#[derive(Debug,Clone)]
pub struct MarketDataTick {
    pub symbol: String,
    pub price: f64,
    pub volume: u32,
    pub timestamp: u32,
}
fn start_producer(buffer: Arc<RingBuffer<MarketDataTick>>) {
    thread::spawn(move || {
        let symbols = vec!["AAPL", "GOOG", "MSFT", "AMZN"];
        let mut tick_count:u32 = 0;

        loop {
            let symbol = symbols[tick_count as usize % symbols.len()].to_string();
            let tick = MarketDataTick {
                symbol,
                price: 100.0 + tick_count as f64,
                volume: 1000 + tick_count,
                timestamp: tick_count,
            };
            while buffer.push(tick.clone()).is_err() {
                // Buffer is full, handle accordingly
                println!("Buffer full, retrying...");
                thread::sleep(Duration::from_millis(1));
            }
            println!("Produced tick {}", tick_count);
            tick_count += 1;
            // Simulate data arrival rate
            thread::sleep(Duration::from_millis(10));
        }
    });
}
fn start_consumer(buffer: Arc<RingBuffer<MarketDataTick>>) {
    thread::spawn(move || {
        loop {
            match buffer.pop() {
                Some(tick) => {
                    println!("Consumed tick: {:?}", tick);
                    // Simulate processing time
                    thread::sleep(Duration::from_millis(15));
                }
                None => {
                    // Buffer is empty, wait for new data
                    thread::sleep(Duration::from_millis(1));
                }
            }
        }
    });
}
fn main() {
    let buffer_capacity = 100;
    let ring_buffer = Arc::new(RingBuffer::new(buffer_capacity));
    // Start producer and consumer threads
    start_producer(ring_buffer.clone());
    start_consumer(ring_buffer.clone());
    // Let the simulation run for a while
    thread::sleep(Duration::from_secs(5));
}

State Pattern

State is a behavioral design pattern that lets an object changes its behavior when its internal state changes. It's a pretty popular pattern from the Gang of Four 23 popular design patterns. I first learned this pattern from Head-First Design Pattern, so lets see what it says:

The state pattern allows an object to alter its behavior when its internal state changes. The object will appear to change its class.

Imagine you have a trading system, and its behavior depends on its current state. Risk management is an essential part of any trading system, and you have integrated it into your system to enforce the right controls to adjust the system behavior based on current level of risk.

The state pattern can represent different risk states. Let’s consider an example where we have a "normal operation" state, where trading activities proceed as usual. If you take on a significant amount of risk and the VaR (Value at Risk) of your current position approaches the VaR limit (we'll discuss VaR later, but for now, remember that VaR measures risk in dollar terms), you enter a different state, let's call it the "Warning Level" state. This state limits your trading, increases monitoring, and sends email notifications about any additional positions. If you continue and exceed the VaR by a certain amount, it triggers another state, called the "Limit Breach" state. This state blocks new trades, may close existing positions, and allows for hedging. If you cannot close positions for some time and the situation worsens, it enters the "Shutdown Trading" state and starts shutdown procedures.

The benefits of using state pattern is that your system adapts to changing risk levels in real-time and changes its behavior. Risk policies, actions and controls are encapsulated within each state. The concepts and UML diagram of state pattern is similar to the strategy pattern.

Key Concepts of the State Pattern

• Context: The object that changes how it behaves based on its internal state. Context keeps a reference to one of the specific state objects and lets it handle all the work related to that state.

• State: An interface that defines the behavior associated with a particular state of the Context. The State interface declares the state-specific methods.

• Concrete States: Implementations of the State interface that define the behavior for each state.

  

Example: Risk Management system

We'll model a risk management system that transitions through various states based on the Value at Risk (VaR) of the current trading positions. The states we'll implement are:

• NormalOperationState: Trading proceeds as usual.

• WarningLevelState: Trading is limited, monitoring is increased, and notifications are sent.

• LimitBreachState: New trades are blocked, existing positions may be closed, and hedging is allowed.

• ShutdownState: Trading is halted, and shutdown procedures are initiated.

Define a State: RiskState trait

We'll define a RiskState trait that declares the methods each state must implement.

pub trait RiskState: fmt::Debug {
    fn check_var(&self, context: &RiskManager) -> Option<Box<dyn RiskState>>;
    fn enter_state(&self, context: &RiskManager);
    fn exit_state(&self, context: &RiskManager);
    fn send_command(&self, context: &RiskManager);
}

check_var method will be called to check the current VaR and decide if a state transition is needed. enter_state and exit_state methods can perform actions when entering or exiting a state (e.g., sending notifications, adjusting trading limits). send_command will be used to send commend to trading engine based on the state.

Define the Context": RiskManager

pub struct RiskManager {
    state: Box<dyn RiskState>,
    pub var_limit: f64,
    pub warning_level: f64,
    pub current_var: f64,
    pub positions: HashMap<String, f64>, // Position ID -> VaR contribution
    trading_engine_sender: Sender<TradingEngineCommand>,
}

The RiskManager struct will hold the current state, VaR limits, current VaR, and methods to update VaR and handle state transitions.

pub enum TradingEngineCommand {
    ExecuteTrade,
    NoTrade,
    StopEngine,
}

TradingEngineCommand is the enum that we send as command to TradingEngine thread using mpsc channel.

pub struct TradingEngine;
impl TradingEngine {
    pub fn start(receiver: mpsc::Receiver<TradingEngineCommand>) {
        thread::spawn(move || {
            println!("Trading engine started.");
            loop{
                let cmd = receiver.recv().unwrap();
                match cmd {
                    TradingEngineCommand::ExecuteTrade => {
                        println!("Executing trade");
                    }
                    TradingEngineCommand::StopEngine => {
                        println!("Stopping trading engine.");
                        // Perform cleanup if necessary
                        break;}
                    TradingEngineCommand::NoTrade => {
                        println!("No trade to execute.");}
                }
            }
            println!("Trading engine stopped.");
        });
    }}

Let's implement the RiskManager. You can think of the RiskManager as being between the trading signals and the actual trading engine. The RiskManager receives the trade, checks its risk and state, and then sends a command to the trading engine to either trade, hold, or stop. Let's go through this step by step.

• update_var: Recalculates the current VaR based on positions.

• add_position and remove_position: Modify positions and update VaR.

• check_state: Checks if a state transition is needed based on the current VaR.

• change_state: Handles the transition between states, calling exit_state on the old state and enter_state on the new state.

• send_command: Based on the state, it sends the command to the trading engine thread.

impl RiskManager {
    pub fn new(var_limit: f64, warning_level: f64, trading_engine_sender: Sender<TradingEngineCommand>) -> Self {
        let mut manager = RiskManager {
            state: Box::new(NormalOperationState{cmd: TradingEngineCommand::ExecuteTrade}),
            var_limit,
            warning_level,
            current_var: 0.0,
            positions: HashMap::new(),
            trading_engine_sender,
        };
        manager
    }

    pub fn update_var(&mut self) {
        self.current_var = self.positions.values().sum();
    }

    pub fn add_position(&mut self, position_id: &str, var_contribution: f64) {
        self.positions.insert(position_id.to_string(), var_contribution);
        self.update_var();
        self.check_state();
        self.send_command();
    }

    pub fn remove_position(&mut self, position_id: &str) {
        self.positions.remove(position_id);
        self.update_var();
        self.check_state();
        self.send_command();
    }
    pub fn send_command(&self) {
        self.state.send_command(&self);
    }
    pub fn check_state(&mut self) {
        match self.state.check_var(self){
            Some(state) => self.change_state(state),
            None => (),
        }
    }

    pub fn change_state(&mut self, new_state: Box<dyn RiskState>) {
        self.state.exit_state(self);
        self.state = new_state;
        self.state.enter_state(self);
    }

    pub fn should_shutdown(&self) -> bool {
        self.current_var >= self.var_limit * 1.2
    }
}

Implement Concrete States: NormalOperationState

#[derive(Debug)]
struct NormalOperationState{
    cmd: TradingEngineCommand
}

impl RiskState for NormalOperationState {
    fn check_var(&self, context: &RiskManager) -> Option<Box<dyn RiskState>> {
        if context.current_var >= context.warning_level && context.current_var < context.var_limit {
            Some(Box::new(WarningLevelState{cmd: TradingEngineCommand::ExecuteTrade}))
        } else if context.current_var >= context.var_limit {
            Some(Box::new(LimitBreachState{cmd: TradingEngineCommand::NoTrade}))
        } else {
            None
        }
    }

    fn enter_state(&self, _context: &RiskManager) {
        println!("Entering Normal Operation State");
        // Reset limitations
    }

    fn exit_state(&self, _context: &RiskManager) {
        println!("Exiting Normal Operation State");
    }
    fn send_command(&self, context: &RiskManager) {
        context.trading_engine_sender.send(self.cmd.clone());
    }
}
fn send_email_notification(message: &str) {
    println!("Sending email notification: {}", message);
}

The WarningLevelState, LimitBreachState, and ShutdownState work pretty much the same way, so we don't need to spell them out here.

Client

Let's write a main function that simulates adding positions and triggering state transitions.

fn main() {
    let var_limit = 100.0;
    let warning_level = 80.0;
    let (trading_engine_sender, trading_engine_receiver) = mpsc::channel();
    TradingEngine::start(trading_engine_receiver);

    let mut risk_manager = RiskManager::new(var_limit, warning_level, trading_engine_sender);

    risk_manager.add_position("Position1", 30.0);
    println!("Current VaR: {}", risk_manager.current_var);

    risk_manager.add_position("Position2", 40.0);
    println!("Current VaR: {}", risk_manager.current_var);

    risk_manager.add_position("Position3", 20.0);
    println!("Current VaR: {}", risk_manager.current_var);

    risk_manager.add_position("Position4", 15.0);
    println!("Current VaR: {}", risk_manager.current_var);

    risk_manager.add_position("Position5", 35.0);
    println!("Current VaR: {}", risk_manager.current_var);
    // Simulate the passage of time and check if shutdown is needed
    risk_manager.check_state();
    std::thread::sleep(std::time::Duration::from_secs(2));
}

The RiskManager begins in the NormalOperationState. As you add positions and the current_var goes over the warning_level, it moves to the WarningLevelState. If the current_var goes beyond the var_limit, it shifts to the LimitBreachState. And if it surpasses 120% of the var_limit, it enters the ShutdownState. You can check out the full implementation here: https://github.com/siddharthqs/design-patterns-in-rust

The RiskManager interacts with the trading engine appropriately based on its current state, ensuring that state-specific jobs are performed correctly.

Template Method Pattern

The Template Method Pattern is a behavioral design pattern that defines the skeleton of an algorithm in a method, deferring some steps to subclasses without changing the overall algorithm's structure. This pattern lets subclasses redefine certain steps of an algorithm without altering its structure.

This pattern is all about creating a template for an algorithm and allow subclasses to provide concreate implementation for some of the steps. It's like a blueprint for an algorithm that can be customized by subclasses. You can place your own custom logic in the template method and override the steps that need to be customized.

Key Concepts

• Abstract Class or Trait: Defines the template method and declares abstract methods representing steps that can vary.

• Template Method: A method that defines the skeleton of an algorithm, calling abstract methods that subclasses will implement.

• Concrete Subclasses : Provide implementations for the abstract methods, customizing the behavior of the algorithm.

In Rust, the Template Method Pattern can be implemented using traits with default method implementations. This lets concrete types override specific behaviors. It's one of the most commonly used patterns!

In nearly every financial application, we use a Monte Carlo simulation framework, whether it's for pricing derivatives, managing risk, or generating simulations for backtesting. In this chapter, we'll implement the Monte Carlo Simulation using the template method pattern.

Example: Monte Carlo Simulation

In a nutshell, the concept of Monte Carlo simulation is pretty straightforward; it involves generating random numbers and averaging of the outcome.

We'll define a trait MCSimulation that provides default implementations for generating random numbers and a template method simulation(). The simulation() method first calls generate_random_numbers() and then calls generate_path(). The generate_path() function does not have an implementation in the trait and must be provided by concrete structs like GeometricBrownianMotion.

By using the Template Method Pattern, we can define the skeleton of the simulation algorithm while allowing subclasses to customize specific steps.

Defining the MCSimulation Trait

Let’s define the MCSimulation trait that includes a default implementation for generating standard normal random numbers, a template method simulation() and an abstract method that must be implemented by the structs that implement the trait.

pub trait MCSimulation {
    /// Generates a vector of standard normal random numbers.
    fn generate_random_numbers(&self, num: usize) -> Vec<f64> {
        let mut rng = rand::thread_rng();
        (0..num).map(|_| rng.sample(StandardNormal)).collect()
    }
    /// Template method that runs the simulation.
    fn simulation(&self) -> Vec<f64> {
        let random_numbers = self.generate_random_numbers(self.get_number_of_steps());
        self.generate_path(&random_numbers)
    }
    /// Abstract method to generate the path based on random numbers.
    fn generate_path(&self, random_numbers: &[f64]) -> Vec<f64>;
    fn get_number_of_steps(&self) -> usize;
}

simulation() is the template method that use generate_path(), an abstract method to be implemented by concrete structs, defining how to generate the simulation path.

Implement StochasticProcess Trait

We define StochasticProcess trait for the underlying asset's stochastic process.

pub trait StochasticProcess {
    fn drift(&self, dt: f64) -> f64;
    fn diffusion(&self, dt: f64) -> f64;
}

Geometric Brownian Motion

GeometricBrownianMotionimplements StochasticProcess

pub struct GeometricBrownianMotion {
    pub initial_value: f64,
    pub risk_free_rate: f64,
    pub volatility: f64,
    pub time_steps: usize,
    pub maturity: f64,
}
impl StochasticProcess for GeometricBrownianMotion {
    fn drift(&self, _dt: f64) -> f64 {
        (self.risk_free_rate - 0.5 * self.volatility.powi(2))* _dt
    }
    fn diffusion(&self, _dt: f64) -> f64 {
        self.volatility * _dt.sqrt()
    }
}

Implementing MCSimulation

Now, let's have GeometricBrownianMotion implement the MCSimulation trait. It defines how to generate a path based on the Geometric Brownian Motion model.

impl MCSimulation for GeometricBrownianMotion {
    fn get_number_of_steps(&self) -> usize {
        self.time_steps
    }
    fn generate_path(&self, random_numbers: &[f64]) -> Vec<f64> {
        let dt = self.maturity / self.time_steps as f64;
        let mut s = self.initial_value;
        let mut path = Vec::with_capacity(self.time_steps + 1);
        path.push(s);
        for &dw in random_numbers {
            let drift = self.drift(dt);
            let diffusion = self.diffusion(dt) * dw ;
            s = s * (drift + diffusion).exp();
            path.push(s);
        }
        path
    }
}

Using the Simulation in Client Code

fn main() {
    let gbm = GeometricBrownianMotion {
        initial_value: 100.0,
        risk_free_rate: 0.05,
        volatility: 0.2,
        time_steps: 1000,
        maturity: 1.0,
    };
    let path = gbm.simulation();
    println!("Generated GBM path: {:?}", path);
}

We call the simulation() method, which generates a path using the Template Method Pattern.

By using the Template Method Pattern, we've created a flexible and extensible Monte Carlo simulation framework in Rust. The MCSimulation trait defines the overall simulation process, while concrete structs like GeometricBrownianMotion provide specific implementations for generating simulation paths. You can extend this framework to include other stochastic processes, such as the Heston model or the Cox-Ingersoll-Ross model, by implementing the generate_path() method accordingly.

Limit-Order Book (LOB)

Todo!

Conclusion

Design patterns like Strategy, Decorator, Observer, and Factory Method provide proven and adaptable solutions that help developers create strong and efficient trading systems. By learning and using these patterns, developers can greatly improve the quality, maintainability, and performance of algorithmic trading systems. By abstracting the creation of objects, by encapsulating execution and trading strategies separately, we can build systems that are easier to maintain, extend, and adapt to changing market conditions.

References

https://refactoring.guru/design-patterns/strategy

https://rust-unofficial.github.io/patterns/patterns/index.html

Rust Atomics and Locks-O'Reilly Media (2023)-Mara Bos

Check the data type of a variable

Rust provides a function std::any::type_name::<T>() that returns the name of the type as a string.

use std::any::type_name;
fn type_of<T>(_: &T) -> &'static str {
    type_name::<T>()
}
let s = String::from("Hello, World!");
let x = s.chars().nth(0);
println!("x is of type: {}", type_of(&x)); 
//x is of type: core::option::Option<char>

Using std::fmt

The fmt module provides functionality for formatting and printing, including traits like Display and Debug which are used to control how types are formatted when printed. Let’s see a example:

use std::fmt;
struct Point {
    x: i32,
    y: i32,
}
impl fmt::Display for Point {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "({}, {})", self.x, self.y)
    }
}
impl fmt::Debug for Point {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Point {{ x: {}, y: {} }}", self.x, self.y)
    }
}
fn main() {
    let point = Point { x: 1, y: 2 };
    println!("Display: {}", point);
    println!("Debug: {:?}", point);
}

fmt::Formatter contains the state necessary for formatting. It is passed as a mutable reference to the fmt method when implementing formatting traits like Display and Debug. It provides methods to write formatted data to the output. fmt::Result is an alias for Result<(), std::fmt::Error>. It is the return type for the fmt method, indicating whether the formatting operation was successful or if an error occurred.

Minimum and Maximum

Use std::cmp::min(a,b) to compare two values a and b. This function works with any type that implements the Ord trait, such as integer, float and other comparable types. Similarly cmp::max() works.

For float-point numbers f32 or f64, you can use it’s function min.

let min  = std::cmp::min(a,b);
let max  = std::cmp::max(a,b);

a.min(b);
a.max(b);

The cmp method in Rust returns an ordering result which can be Less, Equal or Greater.

//Signature of cmp
fn cmp(&self, other: &Self) -> Ordering
let comparison = a.cmp(&b);
match comparison {
    std::cmp::Ordering::Less => println!("a is less than b"),
    std::cmp::Ordering::Equal => println!("a is equal to b"),
    std::cmp::Ordering::Greater => println!("a is greater than b"),
    }

Vector

A Vec<T> in Rust is a growable, heap-allocated, contiguous sequence of elements of type T.

let mut v: Vec<i32> = Vec::new(); //Empty Vector
let v = vec![1, 2, 3]; //Using the vec! Macro 
v.retain(|&x| x % 2 == 0); // Keeps only even numbers
//Remove all instances of a value in place 
v.retain(|&x| x != val);

//Iterating Over Vectors
for val in &v { //Iterating by Reference
    println!("{}", val);
}
for val in &mut v { //Iterating by Mutable Reference
    *val *= 2;
}
for val in v { //Consuming the Vector
    println!("{}", val);
}
// v cannot be used after this point
let v1 = vec![1, 2, 3];
let v2 = v1; // v1 is moved to v2
// v1 is no longer accessible
//Only one mutable reference or any number of immutable references.

Minimum, Maximum and Sum of a Vector

Use the iter() method combined with the max() and min() functions, min() and max() are methods from the Iterator trait that return Option<&T>. The result will be Some(value) if the vector is non-empty, or None if the vector is empty. The values need to be unwrapped using pattern matching.

let mut v = vec![80, 50, 10, 20, 45, 30];
match v.iter().min() {
    Some(min_value) => println!("Minimum value: {}", min_value),
    None => println!("The vector is empty!"),
}
// Using let Some
if let Some(max_value) = v.iter().max() {
    println!("Maximum value: {}", max_value);
}
// Sum the elements using sum()
let total: i32 = v.iter().sum();
let total: i32 = v.iter().fold(0, |acc, &x| acc + x);

Slicing of Vector

Let’s say you want to iterate over a vector starting from second element (i.e., excluding the first element):

let mut v = vec![80, 50, 10, 20, 45, 30];
//Using Slice of vector
for i in &v[1..] {
    println!("{}", i);
}

//Using Mutable Slices for Modification
for i in &mut v[1..] {
    *i += 5;  // To modify each element dereference i with *i
}

//Using Index-Based Looping
for idx in 1..v.len() {
    println!("Index: {}, Value: {}", idx, v[idx]);
}
//Using Enumerate with Skip
for (idx, val) in v.iter().enumerate().skip(1) {
    println!("Index: {}, Value: {}", idx, val);
}

&v[1..] : This creates an immutable slice of v starting from index 1 to the end.

Sorting

The sort() method sorts the elements of the vector in place in ascending order. The elements must implement the Ord trait.

If you want to sort with a custom comparator, you can use the sort_by() method, which takes a closure to determine the order of elements.

v.sort();  // Sorts in ascending order
v.sort_by(|a, b| b.cmp(a)); // Sorting in descending order

Let’s say you want to sort vector of vectors based on some conditions such as the first element of each inner vector, or sum of elements etc., you can use sort_by() method. Here are a few examples:

// Sorts by the first element
vec_of_vecs.sort_by(|a, b| a[0].cmp(&b[0]));
// Sorts by the length of inner vectors
vec_of_vecs.sort_by(|a, b| a.len().cmp(&b.len()));
// Sorts by sum of elements
vec_of_vecs.sort_by(|a, b| a.iter().sum::<i32>().cmp(&b.iter().sum::<i32>()));

HashMap

The HashMap in Rust is a key-value store from the std::collections module. You can use the get() method, which returns an Option.

let mut m:HashMap<i32,i32> = HashMap::new(); //Declaring a HashMap
m.insert(1,1); //Inserting Values into a HashMap
m.insert(2,2);
let n = 2;
//Accessing Values from a HashMap using match
match m.get(&n){
    Some(value) => {println!("Value for 'n': {}", *value);}
    None => { //Key does not exist
        let value = 2;
        m.insert(n,value);
        }
    }
//Accessing Values using if let
if let Some(value) = m.get(2) {
    println!("Value for 2: {}", value);
} else {
    println!("Key not found");
}

//Using contains_key
if m.contains_key(2) {
    println!("Key exists");
} else {
    println!("Key does not exist");
}
// Removing Values from a HashMap
let removed_value = m.remove(2);
println!("{:?}", removed_value);

Find the key with the maximum value

You can use the iter() method to iterate over the key-value pairs and use the max_by_key() method to find the pair with the highest value.

// Find the key with the maximum value
if let Some((k, v)) = m.iter().max_by_key(|entry| entry.1) {
    println!("max key and maximum value is: {}: {}", k, v);
} else {
    println!("The HashMap is empty.");
}

max_by_key(|entry| entry.1) finds the key-value pair with the maximum value. Here, entry is a tuple (&key, &value), and entry.1 accesses the value.

.entry()

It returns an Entry enum, which represents either a vacant entry (Vacant) if the key is not present or an occupied entry (Occupied) if the key is already present. or_insert(value) inserts value into the map if the key is not already present and returns a mutable reference to the value in the map (either the existing one or the newly inserted one).

let mut map = HashMap::new();
// Access the entry for the key 'a'
map.entry('a').or_insert(0);
// Now the map contains {'a': 0}
let count = map.entry('a').or_insert(0);
*count += 1; // Increment the count for 'a'

.get_mut()

get_mut() retrieves a mutable reference to a value in a HashMap. It returns an Option<&mut V>, so you can modify the value if the key exists (Some(&mut value)).

let mut m: HashMap<i32, Vec<i32>> = HashMap::new();    
// Initialize the HashMap with empty vectors
for i in 0..5 {
    m.insert(i, Vec::new());
}
//my_vec: Vec<Vec<i32>>
for i in &my_vec {
    if let Some(t) = m.get_mut(&i[0]) { //get_mut() retrieves a mutable reference 
        t.push(i[1]);
   }
}

BinaryHeap

BinaryHeap is a max-heap, meaning the largest element is always at the top.

Heap Operations:

• push(value): Inserts a value into the heap.

• pop(): Removes and returns the greatest value (for max-heap).

• peek(): Returns a reference to the greatest value without removing it.

let mut heap = BinaryHeap::new();
 // Insert elements into the heap
heap.push(5);
let vec = vec![5, 1, 10, 3];
let heap2 = BinaryHeap::from(vec); //vec is no longer usable after this
 //Using collect with an Iterator
let heap3: BinaryHeap<_> = vec.into_iter().collect();

Finding the Kth Largest Element

fn find_kth_largest(nums: Vec<i32>, k: usize) -> i32 {
    let mut heap = BinaryHeap::from(nums);
    for _ in 0..k - 1 {
        heap.pop();
    }
    heap.pop().unwrap()
}

HashSet

A HashSet<T> is an unordered collection of unique values of type T, implemented as a hash table.

let mut set: HashSet<i32> = HashSet::new();
let vec = vec![1, 2, 3, 4];
let set: HashSet<_> = vec.into_iter().collect();
//Using the From Trait
let set = HashSet::from([1, 2, 3, 4]);

//adding elements
set.insert(77);
//Removing Elements
set.remove(&77); //Returns true if the element was present.

//Checking for Elements
if set.contains(&77) {
    println!("Set contains 77.");
}

Set Operations

let set_a = HashSet::from([1, 2, 3]);
let set_b = HashSet::from([3, 4, 5]);
//Returns an iterator
let union: HashSet<_> = set_a.union(&set_b).cloned().collect();
let intersection: HashSet<_> = set_a.intersection(&set_b).cloned().collect();
let difference: HashSet<_> = set_a.difference(&set_b).cloned().collect();
//Returns an iterator over elements that are in either set_a or set_b but not in both.
let sym_diff: HashSet<_> = set_a.symmetric_difference(&set_b).cloned().collect();

//Subset and Superset Checks
if set_a.is_subset(&set_b) {
    println!("set_a is a subset of set_b");
}
if set_a.is_superset(&set_b) {
    println!("set_a is a superset of set_b");
}

Convert a HashSet to a Vec

A HashSet is unordered, so when you collect its elements into a Vec, the order in the resulting vector is arbitrary. This is straightforward using the iter() method and the collect() function.

.cloned() creates an iterator that yields owned copies of the elements.

let vec: Vec<i32> = hash_set.iter().cloned().collect();
//Using into_iter() 
//This moves the elements out of the HashSet and into the vector.
let vec: Vec<i32> = hash_set.into_iter().collect();

String

The Rust String and str types represent text using the UTF-8 encoding form, which means they can contain characters that are more than one byte in size. Because of this, you cannot directly index into a String or &str using the square bracket syntax like s[1] or s[2].

Access Characters at a Specific Position

The chars() method returns an iterator over the characters (char type) of a string. You can use the nth() method to get the character at a specific index. nth() returns an Option<char>, so you need to handle the case where the index is out of bounds. For example, nth(1) retrieves the character at index 1 (second character). This method accounts for multi-byte characters and is safe for UTF-8 strings. The time complexity of nth() is O(n), as it iterates through the string up to the nth character.

let s = String::from("Hello");
let c1 = s.chars().nth(1);
let c2 = s.chars().nth(2);

You can collect the characters into a vector and then index directly. If you need frequent random access the can prefer this method.

let s = String::from("Hello");
let chars: Vec<char> = s.chars().collect();
let c1 = chars[1];

Remove Spaces from a String

let s = String::from("Hello World!");
//Using the replace Method
let s_no_spaces = s.replace(" ", "");

You can iterate over each character, filter out the spaces, and collect the result into a new String.

let s1: String = s.chars().filter(|&c| c != ' ').collect();

let s2: String = s.chars().filter(|c| !c.is_whitespace()).collect();
//Using the retain Method
s.retain(|c| c != ' ');

Methods that consume the string (like replace and filter) create a new String, so you need to assign the result to a new variable.

The is_whitespace() method checks for Unicode whitespace characters, making it robust for international text. Using retain modifies the string in place and can be more efficient since it doesn't create an intermediate collection.

Transforming Strings: Lowercase Conversion and Filtering

let mut s = String::from("Hello, World! 123");
s.retain(|c| c.is_alphanumeric());
s = s.to_ascii_lowercase();

You can iterate over each character in the string, convert it to lowercase, filter out non-alphanumeric characters, and collect the result into a new String.

let s = String::from("Hello, World! 123");
let cleaned: String = s
    .chars()
    .filter(|c| c.is_alphanumeric())
    .map(|c| c.to_ascii_lowercase())
    .collect();

Split a String by Spaces

The split_whitespace() method splits a string slice (&str) on Unicode whitespace characters and returns an iterator over the substrings. The type Vec<&str> means the vector contains string slices referencing the original string s. split_whitespace() automatically ignores leading and trailing whitespace. split_whitespace() splits on any Unicode whitespace character, including spaces, tabs (\t), newlines (\n), and carriage returns (\r).

let words: Vec<&str> = s.split_whitespace().collect();
//If You Need Vec<String>
let words: Vec<String> = s
        .split_whitespace()
        .map(|s| s.to_string())
        .collect();
let words: Vec<&str> = s.split(' ').collect();

split() method can be used with a space delimiter. But consecutive spaces result in empty strings ("") in the output. split(' ') will include empty strings.

let words: Vec<&str> = s.split(' ').collect();
//To filter out empty strings
let words: Vec<&str> = s
        .split(' ')
        .filter(|&s| !s.is_empty())
        .collect();

Joining a Vector of Strings

You can use the join() method on a Vec<String> or a Vec<&str>. This method concatenates the elements of the vector, placing a separator between them.

let words = vec!["Hello", "world"];
let sentence = words.join(" ");
words[1] = "Rust";
// The sentence remains unchanged
println!("{}", sentence);  // Output: Hello world
// Print the modified vector
println!("{:?}", words);  // Output: ["Hello", "Rust"]

The join() method does not consume the vector so you can access or modify it later. join() creates a new, separate copy of the joined string, and any changes to the original vector will not affect the sentence string.

Building a Character Frequency HashMap from a String

let mut freq_map = HashMap::new();
for c in s.chars() {
    *freq_map.entry(c).or_insert(0) += 1;
}

Iterators

Iterators are lazy, they don't compute anything until they are consumed.

Ownership, reference and mutability

Passing references

Use & (e.g., &v for immutable reference, &mut v for mutable reference). If you pass &v, you are passing a borrowed immutable reference of v. When accepting a reference in a function, declare the parameter as &Type to signify that the function will accept an immutable reference.

fn my_function(v: &Vec<i32>)
fn modify_vector(v: &mut Vec<i32>)

&mut v creates a mutable reference to the vector v. You can modify v directly through this reference. The original v remains owned by the original scope, but you have mutable access to it.

Conclusion

Keep practicing to get better at Rust. If any syntax or code that you use a lot in your daily coding, please leave a comment! I'll add new sections based on your feedback!

Implicit Insertion

When you use operator[] to access a key that does not exist in the map, std::map will implicitly insert a new element with that key and a default-constructed value. This can lead to unintended side effects.

std::map<char, int> my_map;
std::cout << "Size of map: " << my_map.size() << std::endl; 

int value = my_map['a']; // 'a' is not in the map, so it gets added with value 0.    
std::cout << "Value for key 'a': " << value << std::endl;
std::cout << "Size of map: " << my_map.size() << std::endl; // Size is now 1

Issue: This implicit insertion can lead to bugs if you are only intending to check for the existence of a key or to retrieve a value without modifying the map and expect a exception if key is not present.

Even if you are performing read-only operations on the map, using operator[] can unintentionally modify the map by inserting new elements.

int main() {
    std::map<char, int> my_map;

    // Read-only check
    std::cout << "Size of map: " << my_map.size() << std::endl;
    if (my_map['a'] > 0) {
        std::cout << "'a' is present with a positive value" << std::endl;
    }
    std::cout << "Size of map: " << my_map.size() << std::endl; // Size is now 1, even though we intended to just check
    std::cout<<"Value at a: "<<my_map['a']<<std::endl;
}

Issue: The map is modified even when the intention was just to check the value.

Default Constructor is called

When a new key is added implicitly using operator[], the value is default-constructed. For primitive types like int or double, this means zero initialization. However, for user-defined types, the default constructor is called, which might not be the desired behavior.

struct MyStruct {
    MyStruct() {
        data =10;
        std::cout << "Default constructor called" << std::endl;
    }
    int data;
};
int main() {
    std::map<char, MyStruct> my_map;
    MyStruct value = my_map['a']; // Default constructor for MyStruct is called
    std::cout<<"Value of data: "<<value.data<<std::endl;
    std::cout << "Size of map: " << my_map.size() << std::endl; // Size is now 1
}

Issue: This default construction might have performance implications or unintended side effects if the default constructor does more than simple initialization.

Performance Overhead

In scenarios where operator[] is used for repeated lookups and the key does not exist, it results in multiple implicit insertions with default values, which can add unnecessary performance overhead.

int main() {
    std::map<char, int> my_map;
    std::cout << "Size of map: " << my_map.size() << std::endl;
    for (char c = 'a'; c <= 'z'; ++c) {
        int value = my_map[c]; // Each lookup inserts a default value if the key does not exist
    }

    std::cout << "Size of map: " << my_map.size() << std::endl; // Size is now 26
}

Issue: This can lead to a larger map with many unnecessary default-constructed values, impacting memory usage and performance. In this example you end up 26 elements in the map.

What you should use instead

To avoid the issues with operator[], you can use the following alternatives:

at Method

at method throws an exception if the key does not exist. You should use this when you expect the key to be present.

int main() {
    std::map<char, int> my_map;
    try {
        int value = my_map.at('a'); // Throws std::out_of_range if 'a' does not exist
    } catch (const std::out_of_range& e) {
        std::cout << "Key 'a' not found" << std::endl;
    }
}

find Method

find method returns an iterator to the element or end() if the key does not exist. you can use this as well for existence checks without modifying the map.

int main() {
    std::map<char, int> my_map;
    auto it = my_map.find('a');
    if (it != my_map.end()) {
        int value = it->second;
        std::cout << "Value for key 'a': " << value << std::endl;
    } else {
        std::cout << "Key 'a' not found" << std::endl;
    }
}

Conclusion

The operator[] in std::map or in std::unordered_map offers easy element access, but it may cause unexpected results. For read-only operations or when checking for the existence of a key, it's better to use methods like at, find, or count to avoid potential issues. Understanding these nuances will help you use std::map more effectively and prevent unintended side effects in your C++ code.

Overview of POV

The POV is an order execution trading strategy that allows traders to execute the desired percentage of the total market volume called participation rate over a specified period. It's an adaptive strategy designed to adjust the order size dynamically based on prevailing or forecasted market volume in real-time.

Unlike TWAP / VWAP, which follows a fixed schedule, the POV algorithm changes its trading in real-time to ensure a consistent participation rate. The POV, however, has an intuition of the VWAP algorithm, but in real-time. The VWAP algorithm trades more when the volume is historically high and less when the volume is historically low. The POV algorithm trades more when the market trades more and less when the market trades less.

How POV Works

Assume a trader intends to execute 5% of total market volume during an hour using a POV algorithm. The algorithm will seek to trade 500 shares for every 10,000 shares that trade in the following hour. The algorithm will attempt to submit an order for 50 shares for every 1000 shares, e.g., if 950 shares trade in 1 minute, the algorithm will send a limit order or market order (for more aggressive execution) for 50 shares (total volume traded would be 950+50 = 1000). And if another 1000 shares trades in the market during the following 30 minutes, the algo will have traded only 100 shares in 31 mins (1+30 mins). Generally, for the POV algorithm, the schedule is updated on a tick-by-tick basis, and it waits for trading to occur and then participates only afterward.

Market Impact and Considerations for the POV

Since the POV algorithm must trade more aggressively to avoid falling behind schedule, it often uses market orders rather than limit orders, placing orders only after the trading volume has happened. Spread expenses increase as a result. The POV is volume-responsive and ultimately becomes a liquidity taker rather than a liquidity generator. It trades immediately following the volume surge, responds to the previous volume, and has the potential to significantly affect the price if another trader has made a similar move. Another increase in volume would result from this, which would encourage trading by additional parties. The POV would trade more aggressively to maintain the target rate due to this feedback cycle at the expense of higher market impact costs.

Inability to Guarantee Completion

POV algorithms cannot guarantee that an order will be executed within the specified time frame. Market conditions, particularly low liquidity, may result in partial fills or the inability to complete the order within the desired period.

Not Suitable for All Securities

Illiquid stocks have short and isolated bursts of activity than more liquid stocks. The POV would respond to such spikes quickly and would have more spread and market impact cost. Intuitively, the POV should be more effective than TWAP / VWAP with the higher volume. However, it ends up being reactive and suboptimal.

Implementation Challenges

POV algorithm implementation and maintenance can be challenging and need careful calibration. Traders must constantly evaluate and modify parameters; they must pay attention to order sizes, the number of open orders, the intervals between orders, the nature of market dynamics, order book sizes, and hidden liquidity. Additional risk management and real-time monitoring are crucial to navigate the complexities introduced by the interaction of Iceberg Orders.

Conclusion

While the POV has several advantages over other algorithms, such as TWAP and VWAP, it also has limitations and challenges. Nevertheless, with careful calibration and monitoring, POV can be a powerful tool for traders seeking to navigate the complexities of algorithmic trading. Traders must pay attention to its implementation details and can use its concept along with other algorithms, such as VWAP, to design custom execution trading algorithms.

A standard way to measure latency is by determining the time it takes a given data packet to travel from source to destination and back, the round-trip time (RTT).

The data packet in the automated trading system is the message. A message is a standardized packet of data that enables a trader and an automated trading venue to communicate with each other. It is a primitive unit of valuable info that enables all the pre-trading, trading and post-trading activities. Messaging is the only way to affect the price formation process. Each message is time-stamped when it passes through a specific subsystem of an automated trading system.

Types of Latency

Here are the primary types of latency encountered in automated trading:

Communication latency

Communication latency is the delay introduced during the transmission of data between different components of a trading system. This can be reduced by buying co-location services. It encompasses the time it takes for data to travel from the trader's system to various destinations, such as the exchange's servers, liquidity providers, or other trading platforms.

• Reducing Communication Latency: Traders often seek to minimize communication latency by leveraging co-location services. Co-location involves placing their trading servers physically close to the exchange's servers.

Market-Feed Latency

Market-feed latency refers to the delay in receiving market data updates from the exchange. This includes information about prices, trade executions, order book updates, and other critical market information.

• Reducing Market-Feed Latency: To mitigate market-feed latency, traders often opt for proprietary data feeds provided by the exchange or a few data vendors. These proprietary feeds are optimized for speed and can provide market information faster than publicly available feeds for retail traders.

Trading Platform Latency

Trading platform latency encompasses the delay introduced within the trading platform itself. This latency is largely beyond the control of individual traders and is determined by the design and infrastructure of the trading platform they are using.

• Trading Platform as a Given: Traders must accept trading platform latency as a given factor that is inherent to the platform they choose to use. While they can optimize their strategies and use low-latency communication channels, they may not have direct control over the latency introduced by the platform itself.

Latency moves volatility moves.

Matching engine processing time

One of the critical aspects of automated trading is the speed at which orders are matched and executed within a trading platform's matching engine. The matching engine is the heart of an exchange or trading platform. Traders seek platforms with low-latency matching engines to ensure that their orders are executed swiftly.

Best Bid and Offer (BBO) Messages

Messages that result in changes in the Best Bid and Offer (BBO) are of utmost importance in terms of the price formation process. The BBO represents the highest bid price and the lowest ask price in the order book. It takes more time for a platform to process BBO messages than a message that changes the lengths of existing queues.

Depth Messages

Conversely, messages that add depth to the order book by introducing small orders or trading volume without changing the BBO have a less immediate impact on price formation. It doesn't move the mid-point price and takes less time to update the existing queues.

In conclusion, latency is a fundamental consideration in automated trading systems and the role of matching engine processing time and the different message types is essential for traders operating in automated trading environments. Communication latency and market-feed latency can often be managed by selecting the right services and infrastructure, while trading platform latency is a factor that traders must consider when choosing their trading platform.

Introducing Option

Option is an enumeration (enum) type that represents either a value or the absence of a value. It can either be "Some" or "None," helping you navigate the potentially missing or invalid values. It is defined by the standard library as follows:

enum Option<T> {
    Some(T),
    None,
}

The Option enum is so useful that it’s even included in the prelude. In other words, if a value has a type that isn’t an Option, you can safely assume that the value isn’t null.

Option Variants

The option has two variants:

1. Some(T): Represents a value of type T. <T> is a generic type parameter. It indicates that a value is present.

2. None: Represents the absence of a value.

Some and None are used directly without the Option:: prefix as they are included in the prelude as well.

The Null-less Rust Coast

Rust is free of the "null" feature that many other languages have. Null is a value that means there is no value there. In languages with null, variables can always be in one of two states: null or not-null. The problem? If you try to treat null as not-null, you’ll get an error and since null or not-null is a common property, it's easier to fall into this.

Sailing with Option

In Rust, Option is the ship, safeguarding you from these treacherous waters. Let's embark on this adventure with a simple example: Here the fun1 function returns an Option<f64> which means it can either return Some(f64) or None.

fn fun1()->Option<f64>{
    let x = 0.0;
    if x ==0.0{
        return Some(x);
    }
    else{
        return None;
    }
}
fn main(){
    let x = fun1();
    let y = 1.0;
    let sum = x+y;
}

error[E0369]: cannot add `{float}` to `Option<{float}>`
 --> src/main.rs:4:16
  |
4 |     let sum = x+y;
  |               -^- {float}
  |               |
  |               Option<{float}>

The expression x + y tries to add an Option<f64> (x) to a f64 (y). This is not allowed because Rust's type system enforces safety, and it doesn't allow adding an Option to a regular value directly. You would need to handle the Option using match statements or other methods to extract the value from Some (if it exists) and then perform the addition.

The Option has a large number of methods that are useful in a variety of situations like this. You can use methods like unwrap or unwrap_or to handle the Option and get its value.

Methods to work with Option

The Option enum provides several methods and associated functions that allow you to work with optional values (values that can be either Some(value) or None). Here are some of the commonly used methods for Option:

1. unwrap(): This method extracts the value from a Some variant and panics if the variant is None. It should be used with caution, as it can lead to a panic.

2. unwrap_or(default): Extracts the value from a Some variant or returns a default value if the variant is None. It provides a safer alternative to unwrap().

3. unwrap_or_else(fn): Similar to unwrap_or, it takes a closure that computes and returns the default value if needed. This allows for lazy evaluation of the default value.

4. expect(msg): Similar to unwrap(), but it allows you to provide a custom error message that will be displayed if the variant is None.

5. is_some(): Returns true if the variant is Some, indicate the presence of a value.

6. is_none(): Returns true if the variant is None, indicating the absence of a value.

7. map(fn): Applies a function to the inner value if it's Some and return a new Option containing the result. If it's None, it returns None.

8. map_or(default, fn): Applies a function to the inner value if it's Some, return the result. If it's None, it returns the provided default value.

fn main(){
    let x = fun1();
    println!("{}",x.unwrap_or(10.0));
    println!("{}",x.unwrap_or_else(|| {fun2()}));
    println!("{}",x.expect(" It has None"));
    println!("{:?}",x.is_some());
}
fn fun2()->f64{
    100.0
}
fn fun1()->Option<f64>{
    let x = 0.0;
    if x ==0.0{
        return Some(x);
    }
    else{
        return None;
    }
}
----------------------------Output---------------------
0
0
0
true

Pattern Matching with Option

fn main() {
    let x = fun1();
    match x {
        Some(x) => println!("The value is: {}", value),
        None => println!("No value found"),
    }
}

In the code above, we use match to extract and handle the value inside the Option. When Some, we print the value; when None, we indicate that no value was found.

Another example of Deserialize and Serialize JSON

Let's see an example of JSON. You are working with JSON where a few attributes are optional. In the below example, let's assume style is optional, you could get JSON as input where style is absent.

 {
"current_price": 100.0
"style":"American"
}

To handle this case, you could define struct as below:

#[derive(Clone,Debug,Deserialize,Serialize)]
pub struct Contract {
    pub current_price: f64,
    pub style: Option<String>
}
pub struct EquityOption {
    pub current_price: f64,
    pub style: Option<String>,
}

let mut contents = String::new();
file.read_to_string(&mut contents).expect("Failed to read JSON file");
let data: utils::Contract = serde_json::from_str(&contents)
                           .expect("Failed to deserialize JSON")
let mut contract = EquityOption {
    current_price: data.current_price,
    style: Option::from(data.style.as_ref().unwrap_or(&default_style))

error[E0308]: mismatched types
style : Option::from(data.style.as_ref().unwrap_or(&"European".to_string()))
  |                 ^^^^^^^^^^^^^^^^^^^^^^ expected `Option<String>`, found `Option<&String>`
  |
  = note: expected enum `Option<String>`
             found enum `Option<&String>`

Option::from is used to convert a value of one type into an Option of that type.

We used Option<String> in our equity struct that is an Option that can either hold a Some(String) value. The String inside Option<String> is owned, meaning we have exclusive ownership of this String.

The above code gives an error because we get the Option<&String> type. Option<&String> is an Option that can hold a reference to a String (&String) or None. The &String inside Option<&String> is a borrowed reference, meaning it does not have ownership of the data, and it cannot be modified or moved. It points to an existing String owned elsewhere.

Either we can change the definition of our struct and use Option<&String> then we have to make sure about the borrowing and life of the referred string. Or we can clone the string as per our use case.

style : Option::from(String::from(data.style.as_ref()
        .unwrap_or(&"European".to_string())))

.unwrap_or(&"European".to_string()) is used to provide a default value in case the Option is None. If the data.style is Some(String), this part is ignored. If the data.style was None, it creates a new String with the value "European" and returns a reference to it. Note unwrap_or expect &String hence we return a reference & .

Here is youtube video discuss about the Option<&T> and &Option<T>

Conclusion

In conclusion, you typically use Option in Rust represents optional or nullable value and wants to avoid null pointer errors and ensure safer code. You most often use when:

• You expect a function to return a value but it might fail or return nothing, you use Option to handle the result.

• You have a variable that might be initialized with a valid value or remain uninitialized, you use Option to represent its state.

• Market data feed: Simulated or real-time market data stream.

• Signal generation: Analysis of market data to generate trading signals.

• Order execution: Placement and execution of trading orders.

In a simple example of this system, we need at least three separate threads to handle simulating market data, signal generation, and order execution, allowing the trading system to react swiftly to market conditions.

Market Data Simulation

For the sake of simplicity for this blog, let's just simulate a single stock. A common way to represent an order book is by using a struct. Here's an example of a basic struct for an order book:

#[derive(Debug)]
struct Quote {
    symbol: String,
    bids: (f64, u32),
    asks: (f64, u32),
}
#[derive(Debug)]
struct Order {
    symbol: String,
    quantity: i32,
    price: f64,
    order_type: String,
}

In this example, the Quote struct has fields: bids and asks. Each field holds tuples representing the best price levels and quantities. You can further enhance the Quote to hold the vector for bids and asks e.g. bids: Vec<(f64, u32)>.

fn simulate_market_data() -> Quote {
    let normal = rand_distr::Normal::new(0.0, 1.0).unwrap();
    let z = normal.sample(&mut rand::thread_rng());
    let st = 100.0;
    let risk_free_rate = 0.05;
    let volatility = 0.2;
    let dt = 1.0/252.0;
    let bidprice = st*exp(((risk_free_rate - 0.5 * volatility.powi(2)) * dt)+volatility * dt.sqrt()*z);
    let askprice = bidprice + 0.05;
    // Generate random quantity
    let mut rng = rand::thread_rng();
    let bid_quantity = rng.gen_range(100..1000);
    let ask_quantity = rng.gen_range(100..1000);
    let quote = OrderBook {
        symbol: "SPY".to_string(),
        bids: (bidprice, bid_quantity),
        asks: (askprice, ask_quantity),
    };
    return quote
}

In this example, the simulate_market_data function generates random market data and returns Quote.

Channels for inter-thread communication

Channel provides a built-in communication mechanism for inter-thread data exchange. It handles synchronization internally, reducing the likelihood of synchronization errors.

fn market_data(tx: mpsc::Sender<Quote>) {
    loop {
        // Generate market data
        let order_data = simulate_market_data();
        // Send market data to the main thread
        tx.send(order_data).expect("Failed to send market data");
        // Pause the execution for one millisecond
        thread::sleep(Duration::from_millis(1));
    }
}

We will run the market_data function in a separate thread and repeatedly generates market data using simulate_market_data. It sends the generated market data to the main thread via a channel (tx) and wait for one millisecond to send another order data.

Spawn threads

fn main() {
    // Create channels for inter-thread communication
    let (tx, rx) = mpsc::channel();
    let (tx_order, rx_order) = mpsc::channel();
    // Spawn a thread for market data simulation
    thread::spawn(move || {
        market_data(tx);
    });
    // Receive and process market data in the separate thread
    thread::spawn(move || {
        signal_generation(rx,tx_order);
    });
    thread::spawn(move || {
        order_management(rx_order);
    });
    loop{}
}

In the main function, two channels are created for inter-thread communication using mpsc::channel(). Then, three separate threads are spawned using thread::spawn, where the market_data, signal_generation and order_management functions are executed. The main thread continuously receives market data from the channel using rx.recv(). Upon receiving market data, it can process it according to the requirements.

Signal generation and order management

fn signal_generation(rx: mpsc::Receiver<Quote>,tx_order: mpsc::Sender<Order>){
    while let Ok(data) = rx.recv() {
        // Process market data
        println!("Received market data: {:?}", data);
        let bid_price = data.bids.0;
        let ask_price = data.asks.0;
        if bid_price<100.0{
            println!("Place BUY Order");
            let order = Order {
                symbol: data.symbol,
                quantity: 1,
                price: bid_price,
                order_type: "LMT".to_string(),
            };
            tx_order.send(order).expect("Failed to send order");
        }
        else if ask_price>100.0 {
            println!("Place SELL Order");
            let order = Order {
                symbol: data.symbol,
                quantity: 1,
                price: ask_price,
                order_type: "LMT".to_string(),
            };
            tx_order.send(order).expect("Failed to send order");
        };
    }
}

Here we are using two channels one for receiving Quotes and another for sending orders.

fn order_management(rx:mpsc::Receiver<Order>){
    while let Ok(data) = rx.recv() {
        // Process Orders
        println!("Received order: {:?}", data);
    }
}

Other considerations

You can use Mutex as well. Ideally, use mutexes to protect shared data and coordinate access to critical sections. You may have multi-step operations that involve multiple threads collaborating to complete a task then Mutexes can be used to coordinate the execution of these operations, ensuring that threads wait for their turn and follow a specific order or synchronization protocol.

Additionally, other synchronization primitives like RwLock, Atomic types, or even higher-level abstractions like message queues or publish-subscribe systems are useful depending on the specific requirements.

Resources

Here is an excellent video by Jon Gjengset to learn more about channels.

And here is another great resource to learn multithreading in Rust, probably the BEST Book on concurrency by Mara Bos. Excellent Read. Thank you, Mara.

There are a few Rust-specific tools like flamegraph that can help identify bottlenecks and areas for optimization while using multithreading. Flamegraphs are used to visualize where time is being spent.

Hope this is helpful!

Time-weighted Average Price (TWAP) Algorithm

TWAP algorithms are designed to execute trades based on a rate proposed to time. As an example, suppose you want to purchase 5000 shares of AAPL over a period of 10 minutes. By using TWAP, you would buy 500 shares during the first minute, followed by another 500 shares during the next minute, and so on. By the beginning of the 10th minute, you should have already bought 4500 shares (90%) and started buying the rest. The process is relatively simple: you create a schedule of 10 orders, and send them at appropriate intervals (e.g. every minute). This can be easily achieved by sending market orders. Let's examine the code to gain a better understanding. In the code below, "total_time" refers to the overall duration of the order, and "interval" refers to the time between each execution interval. By passing the interval as 1, we indicate that the execution should occur every minute.

use std::thread;
use std::time::Duration;

fn twap(symbol: String, quantity: i32,total_time:i32 interval: i32,orderType: String) {
    // Calculate the number of sub-orders needed
    let num_orders = total_time / interval;
    // Calculate the quantity for each sub-order
    let sub_order_quantity = quantity / num_orders;
    // Send the sub-orders at regular intervals
    for i in 1..=num_orders {
        send_order(ticker,sub_order_quantity,orderType);
        thread::sleep(Duration::from_secs(interval*60 as u64));
    }
}

fn main() {
    twap('AAPL',5000, 10,1,'MKT');
}

To maintain the schedule, I opted to use market orders. Market orders allow for immediate buying or selling of securities at the prevailing market price. However, this comes at a cost, as market orders often have higher fees and less favorable prices. Despite being able to achieve the desired trading rate of 500 shares per minute, this approach results in the worst possible execution. Another option to consider is using a Limit Order, where a trader sets a price at which they are willing to buy or sell an asset. The order is then only executed when the market price reaches the specified price or a better one. However, there is no guarantee that a limit order will be executed.

Using limit orders in the TWAP may lead to a situation where a significant number of orders, including large quantities, are left unexecuted, causing a deviation from the desired schedule. Here is the trade-off between market order and limit order, the optimal implementation of TWAP requires both order-type.

It is common to establish acceptable ranges within which the algorithm can deviate from the desired schedule. These ranges define upper and lower limits on the number of shares that the algorithm is permitted to be ahead of or behind schedule at any given point.

Let's consider a range of 10% deviation from the desired schedule, e.g. we are allowed to deviate by 10 % of quantity at a given point in time. In our example at 2 minutes, we should execute 20% of the quantity (1000 shares)), however, we are allowed to deviate 10% so we can have only 10% (500 shares not less). We can start placing limit orders according to our schedule and if our order doesn't get executed and the deviation threshold is breached, we can modify the order type to a market order. A minimum quantity must be executed. Similarly, we have the maximum quantity from the upper bound, the difference between the maximum and minimum quantity is called "maximum working quantity". The maximum working quantity in our case is 1000 shares, it is the key consideration to decide the upper and lower bound along with asset liquidity.

Let's improve our code above a bit.

To implement the algorithm we discussed above, we have two threads. One thread places the order according to the schedule (twap) and the other thread keeps checking pending orders and changes the order type to market and send the order.

use std::sync::{Arc, Mutex};
use std::time::Duration;
use std::thread;
#[derive(Debug)]
struct Order {
    symbol: String,
    quantity: i32,
    price: f64,
    order_type: String,
    status: String
}

fn twap(symbol: String, quantity: i32,total_time:i32, interval: i32, order_book_clone:&Arc<Mutex<Vec<Order>>> ) {
    //let order_book_clone = Arc::clone(&order_book);
    let num_orders = total_time / interval;

    let sub_order_quantity = quantity / num_orders;
    // Send the sub-orders at regular intervals
    for i in 1..=num_orders {
        let order = Order{
            symbol : symbol.clone(),
            quantity: sub_order_quantity,
            price: 0.0,
            order_type: String::from("LMT"),
            status: 
            };
        let mut order_book = order_book_clone.lock().unwrap();
        send_order(&order)
        order_book.push(order);
        println!("{:?}", order_book);
        thread::sleep(Duration::from_secs(interval as u64));
    }
}

fn main() {
    // Create a shared order book
    let order_book = Arc::new(Mutex::new(Vec::<Order>::new()));

    // Clone the order book for use in the threads
    let order_book_clone = Arc::clone(&order_book);

    // Thread to place orders
    let place_order_thread = thread::spawn(move || twap("AAPL".to_string(),5000, 10,1,&order_book_clone));

    // Clone the order book again for use in the threads
    let order_book_clone = Arc::clone(&order_book);

    // Thread to observe and change orders
    let observe_order_thread = thread::spawn(move || {
        loop {
            thread::sleep(Duration::from_secs(interval as u64));
            // Lock the order book and observe the orders
            let mut order_book = order_book_clone.lock().unwrap();
            // Check if there are any orders with status "PENDING"
            let pending_orders = order_book
                .iter_mut()
                .filter(|order| order.status == "PENDING")
                .collect::<Vec<_>>();
            for order in pending_orders {
                order.order_type = '"MKT"
                send_order(&order);
            } 
        }
    });

    // Wait for the threads to finish
    place_order_thread.join().unwrap();
    observe_order_thread.join().unwrap();
}

It is crucial that we promptly identify and rectify any deviation from the upper and lower bounds by implementing necessary changes and minimizing tracking errors. In addition, there are various proprietary implementations of TWAP, some of which integrate other algorithms such as VMAP to improve execution.

Conclusion

TWAP is considered to be a fundamental and indispensable execution algorithm. It is a simple and effective strategy that evenly distributes trading volume over a specific period to minimize market impact. It is important to have a clear understanding of TWAP, as it forms the basis for other more complex execution algorithms used in trading. By understanding TWAP, one can gain insights into other popular execution algorithms such as VWAP, POV, and IS. These algorithms have become essential tools in modern trading, enabling traders to execute large orders with minimal market impact. In conclusion, it is worth investing time and effort to learn the fundamentals of TWAP.

What is a time series database?

A time series database (TSDB) is a database optimized for storing and serving time series data. Time series data is data that is collected over time, such as financial market data, logs, monitoring data, and weather data. TSDBs are designed to make it easy to store, query, and analyze time series data.

Examples of Time series databases

• InfluxDB: InfluxDB is an open-source time series database that is designed for high performance and scalability. It is used by a variety of organizations, including Uber, Twitch, and Yelp.

• Prometheus: Prometheus is an open-source monitoring system and time series database that is used by a variety of organizations, including Google, Facebook, and Netflix.

• TimescaleDB: TimescaleDB is an open-source time series database that is based on PostgreSQL. It provides the performance and scalability of a traditional database with the flexibility and features of a time series database.

• DolphinDB: DolphinDB is a commercial time series database that is designed for high performance and scalability. It is used by a variety of organizations, including Goldman Sachs, Morgan Stanley, and IBM.

• RRDTool: RRDTool is an open-source tool for storing and graphing time series data. It is often used for monitoring systems and network traffic analysis.

• OpenTSDB: OpenTSDB is an open-source time series database that is based on HBase. It is used by a variety of organizations, including Twitter, Facebook, and Yahoo.

TimescaleDB: Postgres for time-series

TimescaleDB is an open-source relational database that extends PostgreSQL with time-series capabilities. TimescaleDB allows developers and organizations to store, query, and analyze time-series data at scale. It also includes features such as automatic data partitioning, advanced indexing for time-series queries, and support for SQL and native time-series functions.

How to install Timescale DB?

The easiest way is to install using Docker. If you are using Windows, you need to install the docker desktop that would get installed in System Drive (C drive). It will create 2 distros docker-desktop and docker-desktop-data. You can see this at %LOCALAPPDATA%/Docker/wsl; and inside you will find vhdx file. If you are running low on System drive space, you can move docker-desktop-data out to another drive.

wsl --shutdown
wsl --export docker-desktop-data D:\docker-desktop\docker-desktop-data.tar
wsl --unregister docker-desktop-data
wsl --import docker-desktop-data D:\docker-desktop\data D:\docker-desktop\docker-desktop-data.tar --version 2

More detail on this can be found on this page.

You can install a TimescaleDB instance from a pre-built container.

• Run the docker image: The TimescaleDB HA Docker image includes Ubuntu as its operating system.

    docker pull timescale/timescaledb-ha:pg14-latest
  

• Run the image from the container

    docker run -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg14-latest
  

•     docker start timescaledb # To start docker container
      docker stop timescaledb # To stop docker container
  

You can use pgAdmin to connect to this container and create tables accordingly.

• Install / Open pgAdmin and add a new server

• Provide the hostname, port, database, username and password to connect to the instance running on the docker container.

  

• You can just use SQL queries like creating a table by executing the SQL query below. You don't need to learn any other structured query language to work with Timescaledb, unlike Influxdb.

  

However, we will of course use Python to connect with Timescaledb using psycopg2 and sqlalchemy.

Connect from Python using Psycopg2

Psycopg2 is the most loved PostgreSQL database adapter for Python, hence worked with TimescaleDB. It was designed for heavily multi-threaded applications that create and destroy lots of cursors and make a large number of concurrent “INSERT”s or “UPDATE”s.

pip install psycopg2-binary

import psycopg2
# Create connection
con = psycopg2.connect(
                host = 'hostname'
                database ='dbname',
                user = 'postgres',
                password = 'password')
# Create cursor
cur = con.cursor()
# SQL
query = 'INSERT INTO stock_intraday (time,symbol,price_open,price_close,
        price_low,price_high,trading_volume VALUES (%s,%s,%s,%s,%s,%s,%s)'
record =(time,symbol,price_open,price_close,
        price_low,price_high,trading_volume)
cur.execute(query,record)
#commit
con.commit()
#close the connection
con.close()

Ingest Pandas Dataframe to TimescaleDB

There are multiple ways to ingest data into our database, however, I want to discuss quickly the simplest and probably the slowest using sqlalchemy. Let's say you have a DataFrame df as below. We can create and insert DataFrame using to_sql:

from sqlalchemy import create_engine
engine = create_engine('postgresql://postgres:pwd@localhost:5432/postgres')
df.to_sql('abc', con=engine, if_exists = 'replace',index =True)

A table gets created based on the data frame with the datatype supported by TimescaleDB.

# reading the table from the db
query = "SELECT * FROM " + table_name
df = pd.read_sql_query(query, engine)

Features of TimescaleDB

At least these feature two worth mentioning here:

Hypertables

Hypertables are PostgreSQL tables that automatically partition your data by time. With hypertables, Timescale makes it easy to improve insert and query performance by partitioning time-series data on its time parameter. Behind the scenes, the database performs the work of setting up and maintaining the hypertable's partitions. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range.

More about hypertables

Continuous aggregates

If you are collecting tick data, e.g. several stock prices per seconds, you might want to aggregate your data into 1 minute, 5 minute or 1-hour instead, with open, close, high, and low candles. Continuous aggregates take raw tick data from the original hypertable, aggregate it, and store the intermediate state in a materialization hypertable. Continuous aggregate views are refreshed automatically in the background as new data is added. Timescale tracks these changes to the dataset and automatically updates the view in the background.

Conclusion

While working with stock data, you can store it in flat files such as CSV, HDFStore object or SQLite. As you start working with thousands of stock prices and with multi-year data (e.g. 5 mins or 15 mins candles) you need to store the time series in the databases. You can transfer several calculations and transformations to the server side where it is most suited. TimescaleDB is the natural choice as it is open source, an extension of PostgreSQL, and supports SQL with native support for time series.

Happy Trading!

Here you will not find a well-written blog on how to AWS, instead raw badly written notes on various services. I invite you to add your notes on more services or suggestions. Here we go:

SERVICES PROVIDED BY AMAZON:

NETWORKING & CONTENT DELIVERY:

Lambda, EC2, VPC, Route53

STORAGE:

S3 (Simple storage service), Glacier, EFS (Elastic File service), Storage Gateway

DATABASES:

RDS (Relational Database Service), DynamoDB, Redshift, Elasticache

MIGRATION:

Snowball, DMS (Database migration services), SMS (Server Migration Service)

ANALYTICS:

Athena, EMR (Elastic Map Reduce), Cloud Search, Elastic Search, Kinesis, Data Pipeline

SECURITY & IDENTITY:

IAM (Identity Access Management), Inspector, Certificate Manager, Directory Service, WAF (Web Application Firewall), Artifacts

MANAGEMENT TOOLS:

Cloud Watch, Cloud Formation, Cloud Trail, Opsworks, Config, Service Catalog, Trusted Advisor

IAM (IDENTITY ACCESS MANAGEMENT)

IAM allows you to manage users and their level of access to the AWS Console.

IAM gives you:

• Centralized control of AWS account.

• Shared access to your AWS account.

• Granular Permission means enabling different levels of access to different users within the organization.

• Identity Federation (including Active Directory, FB, LinkedIn etc)

• Multifactor Authentication- provides additional security for AWS account settings and resources.

• Provide temporary access to users/devices, and services.

IAM consists of the following:

• Users

• Groups: Collection of users under one set of permission.

• Roles: Used to define a set of permissions.

• Policy Documents: Defines one or more permission, policies attached to a user, group or role.

Key Points

• IAM is universal: It doesn’t apply to regions.

• The “root account” is simply the account created when you first set up your AWS account. It has complete Admin access.

• New Users have no permission when first created.

• New Users are assigned Access Key ID & Secret Access Keys when first created.

• These keys are not the same as a password, and you cannot use the Access Key ID & Secret Access Key to log in to the AWS Management Console.

• You can use this to access AWS via the APIs and Command Line Interface(CLI) from your local desktop.

• You only get to view Access Key ID & Secret Access Key once. If you lose them, you have to regenerate them, therefore, save them in a secure location.

• Always set up Multifactor Authentication (MFA) on your root account.

• You can create and customize your password rotation policies.

SERVERLESS COMPUTING

Serverless allows you to run application code in the cloud without worrying about managing any server. AWS handles the infrastructure management task so that you can focus on writing code. Management tasks handled by AWS are capacity provisioning, patching, Auto Scaling and High availability.

Advantages of Serverless:

• Speed to market (without managing infrastructure)

• Super scalable.

• Lower cost.

• Focus on code only.

LAMBDA

Lambda is a serverless computing service, which allows you to run your code without any AWS service. Supported Languages are Node.js, Java, Python, C#, Go, and Ruby.

Lambda Pricing: Pricing is based on the number of requests, their duration and the amount of memory used by the Lambda function

1. Number of requests: First 1 million requests are free. $0.02 per 1 million requests thereafter.

2. Duration**:** Charged in 1ms increment. This is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 100ms.

3. Price per GB-second: The price depends on the amount of memory you allocate to your function. You are charged $ 0.00001667 for every GB-second used.

Lambda is an Event-Driven and serverless application using an Event-Driven Architecture.

The lambda function can be automatically triggered by other AWS services or directly from any web or mobile app. These events could be changes made to data in S3 or DynamoDB table.

AWS services that can invoke Lambda functions are DynamoDB, Kinesis, SQS, Application Load Balancer, API Gateway, CloudFront, S3, SNS, SES, CloudFormation, CloudWatch, CLoudCommit, CloudPipeline etc.

Key Points:

• Cost-Effective

• Continuous Scaling

• Lambda function is independent: Each event will trigger a single function.

• Event-Driven

• Serverless Technology.

Version Control with Lambda:

Versioning: When versioning is used in AWS Lambda, you can publish one or more versions of your lambda function. As a result, you can work with different variations of your Lambda function in your development workflows, such as development, beta and production.

Each Lambda function version has a unique Amazon Resource Name (ARN). After you publish a version, it is immutable (that is it can’t be changed).

AWS Lambda maintains your function code in the $LATEST version. When you update your function code, AWS Lambda replaces the code in the $LATEST version of the Lambda function.

Qualified/Unqualified ARNs

• You can refer to this function using its ARN. There are two ARNs associated with this initial version:

• Qualified ARN: The function ARN with the version suffix.

arn:aws:lambda:aws-region:acct-id:function:helloworld:$LATEST

• Unqualified ARN: The function ARN without the version suffix.

arn:aws:lambda:aws-region:acct-id:function:helloworld

Key Points:

• Can have multiple versions of lambda functions.

• The latest version will use $latest.

• The qualified version will use $latest, unqualified will not have it.

• Versions are immutable.

• Can split traffic using aliases to different versions.

• Cannot split traffic with $latest, instead create an alias to latest.

Lambda Concurrent execution Limit:

• The default concurrent execution limit is 1,000 per second.

• Sometimes on serverless websites, websites hit the limit at some point.

• If sites hit the limit then invocations are rejected-429 HTTP status code, which means there are too many requests.

• The remedy is to get the limit raised by AWS support.

EC2 (ELASTIC COMPUTE CLOUD)

EC2 is secure, resizable compute capacity in the cloud.

• It’s like a virtual machine, only hosted in AWS instead of your own data center.

• Designed to make web-scale cloud computing easier for developers.

ADVANTAGES:

• Pay only for what you use.

• No wasted capacity: Select the capacity you need and you can grow and shrink the capacity as per the requirements.

Pricing Options:

• On-Demand: By default pricing option. Payment is done by the hour or the second depending on the type of instance you run. Low-cost pricing option.

• Reserved: Reserved capacity for one or three years. These instances operate at a regional level.

• Spot: Enables you to bid whatever price you want for instance capacity, providing even greater savings if your applications have flexible start and end times. Purchased unused capacity at a discount of up to 90%.

• Dedicated: Most expensive option. Used for security purposes**,** by using a space that is not shared by anyone.

Uses of Instances:

On-Demand: Applications with short-term, spiky, or unpredictable workloads that cannot be interrupted. Applications being developed or tested on Amazon EC2 for the first time

Reserved: Predictable usage or steady state, Specific capacity requirements, Pay upfront.

Spot Instance: Applications that have flexible start and end times, that are only feasible at very low compute prices and suitable for the urgent need for large amounts of additional computing capacity.

Dedicated hosts: Compliance requirements and regulatory requirements that may not support multi-tenant virtualization. Licensing which does not support multi-tenancy or cloud deployments.

EC2 INSTANCE TYPES:

Instance type determines the hardware configuration and capabilities of the host computers when an instance or virtual machine is running. Instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources for your applications. Each instance type includes one or more instance sizes, allowing you to scale your resources to the requirements of your target workload.

Select an instance type based on the requirement of your application.

For more detail: https://aws.amazon.com/ec2/instance-types/

ROUTE 53

Route 53 is Amazon’s DNS (Domain Name System) service. Allow you to map your domain names to:

• EC2 instances

• Load Balancers

• S3 Buckets

ELASTIC LOAD BALANCERS

Elastic Load Balancing automatically distributes incoming application traffic across multiple servers. Capacity easily increased when needed.

Types of Load Balancers:

Application Load Balancers:

Operates at the OSI layer 7, and makes routing and routing decisions based on the information. Best suited for load balancing of HTTP and HTTPS traffic. They are intelligent, and you can create advanced request routing, sending specified requests to specific web servers.

Network Load balancers:

Best suited for load balancing of TCP traffic where extreme performance is required. Operating at the connection level (Layer 4), basically, layer 4 wants super fast performance & super fast speed. Network Load Balancers are capable of handling millions of requests per second while maintaining ultra-low latencies. It is AWS’s most expensive load balancer but is used in production especially when the latency is an issue.

CLASSIC LOAD BALANCERS:

They are the legacy Elastic Load Balancers. Load balancing of HTTP/HTTPS applications and use Layer 7-specific features, such as X-Forwarded and sticky sessions. You can also use strict Layer 4 load balancing for applications that rely purely on the TCP Protocol. Classic Load Balancer is intended for applications that were built within the EC2-Classic network.

X-Forwarded-For Header: Identify the originating IPv4 address of a client connecting through a load balancer

CLOUD FRONT

Cloud front is Amazon’s CDN (Content delivery network).

CDN is a system of distributed servers (network) that deliver webpages and other web content to a user based on the geographic locations of the user, the origin of the webpage, and a content delivery server. CDN is easy and cost-effective way to distribute content with low latency and high data transfer speed.

Explain by an example, Imagine a website running from London, and has users at geographically dispersed all around the world. So, instead of each user accessing the files directly between the users, we introduce this concept called Edge locations. An edge location is simply a collection of servers that are in geographically dispersed data centers. And these are used by CloudFront to keep a cache of copies of your object, and this means that instead of requesting content from a server in London, users can access that content from the edge location. So, once the request is made, the edge location forwards the request to the server in London, so it downloads the files and it caches them locally. This provides a much faster response time. It means that your requests are only going to that local edge location, they’re not going all the way to the main server.

• Edge Location: Location where content is cached and can also be written. Separate to an AWS Region/Availability Zone.

• Origin: This is the origin of all the files that the CDN will distribute. Origins can be an S3 Bucket, an EC2 Instance, an Elastic Load Balancer, or Route53.

• Distribution: This is the name given the origin and configuration settings for the content to distribute using CDN.

  • Web Distribution: Typically used for websites. HTTP/HTTPS

  • RTMP (Real-Time Messaging Protocol): Used for Media Streaming.

Amazon CloudFront can be used to deliver your entire website, including dynamic, static, streaming and interactive content using a global network of edge locations. Requests for your content are automatically routed to the nearest edge location, so content is delivered with the best possible performance.

CloudFront is optimized to work with other Amazon Web Services, like Amazon S3, EC2, Elastic Load Balancer, and Route 53. Amazon CloudFront also works seamlessly with any non-AWS origin server, which stores the original, definitive versions of files.

Objects are cached for a period which is a TTL(Time to Live) i.e; 24 Hrs. Cached objects can be cleared, but you will be charged.

S3 Transfer Acceleration: Enables fast, easy and secure transfers of files over long distances between your end users and an S3 bucket. Transfer Acceleration takes advantage of Amazon CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path.

AWS: DATABASES

RDS (RELATIONAL DATABASE SERVICE)

RDS is used for OLTP (Online Transaction Processing) workload.

OLTP processes data from transactions in real-time like customer orders, banking transactions, payments and booking systems.

Amazon RDS is available on several database instance types – optimized for memory, performance or I/O – and provides you with six familiar database engines to choose from, including Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server.

BENEFITS:

• Easy to set up and operate: No need for infrastructure provisioning, and no need for installing and maintaining database software.

• Highly scalable: Amazon RDS engine types allow you to launch one or more Read Replica to offload read traffic from your primary database instance.

• Automatically backup and allows taking manual backup of the database as a snapshot.

FEATURES OF RDS

MULTI AZ:

Multi-AZ is an exact copy of the production database in another Availability Zone.

AWS handles the replication for you, so when your production database is written to the primary database in a particular AZ, then this write will automatically be synchronized to the standby database which is located in another AZ.

In the event of planned database maintenance, DB Instance failure, or an Availability Zone failure, Amazon RDS will automatically failover to the standby so that database operations can resume quickly without administrative intervention.

RDS type that can be configured as Multi-AZ: PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server.

Multi-AZ is for Disaster Recovery only. It is not primarily used for improving performance. For performance improvement, you need a Read replica.

READ REPLICA:

Read replicas is a read-only copies of your primary database. This is achieved by using Asynchronous replication from the primary RDS instance to the read replica. You use read replicas primarily for a very read-heavy database workload and take the load off the primary database.

Read Replica can be located in the same AZ as the primary database. It can also be in cross AZ i.e; a completely different AZ, or it can even be in cross region, located in a completely different region.

Key Points:

• Scaling Read Performance: Read replicas are used for scaling, not for disaster.

• Require Automatic Backup: Must have automatic backups turned on in order to deploy a read replica.

• Multiple Read Replica: MySQL, PostgreSQL, MariaDB, SQL and Oracle allow you to add up to 5 read replica copies of any database.

RDS Backups

1. Automated Backups:

These are enabled by default. Allows you to recover your database to any point in time within a “retention period”. The retention period can be between 1 -35 days. Automated Backups will take a full daily snapshot and will also store transaction logs throughout the day. When you do a recovery, AWS will first choose the most recent daily backup, and then apply transaction logs relevant to that day. This allows you to do a point-in-time recovery down to a second, within the retention period.

Automated Backup is stored in S3 and you get free storage space equal to the size of your database.

2. Database Snapshots:

DB Snapshots are done manually (i.e; user-initiated). No retention period, manual snapshots are not deleted even after you delete the original RDS instance, including any automated backup. Provide a snapshot of the storage volume attached to the DB instance.

ELASTICACHE

Elasticache is an in-memory cache. This web service makes it easy to deploy, operate, and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from a fast in-memory cache, instead of relying entirely on a slower disk-based database.

Amazon ElastiCache can be used to significantly improve latency and throughput for many read-heavy application workloads (such as social networking, gaming, media sharing and Q&A portals) or compute-intensive workloads ( such as a recommendation engine). Caching improves application performance by storing critical pieces of data in memory for low-latency access.

Types of Elasticache: