# Fly.io and FLAME

We’re Fly.io and we transmute containers into VMs, running them on our hardware around the world. We have fast booting VMs, so why not [take advantage of them](/content/docs/speedrun/index.html)?

I closed out the year publishing a new library and programming model, called [FLAME](/content/blog/rethinking-serverless-with-flame/index.html). If you missed it, FLAME replaces the need for AWS Lambda or Cloudflare Workers and the proprietary services that call proprietary services fractal of architecture patterns that come with them.

Instead, imagine if you could auto scale by wrapping any existing app code in a function and have that block of code run in a temporary copy of your app. That’s what FLAME is all about.

Since the release, I’ve built a few things using this pattern, and the most remarkable thing is just how unremarkable the solutions are. I can write my naive Elixir application on my laptop, deploy it to Fly.io running across the planet in a single command, then scale it out elastically by changing almost no code.

## Measuring Page Speed

I’ve wanted to build a “World Page Speed Test” for a long time. Think Google’s Page Speed Insights, but as viewed from various observers around the globe. To do this correctly, you need to be running a full browser. And those browsers need to be running on servers around the planet. The client must download all scripts, styles, and images to reflect a real-world page load. Doing this at scale is usually hard. Doing this with Elixir, FLAME, and Fly.io is an afternoon of tinkering.

### 🔥🔥 [Try World Page Speed Now!](https://worldpagespeed.fly.dev/) 🔥🔥

So we need a browser. Fortunately, the major browsers support headless drivers through a W3C standard. This allows starting a headless browser and communicating with it via local HTTP. You can drive page navigation, evaluate page JavaScript, and simulate user interaction.

Our Elixir app can start with the basics. We’ll need:

- A process group of nodes and where they are located geographically
- A LiveView page that accepts a URL
- A running `chromedriver` process that can launch headless chrome sessions for us
- The ability to tell chrome to visit a URL and monitor network performance events
- When the page starts loading, we display a loading status
- When the page finishes loading, we display the loading time

This doesn’t take much code. I’ll crib the highlights.

First, we need a way to locate all nodes on the cluster, and the Fly region they belong to. This lets us know where the chrome requests are happening from. The built-in `Node.list()` returns all reachable nodes, but we need metadata alongside their name showing where they are located.

Fortunately, we can use [Phoenix.Tracker](https://hexdocs.pm/phoenix_pubsub/Phoenix.Tracker.html) for this, which provides a process group with metadata:

```elixir

defmodule WPS.Members do
  use GenServer

@tracker WPS.Tracker

def list(group_name \ __MODULE__) do
    Phoenix.Tracker.list(@tracker, group_name)
  end

def start_link(opts) do
    GenServer.start_link(__MODULE__, opts)
  end

def init(opts) do
    my_region = System.get_env("FLY_REGION") || "ord"
    group_name = Keyword.get(opts, :name, __MODULE__)

{:ok, ref} =
      Phoenix.Tracker.track(@tracker, self(), group_name, my_region, %{
        node: Node.self(),
        machine_id: System.get_env("FLY_MACHINE_ID")
      })

{:ok, %{group_name: group_name, ref: ref}}
  end
end
```

Now we need a bit of HTML inside a LiveView which drives our headless chrome when the user hits “go”:

```elixir

def render(assigns) do
  ~H"""
  <h1>World Page Speed</h1>
  <form
    :if={!@ref}
    id="url-form"
    phx-change="validate"
    phx-submit="go"
  >...
  """
end

def handle_event("go", %{"url" => url}, socket) do
  validated_url = URI.to_string(uri)
  ref = make_ref()
  parent = self()

node_times =
    for {region, meta} <- WPS.Members.list(), %{node: node} = meta, into: %{} do
      {node, Browser.Timing.build(validated_url, region)}
    end

{:noreply,
  socket
  |> clear_flash()
  |> assign(ref: ref, uri: uri, form: to_form(%{"url" => validated_url}))
  |> stream(:timings, Map.values(node_times), reset: true)
  |> start_async(:timing, fn ->
    nodes = Enum.map(node_times, fn {node, _} -> node end)
    :erpc.multicall(nodes, fn -> timed_nav(node_times[node()], parent, ref) end)
  end)}
end
```

When the user hits the “go” button on the web page, our goal is to kick off some timed headless chrome navigations.

We’ll simulate a multi-node setup locally for now, with just a single member. We aren’t going to worry about going planet-wide multi-node just yet – though it won’t take any code changes to get there.

After validating the user’s URL, we build up a `%Browser.Timing{}` struct for each member’s region in the cluster. Next, we asynchronously navigate to the page inside a `start_async` call.

Within the `start_async`, we can see some built-in Erlang standard library treasures. `:erpc.multicall` accepts a list of nodes and a function to run. The Erlang VM will run the function on the passed nodes and block until it gets a result from all nodes.

Our local node setup is the only node running in dev, so the function will just run locally. The `timed_nav/3` function looks like this:

```elixir

def timed_nav(%Browser.Timing{} = timing, parent, ref) do
  timing = Browser.Timing.loading(timing)
  send(parent, {ref, {:loading, timing}})

case Browser.time_navigation(timing, @browser_timeout) do
    {:ok, %Browser.Timing{} = timing} ->
      send(parent, {ref, {:complete, timing}})

{:error, {reason, %Browser.Timing{} = timing}} ->
      send(parent, {ref, {:error, {reason, timing}}})

{:error, reason} ->
      send(parent, {ref, {:error, {reason, Browser.Timing.error(timing)}}})
  end
end
```

We first tell it we’re about to start loading. Next, we call into a `Browser.time_navigation` function, which asks headless chrome to navigate to the webpage and give us the timing details.

If the navigation is successful, we send a `{:complete, timing}` message to the LiveView. If it fails, we send an error message. That’s it!

We now have a UI that looks like this:

We’re not really multi-node yet, but the moment we have a cluster and provide a list of real nodes, our `:erpc.multicall` will Just Work across the cluster and message our remote `parent` LiveView thanks to the distributed, location-transparent nature of Erlang’s process messaging.

For the LiveView to handle the timing messages, we only need to implement a few functions:

```elixir

def handle_info({_ref, {:loading, %Browser.Timing{} = timing}}, socket) do
  {:noreply, stream_insert(socket, :timings, timing)}
end

def handle_info({_ref, {:complete, %Browser.Timing{} = timing}}, socket) do
  {:noreply, stream_insert(socket, :timings, timing)}
end

def handle_info({_ref, {:error, {_, %Browser.Timing{} = timing}}}, socket) do
  {:noreply, stream_insert(socket, :timings, timing)}
end
```

## Going Multi-node

So what about going multi-node? Is it really that easy with the Erlang VM? Let’s take our dev prototype and deploy it on Fly.io:

```bash
fly launch
```

Once our app is deployed, we can scale out to any number of regions. For good geographic coverage, let’s spread things out across the planet:

```bash
fly scale count 8 --max-per-region 1 --region bom,fra,gru,hkg,nrt,ord,scl,syd
```

We use `fly scale count` to scale our app to 8 regions across the planet, ensuring we only have a single instance per region.

We’re in business. Let’s try it out at [https://worldpagespeed.fly.dev/](https://worldpagespeed.fly.dev/).

It works! Our nodes clustered automatically and now when our LiveView hits the `start_async` call, `:erlang.multicall` will call each node to visit the provided URL.

## Elastic scaling with FLAME

I often tell people that FLAME is so remarkable in how _unremarkable_ it is when you use it in practice. What is effectively an amazing elastic scale primitive is turned into a boring decision.

“Do I want elastic scale here?”

If the answer is yes, you wrap your code in a `FLAME.call` and you carry on with life.

Let’s make this thing scale:

```elixir

def timed_nav(%Browser.Timing{} = timing, parent, ref) do
  + FLAME.call(WPS.BrowserRunner, fn ->
    timing = Browser.Timing.loading(timing)
    send(parent, {ref, {:loading, timing}})

case Browser.time_navigation(timing, @browser_timeout) do
      {:ok, %Browser.Timing{} = timing} ->
        send(parent, {ref, {:complete, timing}})

{:error, {reason, %Browser.Timing{} = timing}} ->
        send(parent, {ref, {:error, {reason, timing}}})

{:error, reason} ->
        send(parent, {ref, {:error, {reason, Browser.Timing.error(timing)}}})
    end
  end)
end
```

And we’re done.

The fact that we sent our `parent` LiveView pid into a multicall, which sent it across the cluster to another node, who took the `parent` and sent it again to yet another node isn’t an issue. The message will make it back to the parent by design – it’s what the Erlang VM does.

## Hot vs cold starts

To avoid cold starts, we could configure our FLAME pool to always keep a single runner alive. Or we could configure our pool to warm up a number of runners at app start so our own cold deploys don’t cause users to hit a cold FLAME pool.

```elixir
{FLAME.Pool,
 name: WPS.BrowserRunner,
 min: 1,
 max: 50,
 max_concurrency: 20,
 min_idle_shutdown_after: :timer.seconds(30),
 idle_shutdown_after: :timer.seconds(30)}
```

Here we start the app with a single warmed up `BrowserRunner`, and up to a maximum of 50 runners that each can service 20 concurrent web page visits. We also configure a 30s idle shutdown for the general pool operation, as well as the warmed up initial runner. The pool will grow and shrink elastically as load comes and goes.

## Solving the problem vs removing the problem

Just imagine what this would have taken on your FaaS of choice.

To achieve elastic scale, you would need to stand up separate deployment pipelines of various lambdas, configure SNS or SQS queues, and write the glue back in your application.

You would also need to configure various availability zones to run your functions at the edge.

Or you could use FLAME and remove these problems entirely. With Elixir and FLAME you write the code and carry on with life. If you need to scale, you put it into a FLAME and start shipping your next features.

Happy hacking!

–Chris

## Fly.io ❤️ Elixir

Fly.io is a great way to run your Phoenix LiveView apps. It’s really easy to get started. You can be running in minutes.