Author

Name
Jason Stiebs
@peregrine @peregrine

Image by Annie Ruygt

We’re Fly.io. We run apps for our users on hardware we host around the world. Fly.io happens to be a great place to run Phoenix applications. Check out how to get started!

I love ExDocs, I think they are one of the best parts about the Elixir Ecosystem and I frankly cannot shut up about it. My one complaint is that it can sometimes be hard to find exactly what you are looking for using ExDoc’s built-in search.

My favorite example is that I know that Ecto.Repo has a function to dump the generated SQL to a string, but if you search inside of Ecto, you get nothing. That’s because it’s in a different project called ecto_sql. What if we had one spot we could go to and search all of HexDocs?

In this post we will walk through how I downloaded, cleaned up, indexed and searched all of HexDocs using SQLite FTS5 and LiveView! It turned into a project that matches the real world incredibly well, warts and all! So I won’t be walking through every single step, but check it out on Github and try it out hosted here, let’s begin!

Downloadin’ Hex Docs

When you visit an ExDoc package and try using the search box, what you are using is a fully in-browser search experience provided by Lunrjs. When the ExDoc generated page is first visited, it downloads a JavaScript payload that contains the “Search Data”, which is then indexed in the browser via Lunrjs, compressed and stored in the browser SessionStorage. Then there is some more JavaScript that does the autocomplete and rendering of the search fully in the browser.

We’re going to take advantage of that Search Data that comes with every hex package, and it looks like this:

ExDoc generates this for every package and uploads it to HexDocs for you. And because ExDoc has a special place in the world of Elixir, every Elixir project depends on it, including Elixir. So it can’t really depend on any project that’s not Elixir or fully vendored, like Lunr.js.

Which means that when ExDoc creates the above searchData.js file it has to encode the JSON directly. The good news is the “items” all have the same document shape; the, meh news is this filename is different for every deploy.

Garden Variety Extract, Transform and Load Pipeline

This project started as a fun project that scratches an itch for me, but building out an ETL pipeline is as “Real World” as programming gets. Much like the “Real World” this part of the project kept expanding in scope as I went along.

So we don’t get lost in the weeds here is a little road map to keep us on track.

• E. Use the Hex API get a listing of all the packages.
• E. Scrape the Search Page of every package that has documentation looking for a script tag that contains search_items and download that file.
• T. We’re going to strip off all JavaScript in that file till we just have JSON, and then clean it up.
• T. We’re going to parse that JSON using Jason; if that fails, use a hand-rolled JSON parser that works with the cases that don’t work with Jason.
• L. Store this into a SQLITE Table.

E: Get the data

Enum.reduce_while(1..500, [], fn page, acc ->  
  data = API.list!(%{page: page, sort: "updated_at"})

current = Enum.count(data)  
  if current < 100 do  
    {:halt, [data | acc]}  
  else  
    {:cont, [data | acc]}  
  end  
end)  
|> Kernel.++(special_packages())  
|> List.flatten()  

We’re hitting the Hex API page by page and downloading all the data till we get a page less than size 100. I’m also adding in some “special packages” which are just Elixir, Eex, ExUnit, IEX, Logger, and Mix that are hosted on hex but not served via the API. Standard stuff here!

You’ll notice I’m just letting errors happen instead of handling them; this is a “for fun” project but its also practical as it stops this process as soon as something fails so I don’t need to re-run it a ton of times.

E: Scrape

def scrape_json(pkg)  
  pkg["docs_html_url"] <> "search.html"  
  |> grab_html(search_url)  
  |> grab_json(["search_items", "search_data"], url)  
end

def grab_html(url) do  
  {:ok, document} = Req.get!(url).body  
                    |> Floki.parse_document()  
  document  
end

def grab_json(html, fnames, url) do  
  html  
  |> Floki.attribute("script", "src")  
  |> Enum.find(fn str ->  
      Enum.any?(fnames, fn fname ->  
      String.contains?(str, fname)  
    end)  
  end)  
  |> fetch_json(url)  
end

defp fetch_json(nil, _), do: nil  
defp fetch_json(path, url) do  
  Req.get!(URI.encode(url <> path)).body  
end

We grab the search.html, parse the HTML using Floki, look for script tags that contain the file_name and then download that JavaScript File. Pretty straightforward so far!

T: JSON

While reading this next section, it is important to note that given the constraints, I think the Elixir Team did an absolutely incredible job with ExDoc. Engineering is about working within constraints and they killed it. You would be hard-pressed to find any language that gives you machine-readable docs at all for any reason.

Here is where things get a little in the weeds; remember when I said different versions of ExDoc can produce different versions of that JavaScript? Well, it also hasn’t always produced “standards”-compliant JSON. Thanks to the incredible engineering of Browsers and JavaScript engines, that doesn’t matter. In our case Jason expects to work with standards-compliant JSON… Suffice to say ExDoc has been patched and a bunch of the following wouldn’t be necessary with latest ExDoc generated docs.

First we gotta clean up all the JavaScript from around the JSON that we’re after:

def cleanup(<<"<", _::binary>>), do: "" # Html or something  
def cleanup(nil), do: ""  
def cleanup(str) do  
  str = str  
  |> String.trim()  
  |> String.trim_leading("searchNodes=")  
  |> String.trim_leading("searchData = ")  
  |> String.trim_leading("searchData=")  
  |> String.trim_leading("searchNodes = ")  
  |> String.trim_leading("sidebarNodes=")  
  |> String.trim_leading("sidebarNodes = ")  
  |> String.trim_leading("var versionNodes = ")  
  |> String.trim_leading("var versionNodes=")  
  |> String.trim_leading("versionNodes=")  
  |> String.trim_leading("versionNodes = ")  
  |> String.trim_trailing(";\nfillSidebarWithNodes(sidebarNodes);")  
  |> String.trim_trailing(";")  
  |> String.trim()  
  |> cleanup2()  
end

This is pretty self-explanatory; over a decade JS has had many forms and come with many functions. This next part is a little strange…

def cleanup2(str) do  
  case Regex.named_captures(~r/<<(?<binary>[\\d\,\s]*)...>>/, str) do  
    nil -> str  
    %{"binary" => bin} ->  
      ha = bin  
      |> String.trim()  
      |> String.trim_trailing(",")  
      |> String.split(", ")  
      |> Enum.map(fn str -> <<String.to_integer(str)>> end)  
      |> Enum.join("")

Regex.replace(~r/<<([\d\,\s]*)...>>/, str, "\"<>ha<>\