rrrene* About

Designing Token APIs for Architecting Flow in Elixir

In previous articles, we discussed the Token approach and examined when to use a Plug-like Token and when to look at other options.

A Token is basically a struct which contains all information relevant to a certain use-case. If we were to create a time tracking app, we might have a Token for the time a user has entered:

defmodule MyTimeTracker.ManualEntry do
  defstruct session_id: nil,
            time: nil,
            billable: nil,
            # probably lots of other fields ...
end

Creating a Token API means creating functions to interact with our Token. In case of our imaginary time tracking app, this could lead to something like this:

defmodule MyTimeTracker.ManualEntry do
  defstruct session_id: nil,
            time: nil,
            billable: nil,
            # probably lots of other fields ...

  @doc "Creates a time entry for the given `user`."
  def build(user)

  @doc "Reads the entered time for the given `entry`."
  def time(entry)

  @doc "Tries to parse the given `time_string` and add it to the given `entry`."
  def time(entry, time_string)

  @doc "Return `true` if the given `entry` is billable."
  def billable?(entry)
end

Let’s see why this is useful.

The need for Token APIs

The usefulness might not be apparent at first. You might ask yourself:

Why would we need APIs for dealing with a Token? Can’t we just modify the damn thing?

Let’s take a look at why it is preferable to have a standardized API:

By using a Token API to access information, we are also independent in all aspect of our data management, i.e. how we store, look up and index information inside our Token (which can, again, be important when dealing with larger amounts of data).

Other aspects you can deal with through a Token API are access management, caching, implementing domain-specific time-to-live concepts and basically anything you can think of.

Let’s look at a practical example.

Example: An Online Book Store

Let’s say we are building an online store for Elixir programming books. In this store, customers can register, login, curate a wishlist, fill and checkout a shopping cart.

We will use the shopping cart as our example for a Token: the cart is being used during the shopping process, which is the primary use-case of our online store.

Creating a Shopping Cart Token

So what do we “need” from our ShoppingCart Token?

Here’s a good starting point to model this Token:

defmodule MyStore.ShoppingCart do
  defstruct session_id: nil,
            created_at: nil,
            items: nil

  def build(session_id) do
    # we're using the `__MODULE__` special form here, so we don't have to refer
    # to `MyStore.ShoppingCart` every time
    %__MODULE__{
      session_id: session_id,
      items: [],
      created_at: DateTime.utc_now
    }
  end
end

Creating a Shopping Cart API

Most of the time, I try to structure my Token so that I can read most, if not all, values directly from the Token. For writing values, I regularly defer that task to a function.

This pattern can also be found in Plug.Conn, where you can read assigns directly and use assign/3 to write new assigns to the struct:

iex> conn.assigns[:foo]
nil
iex> conn = assign(conn, :foo, :bar)
iex> conn.assigns[:foo]
:bar

This is much more readable than:

iex> conn = %Plug.Conn{conn | assigns: Map.put(conn.assigns, :foo, :bar)}

We can adapt this pattern to our shopping cart.

A put_item/2 function could help us to validate and/or cast items before putting them in the cart:

iex> cart = MyStore.ShoppingCart.build(user)
iex> cart.items
[]
iex> cart = put_item(cart, 43212332)
iex> cart.items
[%Item{...}]
iex> cart = put_item(cart, 87632785)
iex> cart.items
[%Item{...}, %Item{...}]

This is not only much nicer to read than

iex> cart = %MyStore.ShoppingCart{cart | items: [cart.items | item]}

but we can also have put_item/2 look up the item with the given id or see if the user doing the shopping is allowed to purchase the given item (e.g. due to his country’s age restrictions, although there are probably no such age restrictions in our imaginary Elixir programming book store 😉).

Lastly, we can ensure that, if given the id of an item instead of the Item itself, we look it up in order to have a consistent list of Item structs in our ShoppingCart.

With the help of put_item/2, we can validate and normalize inputs more easily.

Combining Tokens and APIs

At some point, our users want to checkout their ShoppingCart. In that final step, we will have to let them choose a method of payment and provide further details.

As you might have guessed, we model the PaymentInformation as another Token:

defmodule MyStore.PaymentInformation do
  defstruct session_id: nil,
            created_at: nil,
            provider: nil,
            card_number: nil,
            # lots of other payment info ...

  def build(session_id) do
    %__MODULE__{
      session_id: session_id,
      created_at: DateTime.utc_now
    }
  end

  # lots of functions for setting payment providers, card numbers, etc.
end

We will now have to bring those two distinct pieces of data together: what a user wants to buy and how they plan to pay for it. In other words: We need an API which takes both a ShoppingCart and a PaymentInformation Token.

defmodule MyStore.Checkout do
  def perform_checkout(shopping_cart, payment_information) do
    # suppose `CheckoutService.call/2` is our checkout API, which is
    # implemented and maintained by another team of engineers
    result = CheckoutService.call(shopping_cart, payment_information)

    case result do
      # `:ok` and `true` both mean that the checkout has gone through
      success when success in [:ok, true] ->
        # TODO: talk to checkout team why we need to check for two values
        :ok

      # one of the engineers from the checkout team mentioned that in case of
      # an error the result is a Map, containing the reason for the failure
      %{} = map ->
        {:error, map["reason"]}

      # if we get neither `:ok`, `true` nor a Map, let's raise to account
      # for the unexpected input
      value ->
        raise "Unexpected return from CheckoutService. " <>
                "Expected :ok or Map, got: #{inspect(value)}"
    end
  end
end

This example demonstrates how we can wrap an inconsistent external API and build a clean interface around it.

As in the example above, our own API might just return :ok or {:error, reason}. It does not necessarily have to have its own Token, although it very easily could:

defmodule MyStore.Checkout do
  defstruct shopping_cart: nil,
            payment_information: nil,
            success?: nil,
            errors: nil

  def perform_checkout(shopping_cart, payment_information) do
    # `build/2` will ensure that our inputs are valid and create a Token
    checkout = build(shopping_cart, payment_information)

    # we are still calling the same checkout API, this time using
    # the validated inputs from the Token
    result =
      CheckoutService.call(checkout.shopping_cart, checkout.payment_information)

    case result do
      success when success in [:ok, true] ->
        # we're using internal Token APIs to modify our Token ...
        add_success(checkout)

      %{} = map ->
        # ... this way, we have a central place that defines what
        # "success" or "failure" means ...
        add_failure(checkout, map["reason"])

      value ->
        # ... and we still raise in case of inputs we can't handle!
        raise "Unexpected return from CheckoutService. " <>
                "Expected :ok or Map, got: #{inspect(value)}"
    end
  end

  # NOTE: this time, our `build` function is private, because we do not
  #       initialize a `Checkout` Token from outside this module
  defp build(shopping_cart, payment_information) do
    # we validate our input data here to ensure that, once we have a token,
    # we can rely on the Token's contents being valid
    valid? = shopping_cart.session_id == payment_information.session_id

    if valid? do
      %__MODULE__{
        session_id: shopping_cart.session_id,
        shopping_cart: shopping_cart,
        payment_information: payment_information
      }
    else
      raise "session_id not matching!"
    end
  end

  # these functions define what "success" ...
  defp add_success(checkout) do
    %__MODULE__{checkout | success?: true, errors: []}
  end

  # ... and "failure" mean.
  defp add_failure(checkout, errors) do
    %__MODULE__{checkout | success?: false, errors: List.wrap(errors)}
  end
end

Wrapping the checkout process in a Token like this has several advantages:

One of my favourite quotes from the Elixir documentation states:

Remember that explicit is better than implicit. Clear code is better than concise code.

This is especially important since APIs have to be stable and provide a clear boundary people can rely on.

In conclusion

Tokens are a great way to provide a common interface to a shared domain.

And while APIs have to be stable, APIs are also constantly evolving, especially during the early stages of development.

Token APIs have to accomodate both: provide a well defined interface and a stable “language” to talk about shopping carts, payment information and checkouts.

Once we start implementing Token APIs that combine multiple Tokens, we have to tread lightly, since they have to rely on the interfaces of the Tokens they combine and provide a stable interface of their own.

This is why we need guiding principles for developing Token APIs.

To paraphrase one of my favourite guiding principles:

Be liberal in what you accept as input and be conservate in what you send as output.

The example above shows how to do just that by wrapping an inconsistent external interface into a cohesive and explicit interface using an internal Token.

Your turn: Liked this post? Retweet this post! 👍