There are however a few foot-guns when approaching forms like that. Some due
to how html forms work, but also some due to the historical use of Ecto.Changeset
to power forms. Changesets are great, but especially combined with LiveView they
can feel limiting.
So let’s consider the following form for creating a groceries list to send to someone via email – sending part non functional. This involves a root level input for the email address and zero or more rows of multiple inputs for defining the list.
To start from the beginning – we’ll need a schema to power our form. There
are schemaless changesets, but the Phoenix.HTML.Form
implementation for
changesets doesn’t support nested forms using schemaless changesets. Given
the example shown here is in memory only we’ll be using an embedded schema,
but a database backed schema works just as well.
Phoenix 1.7 added support for plain maps powering forms, which seems like a viable alternative as well. That option won’t be discussed here as part of updating this blog post though.
defmodule GroceriesList do
use Ecto.Schema
embedded_schema do
field(:email, :string)
embeds_many :lines, Line, on_replace: :delete do
field(:item, :string)
field(:amount, :integer)
end
end
end
Using embeds also allows us to inline the Line
embed, which is described
in more detail in the documentation.
To apply changes to the defined schema and validate the input there’s also the
need for changeset/2
type functions. These should be mostly straight forward to
anyone having worked with ecto
before, so this blog post won’t go into detail
what these functions specifically do. There’s again more to read in the documentation.
defmodule GroceriesList do
[…]
import Ecto.Changeset
def changeset(form, params) do
form
|> cast(params, [:email])
|> validate_required([:email])
|> validate_format(:email, ~r/@/)
|> cast_embed(:lines, with: &line_changeset/2)
end
def line_changeset(city, params) do
city
|> cast(params, [:item, :amount])
|> validate_required([:item, :amount])
end
end
Getting to the meat of the topic – the LiveView itself. Starting from a mostly bare bones implementation.
defmodule GroceriesWeb.ListLive do
use GroceriesWeb, :live_view
@impl true
def render(assigns) do
[…]
end
@impl true
def mount(_, _, socket) do
base = %GroceriesList{
id: "4e4d0944-60b3-4a09-a075-008a94ce9b9e",
email: "friend@example.com",
lines: [
%GroceriesList.Line{
id: "26d59961-3b19-4602-b40c-77a0703cedb5",
item: "Melon",
amount: 1
},
%GroceriesList.Line{
id: "330a8f72-3fb1-4352-acf2-d871803cd152",
item: "Grapes",
amount: 3
}
]}
{:ok, init(socket, base)}
end
end
Usually base
would be fetched from the database. This example again runs
completely with data in memory, so there’s just some hardcoded initial data.
init/2
is a small helper, which generates the initial changeset behind the form,
but does also handle a few things needed just for this being in-memory.
Changing the id also means LV will reset its client side state around
the form properly on successful saves.
defp init(socket, base) do
base = autogenerate_missing_ids(base) # Mimic DB setting IDs
changeset = GroceriesList.changeset(base, %{})
assign(socket,
base: base,
form: to_form(changeset),
id: "form-#{System.unique_integer()}" # Reset form for LV
)
end
Let’s fill render/1
with some actual markup. First the outer form with the
:email
input, event handler configuration and submit button, but also a
<fieldset>
to wrap all the nested inputs.
<.simple_form
id={@id}
for={@form}
phx-change="validate"
phx-submit="submit">
<.input field={@form[:email]} label="Email" />
<fieldset class="flex flex-col gap-2">
<legend>Groceries</legend>
<.inputs_for :let={f_line} field={@form[:lines]}>
<.line f_line={f_line} />
</.inputs_for>
</fieldset>
<:actions>
<.button>Save</.button>
</:actions>
</.simple_form>
The nested inputs themselves are extracted into a function component, which makes things a little easier to follow compared to one huge blob of html. It will also allow us computing assigns per row later.
<div>
<div class="flex gap-4 items-end">
<div class="grow">
<.input class="mt-0" field={@f_line[:item]} label="Item" />
</div>
<div class="grow">
<.input class="mt-0" field={@f_line[:amount]} type="number" label="Amount" />
</div>
</div>
</div>
There used to be the need to call Phoenix.HTML.Form.hidden_inputs_for/1
here
when using a manual for
comprehention with Phoenix.HTML.Form.inputs_for/2
,
but the newer function component Phoenix.Component.inputs_for/1
automatically
adds those. Those hidden inputs submit metadata like primary keys, so ecto can
map any changes back to existing data if available.
With the markup and the data backing the form out of the way we can get started making adding and removing lines work.
Lets start with adding new lines to the list. For that we’ll add a button within
the <fieldset>
wrapping the groceries list.
<fieldset class="flex flex-col gap-2">
[…]
<.button class="mt-2" type="button" phx-click="add-line">Add</.button>
</fieldset>
The phx-click
handler will send an event to the server to add a new line. How
to do that however is already a tricky topic, given how changesets work. Explaining
those requires a quick tangent:
Ecto.Changeset
in LiveView
There are two things to understand about Ecto.Changeset
s, which make working
with it feela bit bend over backwards.
The changeset API is a very functional one. Given a base and an input of of how things are meant to look like at the end ecto figures out all the necessary changes to get to that endresult. One can then validate all the changes to prevent disallowed invariant and eventually can check if the changes are good to go or not.
The API however is not stateful in the sense that after such a round of validation one could go back and add more changes and have the changeset know which errors won’t apply anymore. A changeset doesn’t keep track of which validations were applied end how, it only keeps their results – errors and metadata – around.
So whenever there is new input with changes to be validated the expectation is that a new changeset is created, which again is run through the same validation paths as the previous.
For forms in LiveView this means you don’t want to store data in a changeset,
which won’t be reflected back to a new changeset by how the form on the client is
updated. Creating a new changeset from just the form params
should always work.
Changesets modify associations or embeds through a set based approach. The input to a changeset can just include how the list of associations or embeds is meant to look like after changes are applied and ecto figures out which items need to be added, which need to be update, which had no changes or which need to be deleted.
This is great for non-interactive clients, which have no means of modifying that list over time by e.g. by applying “delete item a” and later applying “delete item b”. This also overlaps with the previous point of being meant to create a new changeset each time the set of updates to be applied changes.
Any interactivity we add to a form with LiveView will be imperative however. We
also won’t have access to the forms params
in the related event handlers. That
requires updating the existing changeset on the LiveView, but also constrains how
that can happen.
Both of those facts about changeset – which to be fair were never build to power interactive frontend forms – don’t map too well to what LiveView allows people to build forms.
The mentioned properties of changesets means we’ll need to be thoughful in what we do for handling the additional events related to our LiveView form.
def handle_event("add-line", _, socket) do
socket =
update(socket, :form, fn %{source: changeset} ->
existing = Ecto.Changeset.get_embed(changeset, :lines)
changeset = Ecto.Changeset.put_embed(changeset, :lines, existing ++ [%{}])
to_form(changeset)
end)
{:noreply, socket}
end
In the event handler we want to add a line to our form, but also don’t want to
loose any existing changes present in the form, but not yet applied to our
base
data. We use get_embed
to get whatever the changeset considers the
current list of lines, including all known changes and then append a new item.
That new item doesn’t have changes, so it’s an empty map. The modified list is
then passed to put_embed
to be set on the changeset.
This feels like imperative editing the changeset in place and it is. But the goal
really is that the re-rendered form on the client includes new inputs for this
new item. So that they’re visible to the user and subsequent phx-validate
events
include those params
to be able to build a changeset. No other code of ours
should need to look at that added item in the changeset again.
The second step is the inverse to the previous: removing lines. This one also has a few complexities:
The usual answer to identifying data especially database records would be using
primary keys, most often the id
of a record. The form we’re looking at however
allows adding new lines, which might only get a fixed id assigned when persisted.
So there might be many lines without an id
yet. Also not every database record
has a primary key.
A more flexible approach is using @f_line.index
, which inputs_for
sets.
That value is available and works with any schema without our code needing to
depends on any of their details, which is great.
HTML forms cannot send a value of “empty list”. The encodings for form data only allow for sending data on a form, but not the lack of data.
If there are two existing lines for our form, we delete the inputs for the first line and submit the form everything will work just fine.
If we however remove all the line inputs and submit the form the params
would look
like %{email: "…"}
instead of %{email: "…", lines: []}
. To ecto %{email: "…"}
means “no changes to lines” rather than “delete existing lines”. That’s obviously
not what we want.
There are few ways to work around that issue, but the cleanest is to be more more explicit about deletions. Instead of removing inputs for existing lines from the form immediatelly, we instead update a flag on to be deleted lines, which makes them be deleted when the form is saved.
For that to happen we need to go back and update the schema and its changeset/2
function slightly to make the delete on save part work.
embeds_many :lines, Line, on_replace: :delete do
[…]
field(:delete, :boolean, virtual: true)
end
[…]
def line_changeset(city, params) do
changeset =
city
|> cast(params, [:item, :amount, :delete])
|> validate_required([:item, :amount])
if get_change(changeset, :delete) do
%{changeset | action: :delete}
else
changeset
end
end
With that out of the way we can add the button to remove a line as well as the
accompanying event handler. We’ll also add a new computed assign to the line/1
function component, so the few places needing the be adjusted for lines marked
to be deleted have a single assign to use.
assigns = assign(assigns, :deleted, Phoenix.HTML.Form.input_value(assigns.f_line, :delete) == true)
For educational purposes I only dropped the opacity on existing rows flagged for deletion, instead of hiding them completely. Feel free to adjust as needed.
<div class={if(@deleted, do: "opacity-50")}>
[…]
<input
type="hidden"
name={Phoenix.HTML.Form.input_name(@f_line, :delete)}
value={to_string(Phoenix.HTML.Form.input_value(@f_line, :delete))}
/>
<div class="flex gap-4 items-end">
[…]
<.button
class="grow-0"
type="button"
phx-click="delete-line"
phx-value-index={@f_line.index}
disabled={@deleted}
>
Delete
</.button>
</div>
</div>
The event handler for deleting a line is conceptionally similar to the one for
adding one. We again use the get_embed
to fetch all current lines, split out
the one to delete and check if it’s one already existing in base
or not.
Here we check for the presense of an id
, which is unfortunate, but for embeds
there’s no good way to check if a line was part of base
or was added to the form
later. For database backed schemas you can consider using Ecto.get_meta(schema, :state)
.
For existing lines the event handler then marks the line as :deleted
, while any
other lines are fine to just be removed from the changeset immediatelly.
def handle_event("delete-line", %{"index" => index}, socket) do
index = String.to_integer(index)
socket =
update(socket, :form, fn %{source: changeset} ->
existing = Ecto.Changeset.get_embed(changeset, :lines)
{to_delete, rest} = List.pop_at(existing, index)
lines =
if Ecto.Changeset.change(to_delete).data.id do
List.replace_at(existing, index, Ecto.Changeset.change(to_delete, delete: true))
else
rest
end
changeset
|> Ecto.Changeset.put_embed(:lines, lines)
|> to_form()
end)
{:noreply, socket}
end
We went to great length to consider how forms and changesets work before
implementing adding and removing lines. For validating and saving the whole form
this pays off, as the event handlers for phx-change
and phx-submit
of our form
won’t look any different as they would for most other LV forms:
def handle_event("validate", %{"groceries_list" => params}, socket) do
changeset =
socket.assigns.base
|> GroceriesList.changeset(params)
|> struct!(action: :validate)
{:noreply, assign(socket, form: to_form(changeset))}
end
def handle_event("submit", %{"groceries_list" => params}, socket) do
changeset = GroceriesList.changeset(socket.assigns.base, params)
case Ecto.Changeset.apply_action(changeset, :insert) do
{:ok, data} ->
socket = put_flash(socket, :info, "Submitted successfully")
{:noreply, init(socket, data)}
{:error, changeset} ->
{:noreply, assign(socket, form: to_form(changeset))}
end
end
The params
supplied by our form will include all the details we need to
validate the top level inputs, but also any lines. We won’t loose any not
yet saved lines on validation and also will have any deleted lines be
properly deleted, even if there are no more lines left on the form.
From here there could be additional features added like for example ignoring newly added lines, which have none of their inputs filled.
If you want to play with this you can look at this example repo, which actually stores data in the database: https://github.com/LostKobrakai/one-to-many-form
2023-02-27: Updated to work with phoenix 1.7.0 form changes.
2023-01-17: Replaced usage of Ecto.Changeset.get_field/3
in "add-line"
and
"delete-line"
event handlers with custom function using Ecto.Changeset.get_change/3
with a fallback of Ecto.Changeset.get_field/3
. This fixes a bug, where adding
or removing a line would make changes in other lines be “forgotten”.
2023-07-26: Replaced usage of the Ecto.Changeset.get_change/3
and fallback
to Ecto.Changeset.get_field/3
helper with Ecto.Changeset.get_embed/3
as
released with ecto 3.10.
So people can now find me under the account name @lostkobrakai@hachyderm.io
.
So far so good, but it kinda rubed me the wrong way to be @lostkobrakai
of some
random Mastodon instance. What if it goes away, will people still find my profile?
In the end these instance often depend on volunteers, which are in my opinion
not to blame if they would like to move on or stop doing what they do today.
However I’d really like to have my profile be linked to my own domain, even though
I don’t run my own instance (yet?).
Luckily that’s actually rather simple to do as explained to me when I asked on the ElixirForum.
The answer is webfinger – a http based protocol to “to discover information about people or other entities on the Internet”1.
The protocol works by having a known path on a website:
GET /.well-known/webfinger
That path can be queried with a ?resource=…
query string. Websites supporting
webfinger can return any suitable information they know about the resource in
the JSON Resource Description (JRD) format, a standardized schema for
data encoded as JSON. More details on it can be found in the RFC.
Given I recently moved my website to be powered by phoenix I did implement this as a simple controller.
I knew I wanted to have ETag support, so I pulled in {:etag_plug, "~> 1.0"}
first and put that on the controller. ETags are a mechanism to skip sending
data when the client already has the data cached locally, which I hadn’t really
touched much, so this was a good excuse to see how it goes.
defmodule MyAppWeb.WebfingerController do
use MyAppWeb, :controller
plug ETag.Plug
end
Next the RFC for webfinger expects servers to return a 400
, if the request
misses the required ?resource=…
query parameter. I handled this one in a
plug as well, though a custom one this time.
defmodule MyAppWeb.WebfingerController do
use MyAppWeb, :controller
[…]
plug :resource_required
defp resource_required(conn, _) do
if conn.query_params["resource"] do
conn
else
conn
|> send_resp(:bad_request, "")
|> halt()
end
end
end
Clients are also allowed to add filters using the ?rel=…
query parameter, but
I didn’t implement supporting that feature. The RFC makes this optional, as the
server may just ignore those parameters and still work to the specification.
So the last thing to do is figuring out if our server knows about the requested resource. This I handled in the controller action:
defmodule MyAppWeb.WebfingerController do
use MyAppWeb, :controller
[…]
@aliases ["acct:lostkobrakai@kobrakai.de", "acct:lostkobrakai@hachyderm.io"]
def finger(conn, %{"resource" => resource}) do
case resource do
r when r in @aliases ->
data = %{
subject: "acct:lostkobrakai@kobrakai.de",
aliases: [
"acct:lostkobrakai@hachyderm.io",
"https://hachyderm.io/@lostkobrakai",
"https://hachyderm.io/users/lostkobrakai"
],
links: [
%{
rel: "http://webfinger.net/rel/profile-page",
type: "text/html",
href: "https://hachyderm.io/@lostkobrakai"
},
%{
rel: "self",
type: "application/activity+json",
href: "https://hachyderm.io/users/lostkobrakai"
},
%{
rel: "self",
href: "https://kobrakai.de"
},
%{
rel: "http://ostatus.org/schema/1.0/subscribe",
template: "https://hachyderm.io/authorize_interaction?uri={uri}"
}
]
}
response = Phoenix.json_library().encode_to_iodata!(data)
conn
|> put_resp_content_type("application/jrd+json")
|> send_resp(200, response)
_ ->
send_resp(conn, :not_found, "")
end
end
end
As you can see I just hardcoded the values, but this could easily become more
flexible when needed. One thing of note however is the returned content type of
application/jrd+json
.
With phoenix we can do proper content type negotiation,
so in the router I didn’t just add the router under the :browser
pipeline, but
I created a custom pipeline:
pipeline :webfinger do
plug :accepts, ["jrd", "json"]
end
scope "/", MyAppWeb do
pipe_through :webfinger
get "/.well-known/webfinger", WebfingerController, :finger
end
The "jrd"
format isn’t known by the :mime
application, so I’ve added the
necessary config to extend it as well. Make sure to recompile it using
mix deps.compile mime --force
. Otherwise you might run into compile time errors
around mismatched compiled and runtime configuration.
config :mime, :types, %{
"application/jrd+json" => ["jrd"]
}
With that everything is in place for me to be discovered under my own domain
as @lostkobrakai@kobrakai.de
:
https://kobrakai.de/.well-known/webfinger?resource=acct:lostkobrakai@kobrakai.de
This should allow people to find me without them needing to know which Mastodon instance actually hosts my account at the time.
]]>defmodule Components do
use Phoenix.Component
def alert(assigns) do
~H"""
<div class="alert">
<h3 class="alert__title"><%= @title %></h3>
<p class="alert__body"><%= @body %></p>
</div>
"""
end
end
<!DOCTYPE html>
<html lang="en">
<head>
<title>Document</title>
</head>
<body>
<.alert title="Attention" body="Something went wrong!" />
</body>
</html>
That approach works great for abstracting markup and making templates more expressive. But it’s only useful to a project interested in this specific markup. I think a lot of the frustrations of people with frameworks like bootstrap came from the fact that it sounds nice to share components, but as soon as it comes to customizations it’s easy to get into a big mess fast – especially where functionality and how things look are coupled. It would be nice to not only be able to share implementations of components, but also to be able to create higher level components, which share logic and functionality, but without coupling it to markup yet.
Components are considered “renderless” when they’re not rendering any markup on their own, but delegate rendering of information to how the user of the component considers it useful. This approach has been used in client side systems like react or vue for a long time already, but with heex it’s now usable within Phoenix as well.
As an example consider a pagination component. Usually pagination is constraint
only by a few datapoints, like e.g. current_page
and total_pages
. A function
component could transform those two values to all those many intermediate values
needed to actually render a pagination. Sometimes it just makes sense
to abstract (complex) functionality without coupling to a specific and fixed way
to render the data and having a gazillion options to pass around won’t
make anybody happy. That’s where a renderless component can help out.
defmodule Components do
use Phoenix.Component
use Phoenix.HTML
def pagination(assigns) do
%{
current_page: current_page,
total_pages: total_pages
} = assigns
lower_bound = max(1, current_page - 3)
upper_bound = min(total_pages, current_page + 3)
pages = lower_bound..upper_bound//1
setup = %{
is_first: current_page == 1,
is_last: current_page == total_pages,
pages: pages,
current_page: current_page,
total_pages: total_pages
}
assigns = assign(assigns, :setup, setup)
~H"""
<%= render_slot(@inner_block, @setup) %>
"""
end
end
<!DOCTYPE html>
<html lang="en">
<head>
<title>Document</title>
</head>
<body>
<.pagination current_page={7} total_pages={20} let={setup}>
<ul class="pagination">
<%= unless setup.is_first do %>
<li class="page-item">
<%= link "«", to: Routes.some_index_path(@socket, :index, %{page: 1}), title: "Go to first" %>
</li>
<% end %>
<%= for page <- setup.pages do %>
<li class="page-item">
<%= link page, to: Routes.some_index_path(@socket, :index, %{page: page}) %>
</li>
<% end %>
<%= unless setup.is_last do %>
<li class="page-item">
<%= link "»", to: Routes.some_index_path(@socket, :index, %{page: setup.total_pages}), title: "Go to last" %>
</li>
<% end %>
</ul>
</.pagination>
</body>
</html>
This works great – though a critic might still say this can be done by precomputing
assigns
.
There’s however an interesting case where those renderless components really shine.
For react there exists a quite interesting library called downshift, which implements a autocomplete/combobox/searchable select as a renderless component, while taking care of the necessary accessibility requirements to such a component, supplying necessary aria events to users to add to their markup.
I don’t yet see something like downshift coming to HEEx soon. It also deals with event listeners and many of them should be handled on the client side with no server involved.
What could be interesting however is having less interactive components be supplied with proper aria markup, e.g. for validation errors on forms or open/closed state on toggleable content.
I quickly toyed with something like this today and it’s for sure an interesting idea for making it easy to do the correct thing:
<.form for={@changeset} let={f}>
<.field_context form={f} field={:name} let={field}>
<label for={field.id}><%= field.label %></label>
<input
type="text"
id={field.id}
name={field.name}
class={["input", if(field.errors, do: "has-error")]}
{field.input.aria} />
<%= if field.errors do %>
<p class="errors" {field.validation.aria}>
<%= for error <- field.errors %>
<span><%= error %></span>
<% end %>
</p>
<% end %>
</.field_context>
</.form>
As far as I know there’s some work happening on form handling with live view by
the Phoenix team at this time. Maybe this can be a time to think not just about
how to update phoenix_html
helpers to HEEx, but also go beyond that.
Jason
, the default json library of phoenix.
Looking at the README of Jason
this is quickly resolved by doing something like
this:
defmodule User do
@derive Jason.Encoder
defstruct [:id, :name, :title, :coordinates]
end
This does work, but it also has quite a drawback: Protocol implementations are module based and therefore global. One cannot have a given struct encode to multiple different json representations. This might not sound that problematic at first, as one usually hits that problem trying to get the struct to convert to the first json form required in a project – though that might not stay to be the only one.
The User
struct of the past section could be used in multiple parts of an
application. Let’s consider the application holding a blog, which users can publish on,
and also a map of registered users to find people by location. Both should be powered
by individual api endpoints.
By implementing Jason.Encoder
the users can be encoded to json directly in the
controller:
# BlogController
def index(conn, _) do
json(conn, %{authors: Users.list_users()})
end
# MapController
def index(conn, _) do
json(conn, %{users: Users.list_users()})
end
This will encode all users to json, but including all struct fields on both endpoints, even though only the blog is concerned about titles and coordinates only being relevant to the map component.
If we want to remove the :coordinates
field from the authors this would be possible
by adjusting how the Jason.Encoder
implemenation works. Though it would remove
the coordinates for the map’s users list as well. Additionally it’s not great to need
to adjust a core business logic module to cater to a need of the web api.
A great solution to the problem here is using the phoenix view layer instead of
the Jason.Encoder
protocol. Often the view layer of phoenix is seen as a part
only needed for HTML based websites or even more specific for handling templates,
but the view layer is very useful even beyond those use-cases.
The phoenix view layer has two important pieces to it. The template engines
(.eex, .exs, .leex, .heex, …), which turn template files into functions on the
view module – not so important here – and format encoders. Format
encoders turn the values returned by MyAppWeb.SomeView.render/2
functions
into iodata to send back as the http response.
That’s how a map returned from such a function is turned into a json string
if the format is .json
or how for .html
the html encoding is applied. One
can even add custom format encoders (e.g. for .xml
or .mjml
).
How would using the phoenix view layer look like for our example. Let’s start with a view module for each of the controllers.
# BlogView
def render("index.json", %{authors: authors}) do
%{
authors: render_many(authors, __MODULE__, "author.json", as: :author)
}
end
def render("author.json", %{author: author}) do
%{
id: author.id,
name: shorten_firstname(author.name),
title: author.title
}
end
defp shorten_firstname(name) do
[first, rest] = String.split(name, " ", parts: 2)
<<letter::binary-size(1), _rest>> = first
"#{letter}. #{rest}"
end
This shows not only how the returned fields can be limited, but also how the format of a field can be adjusted to what needs to be returned. Using views for converting the struct to the returned map of data provides a lot of flexibility and also a place to segment such endpoint specific implementation details into.
This not only prevents controllers to get more complex, but also view modules can be used by multiple controllers. So composition and reuse is supported.
# MapView
def render("index.json", %{users: users}) do
%{
users: render_many(users, __MODULE__, "user.json", as: :user)
}
end
def render("user.json", %{user: user}) do
%{
id: user.id,
name: user.name,
coordinates: user.coordinates
}
end
This one shows the map view. There doesn’t seem to be much new here, but consider that the coordinates are just returned as is, even if in a real world implementation it’s likely a struct as well.
Using the view layer makes most sense for domain models of an application.
Auxiliary structs, which mostly represent complex values like coordinates or
Decimal
structs still benefit from global Jason.Encoder
implementations given
there’s hardly any use in encoding just parts of the data they hold. Those structs
are more akin to a single value represented by a struct of details and less a
container of multiple distinct pieces of data.
Phoenix views are generally a great way to handle the transformation from
application level data to an exchange format sent to other parties. It’s kind of
the inverse of Ecto.Changesets
, which bring data into a system, while views
provide data to the outside. Both are in my opionion nice ways of forming an so
called anti-corruption layer in an application to separate the outside world
from core data formats and making outside change easier to handle.
Starting with phoenix 1.6 the view layer got extracted from phoenix the framework, so it can be included wherever useful, similar to how ecto can be useful even without a database.
]]>$ mix phx.gen.live JobBoard Company companies name:string website:string
This does generate a liveview module MyAppWeb.CompanyLive.Index
, which for the
liveaction :index
can render a nice list of companies.
But usually things do not stop here. Let’s say the next requirement is adding the number of jobs published per company in that table on the index page. A quick look into the liveview module will reveal that it calls into its related context module to fetch companies’ data for rendering.
def list_companies() do
Repo.all(Company)
end
Assuming jobs are related to companies in the database using foreign keys one obvious way of handling the requirement would be to preload jobs from the database and count them up in the template:
def list_companies() do
Company
|> Repo.all()
|> Repo.preload(:jobs)
end
# In the template
<td><%= Enum.count(company.jobs) %></td>
This however has the downside of loading much more data from the database than required. So another approach could be loading the number of jobs directly from the database instead of counting jobs within elixir.
def list_companies() do
jobs_by_company =
from j in Jobs,
group_by: j.company_id,
select: %{
company_id: j.company_id,
job_count: count(j.id)
}
companies_query =
from c in Company,
join: jc in subquery(jobs_by_company),
on: c.id == jc.company_id,
select: %{c | job_count: jc.job_count}
Repo.all(companies_query)
end
# In the company schema
field :job_count, :integer, virtual: true
# In the template
<td><%= company.job_count %></td>
This works great. The database sums up the number of jobs per company and it’s simply stored in a virtual field on the company schema. No longer are whole jobs loaded just to count them up.
Let’s imagine another requirement. The JobBoard in the meantime got a new feature: people can submit rating for companies, any company not just the ones publishing jobs. Still the companies index shall show the average rating of each company. That rating itself is part of a completely different context and there’s no foreign key relationship on the database level. Querying that separate context from within the JobBoard context doesn’t feel like a great solution. Those companies are only related to ratings on the UI level.
Given the last approach is no longer a great one the data loading needs to become multiple steps:
companies = JobBoard.list_companies()
ratings = Comparator.list_ratings(companies)
This seems simple, but is a bit convoluted on the template level:
<%= for company <- @companies do %>
<% rating = Map.get(@ratings, company.id) %>
…
<% end %>
One would love to just be able to do company.rating
instead, but there is no
:rating
field on the company schema. Searching the ratings for the
current company in the iteration each time doesn’t seem nice.
The approach I recently adopted is using live components to compose the various places to fetch data from. Instead of taking all companies, loading all the related ratings in one place, just to bring together individual companies with their individual rating in a completely other place – live components allow for all that to happen in one place.
<%= for company <- @companies do %>
<% live_component @socket, Row, id: company.id, company: company %>
<% end %>
defmodule MyAppWeb.CompanyLive.IndexRowComponent do
use MyAppWeb, :live_component
@impl true
def render(assigns) do
~L"""
<tr>
<td><%= @company.name %></td>
<td><%= @rating %></td>
</tr>
"""
end
@impl true
def preload(list_of_assigns) do
companies = Enum.map(list_of_assigns, & &1.company)
ratings = Comparator.list_ratings(companies)
Enum.map(list_of_assigns, fn assigns ->
Map.put(assigns, :rating, Map.get(ratings, assigns.company))
end)
end
end
The preload/1
callback for those components allows for data fetching to happen
on the whole set of companies – preventing N+1 query issues – while the template
simply works on per company data, which does hold a simple @rating
. No need to
deal with mapping individual ratings back to companies in the template itself.
Besides making templates cleaner I also like the idea of letting the liveview itself only deal with fetching companies, which is it’s job. While metadata is only added if actually needed. Imagine the rating column is not always enabled. If the component loading ratings is not rendered nothing is loaded. It allows in my opinion for a much better locality as there’s not just one “controller” (the liveview) and one view layer (the template), but it’s layers of components, which load just the data they themselves need.
Having the preload/1
callback the danger of n+1 queries within loops is
mitigated. The only downside left however is multiple different components fetching
the same data on demand. Here n
would be the number of components though instead
of the number of loop iterations.
ecto
3.5 update introduced the parameterized type Ecto.Enum
, which
is a great way to use atoms in elixir for signifying things like statuses or
types of data. In the database those values are stored as plain strings, so
they’re easy to manage there as well – until it comes to sorting by those columns.
The simplest solution for sorting not alphabetically, but by e.g. the order the
statuses are applied, is likely to use an proper enum
column in the database or
joining a table in the db, which holds an index for each possible value in a secondary
column.
But what about arbitrary sorting, which cannot be placed in the db?
For postgres there’s the possibility to build that join table dynamically with
unnest
instead of an actual table. The usual answer one finds on the internet
most often involves using VALUES
lists, but they don’t really work well with
ecto queries, as they’re not easily parameterized. unnest
luckily can be:
defmodule MyApp.QueryHelpers
@doc """
Unnest into an table format.
## Example
import MyApp.QueryHelpers
status = [:waiting, :running, :done]
order = [1, 2, 3]
from jobs in "jobs",
join: ordering in unnest(^status, ^order),
on: jobs.status == ordering.a,
order_by: ordering.b
"""
defmacro unnest(list_a, list_b) do
quote do
fragment("SELECT * FROM unnest(?, ?) AS t(a, b)", unquote(list_a), unquote(list_b))
end
end
end
As you can see unnest
does work with plain lists, which can be passed as
parameters and therefore work with fragment
. This allows to build up an table
of arbitrary size (more columns could be done by adding additional arities for
the unnest
helper), which can be joined to the data in the database.
Given we’re manually supplying the list of “positions” this can not only be used for read operations, but e.g. also for updates for things like position columns for data ordered in the db.
@spec update_order(%{produce_id :: integer => new_position :: integer}) :: result :: term
def update_order(new_order) do
{ids, positions} = Enum.unzip(new_order)
query =
from p in Product,
join:
positions in unnest(
type(^ids, {:array, :integer}),
type(^positions, {:array, :integer})
),
on: p.id == positions.a,
update: [set: [position: positions.b]]
Repo.transaction(fn ->
# Create deferable constraint like that (not `unique_index/3`):
#
# execute("""
# ALTER TABLE products
# ADD constraint products_position_unique unique (position) deferrable;
# """, "")
#
# This allows for all rows to update before checking if everything is
# still unique.
Repo.query!("SET CONSTRAINTS products_position_unique DEFERRED")
Repo.update_all(query, [])
end)
end
]]>phoenix_live_view
people are often worried about their LiveViews
being mounted twice for fresh requests. I regularly see people asking for ways around
this fact or even why this is needed in the first place.
A fresh request by an user to a website is always a plain http request. It hit’s
MyAppWeb.Endpoint
and is served by the plug based pipeline setup on the server.
LiveView’s will be rendered in what is called “static render”. That means
instead of starting a long running process any LiveView will run mount/3
(+ handle_params/3
) once and the resulting markup is rendered into the http
response.
When the client received that response to the http request it’ll run the app.js
and start connecting to the websocket endpoint for LiveView. This time any LiveView
will be started in a proper long running process, it will mount and render again
and the client side will replace (merge into) what it got from the http response.
After that the server knows which markup the client received (and which templates)
it got, so it can start doing its extensive diffing for subsequent changes to the dom.
Given the websocket connection can only start after the client did receive the initial http response (and the javascript) it’s quite obvious that there need to be at least two steps. Browsers don’t have means to directly start with the websocket connection.
After this one initial http request however one can use live_redirect
and live_patch
to move from one LiveView enabled page to another without needing the http request.
Any changes can be transfered purely over the existing websocket connection, now that
it’s established and the js for it resides on the client.
On the intial request: No, we can’t.
For subsequent request: See live_redirect
and live_patch
.
This is more complex, as there are various layers to it.
First of all the initial http request and all its fetched data is long cleaned up at the time the websocket connection connects to the server. So there’s no way to share data between them directly.
Then there’s the session
which allows sharing data from the plug pipeline of the
static render to any later render. The data of the session
is transfered by
placing it encoded within the html send as response by the http request. So it
should not hold much information or it will bloat the http response. It’s a solution
for passing around id’s to certain things, which were already validated by the plug
pipeline, but not for bigger amounts of data.
The only way to keep data around without sending it to the client is caching.
It’s still not a simple “yes”.
As we’ve already seen an inital LiveView page’s loading involves two connections to the server. Therefore one shouldn’t try to treat them like one request, but it’s much more similar to an user requesting an non LiveView page and then immediately hitting refresh. This analogy will bring up some important considerations for the time the first of those requests might want to cache data.
Will the reload happen? a.k.a. Will the LiveView js connect (to this node) at all?
There could e.g. be a problem with loading the javascript for liveview.
In a distributed setup it could just be the load balancer routing the request
to different nodes.
When will the reload happen? a.k.a. When will the LiveView js connect?
The user could be on a slow network and the websocket connection will only succeed minutes later.
There’s also another point for LiveView specifically:
So the answer if caching helps really depends.
One way to avoid the whole problem of fetching expensive data twice is not fetching and therefore rendering that data on the initial static render and instead e.g. rendering some loading spinner or similar. On the mount via the websocket connection the actual data would then be loaded.
This has the downside that any issue preventing the websocket connection will mean that content is not being available to the user. This is similar to e.g. the consideration in a traditional SPA context.
]]>Supervisor.Spec
based syntax to the newer child spec based syntax, people trying to integrate with erlang libraries or because they added more parameters to their start_link
functions and now wonder why it fails when trying to supervise the process.
To start, it’s important to know what a child spec is. It’s a map of data, which is used by supervisors to determine how they should start and interact with a certain supervised child. The format is documented in the Supervisor
docs for elixir as well as the :supervisor
docs for erlang, therefore I’ll keep my explanations short. There are 6 keys: :id
, :start
, :restart
, :shutdown
, :type
and :modules
. Besides :id
and :start
the keys are optional with defaults.
The child spec for an run-of-the-mill GenServer
essentially looks like this:
%{
id: Stack,
start: {Stack, :start_link, [[:hello]]}
}
:id
is the id the process will have as a child of a supervisor. Per Supervisor
instance the id needs to be unique. :start
is a mfa()
tuple for what the supervisor shall call to start the child.
Up until here this is information applicable to both erlang and elixir. But elixir did build on top of child specs.
In Elixir the change to child specs for configuring supervisor children was used to add standardized conveniences to that configuration. The idea here being to bring locality of information into the mix.
Before the change a module MyApp.Worker
would implement for example an GenServer
, but the supervisor MyApp.Supervisor
would need to configure the process being started as type: :worker
and with restart: :permanent
. This is ok’ish for processes you own the implementation for, but say the process comes from a library. If the library now needs to change and put the worker nested under an internal supervisor it’s a breaking change because users of the library need to change to type: :supervisor
. There’s a disconnect between the source of truth for “What type of process am I?” and the code implementing the process.
I know of some (erlang) libraries, which implemented their own ways of returning a child spec map. Users could call a function with arguments, and get the full child spec in return. Elixir however took the idea and wrapped stdlib tooling around it.
Instead of a list of child spec maps the elixir Supervisor
can additionally deal with children being setup via a module name or a tuple of {module_name, arg}
:
children = [
# Module only
MyApp.Supervisor,
# Tuple
{Registry, keys: :unique, name: Registry.ViaTest},
# One-off way to build a child spec map
:poolboy.child_spec(name, pool_args, worker_args),
# Inline child spec map
%{id: Stack, start: {Stack, :start_link, [[:hello]]}}
]
The additional options of a module or a tuple are stdlib means of doing a similar thing as the :poolboy
example:
{Registry, keys: :unique, name: Registry.ViaTest}
is a shortcut for Registry.child_spec(keys: :unique, name: Registry.ViaTest)
MyApp.Supervisor
is a shortcut for {MyApp.Supervisor, []}
and therefore MyApp.Supervisor.child_spec([])
Both are ways of building a child spec map by calling child_spec/1
on the respective module.
This is how an implementation of that function might look like:
def child_spec(init_arg) do
%{
id: __MODULE__,
start: {__MODULE__, :start_link, [init_arg]}
}
end
start_link
and why do I never see a child_spec/1
functions?
The most common processes we interact with in elixir are build using use Supervisor
or use GenServer
. Both of them generate the needed child_spec/1
automatically. It’s overridable however if you need to alter the behavior.
Supervisor.Spec
The old way of configuring children looked like this:
children = [
worker(MyWorker, [arg1, arg2, arg3]),
supervisor(MySupervisor, [arg1])
]
The difference between supervisor/3
and worker/3
will be handled by the newly used child_spec/1
function, so that becomes irrelevant knowledge to the supervisor itself.
However the [arg1, arg2, arg3]
on both of the old functions meant it would start the process with MyWorker.start_link(arg1, arg2, arg3)
. That’s no longer the case with the automatically generated child_spec/1
implementations.
children = [
{MyWorker, [arg1, arg2, arg3]},
{MySupervisor, [arg1]}
]
This will call MyWorker.start_link([arg1, arg2, arg3])
. Instead of start_link/3
start_link/1
is called with a list. There are two ways to fix this:
One is to manually implement child_spec/1
for MyWorker
altering the :start
value – notice the missing square brackets.
def child_spec(init_arg) do
%{
…
start: {__MODULE__, :start_link, init_arg}
}
end
The other is to modify MyWorker
to have a start_link/1
function, which can handle the list inputs.
start_link
?
If your process shall only be started as a child of a supervisor, then implementing child_spec/1
can be enough. It can directly link the :start
parameter to another module like GenServer
. However it’s common best practice to have start(_link)/x
functions as well, to be able to start those processes outside a supervision tree.
Given child_spec/1
is an elixir based convention it’s something not always present for erlang libraries. For erlang libraries look out for similar functions like the one I showed for :poolboy
. Otherwise you can either build the child spec map in a private function on the supervisor or build your own elixir module implementing child_spec/1
, which can be used in supervisors as child, but configures :start
, so the erlang library is started.
Ecto.Type
There are three conceptional types of data ecto deals with.
field :some_name, :date
being a %Date{}
.
Turning user input into the runtime data type is using Ecto.Type.cast/2
and is therefore referred to as “casting”.
The following shows a concrete example of those three types of representations for one underlying value.
# Plain string, describing a date in iso8601,
# but needs to be parsed to be sure.
input_date = "2020-07-20"
# Using proper elixir type of a %Date{} struct,
# which one can be sure about being a proper date.
runtime_date = ~D[2020-07-20]
# This is the same as the runtime_date, as ecto's database
# drivers are expected to be able to handle that directly.
# That usually isn't the case however for custom ecto types.
database_date = ~D[2020-07-20]
Conversion between those types of values is provided by Ecto.Type
– natively or using custom implementation of its behaviour.
# input -> runtime
runtime_date = Ecto.Type.cast(:date, input_type)
# database -> runtime
runtime_date = Ecto.Type.load(:date, database_date)
# runtime -> database
database_date = Ecto.Type.dump(:date, runtime_date)
There’s no way of going from the runtime value to a possible input value, but that’s not needed, at least in the context of ecto.
Ecto.Changeset
Ecto.Changeset
exposes multiple functions for casting to a whole set of values of user input: cast/4
, cast_assoc/3
and cast_embed/3
.
However changes applied on a changeset are not always “user input” and therefore don’t always need to involve casting the changes. E.g. in an event sourcing system one might handle events, whose data has been validated before and is no longer expressed by constrainted data types. Those can be applied by another set of functions on Ecto.Changeset
: change/2
, put_change/3
, put_assoc/4
and put_embed/4
. For those functions values for fields are expected to already be “runtime format” values,
which don’t need to be casted.
Needs Casting | No Casting | |
---|---|---|
Schema Values |
cast/4 |
change/2 /put_change/3 |
Assoc Values |
cast_assoc/3 |
put_assoc/4 |
Embed Values |
cast_embed/3 |
put_embed/4 |
Both cast_assoc/3
and put_assoc/4
as well as their embedded counterparts compare preloaded data with the input provided.
From there a set based comparison is done. All new items in the input are created. All existing items are updated if needed. All no longer available items are deleted. Items not present in both previous data or input are ignored – that’s only really possible for assocs though. Matching of items is done based in primary key.
All those APIs are not suitable for creating “just an relationship” of already existing records in the db. For creating relationships for existing records or updating a parent and maybe a single somehow special assoc it’s usually simpler to opt for other solution like manually updating those separately. Likely using Ecto.Multi
.
Most of the times the usual answer comes as a surprise to people, as usage of Ecto.Schema.many_to_many
is almost actively discouraged. I’ll try to bring some insights into why this is the case and what’s the better alternative.
many_to_many
in ecto
For quite some time ecto didn’t even come with many_to_many
relationships and people instead modeled them explicitly using plain belongs_to/has_many relationships on all tables involved. But due to popular demand explicit support was eventually added. (1)
There’s one caveat though. It was added with the intent of hiding away the implementation detail of needing a join-table to create the relationship between the two many-to-many schemas. Because of that the join table can only support two foreign key columns and nothing else on the table. Those two are everything needed for modeling the relationship in the db and the only amount of information, which can be used without additional APIs for retrieving it.
This is a serious limitation for almost all many-to-many relationships out there. I’ll list a few examples:
User <-> Company
: This is likely to become a more featureful relationship in the future. There might be role assignments or time based access gates added. Photo <-> Album
: The need to have photos in a certain order per album might come up. User <-> Group
: Even just showing when a user was added to a group needs additional fields on the join table. All of the above are better modeled with an explicit schema for the join table. This way additional fields can be added to the relationship without a problem (or any new/special api to learn). For quick access and preloading purposes you can still have relationships between the outer schemas of the relationship as well.
# Standard belongs_to/has_many
Company has_many CompanyUsers
CompanyUser belongs to Company
CompanyUser belongs to User
User has_many CompanyUsers
# But also has_many through
Company has_many Users through CompanyUsers
User has_many Companies through CompanyUsers
Documentation and Actual code examples
many_to_many
in ecto is probably not going anywhere. But now you’re aware of it’s limitations. Unless you really don’t care about the join table and you’re quite sure this won’t change in the future it’s better to just model the relationship explicitly and use has_many
‘s through
option to get all the convenience of loading relationships via the join schema if that’s all you need.
1: http://pages.plataformatec.com.br/ebook-whats-new-in-ecto-2-0
]]>I've thought that @rails deployment on a non-Docker or non-Heroku production server was tricky. But that is nothing compared to @elixirphoenix.
A million buggy how-tos and deployment tools. Not a single one works. Not a single README works. Always something "small" is missing.
At the end of a two day hunt for a https://t.co/nY1mnbHhzA like tool I ended up with the advise of pros in an Elixir Slack channel to "just write a Bash script which glues everything together".
— Stefan Wintermeyer (@wintermeyer) March 17, 2020
Thoughts like this are not really rare for people new to elixir. And they’re not totally wrong in their assessment either. Doing such a comparison however expects that a similar level of “just handle this for me” is actually possible for elixir tools to provide and if so that it’s a good idea to do so. This is an attempt in clearing up a bit of the missing pieces in comparing elixir deployment to deployment tools of other languages.
Deployment in the context of elixir usually refers to the need of talking elixir source code usually in a mix project and bringing it’s functionality to a server somewhere on the internet and running the project on it.
In interpreted languages – like ruby – this is usually done by having the server use the language’s runtime and shipping source code directly to the server. The source code is then executed using the installed runtime.
The same can be done in elixir as well. With erlang, elixir and mix on the server one can run a mix project with MIX_ENV=prod mix …
. This is usually not how deployment is handled in the community though. People tend to use releases, which I’ll discuss in more detail later.
First I’d like to separate “deployments” up into a few sub tasks, which need to be done as part of a deployment. A few key tasks are:
This is not meant to be a comprehensive list of everything a deployment handles, but those are tasks useful to keep in mind for the rest of the blog post.
Releases on the beam refer to a set of compiled files/folders, which form a self-contained artifact specifically for deployments. Often it comes as a tar archive, which can be unarchived on a production server and be started there using plain old executable shell scripts.
Self-contained for releases means it contains the beam vm, all applications of a project with all their compiled beam files, priv folders and scripts to start/stop things. The server itself doesn’t need erlang/elixir installed at all (unless bundling of erts files is disabled). No need to version control a language runtime.
In a mix project a release is built by running mix release
. There’s great documentation available on how it works, so I won’t go into much detail here. For phoenix projects make sure to also consult its documentation.
How does this relate to the difficulty of kitchen sink deployment tools especially if we seem to have a great way to bundle everything up neatly?
The answer is that while self-contained a release is build for a specific system. Things like OS, various system libraries as well as certain NIF resources need to be the same on the system building a release as on the system running it.
To fulfill this requirement there are essentially three options:
Option 1 and 3 are not really popular. Often people are not running their server’s flavor of linux in development and a production server should not be bothered with compiling releases, but with running them. There exist tools using those options, but they often don’t have many users and are not quick to pick up in a general sense.
Option 2 is the one, which most people implement, but it highly depends on the infrastructure one has available. With a CI/CD environment ready it’s usually quite simple to have it automate a docker container, which mimics production, to build a release. If that’s not the case it might be possible to run docker locally to do the same. If docker is a problem then there might be other means to have a build system running locally or remote to do the job. But one needs to find some way to have a system for building the release.
Automating Option 2 is quite difficult for a generalized tool. Without limiting to a certain kind of project, infrastructure and possibly other constraints it’s not possible to generalize what needs to be done where.
There is quite some complexity in infrastructure involved in getting a release to be build, once this is handled though a mix release
should do the job – with phoenix one additional command might be needed to trigger the assets pipeline. But what about the other parts of deployment listed in the initial section of the blogpost. How is the release uploaded, started, stopped, restarted, ….
This is where erlang traditionally as well as elixir don’t need to be involved anymore. Any system, which can handle plain shell scripts, can handle releases. From bash scripts to ansible or heck even capistrano. On the server there is upstart, initd, systemd, docker and further tools to handle the lifecycles of services. There are a variety of solutions out there. One just has to choose which one to employ.
In professional environments this is also the place where integration with existing tooling becomes relevant. There might be multiple projects of various languages to be handled. Therefore general purpose tools are actually a good investment and might even be a requirement.
The last two sections hopefully explain a bit why there are few batteries included tools for handling complete deployments in elixir. The complex, elixir-specific parts are mostly in replicating ones specific server setup – there are e.g. docker solutions for certain os/architectures, but not for others – and the parts, which can be generalized are actually better handled by more general server management/deployment tooling, which can be used in language agnostic manner.
For most companies deployment tools, CI/CD Pipeline and servers are already present, so it’s often more a matter of adjusting the elixir workflow to work within those existing systems than it’s a case of a tool being able to prescribe how deployment has to work with elixir and doing everything in its way.
Sadly this means one-off and less infrastructure heavy deployment solutions are currently not supported out of the box to the degree one might hope for. Falling back to installing elixir on the server and running source code can be a solution for those kinds of cases though.
]]>def index(conn, _params), do: […]
or def create(conn, %{"entity" => entity_params}), do: […]
. This seems like a nice default when starting out working with phoenix. All you need to handle a request is the connection and its params after all, right? With me writing this post I obviously came to a different conclusion.
Quite early in a project of mine I noticed, that I handled most of the recurring tasks in plugs running before the actual controller action. For example the usual Get an entity by its ID supplied as a url param is something I use a plug for instead of doing the Context.get_entity(id)
call in all 4 actions of show, edit, update, delete
. The results of those plugs are accumulated in the conn.assigns
map, so I can use them later on. This use them later on is elegantly solved in views, because all assigns are available as @entity
in templates, but that’s not the case for controller actions. At first is just felt like replacing entity = Context.get_entity(id)
with entity = conn.assigns.entity
lines, which didn’t really feel like much of an improvement. That’s something I wanted to deal with like with params: Pattern matching in the function head.
Luckily there’s a way in phoenix to make that happen by putting this into the controller module:
def action(conn, _) do
args = [conn, conn.params, conn.assigns]
apply(__MODULE__, action_name(conn), args)
end
For a bit of technical background: You might already know that you can use plugs in the body of a controller module, but the controller by itself is also a plug. Using use MyAppWeb, :controller
will make Phoenix.Controller
setup the necessary callbacks for it being a plug.
Now when a controller is called by the router it’s not directly executing the action callback like index/2
and others, but it’s executing the call/2
function of the controller plug and that’s calling action/2
of the controller (set up for us by default), which then in turn checks which action callback needs to be called. That function is overridable, allowing the user to modify how action callbacks are called (and if at all). This is what I did for the above code snippet.
With it in place my controllers mostly look similar to this:
plug :entity_by_id when action in [:show, :edit, :update, :delete]
def edit(conn, %{"entity" => params}, %{entity: entity}) do
with {:ok, entity} <- Context.update_entity(entity, params) do
[…]
end
end
defp entity_by_id(%{params: %{"id" => entity_id}} = conn, _) do
case Context.get_entity(entity_id) do
%Entity{} = entity -> assign(conn, :entity, entity)
_ -> not_found(conn)
end
end
Using such a setup my controller actions usually don’t need to match anything out of the supplied params anymore while most do match something out of the assigns. All of the setup work can be done in self contained plugs, while controller actions are mostly concerned with handling only the action that’s supposed to happen.
I’ve been using this in my work for a while now and it makes controller actions quite a bit cleaner and extracting things to a plug — even one local to the single controller — quite a bit more enjoyable. In new projects I usually put the snippet in the my_app_web.ex
file, so any controller in the project works with such 3-arity actions. I’m not sure if this would ever find its way to be a phoenix default or the phoenix generators using plugs in their controller scaffold, but I feel it could make plugs and assigns quite a bit more approachable for beginners as well.
One recent result of this awareness was the addition of address handling and retail-stores into an application. Stores are a key element of that application, so they’re a rather high level element. Addresses on the other hand are just a bunch of strings and location data—as well as not really limited in usage for stores—therefore they’re rather low level.
I quickly added the two contexts retailers
and addresses
to my application as well as schemas for Store
and Address
. But when it got to adding the association between both I got kinda stuck. The Address
shouldn’t need to know about a more high level concept of stores yet a association of has_one :address, Address
on the Store
schema would mean putting a store_id
column in the addresses
table. At the same time belongs_to :address, Address
is incorrect as well.
This is where ectos abstract tables feature comes into play. It’s a way to define a low level abstract schema Address
which is not linked to any specific table. It’s simply a blueprint, which each higher level component like my Store
schema can use in combination with its own addresses table like this: has_one :address, {"store_addresses", Address}, foreign_key: :assoc_id
. So this new table can have it’s assoc_id
linking to the stores
table without interfering with any other addresses I might need to store for that application. The documentation about the feature also mentions this as being more performant and overall a better design for the database. At runtime everything stays as it was. Each Address
struct will still be the same—besides the primary key.
So being aware of those dependencies between contexts and they’re place within the application actually lead me to a better database architecture and told me that I’m working with the concept of polymorphic associations even before I had the actual use-case of sharing the address schema with multiple higher level elements.
So the TLDR would be: If neither belongs_to
nor has_one/has_many
seem to fit your use case maybe take a look at abstract tables.
The entity table houses the latest status value, so a simple solution to the problem would be to keep another table to track the history of old status values. My naive first approach would be to save the old status to a separate table every time it does change from its previous value. Here is an example ecto schema illustrating this approach:
schema "entities" d
field :status, :string
has_many :old_statuses, OldStatuses
end
This would allow me to keep the history of states with metadata like “when was the status changed” or “… by whom”. What I didn’t like about this approach is the indirection between the current status and the older ones. That’s why I went for a slightly more sophisticated solution of using only the separate table to store the current as well as old statuses of an entity.
So we need to have two separate tables:
create table(:entities) do
add :name, :string
timestamps()
end
create table(:entity_status) do
add :name, :string
add :entity_id, references(:entities)
timestamps()
end
And their related ecto schemas:
# […] entity.ex
schema "entities" do
field :name, :string
field :status, :string, virtual: true
has_many :status_history, EntityStatus
timestamps()
end
# […] entity_status.ex
schema "entity_status" do
field :name, :string
timestamps()
end
The above ecto schemas look quite similar to my initial idea, with the exception of :status
being a virtual field. The more interesting part is saving to the database and retrieving the record again.
You are probably encapsulating your business logic via some kind of module — like a context module in phoenix — with a create_entity/1
function and a update_entity/2
, along with some functions for querying data from the database like get_entity/1
. Let’s start with inserting into the database.
For this part I’ll use Ecto.Multi
since it allows to change multiple tables of data within a single transaction with its nice API. Below is a somewhat complicated example of what you might normally do using a changeset/2
call and Repo.insert
:
def create_job(attrs \\ %{}) do
with %{valid?: true} = ch <- Entity.changeset(%Job{}, attrs),
{:ok, %{job: job}} <- build_multi(:insert, ch) do
{:ok, job}
else
%{valid?: false} = changeset ->
apply_action(changeset, :insert)
{:error, :entity, changeset, _} ->
{:error, changeset}
{:error, _, _, _} ->
Entity.changeset(%Job{}, attrs)
|> add_error(:status, "couldn't be created")
|> apply_action(:insert)
end
end
This part is mostly so complex because I wanted to keep the return value of {:ok, entity} | {:error, changeset}
.
Let’s look at the actual Ecto.Multi
part of the code. This is the part I simply reused for create as well as update tasks. That’s the reason why I did pass the :insert
above when invoking build_multi/2
.
defp build_multi(action, changeset) do
status = get_change(changeset, :status)
Ecto.Multi.new()
|> entity_multi(action, changeset)
|> status_multi(status)
|> Repo.transaction()
end
defp entity_multi(multi, :insert, changeset) do
Ecto.Multi.insert(multi, :entity, changeset)
end
defp entity_multi(multi, :update, changeset) do
Ecto.Multi.update(multi, :entity, changeset)
end
defp status_multi(multi, nil), do: multi
defp status_multi(multi, status) do
Ecto.Multi.run(multi, :status, fn %{entity: entity} ->
Ecto.build_assoc(entity, :status_history)
|> EntityStatus.changeset(%{"name" => status})
|> Repo.insert()
end)
end
This will go ahead and “upsert” the entity — either insert it or update an existing one. Only if the status field did change will a new row of the EntityStatus
struct be added into the db.
The function to update an entity is really the same as with creating one like above. Only make sure to pass the existing struct and to invoke build_multi/2
with :update
.
Now things are stored in the database, but we still need to retrieve them again. As you might know virtual fields can’t be simply fetched from the database like any other column. So we need to make sure this does happen the way we need it to. In a freshly generated phoenix context module or also in lots of other ecto projects one does simply query for a module like so: Repo.get(Entity, id)
. All those calls to Entity
need to be replaced with a function call to entity_query/0
. This will allow us to fetch the latest status in each of those instances.
defp entity_query do
from entity in Entity,
join: status1 in assoc(entity, :status_history),
left_join: status2 in EntityStatus,
on: (entity.id == status2.job_id) and
(status1.updated_at < status2.updated_at or
(status1.updated_at == status2.updated_at and
status1.id < status2.id)),
where: is_nil(status2.id),
select: %{entity | status: status1.name}
end
This will join the entity with only the most recent status row and it’ll populate the :status
virtual field with the name of the status. The field :status_history
can still be used like any other has_many
field.
The reason for joining the statuses table twice as status1
and status2
is so we are able to compare the many status associations of the entity to select only the most recent one. The left_join
will only join NULL
values in case its conditionals are not meet. The where
clause skips over those rows essentially filtering out all the joined rows of older statuses.
I’ve also noticed people suggesting subqueries to query for the latest status — which should work as well with ecto — but I went with the fancy solution once again. Be sure to try both if you are worried about performance.
So that’s it. We’ve extracted the whole status handling into a fully separate ecto schema, while still keeping the convenience of updating a textual status on the entity itself, so on the frontend one could simply use a select field for status changes.
]]>View
module (see here if you don’t know how), add the related .eex
templates, and it just works. What if you need slightly more customization to render your templates? Maybe you need more than a single “layout” template or you want to render a different partial depending on values in the assigns
map. In such cases you can use the render/2
function in your view module.
If you’ve read the Phoenix Guides you’ve probably already seen examples of the render/2
functions in View
modules. They’re used when JSON data needs to be rendered.
defmodule HelloWeb.PageView do
use HelloWeb, :view
def render("index.json", %{pages: pages}) do
%{data: render_many(pages, HelloWeb.PageView, "page.json")}
end
[…]
end
You can do something similar for your HTML templates as well, but you cannot simply return a map of data like in the above example. You will want to render the template within your customizations. For that you can use the private function render_template/2
(docs) of your View
module.
defmodule HelloWeb.EventView do
use HelloWeb, :view
def render("show.html", %{event_status: status} = assigns) do
template =
case status do
:public -> "public.html"
:presale -> "presale.html"
:past -> "bygone.html"
_ -> "not_public.html"
end
render_template template, assigns
end
end
render_template/2
is the actual function to render a template file. If you don’t add your own render/2
function, then this function is called behind the scenes. Using this function you’re now free to compose how exactly you’d like your template files to be rendered. E.g. in the above example there wouldn’t even need to be a index.html.eex
because the template is replaced by different ones depending on the event’s status.
Another example is the need for composed “layout” template in phoenix. In an imaginary application there’s the admin layout and the customer facing one. Without any customization one would have a app.html.eex
and a admin.html.eex
, which is totally fine. But in a case where both layouts share the whole <head>
setup as well as the linked javascript files changes would need to be duplicated in both files. Ideally there would a wrapper.html.eex
for the outer html and app.html.eex
/ admin.html.eex
would only hold the actually different markup for headers or footers.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<title>Hello!</title>
<link rel="stylesheet" href="<%= static_path(@conn, "/css/app.css") %>">
</head>
<body class="helvetica">
<%= render_template @layout_template, assigns %>
<script src="<%= static_path(@conn, "/js/app.js") %>"></script>
</body>
</html>
<header>App View</header>
<main role="main">
<%= render @view_module, @view_template, assigns %>
</main>
<footer>© SomeApp</footer>
defmodule HelloWeb.LayoutView do
use HelloWeb, :view
@wrappedLayouts ["app.html", "admin.html"]
def render(template, assigns) when template in @wrappedLayouts do
render_template "wrapper.html",
Map.put(assigns, :layout_template, template)
end
end
Here render_template/2
is used within the wrapper.html.eex
to render any template supplied via :layout_template
just like they have been rendered automatically before. But there’s no more need to duplicate the whole <head>
section or any of the appended scripts in those templates. That’s all keep in the single place of the wrapper.html.eex
.
Die Hauptschuld dafür hat — zumindest für meine Anwendungsfälle — vor allem der Zusatzaufwand, der in die Pflege einer solchen Bibliothek fließt. Da viele Styleguide Generatoren auf Node basieren werden in den meisten Fällen auch JavaScript Templating Sprachen verwendet. Eine direkte Verwendung von der PHP Seite ist daher meist nicht direkt möglich. Damit bleibt erstmal nur das manuelle Abgleichen von Library und Live-Umgebung.
Mein zweiter Kritikpunkt an einigen Systemen ist zudem die Datenstruktur. Diese ist meist eher starr aufgebaut, als wäre der Styleguide ein geschlossenes Projekt. Möchte man einen Living Styleguide ist jedoch genau das nicht von Vorteil und man müsste zwischen dem eigentlichen Projekt und dem Styleguide einige Ordner und Strukturen querverlinken. Manche Systeme gehen sogar soweit, dass Templates oder Metadaten komplett als JSON Dateien abgespeichert werden, was eine potentielle externe Nutzung noch komplexer macht.
Vor kurzem bin ich dann auf Fractal gestoßen. Fractal läuft, wie so viele andere Libraries auch, unter Node.js. Somit ist das Problem der Templating Sprache ebenso vorhanden. Der Punkt der mich überrascht hatte war der Aufbau der Dateistruktur dahinter. Fractal geht hier einen großen Schritt weiter als andere Systeme. Während die minimal Version einer Komponente nur aus einer einzigen Template Datei besteht lässt sich die Definition auch zu einem Ordner mit folgenden Dateien ausdehnen, wenn nötig.
├── components
│ ├── _preview.hbs
│ ├── blockquote
│ │ ├── blockquote.config.yml
│ │ ├── blockquote--fancy.hbs
│ │ ├── blockquote.hbs
│ │ ├── blockquote.scss
│ │ ├── modal-quote.js
│ │ ├── screenshot.png
│ │ └── README.md
Diese Struktur ist schon für sich allein sehr übersichtlich und lässt dem Nutzer sehr viele Möglichkeiten eine Komponente umfassend zu beschreiben. Das Frontend setzt hier noch das i-Tüpfelchen, in dem es die verfügbaren Daten noch mit einem angenehmen Interface aufbereitet.
{% primaryImage “/images/blog/fractal-processwire/intro.png”, “” %}
Bei der Installation von Fractal stellt man auch sofort fest, dass hier im Sinne der Flexibilität einiges anderes läuft als bei Alternativen. Eine der ersten Einstellungen, die man trifft: Wo im Dateisystem sollen Komponenten, Dokumentation oder Assets abgelegt werden. Das macht es einfach das System neben z.B. einer CMS oder anderen Projektstruktur zu integrieren und auf bereits bestehende Ordnerstrukturen zurückzugreifen.
Zur Verwendung als Living Styleguide System fehlte nun nur eine Templating Option, die sowohl in Fractal als auch im PHP CMS verfügbar ist. Die einzige Templating Sprache die Node und PHP offiziell supportet ist Mustache, die nicht den Featureumfang bietet den ich benötige.
Geht man nun von PHP aus, dann ist die Templating Sprache der Wahl im Moment eigentlich Twig. Es gibt zwar mit twig.js eine JavaScript Adaption, allerdings ist man hier von davon Abhängig, das Features in beiden Systemen funktionieren. Die Basis-Implementation in Fractal mit twig.js unterstützt zudem keine @component Syntax, die anstelle von Pfadangaben verwendet wird.
Nach einen kurzen Chat mit dem Entwickler von Fractal im eigenen Slack-Channel kam mir dann der Gedanke das man Templates auch direkt in PHP rendern lassen könnte. Ein kurzer Blick auf npm ergab, dass das Packet node-twig genau das tat, fehlte somit nur der Adapter für Fractal den ich inzwischen erstellt habe. Der Adapter ist nun auf GitHub verfügbar, genauso wie das dazugehörige composer Packet.
Nun kommen wir zum Schluss noch zu dem Punkt ProcessWire, der ja schon in der Überschrift steht, bisher aber nicht angesprochen wurde. Nachdem ich fast ausschließlich mit ProcessWire arbeite ist natürlich auch die Beispiel-Anbindung mit ProcessWire umgesetzt. Ich denke es wäre jedoch durchaus auch in anderen Systemen möglich die nötigen Anpassungen zu treffen.
Im folgenden gehe ich von folgender Struktur aus, wobei diese durchaus auch noch an individuelle Bedürfnisse anpassbar ist.
├── project-root
│ ├── docs
│ ├── fractal
│ │ └── docs
│ ├── site
│ │ ├── templates
│ │ │ ├── views
│ │ │ │ └── basic-page.twig
│ │ │ └── basic-page.php
│ │ ├── init.php
│ │ └── fractal-handles.php
│ ├── wire
│ ├── fractal.js
│ ├── composer.json
│ ├── package.json
│ └── index.php
docs/
In diesem Ordner wird die statische Version von Fractal erstellt.
fractal/
Hier kommen die Quelldaten für die Fractal Dokumentation unter, aber auch Dinge wie Veränderungen am Fractal Theme können hier abgelegt sein.
site/templates/views/
Das ist der Ordner in dem alle Komponenten abgelegt sind, wie es oben beschrieben ist. Somit sind die Templates, aber auch die Metadaten auch als Teil des CMS zu verstehen.
site/init.php & site/fractal-handles.php
Hier wird Twig konfiguriert und dafür vorbereitet auch im CMS Kontext mit dem @handle Kontext von Fractal umgehen zu können.
fractal.js
Die Konfigurations-Datei für Fractal.
composer.json & package.json
Sorgen dafür, dass die benötigten Bibliotheken und Adapter installiert werden mit composer install sowie npm install.
Mit diesem Setup hat man in ProcessWire sehr einfach die Möglichkeit auf bereits definierte Komponenten zuzugreifen und diese mit den nötigen Daten zu befüllen.
<?php
echo $twig->render('@basic-page', array(
'title' => $page->title,
'editable' => $page->editable()
);
Gleichzeitig greift Fractal auf die selben Templates zu und rendert diese mit Daten die in den Kontext-Dateien definiert sind.
name: basic-page
context:
title: Hello World
editable: false
Das Setup ermöglicht es alle Teile des Interfaces unabhängig von der eigentlichen Applikation anzusehen und zu verwalten. Das Gestalten der einzelnen Module rückt viel mehr in den Vordergrund und das Backend ist im optimalen Fall rein für die Kombination der bereits bestehenden Komponenten zuständig.
Um das System auch nicht nur in der Theorie sondern auch an einem praktischen Beispiel zu testen hab ich meine private Homepage komplett auf Twig mit Fractal umgestellt und ein wenig mit Reverse-Engineering die einzelnen Komponenten herausgetrennt. Im Idealfall macht man das natürlich nicht nachträglich ;)
{% primaryImage “/images/blog/fractal-processwire/example.png”, “” %}
Mit den Änderungen ist nun mein ProcessWire “Template-File” ein reiner Controller, wie man es von MCV Frameworks kennt und von meinem HTML/PHP Spaghetti Code ist nur ein kleiner Block übrig geblieben, der das oben angezeigte Twig Template mit den richtigen Daten befüllt.
<?php
echo $twig->render('@index', array(
'headline' => $page->headline,
'subline' => $page->subhead,
'body' => $page->body,
'projectsHeadline' => __('Projekte'),
'projects' => $projects,
'footerLinks' => $pages->find("template=contact|impressum, sort=sort")->explode(array('title', 'url')),
'photoUrl' => 'https://foto.kobrakai.de/',
'photoTitle' => __('Meine Fotografien'),
'images' => $images,
'photoJson' => $galleryJSON
));
Alles in Allem bin ich mit dem System bisher super zufrieden. Fractal bietet neben dem oben Beschriebenen noch einiges mehr an Möglichkeiten zur Anpassung und weitere coole Features. Dadurch, dass die Twig Templates nicht direkt aus ProcessWire geladen und gerendert werden ist man total frei darin wie man die nötigen Daten generiert und an welchem Fleck man dann tatsächlich Markup generiert und ausgibt. Man könnte das System somit sogar ohne Probleme Stück-für-Stück in ein bestehendes Projekt einarbeiten oder auch nur für einige Komponenten und den Rest manuell erstellen. Meine nächsten Projekte werden auf jeden Fall in der Kombination starten und dann wird sich herausstellen, wie gut es sich im Arbeitsalltag schlägt.
Links:
Fractal: http://fractal.build/
ProcessWire: http://processwire.com/
Twig-Adapter: https://github.com/LostKobrakai/twig
PHP-Companion: https://github.com/LostKobrakai/frctl-twig
Bootstraper für ProcessWire: https://github.com/LostKobrakai/processwire-fractal