Tip

This article is part of the Agenix series

• Getting Started with Agenix
• Using Agenix with Home Manager
• 👉 Using Agenix with devShells

Nix with Flakes allows you to easily centralize project environments and setup right in your repository. Often, your README will tell the reader "get an API key for FOOSERVICE from a teammate"

Just like NixOS modules and home-manager, we can use Agenix to encrypt and store secrets without exposing them to our git repository.

flake-parts

The easiest way to use Agenix with devShells is to use the agenix-shell module for flake-parts.

Let's start with a basic flake using flake-parts.

{
  description = "Agenix example for devShells";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+   flake-parts.url = "github:hercules-ci/flake-parts";
  };

- outputs = { self, nixpkgs }:
+ outputs = { self, nixpkgs, flake-parts }: 
+   flake-parts.lib.mkFlake {inherit inputs;} {
+     imports = [];
+     systems = ["x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin"];
+     perSystem = { pkgs, ... }: {
+       devShells.default = pkgs.mkShell {
+         packages = [];
+       };
+     };
+   }
+ };
}

Add Agenix and agenix-shell

Lets add flake-parts and the agenix-shell module.

{
  description = "Agenix example for devShells";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
+   agenix.url = "github:ryantm/agenix";
+   agenix-shell.url = "github:aciceri/agenix-shell";
  };

- outputs = { self, nixpkgs, flake-parts }:
+ outputs = { self, nixpkgs, flake-parts, agenix, agenix-shell }: 
    flake-parts.lib.mkFlake {inherit inputs;} {
-     imports = [];
+      imports = [
+        agenix-shell.flakeModules.default
+      ];
      systems = ["x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin"];
-      perSystem = { pkgs, ... }: {
+      perSystem = { pkgs, config, lib ... }: {
        devShells.default = pkgs.mkShell {
-         packages = [];
+         packages = [agenix.packages.${system}.default];
+         shellHook = ''
+           source ${lib.getExe config.agenix-shell.installationScript}
+         '';
        };
      };
    }
  };
}

Configure your secrets recipients

You can reference the section we described last time.

This time, we're going to encrypt a file called api-keys.age.

Create your secrets files

Create your secrets files by running agenix -e <key-name>.age in the same directory as secrets.nix. Fill them in with the relevant values.

# service-a-api-key.age
foo

Use your secrets

{
  description = "Agenix example for devShells";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    agenix.url = "github:ryantm/agenix";
    agenix-shell.url = "github:aciceri/agenix-shell";
  };

  outputs = { self, nixpkgs, flake-parts, agenix, agenix-shell }: 
    flake-parts.lib.mkFlake {inherit inputs;} {
       imports = [
         agenix-shell.flakeModules.default
       ];
      systems = ["x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin"];
+     agenix-shell = {
+       secrets = {
+         SERVICE_A_API_KEY.file = ./service-a-api-key.age;
+         SERVICE_B_API_KEY.file = ./service-b-api-key.age;
+         SERVICE_C_API_KEY.file = ./service-c-api-key.age;
+       };
+     };
       perSystem = { pkgs, config, lib ... }: {
        devShells.default = pkgs.mkShell {
          packages = [agenix.packages.${system}.default];
          shellHook = ''
            source ${lib.getExe config.agenix-shell.installationScript}
          '';
        };
      };
    }
  };
}

Your local project secrets are now available when you enter your devShell with nix develop.

Wrapping Up

Agenix seemed complicated to me at first, but once I got it set up up in the three usual scenarios (NixOS, home-manager, and devShells), its simplicity came to light.

I hope this series helps you take advantage of Agenix as well.

Full Example

flake.nix

{
  description = "Agenix example for devShells";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    agenix.url = "github:ryantm/agenix";
    agenix-shell.url = "github:aciceri/agenix-shell";
  };

  outputs = { self, nixpkgs, flake-parts, agenix, agenix-shell }: 
    flake-parts.lib.mkFlake {inherit inputs;} {
       imports = [
         agenix-shell.flakeModules.default
       ];
      systems = ["x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin"];
      agenix-shell = {
        secrets = {
          SERVICE_A_API_KEY.file = ./service-a-api-key.age;
          SERVICE_B_API_KEY.file = ./service-b-api-key.age;
          SERVICE_C_API_KEY.file = ./service-c-api-key.age;
        };
      };
       perSystem = { pkgs, config, lib ... }: {
        devShells.default = pkgs.mkShell {
          packages = [agenix.packages.${system}.default];
          shellHook = ''
            source ${lib.getExe config.agenix-shell.installationScript}
          '';
        };
      };
    }
  };
}

secrets.nix

let
  local = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP55ETmYHSCjtvDZ/SDoHDTblYZPD2XDmObLMQvc+9xR mitchell@nixos";
  remote = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII5SFrZIaTh42TWQKSXeGRhBZ5CAvJWoJov+eiaUbwxa root@nixos";
in {
  "service-a-api-key.age".publicKeys = [local remote]; 
  "service-b-api-key.age".publicKeys = [local remote]; 
  "service-c-api-key.age".publicKeys = [local remote]; 
}

I think the most we did was manually calling gcc or were given a Makefile that no one comprehended... or was that just me 🤔.

I recently read through Modern C by Jens Gustedt and I'm currently reading through Tiny C Projects by Dan Gookin (did you know he wrote the very first "For Dummies" book??). A little unusual for me, I'm actually trying to code the exercises, which has brought me to C build systems.

I was perfectly fine compiling my tiny projects with clang -Wall greeter.c -o greeter, but since I'm also infected with the Nix mind virus, I wanted to be able to package it up in a Nix flake.

This lead me to a conundrum. I wasn't sure how to share the build configuration between the nix-less project and the Nix flake.

The correct thing to do is to learn CMake and make, but I'm also interested in Zig and planned on doing these tiny C projects in Zig as well, so might as well try it out.

build.zig

This resulting build.zig is more or less the generated file from zig init, minus all the comments and anything related to tests.

Really, all we need to do is add an executable to the build graph, add a C source file, and link libc.

Now, we can run zig build run to run our code in one step!

const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    // the main event 👇
    const exe = b.addExecutable(.{ .name = "chapter2", .target = target, .optimize = optimize });
    exe.addCSourceFile(.{ .file = .{ .src_path = .{ .owner = b, .sub_path = "./main.c" } } });
    exe.linkLibC();

    b.installArtifact(exe);

    const run_cmd = b.addRunArtifact(exe);
    run_cmd.step.dependOn(b.getInstallStep());

    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);
}

Nix

As a bonus, we'll check out the Nix flake. With this, we can run our code in one step with nix run .#chapter2.

flake.nix

{
  description = "Tiny C Projects";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  };

  outputs = {nixpkgs, ...}: let
    inherit (nixpkgs) lib;
    systems = [
      "aarch64-darwin"
      "x86_64-darwin"
      "x86_64-linux"
      "aarch64-linux"
    ];
    perSystem = func: lib.genAttrs systems (system: func {pkgs = nixpkgs.legacyPackages.${system};});
  in {
    packages = perSystem ({pkgs, ...}: {
      chapter2 = pkgs.callPackage ./chapter2/package.nix {};
    });
    devShells = perSystem ({pkgs, ...}: {
      default = pkgs.mkShell {
        packages = with pkgs; [
          zig_0_14
          zls
          clang-tools
          clang
        ];
      };
    });
  };
}

package.nix

Each chapter has its own separate package.nix file.

All you have to do is include zig.hook in nativeBuildInputs. This configures all the standard phases to instead call the relevant Zig commands.

{
  pkgs,
  zig,
  ...
}:
pkgs.stdenv.mkDerivation {
  name = "chapter2";
  version = "0.1.0";
  src = ./.;
  nativeBuildInputs = [
    zig.hook
    pkgs.zig_0_14
  ];
}

The code

You can find my solutions on my GitHub.

Tip

This article is part of the Agenix series

• Getting Started with Agenix
• 👉 Using Agenix with Home Manager
• Using Agenix with devShells

Home Manager (also styled as home-manager) allows you to configure your programs and configuration with Nix code, similar to NixOS modules.

Just like NixOS modules, we can use Agenix to encrypt and store secrets without exposing them to our git repository.

Install home-manager

Let's start with a basic home-manager configuration to our flake from last time (sans the NixOS configuration, for brevity).

{
  description = "Agenix example for home-manager";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+   home-manager = {
+     url = "github:nix-community/home-manager";
+     inputs.nixpkgs.follows = "nixpkgs";
+   };
    agenix.url = "github:ryantm/agenix";
  };

  outputs = {
    self,
    nixpkgs,
+   home-manager
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
+   homeManager."mitchell@nixos" = home-manager.lib.homeManagerConfiguration {
+     inhert pkgs;
+     modules = [];
+   };
  };
}

Agenix homeManagerModule

The Agenix module is responsible for decrypting and installing your secrets at runtime.

{
  description = "Agenix example for home-manager";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    agenix.url = "github:ryantm/agenix";
  };
 
  outputs = {
    self,
    nixpkgs,
    home-manager
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    homeManager."mitchell@nixos" = home-manager.lib.homeManagerConfiguration {
      inhert pkgs;
      modules = [
+       agenix.homeManagerModules.default
      ];
    };
   };
}

Configure your secrets recipients

You can reference the section we described last time.

This time, we're going to encrypt a file called api-keys.age.

Create your secrets file

Create your secrets file by running agenix -e api-keys.age in the same directory as secrets.nix. Fill it with the following environment variables.

export SERVICE_A_API_KEY=foo
export SERVICE_B_API_KEY=bar
export SERVICE_C_API_KEY=baz

Use your secrets

We want these environment variables to be available in our shell, so we're going to export them in our zsh configuration.

Let's start with a basic home-manager module that enables zsh.

{
  description = "Agenix example for home-manager";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    agenix.url = "github:ryantm/agenix";
  };
 
  outputs = {
    self,
    nixpkgs,
    home-manager
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    homeManager."mitchell@nixos" = home-manager.lib.homeManagerConfiguration {
      inhert pkgs;
      modules = [
        agenix.homeManagerModules.default
+       ({...}: {
+         home.stateVersion = "25.11";
+         home.username = "mitchell";
+         home.homeDirectory = "/home/mitchell";
+         programs.zsh = {
+           enable = true;
+           initContent = ''
+           '';
+         }
+       })
      ];
    };
   };
}

Now we can configure our age secret and use its path in our zsh configuration.

{
  description = "Agenix example for home-manager";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    agenix.url = "github:ryantm/agenix";
  };
 
  outputs = {
    self,
    nixpkgs,
    home-manager
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    homeManager."mitchell@nixos" = home-manager.lib.homeManagerConfiguration {
      inhert pkgs;
      modules = [
        agenix.homeManagerModules.default
-       ({...}: {
+       ({config, ...}: {
+          age.secrets = {
+            api-keys.file = ./api-keys.age;
+          };
          programs.zsh = {
            enable = true;
            initContent = ''
+             eval $(cat ${config.age.secrets.api-keys.path})
            '';
          }
        })
      ];
    };
   };
}

Those environment variables are now exported in your .zshrc file!

While being an unconventional use case (for API secrets like these, you would probably use them directly in a derivation for a script or a program), this this fully demonstrates how you can use Agenix with home-manager.

Next Time

Home Manager is a good way to store global machine secrets like this for a particular user, but doesn't allow you to handle project level secrets.

Next time we'll se how we can utilize Agenix with devShells.

Full Example

flake.nix

{
  description = "Agenix example for home-manager";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    agenix.url = "github:ryantm/agenix";
  };
 
  outputs = {
    self,
    nixpkgs,
    home-manager
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    homeConfigurations."mitchell@nixos" = home-manager.lib.homeManagerConfiguration {
      inherit pkgs;
      modules = [
        agenix.homeManagerModules.default
        ({config, ...}: {
          home.stateVersion = "25.11";
          home.username = "mitchell";
          home.homeDirectory = "/home/mitchell";

          age.secrets = {
            api-keys.file = ./api-keys.age;
          };
          programs.zsh = {
            enable = true;
            initContent = ''
              eval "$(cat ${config.age.secrets.api-keys.path})"
            '';
          };
        })
      ];
    };
  };
}

secrets.nix

let
  local = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP55ETmYHSCjtvDZ/SDoHDTblYZPD2XDmObLMQvc+9xR mitchell@nixos";
  remote = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII5SFrZIaTh42TWQKSXeGRhBZ5CAvJWoJov+eiaUbwxa root@nixos";
in {
  "api-keys.age".publicKeys = [local remote]; 
}

Tip

This article is part of the Agenix series

• 👉 Getting Started with Agenix
• Using Agenix with Home Manager
• Using Agenix with devShells

NixOS modules make it easy to configure many services with a consistent interface (the Nix language), but configuring confidential options like passwords and API keys this way has two major problems.

• Secrets should not be committed to source control in plain text.
• Values and files used in NixOS modules are copied to the nix store, which is globally readable.

Agenix is a Nix package and CLI utility that allows you to encrypt files to store secrets in your repository and then be able to decrypt them at runtime, giving access to your services.

The key here is that these secrets are decrypted to temporary directories with limited filesystem permissions and are kept out of the nix store.

Let's see what it looks like to use Agenix in these contexts through a Nix flake.

NixOS Modules

Let's build out an example that configures PiHole as an OCI container whose secrets are provided by an environment file.

Add it as an input

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+   agenix.url = "github:ryantm/agenix";
  };
}

Add the CLI tool to your package in your devShell

The agenix CLI tool is what we use to edit our secrets locally.

Now, you can enter your devShell with nix develop and have access to the agenix utility.

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

- outputs = { self, nixpkgs }: {
+ outputs = { self, nixpkgs, agenix }: let
+   pkgs = nixpkgs.legacyPackages.aarch64-linux;
+ in {
+   devShells.aarch64-linux.default = pkgs.mkShell {
+     packages = [
+       agenix.packages.aarch64-linux.default
+     ];
    };
  };
}

NixOS configuration

Let's add the basic NixOS configuration before adding anything Agenix specific.

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

  outputs = { self, nixpkgs, agenix }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
+   nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
+     system = "aarch64-linux";
+     modules = [
+       ./configuration.nix
+     ];
+   };
}

Add the Agenix module

The Agenix module is responsible for decrypting and installing your secrets at runtime.

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

  outputs = { self, nixpkgs, agenix }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        ./configuration.nix
+       agenix.nixosModules.default
      ];
    };
}

Configure the secrets recipients

Agenix controls who can decrypt which secret with a secrets.nix file. This file contains the public keys used to encrypt the data, which also determines which private keys can decrypt the data.

With Agenix, the decryption will happen with the deployed servers ssh keys that are stored in /etc/ssh, so here we make remote equal to the public key at /etc/ssh/ssh_host_ed25519_key.pub

We also set a local key, which is your local computer's ssh public key. This one is used by the agenix CLI utility to encrypt and decrypt the secrets during development

We then configure the pihole.age file use use these two public keys.

let
  local = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP55ETmYHSCjtvDZ/SDoHDTblYZPD2XDmObLMQvc+9xR mitchell@nixos";
  remote = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII5SFrZIaTh42TWQKSXeGRhBZ5CAvJWoJov+eiaUbwxa root@nixos";
in {
  "pihole.age".publicKeys = [local remote]; 
}

Create your secrets file

Create your secrets file by running agenix -e pihole.age in the same directory as secrets.nix. Fill it with the following environment variables.

FTLCONF_webserver_api_password=password

If you try to read this file without decrypting it, you'll see something like this:

age-encryption.org/v1
-> ssh-ed25519 piHZrQ PuyA20t9WXtsZ7EoFS2gYLOgIsDwxtf4eC3nObAReC4
QxeIZNSg8wOzdVAFWKWgFWiXsYpdNRfGLx8wUSP/qDk
-> ssh-ed25519 rglX5A Y1yMBwhebYIL4feWALoFykp0WIWC8hsMtEVQDCgRoyo
zHbuGsYqgdGDdWjiuEjjgEift36XAEksGPAIYbsQnQc
--- w9m5TVJois69mM1HFXbPdmd9zp4DbzYEnmctmU3zGXA
�s�ކ�ٙq���J�d��M�
����Iț����k��r��������!��H/�����M"ҧ2�,W`fs

Pass the secrets path to a service

Now that we have an encrypted secrets file, we can give the path to the (eventually) decrypted file to the NixOS module. Let's add our PiHole service to our configuration, then add the secrets path.

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

  outputs = {
    self,
    nixpkgs,
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";

      modules = [
        ./configuration.nix
        agenix.nixosModules.default
+       ({...}: {
+         virtualisation.oci-containers.containers = {
+           pi-hole = {
+             image = "pihole/pihole:latest";
+             volumes = [
+               "/var/lib/pihole:/etc/pihole"
+             ];
+             hostname = "pihole";
+             ports = [
+               "53:53/tcp"
+               "53:53/udp"
+               "3001:80/tcp"
+               "443:443/tcp"
+             ];
+           };
+         };
+       })
      ];
    };
  };
}

Now we can tell the Agenix NixOS module about our encrypted files

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

  outputs = {
    self,
    nixpkgs,
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";

      modules = [
        ./configuration.nix
        agenix.nixosModules.default
-       ({...}: {
+       ({config, ...}: {
+         age.secrets = {
+           pihole.file = ./pihole.age;
+         };
          virtualisation.oci-containers.containers = {
            pi-hole = {
              image = "pihole/pihole:latest";
              volumes = [
                "/var/lib/pihole:/etc/pihole"
              ];
+             environmentFiles = [
+               config.age.secrets.pihole.path
+             ];
              hostname = "pihole";
              ports = [
                "53:53/tcp"
                "53:53/udp"
                "3001:80/tcp"
                "443:443/tcp"
              ];
            };
          };
        })
      ];
    };
  };
}

Next Time

We built a NixOS module that consumes runtime secrets through a decrypted file that we keep encrypted at rest in our project.

Next time we'll see how we can utilize Agenix with home-manager!

Full Example

flake.nix

{
  description = "Agenix example for NixOS";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    agenix.url = "github:ryantm/agenix";
  };

  outputs = {
    self,
    nixpkgs,
    agenix,
  }: let
    pkgs = nixpkgs.legacyPackages.aarch64-linux;
  in {
    devShells.aarch64-linux.default = pkgs.mkShell {
      packages = [
        agenix.packages.aarch64-linux.default
      ];
    };
    nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";

      modules = [
        ./configuration.nix
        agenix.nixosModules.default
        ({config, ...}: {
          age.secrets = {
            pihole.file = ./pihole.age;
          };
          virtualisation.oci-containers.containers = {
            pi-hole = {
              image = "pihole/pihole:latest";
              volumes = [
                "/var/lib/pihole:/etc/pihole"
              ];
              environmentFiles = [
                config.age.secrets.pihole.path
              ];
              hostname = "pihole";
              ports = [
                "53:53/tcp"
                "53:53/udp"
                "3001:80/tcp"
                "443:443/tcp"
              ];
            };
          };
        })
      ];
    };
  };
}

secrets.nix

let
  local = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP55ETmYHSCjtvDZ/SDoHDTblYZPD2XDmObLMQvc+9xR mitchell@nixos";
  remote = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII5SFrZIaTh42TWQKSXeGRhBZ5CAvJWoJov+eiaUbwxa root@nixos";
in {
  "pihole.age".publicKeys = [local remote]; 
}

On the Elixir community

Reflecting on this year's conference, I realized how comfortable I was the entire time. After 7 years in the Elixir community, I've accumulated a number for friends and acquaintances, something I really never expected from a work adjacent interest like a programming language.

At my first Elixir conference, Lonestar Elixir 2020 (rough year, but also the year I "went pro" with Elixir), I had a couple of acquaintances from my new job at Bleacher Report, but I hadn't really met them yet. I'm a pretty social guy, but traveling across the country to hang out with a bunch of strangers isn't really my comfort zone.

Luckily, I met folks like Todd Resudek who go out of their way to be friendly and introduce you to people they know. I've tried to embody this spirit myself at social gatherings ever since, and Code BEAM America 2025 was no exception.

My talk

This year I gave an update on our progress with the unified language server project.

We are a little more than 6 months into the project and it felt great to finally share how it's going and what to expect.

I spoke to a number of folks on what they're excited about with the project and had a blast exploring the possibilities of the next generation of Elixir tooling.

You can watch my talk on YouTube.

The talks

I thoroughly enjoyed the talks I attended this year (I can admit I do enjoy the hallway track as well...) with a couple of notable talks.

Jason Axelson - Choosing an effective testing structure

Jason Axelson gave a presentation on testing practices from which I almost broke neck from aggressively nodding the entire time. Watch it on YouTube!

One of his suggestions I believe is the most powerful: "Make your tests async" (paraphrased, I didn't write it down because of the aforementioned head banging).

The rationale is basically that by making your tests async, you are forced to iron out a lot of the concurrency kinks in your programs and it helps you really understand your codebase from the bottom up. Improving your own understanding of your codebases will pay dividends when it comes time to handle the unfortunate production incident or training juniors/new teammates.

Later on, I shared with Jason an example repository I had made which demonstrates how to structure a Phoenix application that deals with singleton GenServers (global state) and still maintain async tests.

PS: Isn't it cool that at a conference you can watch a presentation and then later eat lunch (or drink beer) next to them and discuss it??

Jeffrey Matthias and Aidan Obley - Crafting Fully Custom Code Generators

In this talk, Jeffrey and Aidan cover how they use custom generators at Mechanical Orchard to help rebuild legacy programs in Elixir.

In their talk, Jeffrey notes (paraphrased) "Remember, this is your app, you can name the folders what you want and place the files where you'd like".

I think this is an important thing to realize. I love that our community and frameworks have conventions, but sometimes I feel that folks become blinded to what they're "allowed" to do.

The default Phoenix generators might have a certain naming scheme or directory structure, but that doesn't mean it's the ordained way to structure your program (it's mostly superficial anyway.)

See ya next time

I had a blast and I hope to catch ya at the next Code BEAM 👋

This is the culmination of the efforts from Jonatan Kłosko (of LiveBook), Steve Cohen (of Lexical), Łukasz Samson (of ElixirLS), and myself (Mitchell Hanberg of Next LS).

You can read more about it on elixir-lang.org.

We are currently working on merging the three projects (Lexical, ElixirLS, and Next LS) into a single official project, so in the meantime, you can continue to use the language server you are currently using.

If you would like to support my efforts on this project, please sponsor me on GitHub! I have a goal of reaching 100 sponsors, and as of writing I'm a little more than halfway there.

Sponsor link: https://github.com/sponsors/mhanberg

If you sponsor and would like an elixir-tools sticker, please email me your name and address and I'll send one your way (even internationally!)

Schematic

Schematic is a library for validating and transforming data in Elixir, similar to Ecto Changesets and Norm.

I created Schematic early in 2023 while developing the GenLSP library, which I developed to build Next LS. I needed to be able to consume and produce data structures described by the Language Server Protocol JSON Schema specification, which describes data in terms of basic scalar types and certain compound types, similar to some algebraic data types.

I wanted a library that lends itself to expressiveness, composition, and code generation.

Basic schematics

Schematic provides... schematics for basic primitive types in Elixir.

import Schematic

unify(int(), 1)
#=> {:ok, 1}

unify(int(), "one")
#=> {:error, "expected an integer"}

These are just functions, so you can bind them to variables and return them from functions.

defmodule Numbers do
  import Schematic

  def integer(), do: int()

  def validate(value), do: unify(integer(), value)
end

Numbers.validate(1)
#=> {:ok, 1}

Numbers.validate("one")
#=> {:error, "expected an integer"}

List and Tuple schematics

We can also define schematics for lists and tuples.

defmodule Hobbies do
  import Schematic
    
  def validate(values), do: unify(list(str()), values)
end

Hobbies.validate(["foosball", "cooking"])
#=> {:ok, ["foosball", "cooking"]}

Hobbies.validate([1, "cooking"])
#=> {:error, [error: "expected a string", ok: "cooking"]}

Hobbies.validate("foosball")
#=> {:error, "expected a list"}

Map and Struct schematics

Map and struct (or schema in Schematic parlance) schematics are versatile and extendable.

defmodule Animals do
  import Schematic

  def schematic() do
    map(%{
      species: str(),
      genus: str(),
      color: str()
    })
  end

  defmodule Cat do
    def schematic() do
      map(Animals.schematic(), %{
        breed: str(),
        declawed: bool()
      })
    end
  end
end

Schematic.unify(Animals.schematic(), %{})
#=> {:error,
# %{
#   color: "expected a string",
#   species: "expected a string",
#   genus: "expected a string"
# }}

Schematic.unify(Animals.schematic(), %{color: "black"})
#=> {:error, %{species: "expected a string", genus: "expected a string"}}

Schematic.unify(Animals.schematic(), %{color: "black", species: "lupus", genus: "canis"})
#=> {:ok, %{color: "black", species: "lupus", genus: "canis"}}

Schematic.unify(Animals.Cat.schematic(), %{color: "orange", species: "catus", genus: "felis"})
#=> {:error, %{breed: "expected a string", declawed: "expected a boolean"}}

Schematic.unify(Animals.Cat.schematic(), %{
  color: "orange",
  species: "catus",
  genus: "felis",
  breed: "shorthair",
  declawed: false
})
#=> {:ok,
# %{
#   color: "orange",
#   species: "catus",
#   genus: "felis",
#   breed: "shorthair",
#   declawed: false
# }}

Structs can be created from plain maps using the schema schematic. By default it looks for string keys and converts them to atom keys, but that can be disabled using the convert: false option.

defmodule Animals do
  import Schematic

  defstruct [:species, :genus, :color]

  def schematic() do
    schema(__MODULE__, %{
      species: str(),
      genus: str(),
      color: str()
    })
  end
end

Schematic.unify(Animals.schematic(), %{"species" => "lupus", "genus" => "canis", "color" => "grey"})
#=> {:ok, %Animals{species: "lupus", genus: "canis", color: "grey"}}

Optional keys and nullable fields

Optional keys and nullable values can be specified using the optional and nullable schematics. Combining this with a struct, you can create default values for certain keys.

optional keys do not have to be present, but if they are, the value must unify with the given schematic.

defmodule Animals do
  import Schematic

  defstruct [:species, :genus, color: "brown"]

  def schematic() do
    schema(__MODULE__, %{
      optional(:color) => str(),
      species: str(),
      genus: nullable(str())
    })
  end
end

Schematic.unify(Animals.schematic(), %{
  "species" => "lupus",
  "genus" => nil,
})
#=> {:ok, %Animals{species: "lupus", genus: nil, color: "brown"}}

Tranforming keys

While schema will automatically convert string to atom keys, you can also use a tuple for the key specification for key transformation like camelCase to snake_case.

defmodule JobPosting do
  import Schematic

  def schematic() do
    map(%{
      {"startDate", :start_date} => str(),
      optional({"salaryRange", :salary_range}) => str(),
      {"title", :title} => str()
    })
  end
end

Schematic.unify(JobPosting.schematic(), %{
  "startDate" => "Jan 1, 2025",
  "title" => "Chicken Tender Engineer"
})
#=> {:ok, %{title: "Chicken Tender Engineer", start_date: "Jan 1, 2025"}}

You can also use the dump function to transform keys in reverse.

Schematic.dump(JobPosting.schematic(), %{
  title: "Chicken Tender Engineer",
  start_date: "Jan 1, 2025"
})
#=> {:ok, %{"startDate" => "Jan 1, 2025", "title" => "Chicken Tender Engineer"}}

Advanced schematics

Most of your data is rather complex and can be specified further than just a "string", you might have an enumeration, you might actually say a value can be either a Mammal or a Reptile, or convert an ISO timestamp into an Elixir DateTime struct.

oneof

If you want to say a value is "one of" a list of schematics, you can use the oneof schematic. I believe the semantics are similar to an enum or union type from other languages.

In this example, we also demonstrate using literals as schematics.

defmodule HousePet do
  import Schematic

  def schematic() do
    map(%{
      name: str(),
      type:
        oneof([
          "Dog",
          "Cat",
          "Hamster",
          "Fish"
        ])
    })
  end
end

Making an enum of strings is nice, but for a proper union type style schematic, we can use other schematics, even map schematics.

defmodule HousePet do
  import Schematic

  def dog, do: map(%{type: "dog"})
  def cat, do: map(%{type: "cat"})
  def hamster, do: map(%{type: "hamster"})
  def fish, do: map(%{type: "fish"})

  def schematic() do
    oneof([
      dog(),
      cat(),
      hamster(),
      fish()
    ])
  end
end

Schematic.unify(HousePet.schematic(), %{type: "cat"})
#=> {:ok, %{type: "cat"}}

Unfortunately the error message for these kind of failure case is not very good, but will be improved in a future version Schematic.

Schematic.unify(HousePet.schematic(), %{type: "snake"})
#=> {:error, "expected either a map, a map, a map, or a map"}

Value based validations

So far we've covered more structural style of data validation, but we can also do more value based validations that you are probably used to in your Ecto.Changeset code.

We can use the all and raw schematics to accomplish this!

defmodule SpecialNumber do
  def schematic do
    all([
      int(),
      raw(&Kernel.<(&1, 10), message: "must be less than 10"),
      raw(&(Kernel.rem(&1, 2) == 0), message: "must be divisible by 2")
    ])
  end
end

Schematic.unify(SpecialNumber.schematic(), 8)
#=> {:ok, 8}

Schematic.unify(SpecialNumber.schematic(), 15)
#=> {:error, ["must be less than 10", "must be divisible by 2"]}

Schematic.unify(SpecialNumber.schematic(), "15")
#=> {:error, ["expected an integer", "must be less than 10", "is invalid"]}

Transforming Data

You can also use the raw schematic to transform the data as you parse and validate it. Here we read a ISO8601 timestamp and turn it into a DateTime struct.

defmodule Datetime do
  import Schematic

  def schematic() do
    raw(
      fn
        i, :to -> is_binary(i) and match?({:ok, _, _}, DateTime.from_iso8601(i))
        i, :from -> match?(%DateTime{}, i)
      end,
      transform: fn
        i, :to ->
          {:ok, dt, _} = DateTime.from_iso8601(i)
          dt

        i, :from ->
          DateTime.to_iso8601(i)
      end
    )
  end
end

Schematic.unify(Datetime.schematic(), "2024-07-11T17:48:41.972851Z")
#=> {:ok, ~U[2024-07-11 17:48:41.972851Z]}

Dumping Data

Not only can you parse and validate external data into your internal format, you can also dump that data back into the external format.

This will respect any map key transformations that you've declared and as seen above, you can use a raw schematic to arbitrarily control how the data is transformed in each direction.

Schematic.dump(Datetime.schematic(), ~U[2024-07-11 17:48:41.972851Z])
#=> {:ok, "2024-07-11T17:48:41.972851Z"}

Schematic.dump(Animals.schematic(), %Animals{
  species: "lupus",
  genus: "canis",
  color: "grey"
})
#=> {:ok, %{"color" => "grey", "genus" => "canis", "species" => "lupus"}}

Future features

While the oneof schematic handles "union" types (typically represented like Dog | Cat), I would like to add "intersection" types (represented sometimes like Dog & Cat).

Conclusion

I am pretty happy with what I've come up with, and it works for my use cases very well!

You can see Schematic in action in the gen_lsp code base and for an example of how it works with code generation you can check out the lsp_codegen project.

Happy hacking!

btw

I update this periodically as I find new things things that bother me.

noun informal

something that a particular person finds especially annoying: one of my biggest pet peeves is poor customer service.

Here's a list of minor things I find irritating.

• Calling software "dead" or "abandoned"
• Complaining about too many choices, especially when C=2
• Complaining about what other people are doing when it doesn't affect you at all
• Judgment without context
• Speaking in a way that implies you have a ton of experience with something (saying "I do this in all my apps") when really you've done it two times
• The word "just"

We're not really in the market to create an actual "distribution", but we are going to explore how to extract your Neovim configuration into it's own plugin.

Other real distributions actually employ this trick themselves like LazyVim and AstroVim, the main requirement is that you use the lazy.nvim package manager.

Let's get into the code!

Converting your current configuration

Converting your configuration into a plugin is mostly just renaming some files/directories and moving them into their own repo.

Original Configuration

Your original configuration might look something like this, you have a folder that handles your lazy.nvim plugin specs, some custom modules, some ftplugins, and a normal init.lua.

$HOME/.config/nvim/
└── lua/
    ├── custom/
    │   ├── plugins/
    │   │   ├── init.lua
    │   │   ├── elixir.lua
    │   │   └── lsp.lua
    │   ├── terminal.lua
    │   └── treesitter.lua
    ├── ftplugin/
    │   ├── elixir.lua
    │   └── javascript.lua
    └── init.lua

Plugin based Configuration

The steps I took to extract my distribution was to:

• Move the whole thing to a new git repository.
• Rename custom to mydistro and resolve any necessary changes.
• Rename init.lua to plugin/mydistro.lua.

Your new configuration structure should look like this:

$HOME/
├── .config/nvim/
│   └── init.lua
└── mydistro/
    └── lua/
        ├── custom/
        │   ├── plugins/
        │   │   ├── init.lua
        │   │   ├── elixir.lua
        │   │   └── lsp.lua
        │   ├── terminal.lua
        │   └── treesitter.lua
        ├── ftplugin/
        │   ├── elixir.lua
        │   └── javascript.lua
        └── plugin/
            └── mydistro.lua

init.lua

The init.lua file in your dotfiles should look like this:

-- bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.uv.fs_stat(lazypath) then
  vim
    .system({
      "git",
      "clone",
      "--filter=blob:none",
      "https://github.com/folke/lazy.nvim.git",
      "--branch=stable", -- latest stable release
      lazypath,
    })
    :wait()
end

vim.opt.rtp:prepend(lazypath)
require("lazy").setup {
  spec = {
    {
      "myname/mydistro", -- the location on GitHub for our distro
      dev = true, -- tells lazy.nvim to actually load a local copy
      import = "mydistro.plugins" -- the path to your plugins lazy plugin spec
    },
  },
  dev = { path = "~" }, -- the path to where `dev = true` looks for local plugins
  install = {
    missing = true,
  },
}

And we're done!

Why?

Well, one: for fun!

And two: I personally use home-manager to manage my dotfiles, but that requires running home-manager anytime you change them.

This is very annoying when it comes to tweaking your Neovim configuration, so moving my configuration to a plugin that lives outside of home-manager means I can iterate quicker.

Another hypothetical use case is making it easier for someone to try out your Neovim configuration. This could be for someone getting into Neovim for the first time, or perhaps a plugin author trying to help debug an issue you're having.

Happy hacking!

In Neovim, you must create this manually, but it is very easy using autocmds.

-- 1
vim.api.nvim_create_autocmd("LspAttach", {
  group = vim.api.nvim_create_augroup("lsp", { clear = true }),
  callback = function(args)
    -- 2
    vim.api.nvim_create_autocmd("BufWritePre", {
      -- 3
      buffer = args.buf,
      callback = function()
        -- 4 + 5
        vim.lsp.buf.format {async = false, id = args.data.client_id }
      end,
    })
  end
})

1. We create a new autocmd that fires on the LspAttach event. This event is fired when an LSP client attaches to a buffer. In this autocmd, you can easily set configuration that is specific to that buffer and LSP client.
2. We create another autcmd inside the LspAttach callback, this time for the BufWritePre event. This fires when you save the buffer but before it flushes anything to disk. This gives you a chance to manipulate the buffer first.
3. We configure this autocmd to only execute for the current buffer. This is a little more straight forward rather than setting an autocmd that executes for any range of file types.
4. In the callback, we run vim.lsp.buf.format to format the buffer, with the flag async = false. This ensures that the formatting request will block until it completes, so that it completely finishes formatting before flushing the file to disk.
5. We also set the id = args.data.client flag so that the formatting request is only sent to the LSP client that is related to the outer LspAttach request. This ensures that we aren't running the formatting request multiple times in case the buffer is attached to multiple LSPs.

And there we have it, modern format on save for those who want it.

The way I have been using it mostly has been through an SSH connection, even though its sitting underneath my desk, plugged into my monitor, and all my peripherals plugged into a nifty USB hub splitter thing!

I simply couldn't live without the macOS desktop environment. I have so much muscle memory for all the shortcuts and have so many apps that I use that enhance my workflow.

I have never remotely developed for this long before, so I quickly ran into a bunch of paper cuts for which I was able to easily find bandaids.

Tailscale

When you are going to be SSHing into a remote computer, you need to know the IP address of the computer.

Tailscale makes this super easy. You install Tailscale on your local computer and on the remote computer, and they both join your "tailnet".

Now, you have a static IP address that only other computers on your tailnet can access!

$ ssh mitchell@<remote ip>

Welcome to Ubuntu 23.10 (GNU/Linux 6.5.0-10-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

6 updates can be applied immediately.
To see these additional updates run: apt list --upgradable

*** System restart required ***
Last login: Tue Nov  7 07:24:00 2023 from <local ip>

Clipboard

You'll want to make sure that you can still copy/paste to/from your host computer's clipboard (or 'pasteboard' as macOS calls it) from within TUI applications like Vim/Tmux and in shell scripts.

TUI Apps

If you are using a modern terminal emulator (I use Ghostty), you most likely have clipboard working already for TUI applications, as long as they support the appropriate terminal features.

I use Neovim, which does not currently support the OSC-52 terminal feature that enables system clipboard communication, but I use Neovim inside tmux, which does!

And luckily, Neovim has a tmux 'clipboard provider' which is how Neovim actually communicates to the "outside" clipboard.

If you don't use tmux and use Vim/Neovim, there are plenty of plugins that implement it!

Scripting

The way that I have implemented scripting to my local clipboard is to again use SSH!

Since your computers are using Tailscale and are on the same tailnet, this means that your local computer also has a nice and static IP address that you can use.

Here is an example:

$ ssh mitchell@<local ip> pbpaste \
  | some-local-command \
  | ssh mitchell@<local ip> pbcopy

Here, we remotely run the pbpaste command (which, along with pbpaste, are the macOS scripting utilities for using the system clipboard), pipe the ouput into a local command, and then pipe it into ssh, which will send it to the pbcopy command on our local computer.

Now, you can easily wrap that up in an alias or a shell script.

Opening your web browser

If you are used to opening web pages with utilities like the GitHub CLI gh, you'll notice that those don't work anymore. This is because (on Linux), the tool is (most likely) running xdg-open, which is similar to the open command on macOS.

You are running this on the remote computer, which is not logged into a desktop environment, and has no web browser.

Well, we can easily fix this again with SSH!

What I did was write a shell script called xdg-open and put it in my $PATH. This way, tools like gh will actually call my script instead of the built-in xdg-open.

My script looks like this:

#!/usr/bin/env bash

function main() {
	local ip
	local uri

	uri="$1"
	if [[ ! -z "$SSH_CONNECTION" ]]; then
		ip="$(echo "$SSH_CONNECTION" | awk '{ print $1 }')"

		ssh "mitchell@$ip" open "$uri"
	else
		xdg-open "$uri"
	fi
}

main "$@"

We check the $SSH_CONNECTION environment variable to see if we are in an SSH session.

If we are not, we just call xdg-open as usual. This is helpful in case we log onto our PC like a normal person.

If we are, then we parse our local IP address out of the $SSH_CONNECTION variable and run the open command remotely using ssh.

Web Development

Now, if you are a web developer, you typically will be starting a local web server and previewing your site or app in the browser. This gets a little tricky when your browser and web server are not on the same computer!

Tailscale

Luckily, tailscale comes to the rescue again!

With tailscale, you can create a reverse proxy served over https with your existing free plan.

$ tailscale serve https / http://localhost:4000

This will create a reverse proxy that is only available inside your tailnet!

If you want to show off your beautiful website to a client or coworker, you can run tailscale funnel 443 on, which will make your reverse proxy available outside of your tailnet.

SSH Port Forwarding

You can also just tack on some options to the ssh command instead of using Tailscale

$ ssh mitchell@<remote ip> -L 4999:localhost:4999

Signing Git Commits

I sign my git commits using the 1Password SSH agent. This allows me to use an SSH key (instead of a GPG key) and to authorize the usage of the key with Touch ID on my Mac.

As you can imagine, when I went commit my first code on my new PC, I thought I had found a roadblock kill this new workflow.

Luckily, you can actually forward your SSH agent when you make and SSH connection. This is typically for when you are machine hopping and have to ssh to a remote machine from inside an ssh connection, so that you don't have to install your keys on your local computer and the first remote computer.

And, this also works for 1Password Git Commit signing!

Just add an entry in your ~/.ssh/config for your PC to allow forwarding on your local computer:

Host <remote ip>
  ForwardAgent yes

And on the PC, configure the IdentityAgent to use 1Password. I don't actually remember if this is necessary, but when writing this post, I found this config and I don't remember writing it, so it must be important!

Host *
  IdentityAgent ~/.1password/agent.sock

Conclusion

And there we have it!

Now you should be able to develop remotely and not even be able to tell the difference, except for the sheer speed of your overpowered new PC!

You can check out the video below, let me know what you think!

I gave a presentation on gen_lsp and writing OTP process abstractions at CodeBEAM America 2022. You can watch my talk on YouTube.

Today I'd like to announce the release of the first language server built with gen_lsp.

Credo Language Server

Credo Language Server is a persistent server that communicates with text editors via the Language Server Protocol.

This initial release comes with project wide diagnostics and a code action to add a # credo:disable-for-next-line magic comment above any Credo warning.

You can install it today using one of the elixir-tools family of editor extensions.

elixir-tools

elixir-tools is the new home for Credo Language Server and the aforementioned editor extensions.

elixir-tools.nvim

If you write Elixir and use Neovim, there is a chance you are already using my plugin, elixir-tools.nvim (formerly known as elixir.nvim).

The Neovim plugin has been renamed to help it better fit into it's place in the ecosystem and has given rise to the new elixir-tools family.

elixir-tools.vscode

The elixir-tools family keeps on growing!

The release of Credo Language Server naturally mean that there would need to a way to use it with Visual Studio Code, so elixir-tools.vscode was born.

This initial release brings support for Credo Language Server and Elixir filetype and highlighting support.

ElixirLS support has been omitted since it can happily coexist along side elixir-tools.vscode. If there is demand to add ElixirLS support, that can be done, but at this time there is no need.

Slow and Steady Wins the Race

Over the last 10 months I have been slowly chipping away at this project, making sure that every part is built with excellence in mind.

As I built out gen_lsp, I realized that the best way to achieve correctness was to generate most of the code from the official specification.

So, I built the lsp_codegen library.

It includes a handwritten generator that conforms to the LSP specification's JSON Schema and is used to read the LSP metamodel and to generate Elixir code that includes typespecs and structs for all the data structures.

While building the code generator, I realized that the specification's "Or" type was easy to represent using typespecs, but was actually hard to deserialize from a JSON payload into the Elixir data structures.

Inspired by norm and my friend Chris Keathley, I built schematic. This allows me to fully validate, serialize, and deserialize complex data structures.

So, to build Credo Language Server, it just took 3 libraries, 2 editor extensions, and a conference talk. 😅

I will be writing more about gen_lsp and schematic in the future.

What's Next

This is just the beginning! 🤗

I have some exciting ideas planned and can't wait to be able to share them with the Elixir community. If you'd like to stay on the bleeding edge of elixir-tools, feel free to join the Discord.

lazyasdf is my first real¹ attempt at making a TUI with Elixir!

asdf is normally used through a command line interface (CLI), lazyasdf presents you with a terminal user interface (TUI) for working with asdf.

I recently fell in love with lazygit and have since dreamed of writing my own TUI programs, but with Elixir.

The TUI provides a quick and intuitive interface for those familiar with the terminal and for those who prefer a graphical application, but the TUI is so much more approachable in my humble opinion when it comes to making your own 😄.

While I find lazyasdf to be an amazing achievement for myself, it isn't super interesting on its own. Let's dive into the specifics of how I was able to build and distribute a TUI application with Elixir.

Ratatouille

None of this would be possible if it weren't for the library ratatouille by Nick Reynolds.

I am not some genius when it comes to terminals or laying out text, this all comes from Ratatouille, which builds off of termbox, which is a [n]curses alternative.

Ratatouille leverages the Elm Architecture of which many of us have grown familiar. Let's take a look at a small Ratatouille program that showcases most of its features.

#!/usr/bin/env elixir

Mix.install([:ratatouille])

defmodule Todos do
  @behaviour Ratatouille.App

  import Ratatouille.View
  import Ratatouille.Constants, only: [color: 1, key: 1]

  @style_selected [
    color: color(:black),
    background: color(:white)
  ]

  @space key(:space)

  @impl true
  def init(_) do
    %{
      todo: %{
        items: %{"buy eggs" => false , "mow the lawn" => true, "get a haircut" => false},
        cursor_y: 0
      }
    }
  end

  @impl true
  def update(model, msg) do
    case msg do
      {:event, %{key: @space}} ->
        {todo, _done} = Enum.at(model.todo.items, model.todo.cursor_y)

        update_in(model.todo.items[todo], & !&1)

      {:event, %{ch: ?j}} ->
        update_in(model.todo.cursor_y, &cursor_down(&1, model.todo.items))

      {:event, %{ch: ?k}} ->
        update_in(model.todo.cursor_y, &cursor_up/1)

      _ ->
        model
    end
  end

  defp cursor_down(cursor, rows) do
    min(cursor + 1, Enum.count(rows) - 1)
  end

  defp cursor_up(cursor) do
    max(cursor - 1, 0)
  end

  @impl true
  def render(model) do
    view do
      panel title: "TODO" do
        for { {t, done}, idx } <- Enum.with_index(model.todo.items) do
          row do
            column size: 12 do
              label if(idx == model.todo.cursor_y, do: @style_selected, else: []) do
                text content: "- ["
                done(done)
                text content: "] #{t}"
              end
            end
          end
        end
      end
    end
  end

  defp done(true), do: text(content: "x")
  defp done(false), do: text(content: " ")
end

Ratatouille.run(Todos)

You should be able to copy the above snippet into a file, make it executable (chmod + x) and run it!

Ratatouille calls for 3 callbacks in your TUI program, init/1, update/2, and render/1.

• When the program boots up, the init/1 callback is called and the return value becomes your initial model state.

• Whenever the TUI receives user input, the update/2 callback is executed with the message and your current model state.

• When that returns, the runtime will call the render/1 callback with the new model state.

  The render/1 callback is full of macros which translate to element structs, so it's just an ergonomic DSL. Typing out many structs by hand would be a PITA!

Notes

You have probably observed that, while it is high level compared to raw termbox, Ratatouille is still sort of "low level" as an application framework.

We still have to manually track and move our cursor position, as well as index into our data structures to pull out the right data for that position.

Burrito

Now the normal problem with Elixir apps is that you have to have Elixir and Erlang on your machine to run them, as well as keep track of the version of them you have installed to make sure they are compatible, as well as write aliases to run escripts and Mix tasks, yada yada.

This is where Burrito comes in!

Burrito utilizes Zig to bundle up your application, the BEAM, and the Runtime all into one tidy executable that you can distribute at your leisure!

In the end, once we run MIX_ENV=prod mix release, Burrito will create binaries for each of our specified target platforms, and you can just copy those onto your computer and run them

The Burrito project is lead by Digit.

Homebrew

To make any program useful, it is help to be able to install it easily.

Homebrew is the primary way of accomplishing this on MacOS (my preferred operating system) and you can easily host your own collection of Homebrew packages with your own Tap!

Since lazyasdf has some quirky dependencies, the formula (what Homebrew calls a package definition) is a little interesting.

class Lazyasdf < Formula
  desc "TUI for the asdf version manager"
  homepage "https://github.com/mhanberg/lazyasdf"
  url "https://github.com/mhanberg/lazyasdf/archive/refs/tags/v0.1.1.tar.gz"
  sha256 "787da19809ed714c569c8bd7df58d55d7389b69efdf1859e57f713d18e3d2d05"
  license "MIT"

  bottle do
    root_url "https://github.com/mhanberg/homebrew-tap/releases/download/lazyasdf-0.1.1"
    sha256 cellar: :any_skip_relocation, monterey: "f489e328c19954d62284a7154fbc8da4e7a1df61dc963930d291361a7b2ca751"
  end

  depends_on "elixir" => :build
  depends_on "erlang" => :build
  depends_on "gcc" => :build
  depends_on "make" => :build
  depends_on "python@3.9" => :build
  depends_on "xz" => :build

  depends_on "asdf"

  on_macos do
    on_arm do
      resource "zig" do
        url "https://ziglang.org/download/0.10.0/zig-macos-aarch64-0.10.0.tar.xz"
        sha256 "02f7a7839b6a1e127eeae22ea72c87603fb7298c58bc35822a951479d53c7557"
      end
    end

    on_intel do
      resource "zig" do
        url "https://ziglang.org/download/0.10.0/zig-macos-x86_64-0.10.0.tar.xz"
        sha256 "3a22cb6c4749884156a94ea9b60f3a28cf4e098a69f08c18fbca81c733ebfeda"
      end
    end
  end

  def install
    zig_install_dir = buildpath/"zig"
    mkdir zig_install_dir
    resources.each do |r|
      r.fetch

      system "tar", "xvC", zig_install_dir, "-f", r.cached_download
      zig_dir =
        if Hardware::CPU.arm?
          zig_install_dir/"zig-macos-aarch64-0.10.0"
        else
          zig_install_dir/"zig-macos-x86_64-0.10.0"
        end

      ENV["PATH"] = "#{zig_dir}:" + ENV["PATH"]
    end

    ENV["PATH"] = (Formula["python@3.9"].opt_libexec/"bin:") + ENV["PATH"]

    system "mix", "local.hex", "--force"
    system "mix", "local.rebar", "--force"

    ENV["BURRITO_TARGET"] = if Hardware::CPU.arm?
      "macos_m1"
    else
      "macos"
    end

    ENV["MIX_ENV"] = "prod"
    system "mix", "deps.get"
    system "mix", "release"

    if OS.mac?
      if Hardware::CPU.arm?
        bin.install "burrito_out/lazyasdf_macos_m1" => "lazyasdf"
      else
        bin.install "burrito_out/lazyasdf_macos" => "lazyasdf"
      end
    end
  end

  test do
    # this is required for homebrew-core
    system "true"
  end
end

Here we can see all of lazyasdf's dependencies.

It requires

• Elixir/Erlang: self-explanatory

• asdf: self-explanatory

• gcc, make: used to compile the termbox NIF bindings

• Python 3.9: The termbox NIF uses a python script. For some reason it works with 3.9 and not 3.11, so I pinned it at 3.9 🤷‍♂️.

• zig,xz: Burrito uses these two.

  Burrito specifically uses Zig 0.10.0, not 0.10.1, so we have to specify it as a resource and download it from the Zig website. Luckily, they provide pre-compiled binaries for both of our target platforms, so we can just download, untar, and add them to our PATH!

The Python dependency is even more quirky. The termbox scripts use the unversioned python executable, but Homebrew does not link those by default, so we have to manually add the unversioned one to our PATH for it to work.

Voilà!

Notes

Since this is a 3rd party Tap, the bottles that are generated are for an older version of Intel Mac, so those won't be very useful to anybody.

But if I were to merge this formula into homebrew-core, they would be bottled using the secret Homebrew GitHub Actions runners that can bottle it for all the OS's and architectures.

The Future

Ratatouille is incredible as it is today, but there is a lot of room for improvement.

As time allows, I hope to:

• Contribute to Ratatouille to allow more complex UI features like scrollbars and dynamic size information for elements.
• Create bindings for termbox2 (the next iteration of termbox).
• Create a higher level toolkit for building TUIs with Ratatouille, including menus, inputs, dialogs, etc.

Thanks for reading!

The for special form, also known as a list comprehension, is a construct designed for concise and powerful enumerable transformation in Elixir.

It looks very similar to a "for loop" in other languages like JavaScript and C, but rather than being a language construct, it is an expression (like everything else in Elixir). This means that it evaluates to a value that can be bound to a variable. You may have heard this before as "statement vs expression".

last_names = 
  for friend <- friends do
    friend.last_name
  end

Before reading this rest of this article, I suggest you read the list comprehension guide on elixir-lang.org. I will go over some of the same details, but hopefully will go much more in depth.

Generators

The primary ingredient in a comprehension is the generator. The only other place you will see this "left arrow" syntax (lhs <- rhs) is in the with special form.

The right-hand side is the enumerable you want to loop over and the left-hand side is the intermediate pattern you want to match on during each iteration. This is a normal pattern, so you can pattern match like you would anywhere else.

friends = [
  %{first_name: "Joe", last_name: "Swanson"},
  %{first_name: "Greg", last_name: "Mefford"},
  %{first_name: "Erik", last_name: "Plostins"},
]

for %{last_name: last_name} <- friends do
  last_name
end

# ["Swanson", "Mefford", "Plostins"]

Multiple Generators

Comprehensions are very concise in that you can declare multiple generators, allowing you to generate every permutation of both enumerables. A great example is generating a list of coordinate pairs from a range of x values and y values.

A key detail to recognize is that the both values are yielded to the same block, so the result is a flat list.

for x <- 0..99, y <- 0..99 do
  {x, y}
end

# [{0, 0}, {0, 1}, {0, 2}, ...]

The counterexample using Enum.map/2 is not nearly as readable and demonstrates that you need to remember to flatten the outer loop, or else you'll get a list of lists.

Enum.flat_map(0..99, fn x ->
  Enum.map(0..99, fn y ->
    {x, y}
  end)
end)

# [{0, 0}, {0, 1}, {0, 2}, ...]

Generators work with maps too

The comprehension works with anything that implements the Enumerable protocol, so you can iterate through a map as well. The generator will yield each key/value pair of the given map.

dictionary = %{
  "low-latency" => "short delay before a transfer of data begins following an instruction for its transfer",
  "distributed" => "(of a computer system) spread over several machines, especially over a network",
  "fault-tolerant" => "relating to or being a computer or program with a self-contained backup system that allows continued operation when major components fail"
}

for {word, definition} <- dictionary do
  IO.puts "#{word}: #{definition}"
end

As seen above, we can iterate through a Range as well.

for x <- 0..99, y <- 0..99 do
  {x, y}
end

# [{0, 0}, {0, 1}, {0, 2}, ...]

Bitstring generators

The generators we've seen so far have been "list generators". The other type of generator is the "bitstring generator". The bitstring generator allows you to easily loop over a bitstring while correctly parsing bytes.

For a primer on bitstrings, please see the fantastic guide on elixir-lang.org.

This is often very useful when parsing binary protocols like database protocols. Below is an example that demonstrates that each iteration of the comprehension reads the message length and then uses it to know how much more of the bitstring to read next. While the previous examples could be translated into various form of Enum.map/2, this example can only be achieved with normal recursion.

bitstring = <<1, "I", 6, "really", 4, "love", 4, "list", 14, "comprehensions", 1, "!">>

# <<1, 73, 6, 114, 101, 97, 108, 108, 121, 4, 108, 111, 118, 101, 4, 108, 105,
#  115, 116, 14, 99, 111, 109, 112, 114, 101, 104, 101, 110, 115, 105, 111, 110,
#  115, 1, 33>>

for <<message_length::integer, message::binary-size(message_length) <- bitstring>> do
  message
end

# ["I", "really", "love", "list", "comprehensions", "!"]

The example using recursion looks like:

defmodule For do
  def loop(""), do: []

  def loop(<<message_length::integer, message::binary-size(message_length), rest::binary>>) do
    [message | loop(rest)]
  end
end

bitstring = <<1, "I", 6, "really", 4, "love", 4, "list", 14, "comprehensions", 1, "!">>

For.loop(bitstring)

# ["I", "really", "love", "list", "comprehensions", "!"]

To compare the two styles, let's look at the syntax of the generator in the comprehension example and the second function head of loop in the function example.

In the function example, you'll see that we pattern match on the bitstring in a similar manner to who you pattern match on a list. We pull several individual items off the beginning of the bitstring (message_length::integer, message::binary-size(message_length)) and we pattern match on the "rest" of the bitstring with rest::binary.

The list equivalent looks like:

defmodule For do
  def loop([]), do: []

  def loop([_length, message | rest]) do
    [message | loop(rest)]
  end
end

list = [1, "I", 6, "really", 4, "love", 4, "list", 14, "comprehensions", 1, "!"]

For.loop(list)

# ["I", "really", "love", "list", "comprehensions", "!"]

Now if we compare this to the bitstring generator, we can see the similarity (with a caveat). The lhs of the bitstring generator is the pattern match on the individual items and the rhs is pattern matching on the "rest". The caveat is that bitstring here is not actually the "rest" on each iteration, it is still the entire bitstring. I find this the best analogy to describe how the bitstring generator works since it is different enough from the "normal" list generator.

#   👇 individual item         👇 individual item                      👇rest
for <<message_length::integer, message::binary-size(message_length) <- bitstring>> do
  # 👇 this will print the same thing every time
  IO.inspect(bitstring)

  message
end

Chaining generators

List comprehensions allow you to chain generators together by using the lhs value from a generator in the rhs of a subsequent generator.

Here's an example that demonstrates getting a list of all of your friends hobbies:

friends = [
  %{name: "Derek", hobbies: ["Movies", "Hot Sauce"]},
  %{name: "Joe", hobbies: ["Yu-Gi-Oh!", "Tattoos"]},
  %{name: "Andres", hobbies: ["Photoshop", "Oreos", "Cereal"]},
]

for %{hobbies: hobbies} <- friends, hobby <- hobbies do
  hobby
end

# ["Movies", "Hot Sauce", "Yu-Gi-Oh!", "Tattoos", "Photoshop", "Oreos", "Cereal"]

Generators filter non-matching lhs values

If the match expression in the lhs of a generator does not match on the value yielded from the rhs, it will be rejected, and the list comprehension will move on to the next element in the enumerable.

This is slightly surprising behavior at first and should be kept in mind when using list comprehensions. The following example might lead to a bug in your program.

friends = [
    %{"name" => "Derek"},
    %{name: "Joe"}
  ]

for %{name: name} <- friends do
  name
end

# ["Joe"]

If you were to have written this program using Enum.map/2, you would have ran into a function clause error.

friends = [
    %{"name" => "Derek"},
    %{name: "Joe"}
  ]

Enum.map(friends, fn %{name: name} ->
  name
end)

# ** (FunctionClauseError) no function clause matching in :erl_eval."-inside-an-interpreted-fun-"/1
# 
#    The following arguments were given to :erl_eval."-inside-an-interpreted-fun-"/1:
# 
#        # 1
#        %{"name" => "Derek"}
# 
#    (stdlib 3.17.1) :erl_eval."-inside-an-interpreted-fun-"/1
#    (stdlib 3.17.1) erl_eval.erl:834: :erl_eval.eval_fun/6
#    (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2

This behaviour can be useful! If you only wanted to iterate over configuration options that are enabled, you could write something like this:

configs = [
  %{name: :feature_a, enabled: true},
  %{name: :feature_b, enabled: false},
  %{name: :feature_c, enabled: true}
]

for %{enabled: true} = config <- configs do
  config
end

# [%{name: :feature_a, enabled: true}, %{name: :feature_c, enabled: true}]

Filters

Now that we've talked about generator filtering, let's talk about Filters.

So far we have seen one type of "argument" that can be passed to the comprehension, the generator. Another "argument" is the filter! Let's look at a quick example.

In this example, we iterate over a list of employees, filter based on the employees status, and return a list of the employee's names

employees = [
  %{name: "Eric", status: :active},
  %{name: "Mitch", status: :former},
  %{name: "Greg", status: :active}
]

for employee <- employees, employee.status == :active do
  employee.name
end

# ["Eric", "Greg"]

As with generators, filters can use values bound in a previous step of the comprehension. And like generators, you can use multiple filters as well. You can even mix and match them!

employees = [
  %{
    name: "Eric",
    status: :active,
    hobbies: [%{name: "Text Adventures", type: :gaming}, %{name: "Chickens", type: :animals}]
  },
  %{
    name: "Mitch",
    status: :former,
    hobbies: [%{name: "Woodworking", type: :making}, %{name: "Homebrewing", type: :making}]
  },
  %{
    name: "Greg",
    status: :active,
    hobbies: [
      %{name: "Dungeons & Dragons", type: :gaming},
      %{name: "Woodworking", type: :making}
    ]
  }
]

for employee <- employees,
    employee.status == :active,
    hobby <- employee.hobbies,
    hobby.type == :gaming do
  {employee.name, hobby}
end

# [
#   {"Eric", %{name: "Text Adventures", type: :gaming}},
#   {"Greg", %{name: "Dungeons & Dragons", type: :gaming}}
# ]

At this point we can recognize that the list comprehension has the characteristics of a function with variadic arguments. If we were to write our own for using plain functions, we'd have to pass it a list of callbacks to evaluate and a final callback to do the mapping. While we aren't necessarily concerned with how we'd implement for as a plain function, it's important to recognize aspects that are "different" from "normal" constructs in the language.

One of the great things about the list comprehension is that it allows you to operate on Enumerable data structures in fewer passes (usually 1) than when using the Enum module.

We can write our previous example using functions like so:

employees = [
  %{name: "Eric", status: :active},
  %{name: "Mitch", status: :former},
  %{name: "Greg", status: :active}
]

employees
|> Enum.filter(fn employee -> employee.status == :active end)
|> Enum.map(fn employee -> employee.name end)

# ["Eric", "Greg"]

employees
|> Enum.reduce([], fn employee, acc -> 
  if employee.status == :active do
    [employee.name | acc]
  else
    acc
  end
end)
|> Enum.reverse()

# ["Eric", "Greg"]

:lists.filtermap(
  fn employee ->
    if employee.status == :active do
      {true, employee.name}
    else
      false
    end
  end,
  employees
)

# ["Eric", "Greg"]

You can see benchmarks of all of these styles of "filter map" here.

Options

Now that we've covered the basic principles of the list comprehension, we can explore the various options that can be passed to augment it's behavior. The default behavior is to act more or less like Enum.map/2 with regard to the return type.

As of this writing, there are three options available: :uniq, :into, and :reduce

:uniq

:uniq is the least interesting of the available options, but still quite powerful.

It simply ensures that the return result will only contain unique values.

employees = [
  %{
    name: "Eric",
    status: :active,
    hobbies: [%{name: "Text Adventures", type: :gaming}, %{name: "Chickens", type: :animals}]
  },
  %{
    name: "Mitch",
    status: :former,
    hobbies: [%{name: "Woodworking", type: :making}, %{name: "Homebrewing", type: :making}]
  },
  %{
    name: "Greg",
    status: :active,
    hobbies: [
      %{name: "Dungeons & Dragons", type: :gaming},
      %{name: "Woodworking", type: :making}
    ]
  }
]

for employee <- employees, hobby <- employee.hobbies, uniq: true do
  hobby.name
end

# ["Text Adventures", "Chickens", "Woodworking", "Homebrewing", "Dungeons & Dragons"]

You can see benchmarks of all of these styles of "map uniq" here.

:into

:into is where things start to get interesting.

The default behavior for a list comprehension behaves more or less like a "map" operation, meaning that the expression evaluates to a list.

The :into option allows you to instead push the value returned by each iteration into a collectable. A data structure is collectable if it implements the Collectable protocol.

If you aren't familiar with protocols, you have already been using them! The Enum module is a set of functions that operate on data structures that implement the Enumerable protocol. The builtin data structures that implement the Enumerable protocol are the List, Range, Map, and MapSet types.

The builtin data structures that implement the Collectable protocol are List, Map, MapSet, and BitString. The Enum function that you would use to take advantage of this protocol is Enum.into/2.

Let's take a look at some examples.

List

Using a list as the :into actually doesn't change the behavior at all!

employees = [
  %{
    name: "Eric",
    status: :active,
    hobbies: [%{name: "Text Adventures", type: :gaming}, %{name: "Chickens", type: :animals}]
  },
  %{
    name: "Mitch",
    status: :former,
    hobbies: [%{name: "Woodworking", type: :making}, %{name: "Homebrewing", type: :making}]
  },
  %{
    name: "Greg",
    status: :active,
    hobbies: [
      %{name: "Dungeons & Dragons", type: :gaming},
      %{name: "Woodworking", type: :making}
    ]
  }
]

for employee <- employees,
    employee.status == :active,
    hobby <- employee.hobbies,
    hobby.type == :gaming,
    into: [] do
  {employee.name, hobby}
end

# [
#   {"Eric", %{name: "Text Adventures", type: :gaming}},
#   {"Greg", %{name: "Dungeons & Dragons", type: :gaming}}
# ]

Map

But if we use a map, we can see that it pushes each key/value pair into the map that you pass for the option. Usually I use this with an empty map (%{} or Map.new()), but let's look at an example using a non-empty map.

employees = [
  %{
    name: "Eric",
    status: :active,
    hobbies: [%{name: "Text Adventures", type: :gaming}, %{name: "Chickens", type: :animals}]
  },
  %{
    name: "Mitch",
    status: :former,
    hobbies: [%{name: "Woodworking", type: :making}, %{name: "Homebrewing", type: :making}]
  },
  %{
    name: "Greg",
    status: :active,
    hobbies: [
      %{name: "Dungeons & Dragons", type: :gaming},
      %{name: "Woodworking", type: :making}
    ]
  }
]

base_map = %{
  "Mitch" => %{
    name: "Reading",
    type: :learning
  },
  "Greg" => %{
    name: "Traveling",
    type: :expensive
  }
}

for employee <- employees,
    employee.status == :active,
    hobby <- employee.hobbies,
    hobby.type == :gaming,
    into: base_map do
  {employee.name, hobby}
end

# %{
#   "Eric" => %{name: "Text Adventures", type: :gaming},
#   "Greg" => %{name: "Dungeons & Dragons", type: :gaming},
#   "Mitch" => %{name: "Reading", type: :learning}
# }

Here we can observe three things.

• The comprehension evaluates to a map.
• The "Mitch" key and its value were preserved in the final output.
• The "Greg" key's value in the base_map was overwritten by the value yielded during the comprehension with the same key. If our comprehension were to have returned multiple key/value pairs with identical keys, the last one would have won.

This option is very useful for transforming maps. Since iterating over a map with an Enum function turns it into a list of 2-tuples, you always need to pipe the return value into Enum.into/2 or Map.new/1.

employees = [
  %{
    name: "Eric",
    status: :active,
    hobbies: [%{name: "Text Adventures", type: :gaming}, %{name: "Chickens", type: :animals}]
  },
  %{
    name: "Mitch",
    status: :former,
    hobbies: [%{name: "Woodworking", type: :making}, %{name: "Homebrewing", type: :making}]
  },
  %{
    name: "Greg",
    status: :active,
    hobbies: [
      %{name: "Dungeons & Dragons", type: :gaming},
      %{name: "Woodworking", type: :making}
    ]
  }
]

base_map = %{
  "Mitch" => %{
    name: "Reading",
    type: :learning
  },
  "Greg" => %{
    name: "Traveling",
    type: :expensive
  }
}

employees
|> Enum.filter(fn employee -> employee.status == :active end)
|> Enum.flat_map(fn employee ->
  employee.hobbies
  |> Enum.filter(fn hobby -> hobby.type == :gaming end)
  |> Enum.map(fn hobby ->
    {employee.name, hobby}
  end)
end)
|> Enum.into(base_map)

# %{
#   "Eric" => %{name: "Text Adventures", type: :gaming},
#   "Greg" => %{name: "Dungeons & Dragons", type: :gaming},
#   "Mitch" => %{name: "Reading", type: :learning}
# }

Strings and BitStrings

You can build strings and bitstrings with the :into option as well!

This is useful when you want to build a string or a binary out of a list or map all in one pass. Let's take a look at an example of creating an "attribute string" for use with HTML.

attributes = [
  class: "font-bold text-red-500 underline",
  id: "error-text",
  data_controller: "error-controller"
]

for {property, value} <- attributes, into: "" do
  property =
    property
    |> to_string()
    |> String.replace("_", "-")

  ~s| #{property}="#{value}"|
end

# " class=\"font-bold text-red-500 underline\" id=\"error-text\" data-controller=\"error-controller\""

:reduce

My favorite option to the comprehension is :reduce!

Reduce allows us to change the comprehension from behaving like a "map" operation to a "reduce" operation. This means that it will loop over an enumerable, but collect an "accumulator" instead.

Let's take a look at the first example in the Enum.reduce/3 documentation and then convert it to a comprehension. This example produces the sum of a list of integers.

Enum.reduce([1, 2, 3], 0, fn x, acc ->
  x + acc
end)

# 6

We can express this as a comprehension like so:

for x <- [1, 2, 3], reduce: 0 do
  acc ->
    x + acc
end

# 6

There are two immediate things we can observe.

First, the :reduce option takes a value that is to be used as the first value of the accumulator.

Second, the comprehension in this mode includes a slightly different syntax. Here the inside of the block includes the "arg(s) and right arrow" syntax that you see in anonymous functions and case expressions. This is the syntax that allows the comprehensions to yield the accumulator to the block on every iteration.

The additional syntax is the same as the other places you have probably seen it; you can pattern match and pass additional clauses!

directions = [
  left: 2,
  up: 1,
  down: 5,
  right: 6
]

# You can't move below or to the left of 0.
starting_position = {0, 0}

for {dir, movement} <- directions, reduce: starting_position  do
  {x, y} when dir == :left and x - movement > 0 ->
    {x - movement, y}

  {x, y} when dir == :down and y - movement > 0 ->
    {x, y - movement}

  {x, y} when dir == :up ->
    {x, y + movement}

  {x, y} when dir == :right ->
    {x + movement, y}

  position ->
    IO.puts("Not possible to move #{dir} by #{movement} when you care located at #{inspect(position)}")

    position
end

# {6, 1}

Above we can observe that we've written 5 different clauses, pattern matched on the shape of the data, as well as added guard clauses that capture the data in the generator and the accumulator.

The beauty of using a comprehension as a reducer is the ability to use multiple generators and act on them as if they are one level of iteration.

friends = [
  %{name: "Derek", hobbies: ["Movies", "Hot Sauce"]},
  %{name: "Joe", hobbies: ["Yu-Gi-Oh!", "Tattoos"]},
  %{name: "Andres", hobbies: ["Photoshop", "Oreos", "Cereal"]},
]

for %{hobbies: hobbies, name: name} <- friends, hobby <- hobbies, reduce: [] do
  tagged_hobbies ->
    [{name, hobby} | tagged_hobbies]
end

# [
#   {"Andres", "Cereal"},
#   {"Andres", "Oreos"},
#   {"Andres", "Photoshop"},
#   {"Joe", "Tattoos"},
#   {"Joe", "Yu-Gi-Oh!"},
#   {"Derek", "Hot Sauce"},
#   {"Derek", "Movies"}
# ]

To write this without a comprehension it would look something like:

Enum.reduce(friends, [], fn %{name: name, hobbies: hobbies}, tagged_hobbies ->
  new_hobbies = Enum.map(hobbies, fn hobby -> {name, hobby} end)

  new_hobbies ++ tagged_hobbies
end)

Conclusion

If you've made it this far, congrats! The comprehension packs a lot of features into a tiny programming construct and demonstrating all of them is a lot of work!

The comprehension is one of my favorite features of the Elixir programming language, and it was a pleasure to write about every feature in as much depth as I could.

If you have any questions about comprehensions or want to suggest examples or features that I've missed, feel free to reach out on Twitter or email.

Supplementary Information

List Comprehensions in Erlang

It would be remiss to not mention that the list comprehension also exists in Erlang. I am not personally familiar with them, so I won't explain them very much, but I I'll provide an example and a link to learn more about them on your own.

The following examples are from the official Erlang/OTP documentation and can be found here.

[X || X <- [1, 2, a, 3, 4, b, 5, 6], is_integer(X), X > 3].

% [4, 5, 6]

[{X, Y} || X <- [1, 2, 3],  Y <- [a, b]].

% [{1, a}, {1, b}, {2, a}, {2, b}, {3, a}, {3, b}]

A couple features of EEx that make it stand out are:

• Templates can be compiled into functions ahead of time, avoiding needing to read from the filesystem at runtime.
• EEx has the concept of an "Engine" allowing you to compile an existing template into something different than what the builtin engine, EEx.SmartEngine, does. This is what Phoenix LiveView does.

Recently, José Valim tweeted me some advice on how to use the EEx Engine in my library Temple (an alternative HTML library, that uses an Elixir DSL). At first I was very confused at José's suggestion; I don't want to write EEx at all! I want Temple to go from DSL straight to iolist or %LiveView.Rendered{} structs.

Luckily, my coworker Marlus Saraiva messaged me and gave a bit more context on José's suggestion, and suddenly it all clicked!. Hopefully at the end of this post, you'll understand how José's suggestion makes perfect sense and was extremely clear.

This conversation led me to ask myself, how does EEx work anyway? I decided to source dive EEx, LiveView, Surface, and HEEx and will share with you what I learned!

The Source Code

Technically EEx is just a templating syntax that can handle any plaintext format. It doesn't matter if its HTML, JavaScript, or even Elixir (this is how the Phoenix generators work).

EEx templates allow you to execute arbitrary Elixir code inside of special tags and inject the runtime result into the return value. You can use single expressions or even expressions that take blocks. The EEx.SmartEngine also allows you to ergonomically access data passed to the template with the @ syntax.

<!-- single line expression -->
<span><%= @user.name %></span>

<!-- block expression -->
<ul>
  <%= for user <- @users do %>
    <li><%= user.name %></li>
  <% end %>
</ul>

The Compiler

The public API for compiling an EEx file or string consists of the EEx.compile_file/2 and EEx.compile_string/2 functions. Internally, they will call into the EEx.Compiler module.

The EEx.Compiler module is where the coordination happens when compiling our source code. You can find the source code here.

This module's compile/2 function is responsible for calling the tokenizer, traversing them, and passing them to the appropriate callbacks on the engine. By default, EEx uses the builtin EEx.SmartEngine.

Essentially there are two steps:

1. Turn the source into tokens.
2. Turn the tokens into Elixir AST by passing them to the relevant callback on the given EEx.Engine.