A modern REST API in Laravel 5 Part 2: Resource controls

Enable consumers of your API to control and manipulate the data they request

Posted by Esben Petersen on April 15, 2016

tl;dr

Optimus\Bruno will enable you to filter, paginate, sort and eager load related resources through query string parameters. Optimus\Architect will enable the consumer to decide how the eager loaded related resources should be structured. Lastly, Optimus\Genie provides a quick integrated way to implement the two libraries without having to add a lot of new code.

The full code to this article can be found here: larapi-series-part-2

Introduction

All to often API developers do not give much thought into how it is to work with their APIs for a consumer. This article will give some very useful tips as to how you can make it much more flexible to work with for client developers. This is going to simplify development for both you and your front-end devs.

Agenda

In this article I will take you through...

  1. How you can enable consumers to automatically eager load related resources
  2. How you can give consumers controls like filters, pagination and sorting
  3. How you can enable consumers to decide how eager loaded related resources should be returned using data composition

Excited? Lets go..!

The challenge

If you have ever experienced developing a client that uses an API you have probably encountered the problem we will try to solve in this part of our journey for a modern REST API in Laravel 5. When developing a client you will often need different parts of the same data for different scenarios.

Imagine having two resources: /users and /roles. A user can have many roles: users 1 ----> n roles. Now assume that you are tasked with designing two things:

  1. A table list of all users and the name of their role
  2. A dropdown of all users with the role named 'Agent'

Let us try to think about these views one at a time and the data they would need.

User table list

Okay, so this would be a table that lists all users on the site. Now, we want to display all the roles that is attached to each user in the list.

The data we would need to do so would be the role name of all the roles. Something along these lines. 

[
  {
    "id": 1,
    "active": true,
    "name": "Katrine Elbek",
    "email": "demo.kat@traede.com",
    "roles": [
      {
        "name": "Administrator"
      }
    ]
  },
  {
    "id": 2,
    "active": true,
    "name": "Katrine Obling",
    "email": "demo.agent@traede.com",
    "roles": [
      {
        "name": "Agent"
      }
    ]
  },
  {
    "id": 3,
    "active": true,
    "name": "Yvonne",
    "email": "demo.lone@traede.com",
    "roles": [
      {
        "name": "Administrator"
      }
    ]
  }
]

Agent dropdown

So this is a dropdown for selecting the ID of a user that has the role "Agent".

So we need a filtered version of the user data that only has agents in it. Like so: 

[
  {
    "id": 2,
    "name": "Katrine Obling",
    "email": "demo.agent@traede.com"
  }
]

Note here that we do not actually need the roles array of the user. Because we are not displaying any of this data in the view.

Planning the endpoints

Okay, so we now know the data that we will eventually need to construct the two views. Let us just summarize them here.

Table list

[
  {
    "id": 1,
    "active": true,
    "name": "Katrine Elbek",
    "email": "demo.kat@traede.com",
    "roles": [
      {
        "name": "Administrator"
      }
    ]
  },
  {
    "id": 2,
    "active": true,
    "name": "Katrine Obling",
    "email": "demo.agent@traede.com",
    "roles": [
      {
        "name": "Agent"
      }
    ]
  },
  {
    "id": 3,
    "active": true,
    "name": "Yvonne",
    "email": "demo.lone@traede.com",
    "roles": [
      {
        "name": "Administrator"
      }
    ]
  }
]

Agent user dropdown

[
  {
    "id": 2,
    "name": "Katrine Obling",
    "email": "demo.agent@traede.com"
  }
]

Now, one of the first questions we could ask ourselves could be: for the second data set, do we want to filter the users in the API or in the client? Let us just imagine for a second that the endpoint /users always returns the top data set. Then to get all the agents in javascript view we could simply use a filter function. 

let agents = users
              .map((user) => {
                  user.roleNames = user.roles.map((role) => role.name)
                  return user;
              })
              .filter((user) => user.roleNames.indexOf('Agent') !== -1)

So, it is pretty simple to actually get all the agents out of the data set. However, this code does have a few risks.

What happens if the underlying data model of the API changes? Imagine if the Agent role name changed to Super-Agent. Then we would not get the agents any more since we are looking specifically for roles named 'Agent'. What if the name property changed to title? I know these are all contrived examples, however they do demonstrate the risks of relying to heavily on the client to do these sort of things.

Would it not be much better if the client could simply request all the agents from the API?

And what about our users list: do we always want to load the roles whenever we are requesting the users? What about in a situation where we just want the users, but not their roles? Assuming the underlying database model is a relational is it not highly inefficient to load the roles when they are not needed?

Given these goals we can quickly sum up a list of requirements:

  1. We want to be able to load users without their roles
  2. We want to be able to load users with their roles
  3. We want to be able to load users that have the agent role

Let us look at the different strategies we can implement to achieve this.

The bad one: make an endpoint for each data set

One solution could be to simply make an endpoint per type of dataset.

  • /users returns users without their roles
  • /users/with-roles returns users with their roles
  • /users/agents returns users that are agents

Hopefully I do not have to spend to much time explaining why this is a bad idea. When having so many different endpoints for the same underlying data it becomes a real pain in the a$$™ to maintain. Whenever something has to change it will often result in an eksponential amount of changes elsewhere.

The better solution: resource controls

Would it not be so much nicer if we could just do:

  • /users returns users without their roles
  • /users?includes[]=roles returns users with their roles
  • /users?filters[]=isAgent:1 returns users that are agents

Now we not only use the same endpoint but use query strings to filter and manipulate the data we need.

Implementation

To implement this I have written a small library: Bruno. To find more details about this library and its features you can read its README.

So assume we have the application in place and we have a User model and a Role model. With a relation like so User 1 ----> n Role. Then we can write a controller that looks like this: 

<?php

namespace App\Http\Controllers;

use Optimus\Api\Controller\EloquentBuilderTrait;
use Optimus\Api\Controller\LaravelController;
use App\Models\User;

class UserController extends LaravelController
{
    use EloquentBuilderTrait;

    public function getUsers()
    {
        // Parse the resource options given by GET parameters
        $resourceOptions = $this->parseResourceOptions();

        // Start a new query for books using Eloquent query builder
        // (This would normally live somewhere else, e.g. in a Repository)
        $query = User::query();
        $this->applyResourceOptions($query, $resourceOptions);
        $books = $query->get();

        // Parse the data using Optimus\Architect
        $parsedData = $this->parseData($books, $resourceOptions, 'users');

        // Create JSON response of parsed data
        return $this->response($parsedData);
    }
}

A few things are going on here which are worth to notice. 

// Parse the resource options given by GET parameters
$resourceOptions = $this->parseResourceOptions();

parseResourceOptions is a method of the base controller that will read our query strings (?includes[]=roles and ?filters[]=isAgent:1) into a format that we can use. 

$this->applyResourceOptions($query, $resourceOptions);

This is the method that will apply our different resource controls to the Eloquent query builder. In the case of ?includes[]=roles it will run something like $query->with('roles') behind the scenes.

It is important to note that applyResourceOptions is part of the EloquentBuilderTrait that we include in our controller. Why is this not a standard part of the base controller class? Because database logic should not be written in our controllers :-)

Making filters work

So, I may have oversold the syntax of filters a little bit. ?filters[]=isAgent:1 sounded a little to good to be true, right? Actually, the reason the filter syntax is a bit more complex is because it contains a lot of cool functionality. 

{
  "filter_groups": [
    {
      "filters": [
        {
          "key": "isAgent",
          "value": true,
          "operator": "eq"
        }
      ]
    }
  ]
}

The actual syntax of filters.

You can do a lot more cool stuff with filters, and I suggest you check out the syntax in the Bruno repository.

The short version is that you can define several filter groups. Each filter group can contain many filters using operators such as eq (equals), sw (starts with), lt (less than), in and more. Think of each filter group as a parenthesis grouping in an if-statement. So for instance this example 

{
  "filter_groups": [
    {
      "or": true,
      "filters": [
        {
          "key": "email",
          "value": "@gmail.com",
          "operator": "ew"
        },
        {
          "key": "email",
          "value": "@hotmail.com",
          "operator": "ew"
        }
      ]
    },
    {
      "filters": [
        {
          "key": "name",
          "value": ["Stan", "Eric", "Kyle", "Kenny"],
          "operator": "in"
        }
      ]
    }
  ]
}

If thought of as an if-statement would look like 

if (
  (endsWith('@gmail.com', $email) || endsWith('@hotmail.com', $email)) &&
  in_array($name, ['Stan', 'Eric', 'Kyle', 'Kenny'])
) {
 // return user
}

endsWith is a fictitious function

So the query will return all users whoose email ends with either @gmail.com or @hotmail.com and whoose name is either Stan, Eric, Kyle or Kenny. Cool, yeah?

Anywho, enough syntax. Let us make the 'isAgent' filter work. First of all remember to send the correct data with the request. Instead of ?filters[]=isAgent:1 the data is now 

{
  "filter_groups": [
    {
      "filters": [
        {
          "key": "isAgent",
          "value": true,
          "operator": "eq"
        }
      ]
    }
  ]
}

Or as query string 

?filter_groups%5B0%5D%5Bfilters%5D%5B0%5D%5Bkey%5D=isAgent&filter_groups%5B0%5D%5Bfilters%5D%5B0%5D%5Bvalue%5D=true&filter_groups%5B0%5D%5Bfilters%5D%5B0%5D%5Boperator%5D=eq

Most AJAX libraries will automatically convert the JSON data to a query string on GET requests, so you really do not need to be a query string syntax wizard. jQuery even has the function $.param which can do it for you.

The next thing we must do to make it work is to implement a custom filter. This is because there is no isAgent property on our User model. So when the controller applies our filter to the Eloquent query builder it will try to execute $query->where('isAgent', true) which will throw an error (since the column does not exist). Luckily the controller will look for custom filter methods: so let us implement that! 

public function filterIsAgent(Builder $query, $method, $clauseOperator, $value, $in)
{
    // check if value is true
    if ($value) {
        $query->whereIn('roles.name', ['Agent']);
    }
}

Add this method to our UserController and we are almost done. There is just one more thing. Whenever we are making a custom filter method the system will try to look for a relationship on the Eloquent model to join. That means in this case it will try to find a isAgent relationship on the model. This probably does not exist, but there does exist a relationship named roles. So we overcome this by adding the isAgent relationship to the model. 

public function isAgent()
{
    return $this->roles();
}

Making it work with Larapi

So the above example is just a quick and dirty demonstration of how you can use Laravel controller to get resource controls. However, since we are doing a series on creating a Laravel API with my API-friendly Laravel fork let us see how this example would look using the API structure we laid out in part 1.

The controller

In UserController.php the method for GET /users should be defined like 

public function getAll()
{
    $resourceOptions = $this->parseResourceOptions();

    $data = $this->userService->getAll($resourceOptions);
    $parsedData = $this->parseData($data, $resourceOptions, 'users');

    return $this->response($parsedData);
}

So we pass along the resource control options to the service so it can pass it along to the repository. If your repositories extend my Eloquent repository base class Genie it will already have the helper functions build-in needed to use the resource options.

The service

In the user service class, UserService.php, add the method to get users. 

public function getAll($options = [])
{
    return $this->userRepository->get($options);
}

The get method of the repository is build into the previously mentioned repository base class, so we do not even need to implement it.

The repository

Even though the get method is already implemented in the repository base class, we still need to implement the custom isAgent filter. 

<?php

namespace Api\Users\Repositories;

use Infrastructure\Database\Eloquent\Repository;

class UserRepository extends Repository
{
    public function filterIsAgent(Builder $query, $method, $clauseOperator, $value, $in)
    {
        // check if value is true
        if ($value) {
            $query->whereIn('roles.name', ['Agent']);
        }
    }
}

The model

Lastly, we need to make sure the User model can join the isAgent relationship. Add the following relationship to the model. 

public function isAgent()
{
    return $this->roles();
}

That is it

Yeah there is really nothing more to it. Now the GET /users endpoint has support for filtering, eager loading, pagination and sorting. Plus data structure composition.

Data structure composition?!

If you remember in our UserController we have the following line. 

// Parse the data using Optimus\Architect
$parsedData = $this->parseData($books, $resourceOptions, 'users');

So what the heck is Optimus\Architect? It is a library we can use to define the composition of eager loaded relationships. Easier to explain by example. Imagine this user data:

GET /users?includes[]=roles

{
  "users": [
    {
      "id": 1,
      "active": true,
      "name": "Katrine Elbek",
      "email": "demo.kat@traede.com",
      "roles": [
        {
          "id": 1,
          "name": "Administrator"
        }
      ]
    },
    {
      "id": 2,
      "active": true,
      "name": "Katrine Obling",
      "email": "demo.agent@traede.com",
      "roles": [
        {
          "id": 2,
          "name": "Agent"
        }
      ]
    },
    {
      "id": 3,
      "active": true,
      "name": "Yvonne",
      "email": "demo.lone@traede.com",
      "roles": [
        {
          "id": 1,
          "name": "Administrator"
        }
      ]
    }
  ]
}

This data is loaded with the embedded mode, meaning that relationships of resources are nested within. In this case it would be that inside each User is a embedded collection of its Roles. This is the default way to load relationships with Eloquent. Let us check out some other modes.

GET /users?includes[]=roles:ids

{
  "users": [
    {
      "id": 1,
      "active": true,
      "name": "Katrine Elbek",
      "email": "demo.kat@traede.com",
      "roles": [1]
    },
    {
      "id": 2,
      "active": true,
      "name": "Katrine Obling",
      "email": "demo.agent@traede.com",
      "roles": [2]
    },
    {
      "id": 3,
      "active": true,
      "name": "Yvonne",
      "email": "demo.lone@traede.com",
      "roles": [1]
    }
  ]
}

In the ids mode Architect will simply return the primary key of the related model.

GET /users?includes[]=roles:sideload

{
  "users": [
    {
      "id": 1,
      "active": true,
      "name": "Katrine Elbek",
      "email": "demo.kat@traede.com",
      "roles": [1]
    },
    {
      "id": 2,
      "active": true,
      "name": "Katrine Obling",
      "email": "demo.agent@traede.com",
      "roles": [2]
    },
    {
      "id": 3,
      "active": true,
      "name": "Yvonne",
      "email": "demo.lone@traede.com",
      "roles": [1]
    }
  ],
  "roles": [
    {
      "id": 1,
      "name": "Administrator"
    },
    {
      "id": 2,
      "name": "Agent"
    }
  ]
}

In the last mode sideload all the embedded relationships are hoisted into its own collection at the root level. This removes duplicated entries of the embedded mode and can result in smaller responses.

Conclusion

To really build a good and useful API one most think about the consumers of it. Think of it like a business: your customers should love your product. One of the things that makes it a hell to build a client to an API is when the API does not offer flexibility in the returned data set. What related resources should be included (and how)? Sorting, filtering, pagination is all essential when building stuff like lists and data tables (the bread and butter of any SaaS application).

By implementing a few simple libraries like Optimus\Architect, Optimus\Bruno and Optimus\Genie you can rapidly create resources that scale well and easily grant your consumers the flexibility they need whilst keeping your own development flow sane.

The full code to this article can be found here: larapi-series-part-2

All of these ideas and libraries are new and underdeveloped. Are you interested in helping out? Reach out on e-mail, twitter or the Larapi repository