The structure follows the same principle used in GraphQL batching - tag individual requests by a unique identifier, and attach the request details with a simple JSON structure.

• get1 - Is an identifier assigned to the first GET request specified in the batch.
• get2 - Another GET request. This request specified additional URL query string parameters via the query property.
• create1 - Illustrates a POST request to create an entity. The payload property is used to hold the entity create payload.
• update1 - Illustrates a PUT request to update an entity.

Response Structure

The sample above illustrates the schema for the batch response from the service. As with GraphQL, the individual request responses are attached to the same key/tag names specified during the original request. Each response behaves like a union with either a payload or an error property holding the response to the individual request.

Implementation

We decided to implement this as a standalone service instead of adding to the source REST API service. Initial choice was to implement this as a websocket service using Boost:Beast since we are very familiar with the performance and scalability of that stack. For this service, we however decided to take the HTTP2 route and chose the Nghttp2 library that also uses Boost:Asio.

The entire implementation including integration test suite using Qt (which has supported HTTP2 for quite a long time now) was literally a days work. On the server, we implemented a couple of restrictions to prevent mis-use:

• Restrict total payload size to 1000000 chars.
• Restrict maximum number of requests in a batch to 100.

Once the batch payload is parsed using RapidJSON, we use std::async to send requests to the REST service (using Boost:Beast) in parallel, aggregate the results and send back.

Handler Function

The main batch request handler function performs authorization and then parses the payload into datastructures that are fed to the concurrent downstream service invocation code.

  void handleApi( const nghttp2::asio_http2::server::request& req, const nghttp2::asio_http2::server::response& res )
  {
    auto static const methods = std::unordered_set<std::string>{ "POST", "OPTIONS" };
    if ( methods.find( req.method() ) == std::cend( methods ) ) return unsupported( res );
    if ( req.method() == "OPTIONS" ) return cors( res );

    auto bearer = authorise( req, res );
    if ( bearer.empty() ) return;

    auto compress = shouldCompress( req );
    auto compressedBody = isCompressed( req );

    auto context = std::make_shared<http::Context>( bearer, compress, compressedBody );
    context->payload.reserve( 4096 );
    req.on_data([context, &res](const uint8_t* chars, std::size_t size)
    {
      if (size)
      {
        if (context->payload.size() + size >= http::maxApiPayload )
        {
          LOG_INFO << "Rejecting payload that exceeds allowed max size of " << int( http::maxApiPayload );
          context->payload.clear();
          return error( 507, "Payload too big", res );
        }

        context->payload.append( reinterpret_cast<const char*>( chars ), size );
        return;
      }

      if ( !context->payload.empty() )
      {
        const auto& body = context->compressedBody ?
            decompress( context->payload ) : context->payload;
        auto json = model::BatchRequests( context->bearer, body );
        if ( json.requests.size() > http::maxApiRequests )
        {
          return error( 429, "Too many requests in batch", res );
        }

        const auto& [results, time] = execute( json );
        const auto& [out, compressed] = toJson( results, context->compressResponse );
        auto metric = model::Metric{ json.correlationId, "POST",
            "batch", "/batch/api", util::hostname(),
            context->bearer, 200, static_cast<int32_t>( out.size() ),
            std::chrono::system_clock::now(), time };
        db::save( metric );
        write( 200, out, res, compressed );
      }
    });
  }

Deployment

The batch service as with all our other services are built as minimalist docker images (base image) and deployed to various cloud environments. In this particular case this is deployed to AWS.

Note: When setting up the service behind an AWS load balancer, we learned that we cannot use the newer Application Load Balancer, and need to front these services behind a Classic Load Balancer acting as a simple TCP load balancer with SSL termination at the load balancer.