Proto Over HTTPS

Introduction

Recently I announced the release of CL-Protobufs, a full featured protocol buffer library for Common Lisp. It does not include a gRPC package. In this post, I will make an example hello-world server that will take a protocol buffer message as input and respond with a protocol buffer as output. I will describe some of the limitations of the given code and some ways to improve it. 

I will expect a basic understanding of the ASDF build system; if you don’t have one please see an example such as this.

The reader should find the code in my github repo. It contains three files:

  1. hello-world-server.lisp
  2. hello-world-server.proto
  3. hello-world-server.asd

The .asd file is for integration with the ASDF build system. The proto file is where we define the request and response. 

Code Discussion:

The actual implementation is in the hello-world-server.lisp file. Let’s briefly go over the files.

hello-world-server.asd:

The useful information in this file is:

  • defsystem: The Lisp system is called hello-world-server. 
  • defsystem-depends-on: To load the system you will need to load cl-protobufs first. This is so we can compile the proto file into a file that Lisp will understand. Call this the Lisp proto schema file.
  • depends-on: We will use hunchentoot as our web server.
  • module: We have one module src.
    • protobuf-source-file: This is an asdf directive given to us by cl-protobufs It will look for a file hello-world.proto in our current directory and call lisp-protoc on this file.
      • Note: You can specify a different directory to look in with proto-pathname. 
      • Warning: You must have the lisp-protoc plugin compiled and in your $PATH to call this directive successfully. Please go to CL-Protobufs and follow the protoc installation directions.
    • file: A lisp file hello-world-server.lisp

If you have a background using asdf that may have been obvious.The defsystem-depends-on call to cl-protobufs and the protobuf-source-file call may be non-obvious to newer lispers.

hello-world.proto

This is where we place the request and response definitions. This will be compiled into a lisp file by protoc that lisp will be able to interpret with cl-protobufs. Note the package name is hello_world. To read more about protocol buffers please visit the Google developer documentation and to learn more about the API for cl-protobufs please visit the CL-Protobufs readme.

hello-world-server.lisp

This is where the application code lives. The defpackage tells us mostly what we already know, we will use Hunchentoot as our web server library and the cl-protobufs library which will allow us to serialize and deserialize our protocol buffer objects. We will also be using a package: cl-protobufs.hello-world. This package is defined as part of the code generated by the Lisp protoc plugin and contains everything we need to work with the native lisp protocol buffer objects:

  • cl-protobufs.hello-world:request
  • cl-protobufs.hello-world:response

A good tutorial on Hunchentoot can be found here. We define a handler for our server that will be at /hello and will be available on port 4242. We give a parameter request of type string. 

Inside of the request body is where the real protocol buffer work is done. The handler will take in a request which will be of type string. This string will actually hold a simple array, i.e. #(values) which will be the serialized protocol buffer request message ‘cl-protobufs.hello-world:request. As an example: 

(cl-protobufs:serialize-object-to-bytes
  (cl-protobufs.hello-world:make-request :name "fish"))

Will give us a serialized request object whose string name entry is “fish”. When run in the REPL we get:

#(10 4 102 105 115 104) 

We call read-from-string to get the array as an array and then call make-array to get an octet array. In cl-protobufs:deserialize-object-from-bytes we expect an octet-array so this is required. We deserialize the message with deserialize-object-from-bytes. If we received a message with a set name we return a response with a response string “Hello {name}”, otherwise we just return a response with response string “Hello”. Finally we serialize the created response and return it as a string.

Calling with my “fish” request I get the response:

#(10 10 72 101 108 108 111 32 102 105 115 104)

Then calling 

(cl-protobufs:deserialize-object
  'cl-protobufs.hello-world:response
  (make-array 12 :element-type '(unsigned-byte 8) 
                 :initial-contents 
                 #(10 10 72 101 108 108 111 
                   32 102 105 115 104)))

I get 

#S(CL-PROTOBUFS.HELLO-WORLD:RESPONSE :%RESPONSE "Hello fish" 
                                     :%BYTES NIL 
                                     :%%IS-SET #*1) 

As should be expected.  

Limitations and Extensions

The first limitation is the work we have to do to read the request. We shouldn’t have to call read-from-string at all, and then we shouldn’t have to make a new octet array. What we should do is make our call with octets-to-string and then call string-to-octets on the received message. This can easily be done by importing trivial-utf-8. I was using Postman to make the calls in debugging this simple hello-world-server. This should be easy to fix for the next post.

We shouldn’t even have to do that. We should be able to set the message request in the handler and the octets-to-string then deserialize call should be handled for us. The final serialize and utf-8-string-to-bytes call should also be handled like that. This is easily handled by around methods. Given the ingenuity of Lispers, there’s probably even better answers.

Final Remarks

I hope you found this example interesting. This is only a simple server. Also, I haven’t used Huntchentoot very much so I probably wrote some terrible server code. I 

  1. Hope to improve my lisp server code.
  2. Hope to get a gRPC server out someday.

Thank you for reading. I hope to see some interesting lisp code using cl-protobufs!


I would like to thank @rongut, @cgay, and @benkuehnert for reviewing this document.

4 thoughts on “Proto Over HTTPS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s