In our last post we implemented a basic Pub Sub application that stores an
Any protocol buffer message and a list of subscribers. When the Any protocol buffer message gets updated we send the new Any message in the body of an http request to all of the subscribers in the subscribe-list.
Today we will update our service to save all of the state in a protocol buffer message. We will also add functionality to save and load the state of the Proto Cache application.
Note: Viewing the previous post is highly suggested!
Note: We use red to denote removed code and green to denote added code.
syntax = proto3`
We will use proto3 syntax. I’ve yet to find a great reason to choose proto3 over proto2, but I’ve also yet to find a great reason to choose proto2 over proto3. The biggest reason to choose proto3 over proto2 is that most people use proto3, but the
Any proto will store proto2 or proto3 messages regardless.
Our users are publishing
Any messages to their clients, so we must store them in our application state. This requires us to include the any.proto file in our proto file.
This contains (almost) all of the state needed for the publish subscribe service for one user:
- repeated string subscriber_list
- This is the latest
Anymessage that the publisher has stored in the Proto Cache.
- This is the latest
- string username
- string password
anykind of production use this should be salted and hashed.
This message contains one entry, a map from a string (which will be a username for a publisher) to a PubSubDetails instance. The attentive reader will notice that we save the username twice, once in the PubSubDetails message and once in the PubSubDetailsCache map as the key. This will be explained when we discuss changes to the proto-cache.lisp file.
The only difference in proto-cache.asd from all of the other asd files we’ve seen using protocol buffers is the use of a protocol buffer message in a package different from our current package. That is, any.proto resides in the cl-protobufs package but we are including it in the pub-sub-details.proto file in proto-cache.
To allow the protoc compiler to find the any.proto file we give it a :proto-search-path containing the path to the any.proto file.
... :components ((:protobuf-source-file "pub-sub-details" :proto-pathname "pub-sub-details.proto" :proto-search-path ("../cl-protobufs/google/protobuf/")) ...
Note: We use a relative path: “../cl-protobufs/google/protobuf/”, which may not work for you. Please adjust to reflect your set-up.
We don’t need a component in our defsystem to load the any.proto file into our lisp image since it’s already loaded by cl-protobufs. We might want to just to recognize the direct dependency of the any.proto file.
We are adding new user invokable functionality so we export:
- This is merely to save typing. The cl-protobufs.pub-sub-details is the package that contains the functionality derived from pub-sub-details.proto.
*cache*: This will be a protocol buffer message containing a hash table with string keys and
(defvar *cache* (make-hash-table :test 'equal)) (defvar *cache* (psd:make-pub-sub-details-cache))
*mutex-for-pub-sub-details*: Protocol buffer messages can’t store lisp mutexes. Instead, we store the mutex for a
pub-sub-details in a new hash-table with string (username) keys.
This function makes a
psd:pub-sub-details protocol buffer message. It’s almost the same as the previous iteration of
pub-sub-details except for the addition of username.
... (make-instance 'pub-sub-details :password password)) (psd:make-pub-sub-details :username username :password password :current-any (google:make-any)) ...
(defmethod (setf psd:current-any)
(new-value (psd psd:pub-sub-details))
This is really a family of functions:
:around: When someone tries to set the
current-messagevalue on a pub-sub-details struct we want to write-protect the
pub-sub-detailsentry. We use an around method which activates before any call to the psd:current-any setter. Here we take the username from the
pub-sub-detailsmessage and write-hold the corresponding mutex in the
*mutex-for-pub-sub-details*global hash-table. Then we call
call-next-methodwhich will call the main (setf current-any) method.
(defmethod (setf current-any) (new-value (psd pub-sub-details)) (defmethod (setf psd:current-any) :around (new-value (psd psd:pub-sub-details))
(setf psd:current-any): This is the actual defmethod defined in cl-protobufs.pub-sub-details. It sets the current-messaeg slot on the message struct.
:after: This occurs after the current-any setter was called. We send an http call to all of the subscribers on the
pub-sub-detailssubscriber list. Minus the addition of the
psdpackage prefix to accessor functions of
pub-sub-detailsthis function wasn’t changed.
The main differences between the last iteration of proto-cache and this one are:
*-gethashmethod is exported by cl-protobufs.pub-sub-details so the user can call gethash on the hash-table in a map field of a protocol buffer message.
- (gethash username *cache*)
(psd:pub-sub-cache-gethash username *cache*)
- We add a mutex to the
*mutex-for-pub-sub-details*hash-table with the key being the username string sent to register-publisher.
- We return
tif the new user was registered successfully,
- The main difference here is:
(gethash publisher *cache*)
(psd:pub-sub-cache-gethash publisher *cache*)
- We have to use the
psdpackage prefix to all of the accessors to
(defun save-state-to-file (&key (filename "/tmp/proto-cache.txt")) "Save the current state of the proto cache to *cache* global to FILENAME as a serialized protocol buffer message." (act:with-frmutex-read (*cache-mutex*) (with-open-file (stream filename :direction :output :element-type '(unsigned-byte 8)) (cl-protobufs:serialize-to-stream stream *cache*))))
This is a function that accepts a filename as a string, opens the file for output, and calls cl-protobufs:serialize-to-stream. This is all we need to do to save the state of our applications!
We need to do three things:
- Open a file for reading and deserialize the Proto Cache state saved by save-sate-to-file.
- Create a new map containing the mutexes for each username.
- Set the new state into the *cache* global and the new mutex hash-table in *mutex-for-pub-sub-details*.
- We do write-hold the *cache-mutex* but I would suggest only loading the saved state when Proto Cache is started.
(defun load-state-from-file (&key (filename "/tmp/proto-cache.txt")) "Load the saved *cache* globals from FILENAME. Also creates all of the fr-mutexes that should be in *mutex-for-pub-sub-details*." (let ((new-cache (with-open-file (stream filename :element-type '(unsigned-byte 8)) (cl-protobufs:deserialize-from-stream 'psd:pub-sub-details-cache :stream stream))) (new-mutex-for-pub-sub-details (make-hash-table :test 'equal))) (loop for key being the hash-keys of (psd:pub-sub-cache new-cache) do (setf (gethash key new-mutex-for-pub-sub-details) (act:make-frmutex))) (act:with-frmutex-write (*cache-mutex*) (setf *mutex-for-pub-sub-details* new-mutex-for-pub-sub-details *cache* new-cache))))
The main update we made today was defining
pub-sub-details in a .proto file instead of a Common Lisp defclass form. The biggest downside is the requirement to save the
pub-sub-details mutex in a separate hash-table. For this cost, we:
- Gained the ability to save our application state with one call to cl-protobufs:serialize-to-stream.
- Gained the ability to load our application with little more then one call to
We were also able to utilize the setf methods defined in cl-protobufs to create :around and :after methods.
Note: Nearly all services will be amenable to storing their state in protocol buffer messages.
I hope the reader has gained some insight into how they can use cl-protobufs in their application even if their application doesn’t make http-requests. Being able to save the state of a running program and load it for later use is very important in most applications, and protocol buffers make this task simple.
Thank you for reading!
Thanks to Ron, Carl, and Ben for edits!