Serverless on Always Free Tier with fnproject

I’ve started posting articles related to the project that @stantanev and a few of us are working on. This is snapshot of the puzzle that is to build out a APIs on the Oracle Always Free Tier.

As a demonstration of capability, we built a few different APIs using fnproject (https://fnproject.io/) – an open-source container-native serverless platform. As part of Oracle Cloud Infrastructure, there’s Oracle Functions which is the managed Function-as-a-Service based upon this same project.

Let’s take a look at it here and see what it took to get going. Also, this is being deployed into VM.Standard.E2.1.Micro compute shapes (which is 1 OCPU and 1GB of memory) and hence there are some considerations to make sure we get the most out of the kit we have access to (for free).

To explain this a little more, this is an architecture of the OCI Arcade stack that we are building. The main focus (in this blog) is the set of serverless functions that track events in the game. For example, the pacman game tracks things like – when the game starts and is over, when the player starts a new level or when a power pellet is eaten or the effects have wore off. Each event has information about the game instance, the game itself (this is an arcade), the score and some coordinates (x, y, z). Based upon this, there’s some interesting streaming analytics off the back (different story).

The fnproject has plenty of documentation about getting started and installing the project. Here is a link to the installation process – https://fnproject.io/tutorials/install/.

To check out what we’ve built, you can look at the code here – https://github.com/jlowe000/oci-arcade/tree/free-tier/apis/events. From here, there’s a couple of different variations that we played with.

• Under serverless/js, we’ve got a NodeJS version that is a synchronous API that invokes the autonomous database. It uses username / password for authentication. NB: I’ve cut this version out from the main architecture (so deprecated) but left here as an example.
• Under serverless/python, we’ve got a python version that is a synchronous API that invokes the autonomous database to store the events. This one uses OAuth tokens for authentication. NB: This is the one that we are currently using in the Terraform build.
• Under kafka/event-producer/java, we’ve got a Java version that is a Kafka producer. This publishes the events to the Kafka topic. NB: This has been deprecated but here as an example.
• Under kafka/event-producer/python, we’ve got python version that is a Kafka producer and similarly to the Java version – sends the events to a topic. It supports both PLAINTEXT and SASL_SSL / PLAIN for authentication. NB: This is the one that we are currently using in the Terraform build.

NB: We’ll try to keep the code available as examples however even the list above may change as we continue to evolve and build. Use the git repository as the source of truth.

Beyond developing a set of APIs for handling the events or even simplifying the whole deployment process with this architecture, there’s a few things that we needed to consider and look into – and they are based upon the fact that we had limited resources – compute and memory.

Runtime Options

Reference: https://github.com/fnproject/docs/blob/master/fn/operate/options.md

We needed to consider memory. As we packed more things into the compute node, we experienced this …

Requested CPU/Memory cannot be allocated

This is a problem … And it was also dependent on what else was running and consuming at that point in time. This was such a pain – intermittent runtime issues – argh. Hence, we looked at the different options to make it run leaner. We investigated the following:

• In each function, there is a configuration where we can set the memory. By default, its 128MB (expands to 8GB). We needed to go the other way and the minimum is 64MB. For what we were doing, this is plenty.
• In some recent experimentations moving from Oracle Linux 7.8 to 7.9, we got the same outcome – can’t allocate CPU or memory. The fnproject is based upon docker and both the fnserver (that is the daemon process) as well as the serverless functions can be augmented by docker configuration. So, I looked here for how to squeeze more out of docker. I’ve left this as an experiment at the moment – still looking into this. https://docs.docker.com/config/containers/resource_constraints/
• There are CPU related configurations however, we hadn’t stretched the sides of this yet so we hadn’t really looked into this for the purposes of what we were doing.

Debugging

Reference: https://fnproject.io/tutorials/Troubleshooting/

Whilst I was chasing these issues down, the debugging and information levels were were important – more runtime data meant better understanding. There’s a couple of things that helped here.

DEBUG=1

It’s a simple thing that added this in front of the fn command line added debugging to the stdout. This gives you additional debugging on the fn lifecycle. The common pattern that I used was:

echo '{ "instance_id" : 1, "game_id" : 1 }' | DEBUG=1 fn invoke events publishevent

If you are struggling with the application level side about what’s happening, application logging can be a little hidden. The debug flag above is great at a glance but its still black-box testing. Here’s the next level, fnproject supports syslogging which helps by pushing anything from stdout and stderr to syslog. Here’s a sample of the output (from this morning).

Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.producer.kafka - DEBUG - Starting the Kafka producer
Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.metrics.metrics - DEBUG - Added sensor with name connections-closed
Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.metrics.metrics - DEBUG - Added sensor with name connections-created
Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.metrics.metrics - DEBUG - Added sensor with name select-time
Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.metrics.metrics - DEBUG - Added sensor with name io-time
Feb 16 21:43:23 arcade-web app_id=01EY1NVWAWNG8G00GZJ0000001,fn_id=01EY1NYS70NG8G00GZJ0000003[11700] 01EYPCRFGRNG8G00GZJ00001CN - kafka.client - DEBUG - Initiating connection to node bootstrap-0 at oss.streaming.ap-sydney-1.oci.oraclecloud.com:9092

On Oracle Linux, you can install rsyslogd to accept the logs from fnproject via tcp whilst you configure the fnserver daemon or the fn application to send syslog messages to rsyslog at that tcp port. This is something that I’ve been trialling and it’s likely to be baked into the Terraform build.

Other Options

Reference: https://github.com/fnproject/docs/blob/master/fn/operate/options.md

In building this up, we explored a few other scenarios during our travels not specific to running on the Always Free Tier but useful all the same. Here’s a collection of the rest of the stuff.

• CORS support – if you need to call these APIs directly from the browser then there’s a couple of environment variables to set up the allowable origins and headers. We eventually didn’t need this as I mediate this through the NodeJS server api-score (we do some tricks here too).
• Docker Network – we were conscious about creating a docker network for the functions and APIs to run over. This became VERY handy when we added Kafka to the architecture and also running Docker Swarm. We had to define an overlay network such that everything was internally accessible. The fnserver daemon process and the individual functions themselves can be configured to attach to a defined docker network. This was necessary for api-score to connect to the fnserver (front-end) and for two functions to connect to Kafka (back-end).
• FN configuration – we use the configuration heavily especially for variables that can be configured via Terraform. Passing these configurations as Runtime Context Variables made the code easier to work with.

Most of these things here is to support the fnproject on infrastructure. And looking back at what’s been expressed here, much of this goes away when using Oracle Functions (the managed-cloud version of fnproject). The one thing that is apparent when you craft it yourself, you then appreciate the engineering effort and the simplicity when you do use these other cloud services.

If you want to try this out yourselves, you can get an Always Free Tier environment – head to the following link and sign-up. https://www.oracle.com/au/cloud/free/.

To get the whole thing up and running, then head to this previous blog to see how to get the OCI Arcade up and running (and playing) – https://redthunder.blog/2021/02/12/automating-with-oci-oracle-resource-manager/

Have fun learning, experimenting and if you want to contribute to any of these things – reach out. #CommunityMatters #ItTakesAVillage