Triggering an OIC integration via OCI Events – the Oracle Functions Approach

In a two-part blog series I am exploring the available options in triggering an Oracle Integration Cloud (OIC) integration whenever a resource state change occurs within Oracle Cloud Infrastructure (OCI). One example of this event based pattern is the ability to trigger an OIC integration as soon as a file is uploaded to OCI Object Storage, thereby removing the need for any scheduled based integrations that rely on file polling.  

In my previous blog, I provided some context and background on the OCI Event service and the available options that we have in triggering an OIC integration. Namely these are the OCI Notifications, Functions, and Streaming services. My previous blog also explored the first of these patterns, and detailed how this event based pattern can be achieved using the OCI Notification Service. In this follow up article I will cover how we can use Oracle Functions to achieve the same outcome.

Intro to Oracle Functions:

To recap, Oracle Functions is a managed, serverless and elastic Functions-as-a-Service offering that is powered by the open-source Fn Project. You can use Oracle Functions when you want to deploy your code or application, without the administrative overhead of maintaining any infrastructure or operating system (OS) software. Oracle Functions ensures your app is highly-available, scalable, secure, and monitored.

Functions are packaged as Docker containers that encapsulate your code and any related dependencies. These containers are uploaded to the OCI Registry service which provides an Oracle managed Docker registry. The code within these containers can be written in any of the following supported languages; Java, Python, Node, Go, and Ruby. However if you would like to use another language or your existing docker containers, you should know that Oracle Functions allows you to also bring your own Dockerfile(s). When developing your code you can start off locally (developing and testing on your own machine) and when ready; you can push the containerized image(s) and associated metadata to Oracle Cloud Infrastructure.  Once your functions are deployed to OCI on the Functions service, you can trigger them directly (via a REST API or a language specific SDK) or configure them to respond to OCI events.

Once a function is triggered, Oracle Functions will verify the request against the OCI IAM service. If the request passes authentication and authorization checks, then the function service will identify the Docker image to pull from the OCI Registry and execute the function by running the functions image as a container on an instance associated with a subnet configured during creation time. This network setup allows you to use functions to interact with other services that live in the same subnet (e.g. Database as a Service). You can also use functions to interact with OCI resources that are not tied to a subnet such as an Autonomous Database, Object Storage and even Oracle Integration Cloud (which is what we will explore here).

Using Oracle Functions and the open-source frameworks on which it is based, you can ensure that you deploy cloud native – serverless applications with no vendor lock-in, whilst also eliminating operational overhead and ensuring that you pay only for the resources used during function execution time.

Although functions can be used in a variety of use cases, in this particular blog I will only explore the use of Functions in Response to OCI events. For more information on Oracle Functions, you can find the official documentation here.

The Functions Service Approach

In this blog, we will explore the ability to trigger a function based on an OCI event. To do this we will use one of the most common use cases; triggering an OIC integration as soon as a file arrives on OCI Object Storage. This approach will implement the following architecture:

So with the high-level architecture defined, let’s go into the specific steps required to get this setup and get it running!

Note: This architecture approach shares some common setup activities that were already captured in part 1 of this blog series. Although I could make pointers to the original documentation for the various steps, I have decided to replicate the shared steps in this blog post to ensure that readers can follow a single set of instructions to set up the above architecture.  

STEP 1: Getting the Event Payload

In this step, you will need to retrieve the event payload emitted by the OCI Event service whenever an object has been created. This payload is required so we can establish our OIC trigger connection. There are a number of approaches to retrieve this payload but below I have captured what I believe is the simplest:

1. Navigate to the OCI Event service via the cloud console and create a rule selecting Object Storage as the Service Name and Object-Create as the Event Type:

2. Click on the Validate Rule Button and you will be presented with the sample payload of the Event service:

3. Once you copy the event payload, you can cancel the event rule creation. We will come back to the OCI Event service in a subsequent step.

STEP 2: Configure your Oracle Integration Cloud (OIC) integration

Navigate to OIC and create an App-Driven Orchestration with a REST based adapter for the trigger connection. In the Resource Configuration screen of the trigger adapter, provide a relative resource URI that makes sense for your use case, define your action as POST and select the checkbox to configure a request payload. Note, a response payload is not configured as this pattern uses an asynchronous / fire-and-forget method. In your request configuration, attach the CreateObject JSON payload that is emitted from the OCI Event service (retrieved from step 1).

With the trigger connection defined, you can go ahead and build the rest of your integration flow as per your use case. If your flow requires the retrieval of the object that was just created (on Object Storage) you will need to add a GetObject step. If you want to know how to setup a connection to Object Storage from OIC, please check out my Object Storage with Oracle Integration Cloud – Part 1 blog. I also cover the required setup to list and retrieve objects in Object Storage with Oracle Integration Cloud – Part 2. It is perhaps worth pointing out that to retrieve an object form Object Storage you will need to provide the namespace, bucket name and object (resource) name. You can find all of this information in the Event payload that will trigger your OIC flow:

Once your OIC flow is ready, go ahead and activate it and capture your integration’s endpoint URL. We will need this endpoint in a subsequent step.

STEP 3: Configure your bucket to emit events

The next thing you need to do is ensure that your Object Storage bucket is configured to emit events. This option is available on bucket creation but you can also set it for existing buckets by navigating to the bucket and clicking the little edit next to Emit Object Events (see screenshot below). Enabling this option will allow you to leverage object storage events (state changes) through the OCI Event service. 

STEP 4: Configure your Oracle Function

In this step we will create and deploy a little function that will receive an event message which is emitted by OCI Object Storage and invoke an OIC integration passing the entire event message payload. If you don’t want to send the whole CNCF Cloud event to OIC you can always tailor a request payload for your OIC integration and adjust your function to create a request in the same format.

There are a number of ways to get started with Oracle Functions, the simplest of which is to use OCI Cloud Shell (a web browser-based terminal accessible from the Oracle Cloud Console). You can find detailed steps on this approach here. The following diagram depicts the required setup steps:

The above steps will guide you through all the configuration required to setup your tenancy to use functions and create and deploy your first function. I highly recommend going through these steps before returning to this guide as this will ensure your environment is successfully configured and ready for this specific function implementation.

For those of you who would prefer to use a local environment to create and test your function before pushing it to the Oracle cloud, you can follow the steps outlined here.

Assuming that you have configured your fn environment, deployed your first hello-world function and successfully invoked it, you are now ready to deploy this specific application which will trigger an OIC integration based on the arrival of a file on OCI Object Storage.

Please note that I have created this sample application with Python as this is what I am most familiar with, but if you want to, you can choose to re-implement it in your preferred programming language:

1. Create a Function Application:
   1. Log in to the Console as a functions developer and navigate to Solutions and Platform > Developer Services and click Functions.
   2. Ensure that you are in the region and compartment that you intend to use for Oracle Functions
   3. Click Create Application and specify:
      1. A name for the new application. In my case I have named mine stv_OBJStorage2OIC but you can choose any name that makes sense to you.
      2.  The VCN and subnet in which the function will run.
      3. Logging Policy. Setting an appropriate logging policy is important as this ensures that you can store and view the functions logs. This is especially important during the development and unit test stages to ensure that your function is behaving correctly. Until the Oracle Cloud Infrastructure Logging service is available, Oracle Functions provides one of two options to store logs; OCI Object Storage or SYSLOGURL. Here I would recommend exporting the logs to an external logging destination like Papertrail for real-time analysis of the functions logs. For configuration details on setting up either of these approaches please review the Oracle Function Logs documentation. Note: If exporting logs to an external destination, ensure that your security rules and routing rules are setup to support the outbound traffic.
   4. Click Create

2. Create and deploy your Oracle function:

Initialize the function by running the below command:

fn init --runtime python objstore2oic

cd objstore2oic

Edit the python script replacing the existing Hello World code with the following:

import io
import json
import requests
from fdk import response
 
def call_oic(event_json, oicbaseurl, oicusername, oicuserpwd):
    auth = (oicusername,oicuserpwd)
    headers = {"Content-Type": "application/json"}
    try:
        r = requests.post(oicbaseurl,auth=auth, headers=headers, data=event_json)
    except (Exception) as error:
        print('ERROR: In calling OIC', error, flush=True)
        raise
    return 'STATUS: SUCCESS' if r.status_code == 202 else 'STATUS: ERROR - ' + str(r.status_code)
 
 
def handler(ctx, data: io.BytesIO=None):
    try:
        cfg = ctx.Config()
        oicbaseurl = cfg["oic_base_url"]
        oicusername = cfg["oic_username"]
        oicuserpwd = cfg["oic_userpwd"]
    except Exception:
        print('Missing function parameter(s)', flush=True)
        raise
    try:
        event_py = json.loads(data.getvalue())
        event_json = json.dumps(event_py)
    except (Exception) as ex:
        print('ERROR: Bad event payload', ex, flush=True)
        raise
    result = call_oic(event_json, oicbaseurl, oicusername, oicuserpwd)
    
    return response.Response(
        ctx,
        response_data=json.dumps(result),
        headers={"Content-Type": "application/json"}
    )

fdk
requests

fn –v deploy --app stv_OBJStorage2OIC

3. Function Configuration:

As I did not want to store the OIC URL or credentials in the Python code, I have opted to set these up as configuration parameters which I can easily maintain through the cloud console. As per the documentation, user defined parameters come in two different flavors; application-wide or function-specific. Application-wide parameters can be used by all functions within an application, whilst function-specific parameters are only applicable to a single function. In my case I only have a single function within my application and hence have chosen to configure my user-defined parameters at the function level. To replicate my setup you will need to navigate to the newly created function and select configuration.  Once there you will need to setup the following Key-Value pairs:

1. oic_base_url:<The OIC integration endpoint URL from step 2>
2. oic_username: <OIC username>
3. oic_userpwd: <OIC user password>

Note: in this example i am storing my OIC (basic auth) credentials in the configuration of the function. In a non-demo scenario, we would recommend the use of the Oracle Cloud Infrastructure Vault service to store your OIC credentials. With this approach, you can configure your function to retrieve secrets (containing your OIC credentials) from the Vault service at runtime in order to securely authenticate with OIC and access your integration.

4. Test your app

Create a file containing a sample event under the application directory in your client (Cloud Shell or local).  Copy your sample event payload into this file (retrieved from step 1) and name it appropriately. I have named mine objStoreEvent.json. Now invoke your Function to test that it successfully triggers OIC by running the following command:

fn invoke stv_OBJStorage2OIC objstore2oic < objStoreEvent.json

If the function was successfully invoked it should return a Status: SUCCESS message. If not please review the function logs. Depending on how you configured the function application, these may reside on object storage or an external logging tool.

STEP 5: Configuring the OCI Event service

Navigate back to the OCI Event service in the OCI console. Here you will need to create an event rule for the Object Storage service that will listen for Object-Create state changes in a specific bucket and if a matching event is found it will invoke our Oracle Function.

In my setup I have defined the following Rule Conditions, but feel free to change these based on your requirements:

1. Event Type:
   • Service Name: Object Storage
   • Event Type: Object – Create
2. Add Condition > Attribute (filter):
   • Attribute Name: compartmentName
   • Attribute Value: Enter a desired Compartment (this will filter events from a specific compartment)
3. Add Condition > Attribute (filter):
   • Attribute Name: bucketName
   • bucketName: Ender a desired Object Storage Bucket (this will filter events only to the Bucket from where you want to source files for your OIC flow)

Under Actions I have defined the output destination for any matching events as my Oracle Function:

1. Action Type: Functions
2. Function Compartment: <select the compartment where your Oracle Function resides>
3. Function Application: <Select the name of your Oracle Function Application>
4. Function: <Select the name of your Oracle Function>

STEP 6: TEST your flow

At this stage please go ahead and test the flow by dropping a file in you object storage bucket:

Doing so should trigger your OIC flow:

Validate the payload and ensure that the file that you placed on Object Storage is what OIC has processed:

Conclusion

In this blog we explored using the Oracle Functions service to trigger an Oracle Integration Cloud integration flow. This event-driven approach allows us to react to events as they happen within Oracle Cloud Infrastructure, removing the need to poll for files residing on Object Storage using a pre-defined schedule.

Oracle Functions provide a simple yet powerful way to extend OIC implementations and they can be used for far more than just triggering an OIC flow whenever a file arrives. Some of the other use cases that we at Oracle have come across recently are captured below:

As you can see, serverless functions can be used in a variety of ways with Oracle Integration Cloud. The focus of this blog was to explore the ability of functions to trigger an OIC integration as a response to an OCI resource event. In a future blog I will show you how to invoke serverless functions from an OIC orchestration.

Thank you for taking the time to read this blog, I hope you have enjoyed it. Until next time … take care.

On a final note, i wanted to acknowledge and extend a big thank you to my amazing colleges in ANZ and those around the world who took the time to help me get this blog series published. Thank you all!