Do you want to trigger an Oracle Integration Cloud (OIC) integration as soon as a file is uploaded to OCI Object Storage?
This event driven approach allows you to respond to state changes in Oracle Cloud Infrastructure (OCI) in real-time, removing the need to poll Object Storage buckets on a predefined schedule. In a two-part blog series, I will explore how you can achieve this event-driven pattern with OIC. As the name suggests this blog will capture the Notifications Service Approach, while part 2 will provide a guide to using OCI Functions to achieve the same outcome.
OCI Event Service
A supported list of Oracle Cloud Infrastructure services emit events, allowing customers to create comprehensive automation solutions based on resource state changes within OCI. The messages produced by the event service follow the CloudEvents industry standard. These events report on state changes within a resource, for instance; the creation of an object in a bucket, or the completion of a database backup and even the creation of a Data Science model. The event service not only allows you to subscribe to events that you are interested in, but also allows you to specify filter parameters to ensure that you have control in processing only those specific events that you are interested in. These filter rules for instance allow you to process only events from a specific compartment and bucket. Once you have created your filter rules you will need to define an action (I like to think of actions as subscribers). An action or subscriber will be triggered when the event service finds a matching event based on your filter criteria. Today you can trigger one of the following OCI services as destinations for a matching event: Notifications, Streaming and Functions. These destination services present our available options in triggering Oracle Integration Cloud from OCI.
For more information on the OCI Event service including the supported services and associated events, please refer to the official documentation, which you can find here.
So now that we have identified the three available options to trigger Oracle Integration Cloud, let’s take a minute to evaluate them and their suitability to our use case:
OCI Notifications Service: broadcasts messages to one or more consumers using a publish-and-subscribe message pattern. The service can notify downstream systems whenever an event is triggered, an alarm threshold is breached or someone directly publishes a message to the service. The Notification service currently supports the following subscribers:
- Function (OCI FaaS)
- HTTPS (custom URL)
- PagerDuty
- Slack
You can find more about the OCI Notifications Service here.
As far as our use case (that is triggering an OIC flow whenever a file arrives on Object Storage), we can use the HTTPS subscriber method to trigger an OIC flow. In this approach we would create and activate our integration and present the integration’s endpoint URL to the Notifications service as a subscriber. The advantage of this approach is that it is very easy and quick to setup, however it is restrictive in terms of authentication as it only supports Basic Auth.
OCI Functions: is a managed, serverless and elastic Functions-as-a-Service offering that is powered by the open-source Fn Project and sits on top of the Oracle Cloud Infrastructure stack. You can use OCI functions when you want to deploy your code or application, without the administrative overhead of maintaining any infrastructure or operating system software. OCI functions supports code written in a variety of languages such as Java, Python, Node, Go and Ruby. Applications deployed as OCI Functions can subsequently be triggered in response to events. You can find more about the OCI Function Service here.
In regards to our use case, one could write a function that can respond to an Event emitted by an OCI resource (e.g. the creation of a file on Object Storage). Such a function could process the event payload and notify Oracle Integration Cloud. Functions offer the benefit of enhanced flexibility and control. A developer can leverage different authentication patterns when integrating to OIC, or can adjust the event payload to meet specific requirements passing for instance only the relevant information needed for the OIC integration flow. The disadvantage of using functions is that some integration developers may see a big learning curve in understanding and using the service as a deterrent. The reality is that once familiar with the function service, integration developers can leverage a huge amount of flexibility in meeting their various use cases.
OCI Steaming: provides a fully managed, scalable and durable storage solution for ingesting continuous, high-volume streams of data. When configured as a destination for the OCI Event service, messages such as the creation of a file on Object Storage can be written to a topic for subsequent consumption. To find out more about the OCI Streaming service, you can navigate here.
In the scope of our scenario, the OCI event service would provide a very simple way to consume event messages from OCI in OIC, however currently OIC does not support consuming messages form the OCI Streaming service. As this may change in the near future, please check current supportability prior to deciding on one of these approaches.
In this particular blog I have decided to document the first of these approaches, that is using the OCI Notifications Service. I will create another blog shortly that will depict the OCI Functions approach. In this way, developers can understand the required steps in both approaches and have a complete guide to implementing them.
The Notifications Service Approach
The approach will implement the following architecture:
Ok, with the architecture defined, let’s go through the required implementation steps.
STEP 1: Getting the Event Payload
In this step, you will need to retrieve the event payload emitted by the event service whenever an object has been created. This payload is required so we can establish our OIC connection. There are a number of approaches to retrieve this payload:
- The Email Approach:
- Configure your Object Storage bucket to emit events
- Configure a Notification topic with an Email subscriber
- Configure the Event service to receive an event when an object in your bucket is created and publish is to the Notification topic
- Confirm the subscription request once you receive an email from OCI asking you to do so
- With the above steps configured you can upload a file to you bucket. This should trigger the Event and Notifications service, resulting in the receipt of an email containing your expected event payload. This payload will contain your environment specific variables (compartment, bucket, etc.)
- The Event Approach:
- Navigate to the Event service and create a rule selecting Object Storage as the Service Name and Event Type as Object-Create:
- Click on the Validate Rule Button and you will be presented with the sample payload of the Event service:
This approach presents you with an easy and standard way to get the payload of any event within OCI.
- The Documentation Approach:
- You can also retrieve the event payload for the Create Object event from the Oracle Cloud Infrastructure documentation available here (navigate to Object Event Types). Be mindful that, at the moment the above documentation does not publish examples of all of the Object Storage event types.
STEP 2: Configure your Oracle Integration Cloud integration
Navigate to OIC and create an App-Driven Orchestration with a REST based adapter for the trigger connection. In the Resource Configuration screen of the adapter, provide a relative resource URI that makes sense for your use case, define your action as POST and select the checkbox to configure a request payload. Note, a response payload is not configured as this pattern uses the asynchronous / fire-and-forget flow. In your request attach the CreateObject JSON payload that is emitted from the OCI Event service. I am including it below to provide a visual representation for this blog:
{
"eventType" : "com.oraclecloud.objectstorage.createobject",
"cloudEventsVersion" : "0.1",
"eventTypeVersion" : "2.0",
"source" : "ObjectStorage",
"eventTime" : "2020-03-19T12:17:07.369Z",
"contentType" : "application/json",
"data" : {
"compartmentId" : "ocid1.compartment.oc1..aaaaaaaa3r4di4245vse5bpryfexxxx…",
"compartmentName" : "stan_tanev_sandbox",
"resourceName" : "SourceAPFile-20191111-8GOOD.csv",
"resourceId" : "/n/xxx/b/stv_bucket_APINV/o/SourceAPFile-20191111-8GOOD.csv",
"availabilityDomain" : "SYD-AD-1",
"additionalDetails" : {
"bucketName" : "stv_bucket_APINV",
"archivalState" : "Available",
"namespace" : "xxx",
"bucketId" : "ocid1.bucket.oc1.ap-sydney-1.aaaaaaaav6zg4yafzzipiwgadbaummjunujsxxxx…",
"eTag" : "94075dfa-17f8-48e6-a8f3-5ac3e9af06d6"
}},"eventID" : "a85c04d3-8848-d994-d353-6fec23172f52",
"extensions" : {
"compartmentId" : "ocid1.compartment.oc1..aaaaaaaa3r4di4245vse5bpryfexn5nqk2bqiuzjxxx…"
}}
With the trigger connection is defined, you can go ahead and build the rest of your integration flow as per your use case.
If this flow requires the retrieval of the object that was just created you will need to add a GetObject step. If you want to know how to setup a connection to Object Storage from OIC, you can check out my Object Storage with Oracle Integration Cloud – Part 1 blog. I go into the setup required to list and retrieve objects in Object Storage with Oracle Integration Cloud – Part 2. It is perhaps worth pointing out that to retrieve an object form Object Storage you will need to provide the namespace, bucket name and object (resource) name. You can find all of this information in the Event payload that will trigger your OIC flow:
Once your OIC flow is ready, go ahead and activate it and capture your endpoint URL. We will need this for the subsequent steps.
STEP 3: Configure your bucket to emit events
The next thing you need to do is ensure that your Object Storage bucket is configured to emit events. This option is available on bucket creation but you can also set it for existing buckets by navigating to the bucket and clicking the little edit next to Emit Object Events (see screenshot below). Enabling this option will allow you to leverage object storage events (changes of state) through OCI’s Event service.
STEP 4: Configuring the OCI Notifications service
The next step is to create an OCI Notification Topic. To do this, navigate to Application Integration >> Notifications from the main menu. Topic creation is very simple as the only required input parameter is the topic name:
Once the topic is created. You will need to select it and create a HTTPS subscription. Here you will need to copy the URL endpoint of your activated OIC integration and prefix it with your basic auth credentials. The URL should follow the following syntax: https://username:password@OIC_Integration_EndPoint_URL
Note: Please ensure that you escape any special characters (e.g. ‘@’) in your username or password.
Once you do this you will notice that the subscriber will go in a pending state as it will be awaiting confirmation. To activate the notification subscriber, you will need to extract the ConfirmationURL which the OCI Notifications Service sent in the request to OIC. So please navigate to OIC and view the Activity steam of your triggered integration, looking for the Confirmation URL (note the first integration run would be in a failed state):
As can be seen from the above, you can extract the Confirmation URL and paste it into a browser. Once you do this, you will see the following confirmation:
The HTTP subscriber status should now also change to confirmed in OCI:
Note: if you cannot see a triggered instance run in OIC, please download the activity stream and check the icsflow-logs for a Confirmation URL. Alternatively you can resend the confirmation email from the notifications service:
STEP 5: Configuring the OCI Event service
You will find the OCI Event service under the main Menu >> Application Integration tab. Here you will need to create an event rule for the Object Storage service that will listen for Object-Create state changes and perform some filtering and actions.
In my setup I have defined the following Rule Conditions, but feel free to change these based on your requirements:
- Event Type:
- Service Name: Object Storage
- Event Type: Object Create
- Attribute (filter):
- compartmentName: <Filter events only to the desired Compartment>
- bucketName: <Filter events only to the Bucket from where you want to source files for your OIC flow>
Under Actions I have defined the output destination as my target Notifications service topic:
- Action Type: Notifications
- Notifications Compartment: <select your particular compartment>
- Topic: <Select the name of the topic which you created in the previous step>
STEP 6: TEST your flow
At this stage please go ahead and test the event & notifications service by dropping a file in you object storage bucket:
Doing so should trigger your OIC flow:
Validate the payload and ensure that the file that you placed on Object Storage is what OIC has processed:
Conclusion
In this blog we explored using the OCI Notifications service to trigger an Oracle Integration Cloud integration. This event-driven approach allows us to react to events as they happen within Oracle Cloud Infrastructure. With this pattern OIC no longer needs to poll for files on Object Storage using a pre-defined schedule. In the next series of this blog we will explore how to receive events emitted by OCI services and forward them to OIC using the OCI Functions service.
I hope you have enjoyed this blog. Until next time …
RedThunder.Blog 
Thanks for the post!
I am stuck at STEP 4 now. My OIC integration is activated state and it is still not receive the confirmation message. I try resending another confirmation from OCI Subscription, but it cannot receive the message.
LikeLike
Hi Ronan – To triage this step (in smaller “parts”), are you able to replicate calling the integration flow directly via OIC with basic authentication? Is it related to the security? or the format? Are you seeing any errors in the OIC monitoring?
LikeLike
Thanks for the post!
I am now stuck at STEP 4.
My OIC integration is now activated, but it still is not received the confirmation message from OCI Subcriptions.
I try sending another one, but it still shows nothing.
LikeLike
Same (with the previous comment) – to triage this step (in smaller “parts”), are you able to replicate calling the integration flow directly via OIC with basic authentication? Is it related to the security? or the format? Are you seeing any errors in the OIC monitoring?
LikeLike