#DaysOfArm (13 of X)

This is my 13th #DaysOfArm article that tracks some of the experiences that I’ve had so far. And just to recap from the first post (here) on June 12 2021.

It’s been just over 2 weeks since the launch of Ampere Arm deployed in Oracle Cloud Infrastructure (OCI). Check this article out to learn more (here). And it’s been about one week since I started looking into the new architecture and deployment, since I started provisioning the VM.Standard.A1.Flex Compute Shape on OCI and since I started migrating a specific application that has many different variations to it to test it all out.

This is my next learning is another retrospective with the OCI Arcade deployment the full stack is now being deployed on 1 OCPU with 6 GB of RAM in an Always Free Tier tenancy.

Continue reading “#DaysOfArm (13 of X)”

Using OCI Burstable Instance

With the work that I’ve been doing with Open Street Map (here), I’ve been provisioning Pelias (here) – an open-source implementation of geocoding. This architecture is not small (consisting of 10+ docker images, and potentially 100+GB of raw geo data) especially if you are looking to geocode the whole world. The workload (or pipeline) had 4 main stages – download, prepare, import and query.

  • Download – to get the raw data sources
  • Prepare – to get the raw data into a format that can be easily imported
  • Import – to import the data into the elastic search (which is the backend)
  • Query – to accept geocode queries

Each of these stages have different performance characteristics and required different resources. The main thing that I’m looking at here is the use of compute. The need for compute during the prepare and import stages is significantly different from the download and query stages. I’m also not confidently in terms of when or how much I need.

And this is why I configured a burstable instance.

Here’s a couple of things to know …

  • There is a baseline utilisation OCPU. Consider this as a the minimum compute you want. For my scenario, it was primarily how much compute that I needed for the download and query stages.
  • There is full utilisation OCPU. Where this is can be 2x or 8x the baseline utilisation. (in the terms of the documentation – the baseline utilisation can be either 12.5% or 50% of the full utilisation OCPU). For my scenario, it was primarily the prepare and import stages that needed the additional compute.
  • The increased capacity is based upon the CPU utilisation metrics to determine whether to burst.
  • The average CPU utilisation for the month needs to up to the baseline utilisation OCPU.

Burstable Instances billing is known. It doesn’t come with Bill Shock.

You can find out more about Oracle Cloud Infrastructure burstable instances (here). If you want to try this out yourself or work on your own application, sign-up (here) for the free Oracle Cloud Trial. I’d be interested to hear your experiences and learn from others as well. Leave a comment or contact me at jason.lowe@oracle.com if you want to collaborate.

Free In-Memory Data Grid

Here’s the architecture of the OCI Arcade extended with Coherence-CE

I was talking to Tim Middleton who works in the Coherence Development team the other day about different scenarios that we can work on some areas that demonstrate the scale out – two scenarios were obvious (to me) – Apache Spark was one and the other was Coherence – an In-Memory Data Grid.

And with that I built out the OCI Arcade to bring Coherence-CE (Community Edition) into the mix. There’s some lots of cool stuff that you can do with Coherence-CE and the bulk of the content is (here).

To get access to the OCI Arcade with Coherence – you can find these on a git branch in the repositories.

The simple use-case that we used Coherence-CE for was a simple identity key for the instance id (ie each game plays has a unique identifier). The nice thing was that it was all in Javascript. This following fragment was the only thing that I needed to do to integrate Coherence-CE into the app itself.

const opts = new Options()
opts.address = 'oci-cache:1408'

app.get('/id', (req, res) => {
  var game_id = req.query.game_id;
  var session = new Session(opts)
  var map = session.getMap('oci-id')
  setImmediate(async () => {
    console.log('Map size is ' + (await map.size))
    if ((await map.has(game_id)) == false) {
      await map.set(game_id, { id : 1 })
      res.send('{ "id" : 1 }');
    } else {
      res.send('{ "id" : '+(await map.invoke(game_id,Processors.increment('id',1)))+' }')
    }
    await session.close()
  })
});

From the Coherence-CE cluster perspective – there’s docker images from docker hub already (here) that I could use for it. So replacing what I did previously with Kafka in the deployment architecture and putting Coherence-CE in its place was simple. (NB: There were some changes in terms of stopping events being fired to Kafka and also creating the instance id from this method instead of a Date.now() method. Simple things).

There’s plenty of different directions that we can go from here – session-based cache scenarios, high-availability scenarios, in-memory data processing scenarios. There are plenty to extend and learn from.

If you want to try this out yourselves, you can get an Always Free Tier environment – head to the following link and sign-up. https://www.oracle.com/au/cloud/free/.

To get the whole thing up and running, then head to this previous blog to see how to get the OCI Arcade up and running (and playing) – https://redthunder.blog/2021/02/12/automating-with-oci-oracle-resource-manager/. Remember – use the coherence-ce branch.

I acknowledge Mason Borda for their contribution in building this game.

Have fun learning, experimenting and if you want to contribute to any of these things – reach out. #CommunityMatters #ItTakesAVillage

#WorldInnovationDay Hack Information Pack

We are providing to each participant access to Oracle Cloud Infrastructure (OCI) for the hackathon. The following resources are to make you effective with OCI.

Oracle Cloud Infrastructure

From past hackathons, we noticed a trend of what technologies and services are being used by the winning teams. Here are the top resources with some additional references.

  • For a single platform to store JSON, Graph & Spatial, perform ML or deliver Low-Code Apps, use Autonomous Database (here)
  • For establishing infrastructure with your OS of choice, use OCI Compute (here)
  • For Jupyter notebooks and ML packages, use Data Science Cloud (here)
  • For quick self-service and data visualisation, use Analytics Cloud (here)
  • For making sure everything is secured, use Oracle Security (here)
  • For most things you need for #AppDev, use Oracle AppDev (here)
Live Labs

You will need to learn quickly. These hands-on workshops will help you learn more about the services available.

  • For those who are interested in a single platform to store data (here)
  • For those who want to #lowcode (here)
  • For adding Blockchain to your datasets (here)
  • For those who are interested in data science (here)
  • For ML on datasets in the database (here)
  • For data storytelling with data visualisation (here)
  • For building microservices applications (here)
Next Steps

There will be a series of workshops where we will be present on different OCI topics in the week leading up to the hackathon. Make sure you have these in your calendar so you don’t miss out.

And reach out on the #WorldInnovationDay Hack 2021 Slack workspace if you need anything from the team.

Simple, Secure Log Retention using OCI Services

Between the Oracle Cloud Infrastructure (OCI) Audit Service recording administrative operations and Oracle Identity Cloud Service (IDCS) tracking access and user management events, Oracle Cloud provides pretty comprehensive tracking of security events. Recently however, I have been in conversations with a number of organisations who have been seeking longer term retention of audit events for several different reasons, including governance, compliance and forensic requirements. The OCI Audit service supports requesting bulk export of audit events to Object Storage, which allows you manage the retention and archiving of those events yourself, so I started wondering if I could do the same for the IDCS access events. A bit of testing and some simple coding later, and I had events being pulled directly from the IDCS Audit Events API periodically and sent to Object Storage for retention.

In this blog post, I will provide this code as a sample, and discuss some of the techniques and technologies that are available in Oracle Cloud Infrastructure to enable simple, but highly secure and cost-effective, automation of cross-service tasks such as this.

Continue reading “Simple, Secure Log Retention using OCI Services”

Teaching How to Invoke Gen2 Oracle Cloud Infrastructure (OCI) resources via REST APIs

I am thrilled with the Oracle’s Gen2 Cloud Infrastructure architecture, where Oracle completely separates the Cloud Control Computers from the User Code, so that no threats can enter from outside the cloud and no threats can spread from within tenants.

Obviously with more security, there comes more coordination, especially at the moment of invoking OCI resources APIs. Luckily, Oracle did a good job at providing a simple to use CLI and SDK (see here for more information).

For the purpose of this blog, I built a simple NodeJS application that helps demystify the security aspect of invoking OCI APIs. Check this link for examples of running similar code across other Programming Languages.

My NodeJS application manages OCI resources in order to:

  • List ADW instances
  • Stop an ADW instance
  • Start an ADW instance

I started this NodeJS application to list, start and stop ADW resources. However, I designed this application to easily extend it to invoke any other type of OCI resources.

I containerised this application with Docker, to make it easier to ship and run.

This is a picture of the moving parts:

Continue reading “Teaching How to Invoke Gen2 Oracle Cloud Infrastructure (OCI) resources via REST APIs”

Circuit Breaker in Service Mesh – Istio/Envoy

This Lab, logically follows previous steps required to provision and curate a Kubernetes cluster. Please review them before proceeding. If you are in doubt, feel free to contact me directly via https://www.linkedin.com/in/citurria/

Testing BookInfo app with Circuit Breaker based policy

The third and last test in the Service Mesh, is using a Circuit Breaker based pattern. It further protects our microservices in case of certain conditions occur, such as preventing that an unexpected number of requests overflow and affect the microservices in the service mesh.

We might decide to throttle or simply reject new incoming requests when a number of current incoming http requests reaches certain threshold.

For demonstration purposes, we are going to set rules to allow a maximum of 1 request at a time. If more than 1 request comes in, we will prevent it from entering the mesh.

Continue reading “Circuit Breaker in Service Mesh – Istio/Envoy”

Service Mesh 101 – Getting familiar with Istio and Envoy

This Lab, logically follows previous steps required to provision and curate a Kubernetes cluster. Please review them before proceeding. If you are in doubt, feel free to contact me directly via https://www.linkedin.com/in/citurria/

Introducing Service Mesh

Continue reading “Service Mesh 101 – Getting familiar with Istio and Envoy”

Cloud Native App Dev Series…

Welcome to the Cloud Native App Dev Series. We built and constantly maintain this space to help others get up to speed with Cloud Native App Dev technologies and discover the power of Autonomous Cloud Services.

 

  • Kubernetes:
    • Containers 101: Learn how to containerise and ship your applications to Kubernetes using Docker
    • Kubernetes 101: Get familiar with Kubernetes architecture and components.
    • Kubernetes Dashboard: Deep dive into the Kubernetes dashboard components and concepts.
    • Provision Kubernetes: Learn how to install Kubernetes locally or in the Cloud.
    • Microservices Deployments 101: Learn how to deploy your Microservices into Kubernetes.
    • Cheeses App – Self-Healing and Scalability: Discover how simple it is to run and scale your microservices in Kubernetes.
    • Socks-shop Polyglot App: Play with a polyglot-based microservices application in Kubernetes.
    • (Work in progress…) Roll out your Microservices: Learn how to roll out and roll over your Applications in Kubernetes
    • (Work in progress…) Operators and Service Broker: Learn how to consume other Cloud Services as native Kubernetes workloads.

       

  • Serverless:
    • (Work in progress…) Serverless 101: Get up to speed with Serverless.

     

  • Service Mesh (Istio and Envoy):
    • Service Mesh 101: Get familiar with Service Mesh, using technologies like Istio and Envoy.
    • Policy Enforcement: Learn how to enforce simple routing policies in Service Mesh.
    • Circuit Breaker – Network-Latency based routing. Learn how to enforce more complex Service Mesh routing policies, based on network latency.

       

  • PaaS:
    • Build Modern Web development: Use Autonomous Visual Builder to construct modern Web UIs in minutes
    • Autonomous Data Warehouse 101: Learn how to provision Autonomous Data Warehouse in minutes. Load millions of records in seconds and visually analyse the data.
    • (Work in progress…) Running Visualization on Autonomous Data Warehouse: Learn how to connect Data Visualizer to Autonomous Data Warehouse.
    • Chatbot Development: Learn how to build chatbots in the Cloud.
    • (Work in progress…) Blockchain 101: Get familiar with Blockchain concepts and use cases.

 

Congratulations!!! You have enough to start building your own Cloud Native Applications.

We will keep publishing interesting topics, so you better stay tuned! If you have suggestions for new topics, please contact me directly via https://www.linkedin.com/in/citurria/

Thanks for your time.

*Special thanks to Cameron Senese for being a great mentor and a Cloud warrior, helping us demystify the exciting world of Kubernetes and Microservices… As we keep demystifying technologies and patterns, we are going to keep adding interesting topics, so you better stay tuned!

 

Socks-shop Polyglot App in Kubernetes…

This Lab, logically follows previous steps required to provision and curate a Kubernetes cluster. Please review them before proceeding. If you are in doubt, feel free to contact me directly via https://www.linkedin.com/in/citurria/

In this Lab, we will show how to manage a more complex Microservices-based application based on an E-Commerce Socks-shop App (also see here). Similarly, as with the Simpler Cheeses App, we are going to use Weave Scope to gather real-time insight into this more complex application.

This is an e-commerce application that sells socks on web. However, we chose this application because it is not any different to any modern Application. That is, it is based on multiple microservices, where each one use different technologies as programming languages/frameworks, as well as persistent back stores or databases.

That is:


For more information, see: Weave-socks multiple technologies
(and github).

Continue reading “Socks-shop Polyglot App in Kubernetes…”