If you’re running workloads in Oracle Cloud Infrastructure (OCI) then it’s likely you’ll be familiar with Virtual Cloud Network (VCN) resources such as Subnets, Route Tables, Gateways etc. These software defined components allow you to build networks in OCI for you to deploy and run your workloads.
Oracle has documentation that explains VCN access and security features which include things like Security Rules, Security Zones, Local and Network Firewalls, and IAM policies. Security rules are made up of Security Lists and Network Security Groups (NSG’s) and are a foundational element of every VCN and Subnet that you create. They define what traffic is allowed in and out of your subnets and what hosts can talk to one another. When you create a subnet a Security List is automatically created with some default rules:
When it comes to implementing network access controls, you can use Security Lists, Network Security Groups or both. They are virtual firewall features that control traffic at the packet level. I’ll be covering Network Security Group reviews in a later post as I want to focus on Security Lists, specifically how you can easily review and validate rules to ensure they align with your workload, organisational, security and compliance requirements.
Why perform rule reviews?
Because it’s good Security Operations (SecOps). By reviewing rules regularly, misconfigurations and mistakes can be found and remediated reducing the likelihood of a security incident. Many Information Security and compliance standards such as the Payment Card Industry Data Security Standard (PCI DSS), or the National Institute of Standards and Technology (NIST) Special Publication 800-41 require or recommend organisations review firewall rules at least quarterly or bi-annually. To meet your compliance obligations you may also need to document and provide evidence to auditors that rule reviews are regularly performed and all required actions were taken.
Simplifying the review process
If you are planning to perform a review there are a few things which typically need to be well understood first:
- What does the network architecture look like and is there up to date documentation, diagrams, and requirements available.
- What are the approved traffic flows, such as cloud to on-prem, intra-cloud, or to and from the Internet.
- What traffic protocols (TCP / UDP / ICMP) and ports are required.
- Are there any flows that are no longer required as the result of services being moved, redesigned, or decommissioned.
Sometimes it’s not always possible to have all the required information readily available. Finding myself in this situation recently I began thinking about how I could undertake a review of Security List rules by looking at existing workload traffic flows running in OCI.
The OCI Networking Service allows you to enable VCN flow logs, which records each flow or connection that is established to, or from VNIC’s that reside within your subnets. Using the VCN flow telemetry and Security List rule definitions I’ve built a tool to run in OCI that makes it possible to:
- Identify redundant rules in Security Lists because of port range and IP range overlaps:
- Identify Security List rules and Subnets that are not in the path of network traffic flows observed in the VCN flow logs:
- Understand what are the minimum rules required to permit traffic observed in VCN flow logs:
Using the example information above I can easily identify:
- Rules that are potentially too permissive, such as the ingress rule allowing all protocols and ports from 10.0.0.0/8 which also overlaps the stricter ingress rule TCP port 443 from 10.12.15.128/27.
- Potential rule direction mistakes, such as the ingress rules allowing all protocols and ports from the Oracle Services Network (OSN) as I know my workload connections are outbound to the OSN only.
- Which rules may no longer be required, such as the ingress rule for IDAM access from 172.16.0.0/24 as no flow logs match this rule.
- Potential access that should not be permitted, such as the HTTPS traffic from 10.25.15.128/27 as I know this source network is meant to be decommissioned.
- What traffic that should perhaps be allowed but is currently rejected, such as the ingress UDP RDP traffic on port 3389.
The assertions that you will be able to make will differ from mine as they are workload, network and environment specific. In my case I needed to take the information from the report and speak with the relevant teams to understand why the rules existed and why the flows were required.
From here I used the tool & team insights as inputs to inform what rules need to be removed, added, or modified. Because this tool relies on VCN flow log telemetry, minimum rules are calculated as stateful rules based on actual observed flows and scoped to either:
- The CIDR block of another Subnet in the same VCN, or
- A /24 or /16 CIDR block where it does not match any other subnet CIDR’s in the same VCN.
Scoping CIDR’s to a maximum of /24 or /16 was a conscious decision because I did not want to have rules with CIDR’s scoped to /8 or /0 and it makes more sense for a human to interpret smaller CIDR ranges and extend to a /8 if suitable. Therefore it is important that you confirm the exact source and destination CIDR’s, protocols and port ranges required for your workload before making any changes. As an example I might determine my on-prem network 10.0.0.0/8 should have access to my workload on TCP port 4443 and therefore replace the rules for 10.25.15.160/27, 10.69.3.0/24, and 10.81.64.0/24.
Setup & Configuration
To run this tool yourself you’ll first need to enable VCN flow logs for all OCI regions and subnets you wish to evaluate associated Security List rules. I recommend enabling VCN flow logs for all VCN subnets.
For each Subnet, enable the Flow Log. When enabling the first Subnet VCN flow log create a new Log Group. When enabling subsequent VCN flow logs, select the existing Log Group:
Now we need to create Object Storage buckets in each region where we’ve enabled our VCN flow logs. Ensure all buckets have the same name:
Now we need to create service connectors in each OCI region to push our logs from Logging to Object Storage. Select each VCN flow log that you’ve enabled in source connections. Select the Object Storage bucket you created earlier as the target. Ensure you also click “Create default policy” before creating the Service Connector.
And that’s the setup completed. After a few minutes you should see logs in your Object Storage bucket:
Before continuing it is important that you allow enough time for all traffic flows to be observed and logged. As mentioned earlier the tool relies on VCN flow logs and if traffic flows aren’t observed, then it may flag required rules as unused, or omit rules for minimum required traffic flows. In my situation I waited one week (7 days) to be sure all typical traffic flows were captured. After 7 days I had approximately 1,300 files totalling 3GB in my Object Storage bucket.
Running the tool
To run the tool you’ll need to create a MySQL DB System, and Compute Instance in OCI to run the Docker container. Appropriately sizing the DB System and Compute Instance is important to ensure the tool can import and process all the data in a reasonable timeframe.
First create a Virtual Cloud Network with Internet Connectivity using the Wizard:
Now add an Ingress rule in the Security list for the Private Subnet. This will allow our compute instance to talk to the DB System that we will create in the Private Subnet:
For the DB System I created a standalone instance in the private subnet. I initially selected the shape MySQL.VM.Standard.E4.8.128GB with 200GB storage. After running the tool I validated I’d imported nearly 80M records:
mysql> select count(*) from flowlogs;
+----------+
| count(*) |
+----------+
| 79291781 |
+----------+
1 row in set (2.61 sec)and the database system was using over 80% of the 128GB memory allocated:
Looking at my Disk Utilisation I can see I’m using approximately 50% of my 200GB so know that I can likely allocate less disk storage next time I run the tool:
Depending on the number of files in your Object Storage bucket, and how fast you want the script to run, you may want to select a more appropriate shape. If you’re not sure, you can always select a larger shape and review the metrics after the tool has run.
Go ahead and create a MySQL DB System in the private subnet. Note down the username and password you enter as you’ll need this later. I also disabled backups as these were not required.
Now create a compute instance in the public subnet. I selected a Ubuntu 22.04 image, AMD.E4 Flex shape with 4 OCPU’s and 32GB memory.
Update your SSH config file and add an entry for this host. We’re adding a LocalForward so we can view the reports that are generated by the tool without exposing the port to the Internet or having to copy files from the docker container:
Host network-analyzer
Hostname <Public IP Address>
User ubuntu
ServerAliveInterval 60
IdentityFile ~/.ssh/oci_ssh.key
LocalForward 8000 127.0.0.1:8000Connect to the instance and install Docker:
$ ssh network-analyzer
$ sudo apt-get install -y ca-certificates curl gnupg lsb-release
$ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
$ newgrp docker
$ sudo usermod -aG docker $USERNow we need to copy our OCI API keys and config to the instance as we’ll be mapping the .oci folder to the Docker container so it can make calls to OCI to retrieve VCN, Subnet, Security Lists and the Flow Logs stored in Object Storage. You may wish to create a unique read-only user for this purpose.
$ scp -r ~/.oci network-analyzer:.Now we can run the Docker container. In this example I’m only running the tool for the ap-melbourne-1 region. If you want to run it for multiple regions then you can enter them as a comma separated list E.g. “ap-melbourne-1,ap-sydney-1”.
docker run -it --name network-analyser \
--mount type=bind,source=/home/ubuntu/.oci/,target=/root/.oci/,readonly \
-e OCI_REGIONS=ap-melbourne-1 \
-e DB_USERNAME=<username of the db system> \
-e DB_PASSWORD=<password of the db system> \
-e DB_HOST=<IP address of the DB System> \
-e DB_PORT=3306 \
-e OCI_CLI_PROFILE=<OCI profile name> \
-e FLOW_LOG_BUCKET_NAME=<The bucket name holding your VCN flow logs> \
-p 8000:8000 \
scottfletcher/network-advisorDocker will pull the latest container, connect to the DB System, create the DB Schema, download the flow log files from Object Storage and then begin importing and analysing the data:
Unable to find image 'scottfletcher/network-advisor:latest' locally
latest: Pulling from scottfletcher/network-advisor
eaead16dc43b: Pull complete
753dfbed0c96: Pull complete
7af94765911a: Pull complete
7654989bc5b3: Pull complete
3ef60200484e: Pull complete
c10782f37fdf: Pull complete
a2f0acfe530e: Pull complete
e921fc787b87: Pull complete
447c769910e8: Pull complete
439c929a1f70: Pull complete
ae5f4991b66e: Pull complete
413dda94b7b9: Pull complete
da662d9d7c74: Pull complete
dccb605069ec: Pull complete
ca50647e8508: Pull complete
Digest: sha256:2298a5aa3c423b1243891919f7f23543ceee94d01bcc6e33210f3fea32f3d873
Status: Downloaded newer image for scottfletcher/network-advisor:latest
Downloading VCN Flow logs
Downloaded item [####################################] 100%
Importing VCN Flow logs
[# Importing and analysing 80M flow logs from 1,300 files and 16 subnets took approximately 3 hours. The progress and stages will be shown on screen. When the process is completed you will see:
******************************
* Report Generation Complete *
* View Report http://ip:8000 *
******************************
[2022-10-29 00:43:26] INFO WEBrick 1.7.0
[2022-10-29 00:43:26] INFO ruby 2.7.0 (2019-12-25) [x86_64-linux-gnu]
[2022-10-29 00:43:26] INFO WEBrick::HTTPServer#start: pid=10 port=8000And from your local machine you will be able to connect to https://localhost:8000 and view the reports that are generated:
Clicking a report will display it on screen:
You can then inspect the results and undertake your Security List rule review.
If you need to validate any findings in the reports, you can search the flow logs in OCI Logging, or query the DB System directly using the mysql client.
To re-run the tool, simply start another Docker container. Ensure you save the required reports when they’re generated as the reports are ephemeral within the container and the container also drops and recreates the DB System schema when it starts.
Security List rule reviews don’t need to be tedious, time consuming or repetitive. You can use this tool to help identify rule creep, mistakes, and validate the minimum set of rules required to meet your workload, organisational, security, and compliance requirements.
If you have any questions or comments, feel free to connect with me via LinkedIn https://www.linkedin.com/in/scotti-fletcher/
RedThunder.Blog 