Deploying OCI APM Service for Optimal EBS Application Observability


The OCI Application Performance Monitoring (APM) service allows administrators to monitor and observe the E-Business Suite web applications.

It provides deep visibility into the application performance from end-user experience down through to the application server requests.

For many customers, the E-Business Suite (EBS) Application is critical to business operations. With OCI Application Performance Monitoring (APM) service, administrators can:

  • Analyze all end user experience with accessing EBS web and form pages.
  • Trace transactions across various components and isolate problems to the impacting application or infrastructure tier.
  • Has ability to drill into application code and SQL calls to the database
  • Easily Capture End Username for user sessions without modifying application code
  • To search in context, you can use out of box EBS attributes auto generated from traces. These attributes include:
    – EBS Function Name
    – EBS Class Package Name
    – EBS Forms Name
    – and more ….

Considerations

Please consider the following when deploying the APM service to a target EBS environment.

APM JAVA AGENT

  • The APM Java Agent Supports only OACORE and OAFM Managed Servers for EBS
  • The APM Java Agent needs to be installed in subdirectory under the EBS Non-Editioned Filesystem eg. /u01/app/EBS/fs_ne or some other location. Do not install /u01/app/EBS/fs1 or /u01/app/EBS/fs2
  • For FORMS based Managed Servers, Real User Experience Insights (RUEI) is recommended for deployment to capture the EBS forms sessions. This is optional and can be quickly provisioned using the OCI Marketplace Image.

EBS CONFIGURATION WITH NON-SHARED FILESYSTEM

  • APM Java Agent needs to be provisioned on each EBS app host where WebLogic resides.
  • For every Managed Server instance, you must configure the java arguments for OACORE and OAFM. This should be done on each EBS Application host. Please ensure the APM Java Agent JAR file is placed on the local filesystem.
  • DO NOT configure the JAVA Arguments for APM Java Agent JAR file for the Managed Server instance FORMS and FORMS-C4WS

EBS CONFIGURATION WITH SHARED FILESYSTEM

  • Managed Servers can share the same agent.
  • For each EBS Application host, configure the java arguments for every Managed Server instance. This includes OACORE and OAFM. Please ensure the APM Java Agent JAR file is placed on the shared filesystem.
  • APM Java Agent can be provisioned and deployed once, which can then be used by other EBS app hosts.
  • DO NOT configure the JAVA Arguments for APM Java Agent JAR file for the Managed Server instance FORMS and FORMS-C4WS.

SYNTHETIC MONITORING FOR EBS APPLICATIONS WITH INTERNET ACCESS

  • Public Vantage Points monitors EBS Application Endpoints. These endpoints are accessible via the internet, or public network, by end users.
  • There are multiple Public Vantage Points readily available globally which can be used without any additional effort.

AVAILABILITY (SYNTHETIC) MONITORING FOR EBS APPLICATIONS WITH PRIVATE NETWORK ACCESS

  • Private or Dedicated Vantage Points help monitor EBS Application Endpoint that is accessible via private network by end users.
  • A Private or Dedicated Vantage Point instance will need to be deployed as a prerequisite.
  • Please refer to documentation for further details:

Prerequisites

Please complete the prerequisites.

IAM SETUP

The IAM group and policies below are required for the Application Performance Monitoring service.

1. Create Admin Group (eg. om_admins)

IDENTITY & SECURITY > DOMAINS > [Select Domain eg. DEFAULT] > GROUPS CREATE GROUP

NOTE: Then add users or members to this group including yourself

2. Create Policy Name (eg. om_apm_policy)

IDENTITY & SECURITY > POLICIES > CREATE POLICY

ALLOW GROUP om_admins TO MANAGE apm-domains IN COMPARTMENT <EBS_COMPARTMENT>
ALLOW GROUP om_admins TO MANAGE alarms IN COMPARTMENT <EBS_COMPARTMENT>
ALLOW GROUP om_admins TO READ metrics IN COMPARTMENT <EBS_COMPARTMENT>
ALLOW GROUP om_admins TO USE ons-topics IN COMPARTMENT <EBS_COMPARTMENT>

NOTE: Replace <EBS_COMPARTMENT> with your target Compartment.

APM DOMAIN AND SOFTWARE DOWNLOAD

1. To create the APM Domain, navigate to:
OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > ADMINISTRATION

COMPARTMENT: <EBS_COMPARTMENT>
Click on Create APM Domain


NAME: (eg. ebs_erp)
COMPARTMENT: <EBS_COMPARTMENT>
DESCRIPTION: EBS APM Domain

Click Create

The APM domain name ebs_erp is something I used for this demo, however you can use any other name that is meaningful. Just remember to replace it in other steps where I have referenced this as ebs_erp for the APM domain name.

2. After APM Domain has been created, review the configuration details.

OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > ADMINISTRATION > APM DOMAINS > [Select APM Domain eg. ebs_erp] > DATA KEYS

Make a note or copy of the following configuration for your target environment.  This will be different for each APM domain that is created.

– APM Data Endpoint URL : https://aaaa??????????????????????.apm-agt.ap-sydney-1.oci.oraclecloud.com
– Private Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
– Public Key: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

3. Download the APM Java Agent software

OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > ADMINISTRATION > DOWNLOAD APM AGENT > JAVA AGENT

Create A Monitor Configuration

Any endpoint URL can be configured in Availability Monitoring (synthetic monitoring). In this example, we will create the Monitor configuration for the EBS home page.

The following endpoint URL is accessible by users on the internet or public network. This accessibility allows us to connect and simulate the traffic by using the predefined public Vantage points.

If your endpoint URL is only accessible within a private network, then you must first set up the Dedicated Vantage Points. Do this before performing the steps below.

The Monitor configuration can be also configured for scripted actions including Recorded transactions such as Selenium (.side) or REST (.js) based files. For further details please see documentation.

1. Create the Monitor configuration

OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > AVAILABILITY MONITORING > RESOURCES

Set the Scope:

COMPARTMENT: <EBS_COMPARTMENT>
APM DOMAIN: eg. erp_ebs

2. Click on Create Monitor

3. Specify the inputs for the Monitor Definition:

APM_DOMAIN: ebs_erp
Monitor Name: ebs_erp_monitor

Type: Browser
Base URL: https://<EBS_APP_URL>/  (eg. https://erpportal.apps.com)

Verify Response Content: None
Verify SSL: Yes
Override DNS: None

Click Next

4. Specify Inputs for the Run Settings:
Vantage Points = Sydney, Melbourne
Frequency = Interval
Scheduling Policy = All
Interval Time between Runs = 10
Timeout = <DEFAULT>
Retry on Failure – Enable Retry: Yes
Network Measurements: Enable Network Collection: Yes
Protocol = <DEFAULT>
Probe mode = <DEFAULT>
Probe Per Hop = <DEFAULT>

Click Next


5. Specify Inputs for the Availability Configuration:
Enable Availability Collection: Yes
Maximum failures allowed per interval = 0
Minimum runs allowed per interval = 1

Click Next


6. Specify Tags is optional otherwise you can skip.

Click Next


7. Review Configuration
Click Create


8. To review,navigate to

OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > AVAILABILITY MONITORING > RESOURCES > [Select Monitor eg. ebs_erp]

Provision the APM Java Agent

In this section, the oracle-apm-agent software directory will be created and installed on the EBS Application host(s).

For non-shared filesystem, the following steps will need to repeated on each EBS Application host.

For shared filesystem, the following steps will need to be completed once on only one of the EBS Application host.

  1. Copy the APM Java Agent Software to the target EBS Application Host
scp -i oci.ppk apm-java-agent-installer-<version>.jar opc@192.x.x.x:/tmp/.

NOTE: The name of the downloaded file version will be different for more recent version. Replace apm-java-agent-installer-<version>.jar with the APM java agent version you downloaded eg. apm-java-agent-installer-1.15.0.516.jar

2. On target host, run the following to validate the checksum and run as the Application Software owner (eg. oracle)

md5sum apm-java-agent-installer-<version>.jar

Sample Output

8e5239c49828ff4dd7e2693ceaaf2f84  apm-java-agent-installer-1.15.0.516.jar

NOTE: Should match same MD5 checksum value for software download in OCI Console

3. On the EBS Application Host, verify Network Connectivity and APM Domain Data Endpoint is accessible

NOTE – Replace the following Fully Qualified Domain Name (FQDN) Data Endpoint with the target Endpoint. The Endpoint was captured for the newly created APM domain. For example, aaaa??????????????????????.apm-agt.<oci_region>.oci.oraclecloud.com

a) Verify connectivity from EBS app host using NC utility:

nc -zv <DATA_ENDPOINT_FQDN> 443

Example command:

nc -zv aaaa??????????????????????.apm-agt.ap-sydney-1.oci.oraclecloud.com 443

Sample Output should report if successful

Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 192.X.X.X:443.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

b) Verify using CURL command using syntax:

$ curl -v -X POST “https://<DATA_ENDPOINT_URL>/20200101/observations/public-span?dataFormat=apm&dataFormatVersion=1&dataKey=<PUBLIC_KEY>“

Run the command:

curl -v -X POST "https://aaaa??????????????????????.apm-agt.ap-sydney-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=apm&dataFormatVersion=1&dataKey=ZZZZZZZZZZZZZZZZZZZZZZZZZZ"

Sample Output should report if successful:

< HTTP/1.1 200 OK

4. Locate the path to the EBS Non-Editioned Filesystem eg.  as the Application Software owner (eg. oracle).

NOTE – The following is installed here because we don’t want to install in fs1 or fs2 directory. The filesystem will switch during patching. This would mean we have to deploy it in both fs1 and fs2. The risk is that the APM java agent will be out of sync if not maintained.

sudo su - oracle
locate -b fs_ne

The sample output should be something similar

/u01/install/APPS/fs_ne

/u01/install/APPS/fs_ne

5. The option is to provision the APM Java Agent Software to this location.  Alternatively you can deploy to another location.
Note: The following can be set as the target APM_BASE_DIR

APM_BASE_DIR=/u01/install/APPS/fs_ne

Set the additional variables:

DATA_UPLOAD_ENDPOINT_URL=https://aaaa??????????????????????.apm-agt.ap-sydney-1.oci.oraclecloud.com

PRIVATE_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXX
APM_JAR_FILE=apm-java-agent-installer-<version>.jar
APM_SERVICE_NAME=xxx-ebs-erp

NOTE: Set the DATA_UPLOAD_ENDPOINT_URL and PRIVATE_KEY to the values collected from the OCI console. These values were collected when creating the APM Domain.
OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > ADMINISTRATION > APM DOMAINS > eg. ebs_erp > DATA KEYS

Specify the APM_JAR_FILE to the APM java agent software that was downloaded. The more recent version downloaded may be different to the one shown here eg. APM_JAR_FILE=apm-java-agent-installer-1.15.0.516.jar

APM_SERVICE_NAME can be a name you choose to identify the target EBS service. The APM Java agent collects the traces and tag this with the APM Service Name.

6. Run command to create the oracle-apm-agent subfolder and install software in that target location set for APM_BASE_DIR

java -jar ./${APM_JAR_FILE} provision-agent -service-name=${APM_SERVICE_NAME} -destination=${APM_BASE_DIR} -private-data-key=${PRIVATE_KEY} -data-upload-endpoint=${DATA_UPLOAD_ENDPOINT_URL}

The output should report something similar

Action [provision-agent] completed successfully
********************************************
APM Agent has been provisioned successfully. To complete the setup, you need to modify your destination's appserver startup script so that the APM Agent you have just deployed can be started. Here is the general information on the 
changes.

1. Make a backup copy of your appserver startup script.
2. Edit appserver startup script to add -javaagent argument of APM Agent 
bootstrap library to java executable. For example: 
    java -javaagent:<destination>/oracle-apm-agent/bootstrap/ApmAgent.jar ${JAVA_OPTIONS} -cp <class_path> ... ... 
3. Stop and restart your appserver.

Additional configuration such as proxy settings, can be set by editing 
<destination>/oracle-apm-agent/config/<version>/AgentConfig.properties

For more information, see
    https://docs.oracle.com/iaas/application-performance-monitoring/doc/deploy-apm-java-agent.html

7. Verify the oracle-apm-agent directory and APMAgent.jar file has been installed.

ls ${APM_BASE_DIR}/oracle-apm-agent/bootstrap/ApmAgent.jar

Verify the Output

/u01/install/APPS/fs_ne/oracle-apm-agent/bootstrap/ApmAgent.jar

NOTE: Reminder if using non-shared filesystem, please ensure these steps are performed on each EBS Application Host before proceeding.

Deploy APM Java Agent

1. Login to the WebLogic Admin Console for EBS

http://<EBS_WLS_ADMIN_HOST>:7001/console
(eg. http://ebsapp1.apps.com:7001/console)

(username=weblogic or system)

2. Expand Left Menu and Navigate to:

Domain > Environment > Servers > Configuration


3. Select the managed server for OACORE and OAFM that changes are required to be updated for:

  • oacore_server1
  • oafm_server1

Repeat all the above steps for each of the managed servers of the oacore, oafm, services across all EBS Application hosts. This can be performed from the same WebLogic Admin Console.

4. In the Change Center

5. Click Lock and Edit

6. Then select Server Start tab

7. Update the Arguments field and place the following parameter required for instrumentation of the APM java code at the end of the line.

Syntax:
-javaagent:<APM_BASE_DIR>/oracle-apm-agent/bootstrap/ApmAgent.jar

Example:

-javaagent:/u01/install/APPS/fs_ne/oracle-apm-agent/bootstrap/ApmAgent.jar

Then Click Save.

8. In the Change Center, click Activate Changes to apply the changes

Repeat all the above steps for each of the managed servers of the oacore, oafm, services till each associated managed server has the JVM arguments updated.


9. Under Domain Structure Section: Expand navigation tree: Environment > Servers

10. Select Control tab

11. Choose the managed servers that configuration was updated for:

  • oacore_server1
  • oafm_server1

Include other managed servers of the oacore, oafm, services if you have additional instances in your cluster.

12. Click on Shutdown > Force Shutdown Now.

13. Click on Start

14. Verify that the Managed Servers has been restarted OK

Verify APM Server Traces for EBS

Confirm the APM Server Traces for EBS is now collecting and is visible in APM Trace Explorer

1. Generate some workload by logging into the target EBS application

Navigate around in the EBS application.

2. In OCI Console, navigate to:
OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING : TRACE EXPLORER

NAME: (eg. ebs_erp)
COMPARTMENT: <EBS_COMPARTMENT>

3. Confirm the APM server traces are now collected

4. Confirm the EBS Service is displaying as expected.

When the APM Java Agent was provisioned earlier, the APM_SERVICE_NAME eg. xxx-ebs-erp that was set will appear here.

Configure Real User Monitoring

This section will enable trace collection for EBS user sessions.

For non-shared filesystem, the following steps will need to repeated on each EBS Application host.
For shared filesystem, the following steps will need to be completed once on only one of the EBS Application host.

1. On the EBS App Server, update ${APM_BASE_DIR}/oracle-apm-agent/config/<LATEST_VERSION>/AgentConfig.properties

com.oracle.apm.agent.public.data.key=<PUBLIC_DATA_KEY>
com.oracle.apm.agent.rum.enable.injection=true
com.oracle.apm.agent.rum.web.application=<WEB_APPLICATION_NAME>
com.oracle.apm.agent.rum.service.name=<RUM_SERVICE_NAME>

Example – ${APM_BASE_DIR}/oracle-apm-agent/config/1.10.1.3/AgentConfig.properties

com.oracle.apm.agent.public.data.key=ZZZZZZZZZZZZZZZZZZZZZZZZZZ
com.oracle.apm.agent.rum.enable.injection=true
com.oracle.apm.agent.rum.web.application=EBSAppXXX

NOTE: Set the PUBLIC_KEY to the values collected from the OCI console when the APM Domain was created:
OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING > ADMINISTRATION > APM DOMAINS > ebs_xxx_apm > DATA KEYS
The com.oracle.apm.agent.rum.web.application parameter can be set to a name to identify the target EBS Web Application traces for end users that are being collected by the APM Browser Agent.

2. Verify that the following parameters has been configured to capture the EBS username for end user sessions:

  • EBS_USERNAME_INFO: true
  • capture_username: true
egrep -i "ebs_username|capture_username" ${APM_BASE_DIR}/oracle-apm-agent/config/*/ProbeConfig.acml

This should be the expected output

  EBS_USERNAME_INFO: true
  capture_username: true

NOTE: If the parameters are not set with the above values, then edit the file ProbeConfig.acml and update.

Verify APM End User Session Traces for EBS

Confirm the APM End User Session for EBS is now collecting and is visible in APM Trace Explorer.

1. Generate some workload by logging into the target EBS application

Navigate around in the EBS application.

2. In OCI Console, navigate to:
OBSERVABILITY & MANAGEMENT > APPLICATION PERFORMANCE MONITORING : TRACE EXPLORER

NAME: (eg. ebs_erp)
COMPARTMENT: <EBS_COMPARTMENT>

3. Go to the Web Apps tab. Confirm the EBS Web Application is displaying as expected

When the parameter com.oracle.apm.agent.rum.web.application was set previously, the following Web Application Name eg. EBSAppXXX will appear here.

4. Verify Sessions tab

5. Verify Users tab, and then drill down to Show Traces

6. Choose the Trace with the longest running duration and Drilldown into Show trace details

7. Drill into the top span

8. You can review the attributes that were collected for the span details.

The following information:

  • Apdex details will help determine the end user is satisfied, tolerated or frustrated with their current session.
  • Browser details help understand if they are using an compatible browser for optimal user experience. For some applications using the recommended browser type and version may impact user performance and experience.
  • The Device they are accessing the application from whether from their Personal Computer, or Mobile Device
  • Specific EBS Application attributes eg. EBSOAFunc can be use to identify the Application function the end user is trying to access. There are many other EBS attributes collected by default which are unique to OCI APM service. For further information of these EBS attributes that are collected, please refer to Oracle E-Business Suite Dimensions in the documentation.
  • Error Details – if any is reported, Additional error attributes will be be provided with further details.
  • Geolocation is collected, to help determine where an end user is accessing the application from.
  • Client IP can also be collected if enabled. Please see section on Optional APM Configuration for EBS

Optional APM Configuration for EBS

Depending on the requirements, some additional configuration are available to enable in APM to further enhance the operational and observability experience. If so, please refer to the additional configurations steps below:

  • Additional OCI Services providing comprehensive observability and monitoring capabilities for EBS.
  • Capture Client IP Address for End User Sessions for Real User Monitoring (RUM) – see how to blog post
  • Optimise Spans to exclude collection of Insignificant Spans – see documentation
  • General APM Configuration for Apex Thresholds, Custom Metrics, Drilldowns and Spans – see documentation
  • Collect traces for EBS Forms Sessions with Real User Experience Insights (RUEI) integration – see documentation
  • Dashboards – View Out of Box Dashboards or Build Custom Dashboards

Leave a comment