How to tune JVM CMS GC to decrease CPU hike, Full GC and pause time

JVM GC tuning is the first tuning all system/application administrator perform to improve the system performance. Tuning is a gradual process. You cannot magically improve the performance of your application/system with in a day.

JVM GC tuning is a very important part of tuning exercise and need to be done whenever you observe a change in workload profile of your JVM. I am giving here an idea on CMS ( Concurrent Mark Sweep Collector ) of the HotSpot JVM.

Before going into the details, we must understand what is CMS Garbage Collector and how it works.

A CMS Garbage Collector is non-compacting low-latency collector. Below are the phases of a CMS concurrent collection.

1. Initial Mark: This is a stop-the-world phase hence all application threads are stopped. All the reachable live objects associated with stopped application threads are marked during this phase.

2. Concurrent Mark: During this concurrent phase, all application threads are restarted. Using the object references collected during "Initial Mark" phase, all other reference/live objects are identified.

3. Pre-cleaning Phase: This is an optimization phase in which changes to object references made by the application threads during the "Concurrent Mark" phase are identified. This is required as objects which were referred by say thread A during "initial mark" phase, may have been changed and are now referred by say thread B. These changes identified during this phase are then used to update the results from "Concurrent Mark" phase.

4. Remark Phase: This is a stop-the-world phase. CMS must stop all the application threads in this phase and then catch up with the changes the application has gone through. This step is essential so as to avoid collecting any objects that are still referenced.

5. Sweep Phase: During this concurrent phase, all non-referenced objects (i.e. dead objects)  are removed from the heap.

6. Reset Phase: In this concurrent phase, CMS does some housekeeping work so that it is ready for the next Garbage Collection cycle.

Now we know how CMS GC works. Its time to see how to use it in real case scenario. Firstly I am going to give the initial heap setting that I had in my environment.


-d64 -server -Xms6g -Xmx6g -XX:SurvivorRatio=6 -XX:PermSize=256m -XX:MaxPermSize=1024m -Xmn3686m -XX:NewSize=3686m -XX:MaxNewSize=3686m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnable

Even though I was using CMS GC, I was not using it properly. As you can see from the below screenshots that I captured using JConsole
 

In above screenshot, it is evident that CPU utilization was very high.

In this screenshot, the pause time is huge and the number of full GC is high too.
 
So to fix this what I did was, I added few new parameters and updates few existing parameters as below.


-d64 -server -Xms6g -Xmx6g -XX:SurvivorRatio=8 -XX:PermSize=256m -XX:MaxPermSize=1024m -Xmn2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+ScavengeBeforeFullGC -XX:+CMSClassUnloadingEnabled

Now lets understand these parameters.

-Xmn - This parameter defines the size of young generation. I recommend to keep this value between 30%-35% of total heap size i.e. (-Xmx. Keep -Xmx & -Xms same.)

-XX:+UseParNewGC - Enables the use of the modified parallel throughput collector in the young generation.

-XX:CMSInitiatingOccupancyFraction - Set the percentage of the heap that must be full before the JVM starts a concurrent collection in the tenured/old generation.

-XX:+UseCMSInitiatingOccupancyOnly - Enables the feature that all concurrent CMS cycles should start based on  -XX:CMSInitiatingOccupancyFraction=75

-XX:+ScavengeBeforeFullGC - Enables the feature that forces a young generation collection before starting a new CMS cycle or before a full GC is attempted

I hope after making the suggested changes, you will also experience an improve in performance. Do share your experience in the comment section below.

Display Weblogic Server Instances running on a machine

This shell script will display all the instances of Weblogic server (Admin & Managed Servers) running on any physical or virtual machine. This will capture the username which initiated the weblogic instance, PID of the weblogic instance and domain name of the weblogic instance.
The output format will look as below:
******************************************************************************
   User    |    PID      |   Server Name       |          Domain Name
******************************************************************************
   oracle  | 2239       | AdminServer         | /soa/data/domains/OSB_DEV1
   oracle  | 18406     | AdminServer         | /soa/data/domains/UTIL_DEV1
   oracle  | 23326     | OSB_MngdSvr1     | /soa/data/domains/OSB_DEV1
   oracle  | 23888     | BAM_MngdSvr1     | /soa/data/domains/UTIL_DEV1
   oracle  | 23889     | B2B_MngdSvr1      | /soa/data/domains/UTIL_DEV1
   oracle  | 23953     | JMS_MngdSvr1      | /soa/data/domains/UTIL_DEV1

The BASH Shell script to get this output:

#!/bin/bash regex="^([a-zA-Z]*)\s+([0-9]*)\s[0-9]* .*-Dweblogic\.Name=([a-zA-Z0-9_]*).*\s-Ddomain\.home=([a-zA-Z0-9_\.\/]*).*weblogic\.Server$" javap=`ps -ef | grep -v grep | grep weblogic.Server` IFS=$'\n' echo "******************************************************************************" printf "   User    |    PID   |   Server Name   |             Domain Name\n" echo "******************************************************************************" for jp in $javap do if [[ $jp =~ $regex ]]; then         n=${#BASH_REMATCH[*]}         #echo "${BASH_REMATCH[1]} | ${BASH_REMATCH[2]} | ${BASH_REMATCH[3]}    | ${BASH_REMATCH[4]}"         printf "%+10s | %-8s | %-15s | %-30s\n" ${BASH_REMATCH[1]} ${BASH_REMATCH[2]} ${BASH_REMATCH[3]} ${BASH_REMATCH[4]} fi done

BASH Script to convert certificates, import & export it to keystore

Doing repetitive tasks is always boring. For a system administrator, it is very often. Almost every month I have to perform tasks involving keystores and certificates in one or other environment.
So to ease it up, I have written a shell script that will do it for me. It is written based on my need. Anyone can use it as it is or modify it based on their requirements.
This is the first version and not very much tested so you may have hiccups in using it. Do let me know about it. I will fix it. I will also keep on updating based on the feedback that I will get from my team.

What this script do?
This script perform below operations:

• To convert certificate into pem
• To convert certificate into p12
• To import certificate into keystore
• To export certificate from keystore
• To delete certificate from keystore

I always suggest to use PEM while performing any operations related to certificates.The best way is to convert .p7b to .pem format so that complete certificate chain is included. This is what I prefer :)

#!/bin/bash echo "============================================" echo "Press 1: To convert certificate into pem" echo "Press 2: To convert certificate into p12" echo "Press 3: To import certificate into keystore" echo "Press 4: To export certificate from keystore" echo "Press 5: To delete certificate from keystore" echo "============================================" echo "Enter your choice (1-5)" read input convert_cert() {         filename=$1         path=$2         ## Extract input file extension         ext=${filename##*.}         name=${filename%%.*}         if [ $ext == 'p7b' ] || [ $ext == 'pem' ] || [ $ext == 'cer' ] || [ $ext == 'crt' ] || [ $ext == 'der' ]                 then                 ## Check if file exists                 echo "File to convert is at $path/$filename"                 result=`find $path -name $filename | wc -l`                 if [ $result == 0 ]                         then                         echo "File not found!!"                         exit $?                 fi                 ## Convert file into pem                 echo "Converting into pem..."                 echo "Extension of the file is $ext"                 if [ $ext == 'cer' -o $ext == 'der' -o $ext == 'crt' ]; then                         `openssl x509 -in $path/$filename -inform der -noout &> /dev/null`                         if [ $? -eq 0 ] ; then                                 `openssl x509 -in $path/$filename -inform der -out $path/$name.pem`                                 sed -i '/^$/d' $path/$name.pem                                 sed -i '/^subject/d' $path/$name.pem                                 sed -i '/^issuer/d' $path/$name.pem                                 echo "$name.pem generated!!"                         else                                 cp $path/$filename $path/$name.pem                                 echo "$name.pem generated!!"                         fi                                 elif [ $ext == 'p7b' ]; then                                         `openssl pkcs7 -in $path/$filename -inform der -noout &> /dev/null`                                         if [ $? -eq 0 ]; then                                                 `openssl pkcs7 -print_certs -in $path/$filename -inform der -out $path/$name.pem`                                                 sed -i '/^$/d' $path/$name.pem                                                 sed -i '/^subject/d' $path/$name.pem                                                 sed -i '/^issuer/d' $path/$name.pem                                                 echo "$name.pem generated!!"                                         else                                                 `openssl pkcs7 -print_certs -in $path/$filename -out $path/$name.pem` #                                               cp $path/$filename $path/$name.pem                                                 sed -i '/^$/d' $path/$name.pem                                                 sed -i '/^subject/d' $path/$name.pem                                                 sed -i '/^issuer/d' $path/$name.pem                                                 echo "$name.pem generated!!"                                         fi                                 else                                                 echo "This certificate is already in .pem format"                 fi                 else                                 echo "Please provide a certificate in .p7b or .cer or .crt or .der"                                 result=5         fi } if [ $input == 1 ]         then         echo "Enter file name"         read filename         echo "Enter path"         read path         convert_cert $filename $path elif [ $input == 2 ]; then         echo "Enter location of key & certificate"         read path         echo "Enter certificate file name"         read filename         name=${filename%%.*}         echo "Enter key file name"         read keyfilename         echo "Enter Passphrase/Password"         read passphrase         result=`find $path -name $keyfilename | wc -l`         if [ $result == 0 ]         then                 echo "$keyfilename not found!!"                 exit $?         fi         result=1         convert_cert $filename $path > /dev/null         if [ $result == 5 ] ; then                 echo "Please provide a certificate in .p7b or .cer or .crt or .der"                 exit $?         fi         echo "Generating p12 file $result"         echo "openssl pkcs12 -export -in $path/$name.pem -inkey $path/$keyfilename -out $path/$name.p12"         openssl pkcs12 -export -in $path/$name.pem -inkey $path/$keyfilename -out $path/$name.p12 elif [ $input == 3 ]; then         echo "Enter location of certificate"         read path         echo "Enter certificate file name"         read filename         echo "Enter alias name for the certificate"         read alias         echo "Enter location of keystore"         read kpath         echo "Enter keystore file name"         read kfilename         echo "Enter keystore password"         read kpswd         result=`find $path -name $filename | wc -l`         if [ $result == 0 ]         then                 echo "$filename not found!!"                 exit $?         fi         result=`find $kpath -name $kfilename | wc -l`         if [ $result == 0 ]         then                 echo "$kfilename not found!!"                 exit $?         fi         result=1         convert_cert $filename $path > /dev/null         if [ $result == 5 ] ; then                 echo "Please provide a certificate in .p7b or .cer or .crt or .der"                 exit $?         fi         echo "Importing certificate into keystore"         echo "keytool -importcert -alias $alias -file $path/$filename -keystore $kpath/$kfilename -storepass $kpswd"         keytool -importcert -alias $alias -file $path/$filename -keystore $kpath/$kfilename -storepass $kpswd elif [ $input == 4 ]; then         echo "Enter location for exported certificate"         read path         echo "Enter certificate file name for the exported certificate"         read filename         echo "Enter alias name of the certificate to be exported"         read alias         echo "Enter location of keystore"         read kpath         echo "Enter keystore file name"         read kfilename         echo "Enter keystore password"         read kpswd         result=`find $kpath -name $kfilename | wc -l`         if [ $result == 0 ]         then                 echo "$kfilename not found!!"                 exit $?         fi         echo "Exporting certificate from keystore"         echo "keytool -exportcert -alias $alias -file $path/$filename.cer -keystore $kpath/$kfilename -storepass $kpswd"         keytool -exportcert -alias $alias -file $path/$filename.cer -keystore $kpath/$kfilename -storepass $kpswd elif [ $input == 5 ]; then         echo "Enter alias name of the certificate to be deleted"         read alias         echo "Enter location of keystore"         read kpath         echo "Enter keystore file name"         read kfilename         echo "Enter keystore password"         read kpswd         result=`find $kpath -name $kfilename | wc -l`         if [ $result == 0 ]         then                 echo "$kfilename not found!!"                 exit $?         fi         echo "Deleting $alias certificate from keystore"         echo "keytool -delete -alias $alias -keystore $kpath/$kfilename -storepass $kpswd"         keytool -delete -alias $alias -keystore $kpath/$kfilename -storepass $kpswd else         echo "Incorrect choice!!" fi

Save it as certificate.sh.
Give the required permission to the file
$ chmod a+x certificate.sh

Now lets do the test run :)

Test 1:- Convert Base64Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
Base64Encoded.cer
Enter path
.
File to convert is at ./Base64Encoded.cer
Converting into pem...
Extension of the file is cer
Base64Encoded.pem generated!!

Test 2:- Convert DER Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
DEREncoded.cer
Enter path
.
File to convert is at ./DEREncoded.cer
Converting into pem...
Extension of the file is cer
DEREncoded.pem generated!!

Test 3:- Convert p7b Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
p7bEncoded.p7b
Enter path
.
File to convert is at ./p7bEncoded.p7b
Converting into pem...
Extension of the file is p7b
p7bEncoded.pem generated!!

Test 4:- Convert certificate into p12
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
2
Enter location of key & certificate
.
Enter certificate file name
soa.mycompany.com.cer
Enter key file name
soa.mycompany.com.key
Enter Passphrase/Password
changeit
Generating p12 file 1
Enter Export Password:
Verifying - Enter Export Password:
Test 5:- Import certificate in keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
3
Enter location of certificate
.
Enter certificate file name
soa.mycompany.com.cer
Enter alias name for the certificate
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Importing certificate into keystore
.
.
Trust this certificate? [no]:  yes
Certificate was added to keystore
Test 6:- Export certificate from keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
4
Enter location for exported certificate
.
Enter certificate file name for the exported certificate
myexportedcertificate
Enter alias name of the certificate to be exported
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Exporting certificate from keystore
Certificate stored in file <./myexportedcertificate.cer>
Test 7:- Delete certificate from keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
5
Enter alias name of the certificate to be deleted
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Deleting mysoacert certificate from keystore

BEA-000297 Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate

Error -

After importing new certificate in keystore, weblogic server starts throwing below error.

<Jun 3, 2015 3:37:48 PM PDT > <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>
<Jun 3, 2015 3:37:48 PM PDT> <Error><Server> <BEA-002618> <An invalid attempt was made to configure a channel for unconfigured protocol "Cannot convert identity certificate".
>

Reason -

The new certificate issued now a days are using SHA2 hash algorithm.  SHA2 is stronger than SHA1 and generate longer hash. The SHA-2 family consists of six hash functions with digests (hash values) that are 224, 256, 384 or 512 bits: SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256. The default SSL implementation in weblogic (Certicom SSL) cannot handle more then 128-bit.

Solution -

If you are using WebLogic 10.3.2 or lower, you must first upgrade to WebLogic 10.3.3 or higher (ideally, you should upgrade to WebLogic 10.3.6 as there are some SHA2 bug fixes that are included in WebLogic 10.3.6).  If you are unable to upgrade, then you will need to switch to a SHA1 certificate.

If you are using WebLogic 10.3.3 or higher, then use the following steps to enable JSSE SSL which trusts stronger certificates such as SHA2.
    a. Log in to your WebLogic admin console
    b. From left menu, choose Environment -> Servers -> SOA_MngdSvr1
    c. Click the 'Configuration' tab and 'SSL' subtab
    d. Go to bottom of page and click the 'Advanced' hyperlink
    e. Click the 'Lock & Edit' button on top left menu
    f. Go to bottom of page and check "Use JSSE SSL"
    g. Click "Save"
    h. Click 'Activate Changes'

Restart the servers for the change to take effect.

My tips for 1Z0-133 - Oracle WebLogic Server 12c: Administration I

Last week I cleared my "1Z0-133 - Oracle WebLogic Server 12c: Administration I" certification. just like other Oracle certification exam, this exam was also tricky. The easier the question the harder to answer..phew. So I am sharing my tips that will help you in getting a better idea of this certification and may contribute in your success.

How was the certification exam?
     The certification exam was a mix of questions based on theoretical concepts, case study, keen observation of weblogic etc. There was a good percentage of easy questions but to pass the exam you need to make sure you answer difficult questions are correctly. There were 77 questions and passing mark was 64%.

What were the important topics covered?
     A good number of questions were on new features of Weblogic 12c so I would suggest to have a good understanding of them. There were questions based on basic concept of weblogic which we tend to skip very often ;-) Install weblogic environment and do some hand on before attempting the exam. It is must because there were questions based on case studies which cannot be answered merely by going through theoretical concepts.

Which study material to refer?
      I mainly referred Oracle documentation for covering various topics. However I found Oracle weblogic server 12c administration handbook by Sam R. Alapati very useful. I would suggest to go through the complete book to score good in the certification exam.

All the Best for your Certification Exam!!

Configuring LDAP with Jenkins

Recently my colleague faced issue while trying to integrate LDAP with Jenkins. Even though lots of material is available online, he was not able to do it. The steps were clear in the various documents available on internet but somehow it was not working for his installed Jenkins version 1.575.
I followed almost similar approach but with a little twist at the end (while providing values in the textbox) which worked for us. I am documenting the steps below hoping it may help others who are facing the similar issue.

Pre-requisite:-

No#1 - You must have Jenkins installed and running
No#2 - Details of LDAP server [ I integrated with OVD (Oracle Virtual Directory) which in turns connect to LDAP & AD ]
***Below values structure may vary in your environment***
-- OVD URL - ldap://hostname.mycompany.com:3389
-- Admin User DN - uid=jenkinsadmin,ou=Application Admin,o=mycompany.com (This user is required by Jenkins to connect to OVD)
-- User DN - uid=shantans,ou=Internal,o=mycompany.com ( This is one of the many user who will access Jenkins once it is authenticated by LDAP/AD.)


Step#1 - Login onto Jenkins console using the in-built admin user

Step#2 - Navigate to "Manage Jenkins" --> "Configure Global Security"

Step#3 - Select the radio button for LDAP under security realm

Step#4 - For the textbox of "Server" copy the OVD URL. Click on "Advanced"

Step#5 - Now populate other text box with below information:-

root DN: o=mycompany.com
(Root DN is the name of your organization and is represented in DN as value of 'o'. So for my user "uid=shantans,ou=Internal,o=mycompany.com", my root DN is "o=mycompany.com")

User search base: ou=internal
(User search base is the branch under root DN "o=mycompany.com", where the user will be searched. So for my user "uid=shantans,ou=Internal,o=mycompany.com", it appears in "ou=internal" under root DN)

User search filter: uid={0}
Group search base: ou=groups
Group search filter:
Group membership filter:
(Keep above values as it is. If you need access for particular group then you have to provide value for Group search filter and Group membership filter. Otherwise you can leave it blank as above)

Manager DN: uid=jenkinsadmin,ou=Application Admin,o=mycompany.com
Manager Passwors: password
(Manager DN is the admin user mainly an application user which provides Jenkins, connectivity to LDAP)

Click "Save"

Now logout and try to login with any user available in LDAP.

Hope this will help you. Let me know if you face any issue or it has helped you fix your connectivity issue.

Switching the JVM JDK from Sun JDK to Oracle JRockit JDK and viceversa

A JDK JVM is selected during the domain creation. Sometimes it is required to change the JDK from Sun to Oracle JRockit or Oracle JRockit to Sun Hotspot. In this blog I will go through the steps for switching from Sun to Oracle JRockit. Similar steps are required while switching from Oracle JRockit to Sun JDK.

Step#1 - Download latest version of Oracle JRockit from Oracle site. ( I have downloaded p17852847_2831_Linux-x86-64.zip from oracle.support.com)

Step#2 - Copy this zip on your server and unzip it to the desired location.

Step#3 - If you have downloaded .bin file then follow below steps:

• Give executable rights to .bin file (chmod a+x  <jrockit_bin_file>)
• Execute the bin file (./<jrockit_bin_file>)

Step#4 - Bring down complete domain ( Admin + Managed Servers). If you are using nodemanager to start servers, then bring down nodemanagers too on their respective machines.

Step#5 - Modify commEnv.sh file @$WLS_HOME/common/bin

Search for line similar to below in commEnv.sh (Actual value of JAVA_HOME will be different in your environment)

JAVA_HOME="/apps/oracle/jdk/jdk1.6.0_45"

now replace the Sun JAVA_HOME with Oracle JRockit JAVA_HOME

JAVA_HOME="/apps/oracle/jdk/jrockit-jdk1.6.0_71"
Replace JAVA_VENDOR=Sun by JAVA_VENDOR=Oracle

Step#6 - Modify setDomainEnv.sh file @$DOMAIN_HOME/bin

Search for line similar to below in setDomainEnv.sh

BEA_JAVA_HOME=""
SUN_JAVA_HOME="/apps/oracle/jdk/jdk1.6.0_45"

now replace the Sun JAVA_HOME with Oracle JRockit JAVA_HOME

BEA_JAVA_HOME="/apps/oracle/jdk/jrockit-jdk1.6.0_71"

SUN_JAVA_HOME=""

**Modify your JVM parameters for Admin/Managed Server.

Step#7 - Modify config.xml file @$DOMAIN_HOME/config

Search for all occurrence of line similar to below in config.xml

<java-home>/apps/oracle/jdk/jdk1.6.0_45</java-home>

now replace the Sun JAVA_HOME with Oracle JRockit JAVA_HOME

<java-home>/apps/oracle/jdk/jrockit-jdk1.6.0_71</java-home>

**Modify your JVM parameters for Managed Servers in this file as well if you are using node manager to restart them.

Step#8 - Modify nodemanager.properties file @$WLS_HOME/common/nodemanager

Update javaHome & JavaHome with the currect JDK as below

javaHome=/apps/oracle/jdk/jrockit-jdk1.6.0_71

JavaHome=/apps/oracle/jdk/jrockit-jdk1.6.0_71/jre

Step#9 - Delete tmp & cache folder for admin and managed servers.

Step#10 - Start NodeManager, Admin Server and Managed Servers.