Monday, April 4, 2016

Connection Factory in Weblogic adapter disappear



Issue: Connection Factory in Weblogic adapter disappear

Description: When I am trying to create a new connection factory, it disappears when the adapter is updated and changes are activated in the weblogic console.

Error Message: In Admin log below message is seen
JNDI lookup of 'eis/DB/SOAXACP' failed due to: While trying to lookup 'eis.DB/SOAXACP' didn't find subcontext 'eis'. Resolved ''
javax.naming.NameNotFoundException: While trying to lookup 'eis.DB/SOAXACP' didn't find subcontext 'eis'. Resolved ''; remaining name 'eis/DB/SOAXACP'
        at weblogic.jndi.internal.BasicNamingNode.newNameNotFoundException(BasicNamingNode.java:1224)
        at weblogic.jndi.internal.BasicNamingNode.lookupHere(BasicNamingNode.java:268)
        at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:217)
        at weblogic.jndi.internal.BasicNamingNode.lookupIgnorePartition(BasicNamingNode.java:1503)

Cause: The issue is caused due to Plan.xml. It is either corrupted or belongs to different adapter (i.e. using jms adapter plan.xml to save db adapter connection factory)

Solution: Recreate a new Plan.xml and then try to add the new connection factory to it.

Tuesday, January 26, 2016

[EPM HYPERION 11.1.2.4] A 'Not Found' error occurred communicating with the server

ERROR:

After doing fresh installation and setup of EPM 11.1.2.4, getting below error when login on WORKSPACE.

A 'Not Found' error occurred communicating with the server.

 URI: http://localhost:19000/raframework/conf/ToolsConfig.xml

 Status: 404 - Not Found

 Content: text/html; charset=iso-8859-1
 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 <html><head>
 <title>404 Not Found</title>
 </head><body>
 <h1>Not Found</h1>
 <p>The requested URL /raframework/conf/ToolsConfig.xml was not found.</p>
 </body></html>

CAUSE:

OHS is not aware of context "raframework", hence it does not know where to forward the request.

SOLUTION:

Add the context to mod_wl_ohs.conf file as below:


cd $FMW_HOME/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component
cp  mod_wl_ohs.conf mod_wl_ohs.conf.01282016
vi mod_wl_ohs.conf
Add below snippet at the end:--
<LocationMatch ^/raframework>
    SetHandler weblogic-handler
    WeblogicCluster localhost:9000
    WLIOTimeoutSecs 6000
    Idempotent OFF
    WLSocketTimeoutSecs 600
</LocationMatch>

Restart OHS:
cd $FMW_HOME/user_projects/epmsystem1/httpConfig/ohs/bin
/opmnctl stopall
/opmnctl startall


Try to login again on workspace.

Monday, October 12, 2015

How to tune JVM CMS GC to decrease CPU hike, Full GC and pause time

JVM GC tuning is the first tuning all system/application administrator perform to improve the system performance. Tuning is a gradual process. You cannot magically improve the performance of your application/system with in a day.

JVM GC tuning is a very important part of tuning exercise and need to be done whenever you observe a change in workload profile of your JVM. I am giving here an idea on CMS ( Concurrent Mark Sweep Collector ) of the HotSpot JVM.

Before going into the details, we must understand what is CMS Garbage Collector and how it works.

A CMS Garbage Collector is non-compacting low-latency collector. Below are the phases of a CMS concurrent collection.

1. Initial Mark: This is a stop-the-world phase hence all application threads are stopped. All the reachable live objects associated with stopped application threads are marked during this phase.

2. Concurrent Mark: During this concurrent phase, all application threads are restarted. Using the object references collected during "Initial Mark" phase, all other reference/live objects are identified.

3. Pre-cleaning Phase: This is an optimization phase in which changes to object references made by the application threads during the "Concurrent Mark" phase are identified. This is required as objects which were referred by say thread A during "initial mark" phase, may have been changed and are now referred by say thread B. These changes identified during this phase are then used to update the results from "Concurrent Mark" phase.

4. Remark Phase: This is a stop-the-world phase. CMS must stop all the application threads in this phase and then catch up with the changes the application has gone through. This step is essential so as to avoid collecting any objects that are still referenced.

5. Sweep Phase: During this concurrent phase, all non-referenced objects (i.e. dead objects)  are removed from the heap.

6. Reset Phase: In this concurrent phase, CMS does some housekeeping work so that it is ready for the next Garbage Collection cycle.

Now we know how CMS GC works. Its time to see how to use it in real case scenario. Firstly I am going to give the initial heap setting that I had in my environment.


-d64 -server -Xms6g -Xmx6g -XX:SurvivorRatio=6 -XX:PermSize=256m -XX:MaxPermSize=1024m -Xmn3686m -XX:NewSize=3686m -XX:MaxNewSize=3686m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnable

Even though I was using CMS GC, I was not using it properly. As you can see from the below screenshots that I captured using JConsole
 


In above screenshot, it is evident that CPU utilization was very high.


In this screenshot, the pause time is huge and the number of full GC is high too.
 
So to fix this what I did was, I added few new parameters and updates few existing parameters as below.


-d64 -server -Xms6g -Xmx6g -XX:SurvivorRatio=8 -XX:PermSize=256m -XX:MaxPermSize=1024m -Xmn2048m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+ScavengeBeforeFullGC -XX:+CMSClassUnloadingEnabled



Now lets understand these parameters.

-Xmn - This parameter defines the size of young generation. I recommend to keep this value between 30%-35% of total heap size i.e. (-Xmx. Keep -Xmx & -Xms same.)

-XX:+UseParNewGC - Enables the use of the modified parallel throughput collector in the young generation.

-XX:CMSInitiatingOccupancyFraction - Set the percentage of the heap that must be full before the JVM starts a concurrent collection in the tenured/old generation.

-XX:+UseCMSInitiatingOccupancyOnly - Enables the feature that all concurrent CMS cycles should start based on  -XX:CMSInitiatingOccupancyFraction=75

-XX:+ScavengeBeforeFullGC - Enables the feature that forces a young generation collection before starting a new CMS cycle or before a full GC is attempted

I hope after making the suggested changes, you will also experience an improve in performance. Do share your experience in the comment section below.


Friday, October 2, 2015

Display Weblogic Server Instances running on a machine

This shell script will display all the instances of Weblogic server (Admin & Managed Servers) running on any physical or virtual machine. This will capture the username which initiated the weblogic instance, PID of the weblogic instance and domain name of the weblogic instance.
The output format will look as below:
******************************************************************************
   User    |    PID      |   Server Name       |          Domain Name
******************************************************************************
   oracle  | 2239       | AdminServer         | /soa/data/domains/OSB_DEV1
   oracle  | 18406     | AdminServer         | /soa/data/domains/UTIL_DEV1
   oracle  | 23326     | OSB_MngdSvr1     | /soa/data/domains/OSB_DEV1
   oracle  | 23888     | BAM_MngdSvr1     | /soa/data/domains/UTIL_DEV1
   oracle  | 23889     | B2B_MngdSvr1      | /soa/data/domains/UTIL_DEV1
   oracle  | 23953     | JMS_MngdSvr1      | /soa/data/domains/UTIL_DEV1

The BASH Shell script to get this output:

#!/bin/bash
regex="^([a-zA-Z]*)\s+([0-9]*)\s[0-9]* .*-Dweblogic\.Name=([a-zA-Z0-9_]*).*\s-Ddomain\.home=([a-zA-Z0-9_\.\/]*).*weblogic\.Server$"
javap=`ps -ef | grep -v grep | grep weblogic.Server`
IFS=$'\n'
echo "******************************************************************************"
printf "   User    |    PID   |   Server Name   |             Domain Name\n"
echo "******************************************************************************"
for jp in $javap
do
if [[ $jp =~ $regex ]]; then
        n=${#BASH_REMATCH[*]}
        #echo "${BASH_REMATCH[1]} | ${BASH_REMATCH[2]} | ${BASH_REMATCH[3]}    | ${BASH_REMATCH[4]}"
        printf "%+10s | %-8s | %-15s | %-30s\n" ${BASH_REMATCH[1]} ${BASH_REMATCH[2]} ${BASH_REMATCH[3]} ${BASH_REMATCH[4]}
fi
done

Monday, July 6, 2015

BASH Script to convert certificates, import & export it to keystore

Doing repetitive tasks is always boring. For a system administrator, it is very often. Almost every month I have to perform tasks involving keystores and certificates in one or other environment.
So to ease it up, I have written a shell script that will do it for me. It is written based on my need. Anyone can use it as it is or modify it based on their requirements.
This is the first version and not very much tested so you may have hiccups in using it. Do let me know about it. I will fix it. I will also keep on updating based on the feedback that I will get from my team.

What this script do?
This script perform below operations:
  • To convert certificate into pem
  • To convert certificate into p12
  • To import certificate into keystore
  • To export certificate from keystore
  • To delete certificate from keystore
I always suggest to use PEM while performing any operations related to certificates.The best way is to convert .p7b to .pem format so that complete certificate chain is included. This is what I prefer :)
#!/bin/bash
echo "============================================"
echo "Press 1: To convert certificate into pem"
echo "Press 2: To convert certificate into p12"
echo "Press 3: To import certificate into keystore"
echo "Press 4: To export certificate from keystore"
echo "Press 5: To delete certificate from keystore"
echo "============================================"
echo "Enter your choice (1-5)"
read input
convert_cert() {
        filename=$1
        path=$2
        ## Extract input file extension
        ext=${filename##*.}
        name=${filename%%.*}
        if [ $ext == 'p7b' ] || [ $ext == 'pem' ] || [ $ext == 'cer' ] || [ $ext == 'crt' ] || [ $ext == 'der' ]
                then
                ## Check if file exists
                echo "File to convert is at $path/$filename"
                result=`find $path -name $filename | wc -l`
                if [ $result == 0 ]
                        then
                        echo "File not found!!"
                        exit $?
                fi
                ## Convert file into pem
                echo "Converting into pem..."
                echo "Extension of the file is $ext"
                if [ $ext == 'cer' -o $ext == 'der' -o $ext == 'crt' ]; then
                        `openssl x509 -in $path/$filename -inform der -noout &> /dev/null`
                        if [ $? -eq 0 ] ; then
                                `openssl x509 -in $path/$filename -inform der -out $path/$name.pem`
                                sed -i '/^$/d' $path/$name.pem
                                sed -i '/^subject/d' $path/$name.pem
                                sed -i '/^issuer/d' $path/$name.pem
                                echo "$name.pem generated!!"
                        else
                                cp $path/$filename $path/$name.pem
                                echo "$name.pem generated!!"
                        fi
                                elif [ $ext == 'p7b' ]; then
                                        `openssl pkcs7 -in $path/$filename -inform der -noout &> /dev/null`
                                        if [ $? -eq 0 ]; then
                                                `openssl pkcs7 -print_certs -in $path/$filename -inform der -out $path/$name.pem`
                                                sed -i '/^$/d' $path/$name.pem
                                                sed -i '/^subject/d' $path/$name.pem
                                                sed -i '/^issuer/d' $path/$name.pem
                                                echo "$name.pem generated!!"
                                        else
                                                `openssl pkcs7 -print_certs -in $path/$filename -out $path/$name.pem`
#                                               cp $path/$filename $path/$name.pem
                                                sed -i '/^$/d' $path/$name.pem
                                                sed -i '/^subject/d' $path/$name.pem
                                                sed -i '/^issuer/d' $path/$name.pem
                                                echo "$name.pem generated!!"
                                        fi
                                else
                                                echo "This certificate is already in .pem format"
                fi
                else
                                echo "Please provide a certificate in .p7b or .cer or .crt or .der"
                                result=5
        fi
}
if [ $input == 1 ]
        then
        echo "Enter file name"
        read filename
        echo "Enter path"
        read path
        convert_cert $filename $path
elif [ $input == 2 ]; then
        echo "Enter location of key & certificate"
        read path
        echo "Enter certificate file name"
        read filename
        name=${filename%%.*}
        echo "Enter key file name"
        read keyfilename
        echo "Enter Passphrase/Password"
        read passphrase
        result=`find $path -name $keyfilename | wc -l`
        if [ $result == 0 ]
        then
                echo "$keyfilename not found!!"
                exit $?
        fi
        result=1
        convert_cert $filename $path > /dev/null
        if [ $result == 5 ] ; then
                echo "Please provide a certificate in .p7b or .cer or .crt or .der"
                exit $?
        fi
        echo "Generating p12 file $result"
        echo "openssl pkcs12 -export -in $path/$name.pem -inkey $path/$keyfilename -out $path/$name.p12"
        openssl pkcs12 -export -in $path/$name.pem -inkey $path/$keyfilename -out $path/$name.p12
elif [ $input == 3 ]; then
        echo "Enter location of certificate"
        read path
        echo "Enter certificate file name"
        read filename
        echo "Enter alias name for the certificate"
        read alias
        echo "Enter location of keystore"
        read kpath
        echo "Enter keystore file name"
        read kfilename
        echo "Enter keystore password"
        read kpswd
        result=`find $path -name $filename | wc -l`
        if [ $result == 0 ]
        then
                echo "$filename not found!!"
                exit $?
        fi
        result=`find $kpath -name $kfilename | wc -l`
        if [ $result == 0 ]
        then
                echo "$kfilename not found!!"
                exit $?
        fi
        result=1
        convert_cert $filename $path > /dev/null
        if [ $result == 5 ] ; then
                echo "Please provide a certificate in .p7b or .cer or .crt or .der"
                exit $?
        fi
        echo "Importing certificate into keystore"
        echo "keytool -importcert -alias $alias -file $path/$filename -keystore $kpath/$kfilename -storepass $kpswd"
        keytool -importcert -alias $alias -file $path/$filename -keystore $kpath/$kfilename -storepass $kpswd
elif [ $input == 4 ]; then
        echo "Enter location for exported certificate"
        read path
        echo "Enter certificate file name for the exported certificate"
        read filename
        echo "Enter alias name of the certificate to be exported"
        read alias
        echo "Enter location of keystore"
        read kpath
        echo "Enter keystore file name"
        read kfilename
        echo "Enter keystore password"
        read kpswd
        result=`find $kpath -name $kfilename | wc -l`
        if [ $result == 0 ]
        then
                echo "$kfilename not found!!"
                exit $?
        fi
        echo "Exporting certificate from keystore"
        echo "keytool -exportcert -alias $alias -file $path/$filename.cer -keystore $kpath/$kfilename -storepass $kpswd"
        keytool -exportcert -alias $alias -file $path/$filename.cer -keystore $kpath/$kfilename -storepass $kpswd
elif [ $input == 5 ]; then
        echo "Enter alias name of the certificate to be deleted"
        read alias
        echo "Enter location of keystore"
        read kpath
        echo "Enter keystore file name"
        read kfilename
        echo "Enter keystore password"
        read kpswd
        result=`find $kpath -name $kfilename | wc -l`
        if [ $result == 0 ]
        then
                echo "$kfilename not found!!"
                exit $?
        fi
        echo "Deleting $alias certificate from keystore"
        echo "keytool -delete -alias $alias -keystore $kpath/$kfilename -storepass $kpswd"
        keytool -delete -alias $alias -keystore $kpath/$kfilename -storepass $kpswd
else
        echo "Incorrect choice!!"
fi
Save it as certificate.sh.
Give the required permission to the file
$ chmod a+x certificate.sh

Now lets do the test run :)

Test 1:- Convert Base64Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
Base64Encoded.cer
Enter path
.
File to convert is at ./Base64Encoded.cer
Converting into pem...
Extension of the file is cer
Base64Encoded.pem generated!!

Test 2:- Convert DER Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
DEREncoded.cer
Enter path
.
File to convert is at ./DEREncoded.cer
Converting into pem...
Extension of the file is cer
DEREncoded.pem generated!!

Test 3:- Convert p7b Encoded certificate
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
1
Enter file name
p7bEncoded.p7b
Enter path
.
File to convert is at ./p7bEncoded.p7b
Converting into pem...
Extension of the file is p7b
p7bEncoded.pem generated!!

Test 4:- Convert certificate into p12
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
2
Enter location of key & certificate
.
Enter certificate file name
soa.mycompany.com.cer
Enter key file name
soa.mycompany.com.key
Enter Passphrase/Password
changeit
Generating p12 file 1
Enter Export Password:
Verifying - Enter Export Password:
Test 5:- Import certificate in keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
3
Enter location of certificate
.
Enter certificate file name
soa.mycompany.com.cer
Enter alias name for the certificate
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Importing certificate into keystore
.
.
Trust this certificate? [no]:  yes
Certificate was added to keystore
Test 6:- Export certificate from keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
4
Enter location for exported certificate
.
Enter certificate file name for the exported certificate
myexportedcertificate
Enter alias name of the certificate to be exported
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Exporting certificate from keystore
Certificate stored in file <./myexportedcertificate.cer>
Test 7:- Delete certificate from keystore
$ ./certificate.sh
============================================
Press 1: To convert certificate into pem
Press 2: To convert certificate into p12
Press 3: To import certificate into keystore
Press 4: To export certificate from keystore
Press 5: To delete certificate from keystore
============================================
Enter your choice (1-5)
5
Enter alias name of the certificate to be deleted
mysoacert
Enter location of keystore
.
Enter keystore file name
mykeystore.jks
Enter keystore password
changeit
Deleting mysoacert certificate from keystore