06 November 2011

Dealing with Weblogic Stuck Threads

Check below for:
  • Definition or What is a Stuck Thread?
  • The problem or Why are Stuck Threads evil?
  • What you can do to avoid your application completely fail?
  • How to identify the problem?
  • How to workaround the problem?
  • Test: How to create a Stuck Thread?

Definition or What is a Stuck Thread?
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time.
You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck (Stuck Thread Max Time), and by changing the frequency with which the server checks for stuck threads. Check here to see how to change the Stuck Thread Max Time.

The problem or Why are Stuck Threads evil?
WebLogic Server automatically detects when a thread in an execute queue becomes "stuck." Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. If all threads in an execute queue become stuck, the server changes its health state to either "warning" or "critical" depending on the execute queue:
  • If all threads in the default queue become stuck, the server changes its health state to "critical." (You can set up the Node Manager application to automatically shut down and restart servers in the critical health state. For more information, see "Node Manager Capabilities" in Configuring and Managing WebLogic Server.)
  • If all threads in weblogic.admin.HTTP, weblogic.admin.RMI, or a user-defined execute queue become stuck, the server changes its health state to "warning."
So practically, a couple of Stuck Threads might not crash your server preventing it from serving request, but it is a bad sign. Usually, the number of stuck threads will increase and your server  will eventually crash.

What you can do to avoid your application completely fail?
WebLogic Server checks for stuck threads periodically (this is the Stuck Thread Timer Interval and you can adjust it here). If all application threads are stuck, a server instance marks itself failed, if configured to do so, exits. You can configure Node Manager or a third-party high-availability solution to restart the server instance for automatic failure recovery.You can configure these actions to occur when not all threads are stuck, but the number of stuck threads have exceeded a configured threshold:Shut down the Work Manager if it has stuck threads. A Work Manager that is shut down will refuse new work and reject existing work in the queue by sending a rejection message. In a cluster, clustered clients will fail over to another cluster member.
  • Shut down the application if there are stuck threads in the application. The application is shutdown by bringing it into admin mode. All Work Managers belonging to the application are shut down, and behave as described above.
  • Mark the server instance as failed and shut it down it down if there are stuck threads in the server. In a cluster, clustered clients that are connected or attempting to connect will fail over to another cluster member.

How to identify the problem?
The most recommended way is to check the thread dumps. Check Sending Email Alert For Stuck Threads With Thread Dumps post of Middleware magic, to have Thread Dumps mailed to you automatically  when they occur.

Tools to help you with analyzing the Thread Dumps can be:

How to workaround the problem?
After you have identify the code that causes the Stuck Thread, that is the code which execution takes more than the Stack Thread Max Time, you can use Work Manager to execute your code. Work Managers have a Ignore Stuck Thread options that gives the ability to execute long running jobs. See below:
Below are some posts on how to create a Work Manager

Test: How to create a Stuck Thread?
How to create a Stuck Thread in order to test your weblogic settings? Put a breakpoint in a  backing bean or model method that is called with you request. If you wait in the breakpoint for  Stuck Max Thread Time, you notice a Stuck Thread trace will be shown in servers log:
<16 =Ύί 2011 12:28:22 ΉΉ EET> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 
    'weblogic.kernel.Default (self-tuning)' has been busy for "134" seconds working on the 
request "weblogic.servlet.internal.ServletRequestImpl@6e6f4718[
GET /---/---/----/---/days.xhtml HTTP/1.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=DYG5TDTZSnKLTFw5CMMdLCD9sPsZS4Jqlmxj9wdGNyt1BnPcfNrR!-1520792836

]", which is more than the configured time (StuckThreadMaxTime) of "60" seconds. Stack trace:

More digging:

01 November 2011

TROUBLESHOOTING: weblogic.descriptor.BeanAlreadyExistsException

Example Error
weblogic.descriptor.BeanAlreadyExistsException: Bean already exists: "weblogic.j2ee.descriptor.wl.LibraryRefBeanImpl@c76e5f19(/LibraryRefs[[CompoundKey: adf.oracle.domain]])

Dimitris Stasinopoulos posted on his blog regarding this error. Quick tips based on his post:

  • Source of your problem is weblogic-application.xml. Check to see if you have duplicated or other unrelated entries and remove them. 

More details on his post.

05 October 2011

Enable Weblogic to automatically back up its configuration (config.xml)

Check how to set Weblogic take automatically back up of you configuration (config.xml file)  in Hasim's post Weblogic : Back Up Config.xml file.
Reposting some part of his post:

Login to weblogic console
Click Domain –> Configuration –> General (Advanced)
Check Configuration Archive Enabled and Count.
true - backups of the configuration will be made during server boot.
Change will take place after you redeploy module or restart the server

it will create following files in domain.
  • config-booted.xml
  • config-original.xml

29 September 2011

TROUBLESHOOTING: Unable t o create an instance of the Java Virtual Machine Located at path: ... jre\bin\client\jvm.dll

This is happening when you click to start JDeveloper. It throws this error and fails to start it.

The amount of RAM allocated for JDeveloper's JVM is more than your system can provide and as a result JDeveloper will not start.

Reduce the RAM. Check my previous blog post "TROUBLESHOOTING: "Low Memory Warning" of JDeveloper or How to tweak memory settings." on how to do it

29 August 2011

Modify massively ADF BC xml and ADF jspx pages preserving file text format (without loosing SVN history)

The Problem
ADF BC and also ADF pages are xml files. Sometimes you want to massively change all the files of your application. Typical examples are when you want to change the template of all jspx pages of your application or the bundle that all your entities and/or ViewObjects are using. You might easily do it with DOM but then you ll loose the SVN history, because DOM rewrites all the file and the text format will be lost. Other alternative is to do it by hand, but when you have 1000 of files in your application then this might take enough time.

The tool
If you want to do it automatically and modify only the part of the file that is needed (and so preserve SVN history), check my post Modifying xml without changing the text format (and so without loosing SVN history) for a small tool that I developed to help you perform this task.


25 August 2011

Cost of Ownership Analysis: Oracle WebLogic Server vs. JBoss

A very interesting whitepaper by Crimson Consulting Group comparing the cost of ownership between Weblogic and JBoss. Although JBoss is free, the whitepaper has serious claims that on the long run Weblogic is cheaper. Although, this study was sponsored by Oracle, it seems to be very serious and definitely is worthy of a look.

Below some interesting parts from the whitepaper:

JBoss costs more than WebLogic Server after 2 years and as much as 35% more over 5 years.

Key Takeaways
  • JBoss is 35% more costly than WebLogic Server over 5 years, despite its free license.
  • Oracle WebLogic Server becomes less expensive on a TCO basis within two years from acquisition – an advantage that continues to grow with every year of operation.
  • Software licensing is a small portion of the total cost of ownership; people costs in operations drive the bulk of long-term costs.
  • Other issues, such as performance, time-to-value, and customized infrastructure, can have a significant impact on the overall business ROI of an application server deployment.

 Figure 1 illustrates how small the initial costs of Acquisition and Implementation are with respect to the total 5-year costs of an application server deployment. The savings created by not paying for a software license are more than offset by having to invest in employees and consultants for implementation, development of custom scripts and utilities, configuring and testing other open source components, and managing and monitoring the JBoss environment.

Description of Cost Categories Included in Research and Analysis
Description Acquisition This category includes the hard costs for purchase of the application server software and for the hardware platform(s) to run it.
This category includes the labor costs for implementation, installation, configuration, and testing of the application servers and the related infrastructure. Ongoing Application Deployment &amp;amp; Testing Costs This category includes the ongoing labor costs for deploying custom applications from test and staging environments to production environments. It also includes the ongoing interoperability testing and periodic testing for new releases and updates to the application servers and other infrastructure components.
Ongoing Vendor Support Costs
This category includes the hard costs for annual subscription support or maintenance agreements for the application server software, as well as for any additional software required. Ongoing Administration &amp;amp; Management Costs This category includes the ongoing labor costs to configure, manage, and maintain the application servers and the related infrastructure.
Ongoing Monitoring, Diagnostics, &amp;amp; Tuning Costs
This category includes the ongoing labor costs to monitor, tune, and optimize the application servers. Other Cost Considerations This category includes cost considerations identified in the study but not necessarily included in the cost of ownership model. This includes the cost of unplanned downtime, time to market, and backward compatibility considerations.

Table 3 outlines the pro forma costs for a typical application server deployment, consisting of 5 server hosts (server blades with two dual-core processors each), running an average of 4 application server instances per host (one instance per core). The acquisition and on-going costs in Table 3 reflect current list prices for hardware, software and support, less an average discount of 25%, while the people costs for implementation, deployment, testing, administration, and management are based on the results of Crimson’s primary research and resulting cost model.

Table 4 shows the total costs of implementing, configuring, and customizing the two application servers. Key takeaways are:
  • JBoss implementation costs more than twice as much as WebLogic Server implementation.
  • By the end of the implementation phase, the cost of JBoss (inclusive of acquisition cost) is within 33% of the cost of WebLogic Server and operations haven’t started yet.
  • Though we haven’t tried to quantify the business cost of the delay in time-to-value associated with an extra 8.5 weeks of effort, it could clearly be substantial.

Check the whitepaper for more like Ongoing Operations which involve:
  • Application Deployment and Infrastructure Testing Costs
  • Application Server Administration, Management, Monitoring, and Tuning Costs
  • Monitoring, Diagnostics, and Tuning Costs.

  • “Out-of-the-box” configuration and implementation tools are more mature, robust, and efficient for WebLogic Server than for JBoss, with the result that time-to-value is faster, the customization needs lower, and the costs lesser than with JBoss.
  • Similarly, out-of-the-box administration, management, and tuning tools have been through as many development cycles as the core software and are consequently more complete and more productive than their equivalents in the JBoss environment.
  • Oracle takes on the responsibilities and costs of maintaining performance and backwards-integration as the software evolves; users of JBoss take on those responsibilities for themselves.
All these factors combine, with additional software-specific performance issues, to give a very different picture of the total cost of ownership in comparison to the initial acquisition costs. In fact, Crimson’s analysis indicates Oracle WebLogic Server becomes less expensive on a TCO basis within two years from acquisition – an advantage that continues to grow with every year of operation. Over a 3-to-5 year time horizon, the TCO of Red Hat JBoss becomes as much as 35 percent more than WebLogic Server, in spite of its lower acquisition cost.

26 July 2011

OPSS config files: jps-config.xml, jazn-data.xml and cwallet.sso

Reposting a very helpful post OPSS Artifacts Life Cycle in ADF applications of Andre Correa.

After writing about users and groups migration, it looked to me we should also talk about the life cycle of other important entities in secured ADF applications. When you enable security in an ADF application, you see a couple of new artifacts in your JDeveloper workspace, namely jps-config.xml, jazn-data.xml and cwallet.sso.
Have you ever wondered what their purpose is, their life cycle and how they relate to WLS domain security configuration? This article is just about it.
As you might know, secured ADF applications leverage OPSS (Oracle Platform Security Services).
OPSS is a fundamental component within Oracle Fusion Middleware security. It works as an abstraction layer on top of security services providers, shielding applications from all the complexities in dealing with them. For instance, applications can transparently switch between file-based and LDAP-based policy stores. Likewise for credential store services.
Let's take a closer look at each of those artifacts and their life cycles.


This file can be seen as the lookup services registry for OPSS. Among these services are login modules, authentication providers, authorization policy providers, credential stores and auditing services.
Whenever an OPSS-enabled application requires security services, it looks up a JPSContext object where all the necessary services are supposedly configured.
In ADF applications, a workspace-level jps-config.xml is created once ADF security is enabled. It drives services lookup for ADF's BC (Business Components) Tester available in JDeveloper, which is a JavaSE application.
If you want to have security unit tests, you can also easily leverage it.
It is never used once the ADF application gets deployed in a WLS container, even though it is packaged in the ear file. Within a WLS container, a jps-config.xml in /config/fmwconfig is used by all applications in all servers deployed in that WLS domain. There's no such concept of an application-level or server-level jps-config.xml.


This file keeps users, groups and authorization policies for OPSS-enabled applications and is automatically created once ADF security is enabled. I've already covered users and groups life cycles in a previous article. It is important to mention that users and groups are also leveraged by ADF's BC Tester and can be integrated into security unit tests as well.
Authorization policies are, if not the most, one of the most sensitive parts of a secured ADF application, since it governs who has access to what. As you might guess, they are also leveraged by ADF's BC Tester. When the ADF application is deployed into WLS, at startup time, policies are OOTB (Out-Of-The-Box) migrated into the configured policy store, who, by default, is a file called system-jazn-data.xml, located under /config/fmwconfig. You can configure how (and if) policies are migrated through some properties in weblogic-application.xml. Here they are:
This listener is the one actually responsible for pushing the changes to the runtime policy store. Make sure it is present in weblogic-application.xml. Otherwise, you’ll experience a lot of frustration in trying to deploy authorization policies along with your application.
MERGE, OVERWRITE and OFF are exclusive and applicable for deployments and redeployments. And they mean exactly what you might be thinking.
  • MERGE will merge what’s already available in the runtime policy store. This might be particularly useful in some advanced deployments where more than one application share the same application policy stripe.
  • OVERWRITE wipes away the existing application policy stripe and load all policies from scratch.
  • OFF skips policy migration.
Do notice that once the application is undeployed, its policies are also removed from the policy store, unless you set the following property in weblogic-application.xml:
Authorization policies migration will always happen according to weblogic-application.xml configuration, no matter what the deployment method is.


This file keeps credentials used by the application. A subtle and fundamental distinction is important to be made here: credentials and identities are not the same thing. Simply put, in OPSS, identities are what authentication requests are done against, while credentials are securely kept objects that are somehow presented to authentication providers to be matched against identities.
cwallet.sso is encrypted and you cannot browse it or explicitly edit it via JDeveloper. At design-time, different components make use of cwallet.sso and are responsible for creating the necessary credentials in it. Examples are OWSM policy attachments that override the csf-key and ADF connections requiring credentials in the call out.
If you need credentials that can’t be created within JDeveloper, you can either use wlst createCred online command or write some code using OPSS APIs. Both options makes the whole life cycle story a little catchy, because a running WLS container is necessary. You can also disable credentials migration and create them directly in the WLS domain where applications are deployed.
Like authorization policies, credentials are also OOTB migrated into the configured WLS domain credential store on application startup. By default, the credential store is the cwallet.sso file in /config/fmwconfig folder.
The following weblogic-application.xml properties govern how (and if) they're deployed.
As for policies, the same listener migrates credentials. Avoid frustration and make sure the listener is present if you want to migrate or control how your credentials are migrated.
  • MERGE: migrate non-existing credentials only;
  • OVERWRITE: overwrites existing credentials;
  • OFF: skips credentials migration; 

Dig more:
Do not forget to check also another 2 very good posts of the Andre:

25 July 2011

A Case Study in an Oracle Forms Redevelopment Project to Oracle ADF

A very useful  white paper  by Grant Ronald about redeveloping Forms to ADF. Quoting him "This paper covers topics such as design decisions, architecture, best practice, coding standards we well as showing common mappings of Oracle Forms features to Oracle ADF."

More on his blog post.

21 July 2011

How to configure the Active Directory Provider Specific settings in your Weblogic

In a previous article Setting up MS Active Directory as Authenticator Provider in Weblogic, it is described how to set Active Directory Provider.
Now we ll see how to see the Provider Specific settings based on your Actice Directory structure that was not mentioned then.

Open the Active Directory Provider that you want to configure its Specific setting 

On Host and Port provide the related ones.

Open Active Directory Users and Computers from the Administrative Tools

In the WLS's Provider Specific settings, set the Principal field based on the Username, User folder and domain name.
As we can see from above the Principal would be:


As you can notice dc=com is because the .com in the domain name.

In the Credential type the one provided by your Administrator.

In the WLS's Provider Specific settings, set the User Base DN:
ou=Users,ou="Your Organizational Unit",dc=YourDomain,dc=com
based on the following:

 Set the Group Base DN:
ou="Your Organizational Unit", dc=YourDomain,dc=com
based on the following:

Note: use "" when you name contains spaces.

The value of the fields that you depend on the structure of your AD and might vary in your case.

TROUBLESHOOTING: The domain edit lock is owned by another session in non-exclusive mode (WLS deployer)

[Deployer:149163]The domain edit lock is owned by another session in non-exclusive mode - this deployment operation requires exclusive access to the edit lock and hence cannot proceed. If you are using "Automatically Aquire Lock and Activate Changes" in the console, then the lock will expire shortly so retry this operation.

Actually what Deployer is trying to tell you is that it tried to deployed but another user has Lock and Edit the domain . More regarding Lock and Edit in this related post.

In other not to fail but exist gracefully leaving you deployment to be applied when the Lock is released, then use nonexclusivelock option.

Using this option you ll get the following message, if another user has avquire the lock and the script will not fail:
[wldeploy] Operation is pending and will be activated or cancelled when the ongoing edit session is activated or cancelled.

In case if you use a wldeploy ant task, here is an example script

<project name="deploy" default="deploy">
<property file="deploy.properties" />

<!-- Setting TaskDefinition -->
<taskdef name="wldeploy" classname="weblogic.ant.taskdefs.management.WLDeploy">
        <pathelement location="${wldeploy.home}/weblogic.jar"/>
<!-- Deploying Applications  -->
<target name="deploy">
<wldeploy action="deploy"
          usenonexclusivelock="true" />
Dig more:

20 July 2011

TROUBLESHOOTING: Bad Request' for url: '.../bea_wls_deployment_internal/DeploymentService (upon WLS deployment)

weblogic.deploy.api.internal.utils.DeployerHelperException: The source '/tmp/ApplicationGreeceDEV.ear' for the application 'ApplicationGreeceDEV#2.0'
could not be loaded to the server ''.
Response: '400: Bad Request' for url: ''

I know it seems silly but do not waste time. Just restart server.

TROUBLESHOOTING: Changing the source location is not allowed for a previously attempted deployment. Try deploying without specifying the source. (WLS Error upon deployment)

[Deployer:149007]New source location, '/u01/app/oracle/product/Middleware_1034/user_projects/domains/dev/servers/AdminServer/upload/ApplicationGreeceDEV/2.0/app/ApplicationGreeceDEV.ear',
 cannot be deployed to configured application, 'ApplicationGreeceDEV [Version=2.0]'. 
The application source is at
 Changing the source location is not allowed for a previously attempted deployment. Try deploying without specifying the source.

Possible solution
(or  Solution that worked in my case)

-Undeploy the application.

-Go to Admin server's upload folder

eg. upload in my case

cd /u01/app/oracle/product/Middleware_1034/user_projects/domains/dev/servers/AdminServer/upload

-Remove any relate archives or directories
eg. ApplicationGreeceDEV.ear or ApplicationGreeceDEV directory
from upload folder

17 July 2011

How to set the client and server VM option in JDeveloper

How to set it
Right click you project
Go to Run/Debug/Profile
Select the option you prefer from the Virtual Machine drop down list.

Now when you run you application you ll see the switch in the java run.

Strange behavior
When running the BC tester or a typical java app it seems to take the setting. When running ViewController it ignores it. Strange...

What is client and server VM  options
Regarding the  the client and server VM options you can check my other post - Difference between -client and -server JVM command line options.
In 2 words: The client system is optimal for applications which need fast startup times or small footprints, the server system is optimal for applications where the overall performance is most important. In general the client system is better suited for interactive applications such as GUIs. Some of the other differences include the compilation policy,heap defaults, and inlining policy.

JRockit  client and server VM  options
Make sure when you run your Fusion applications over JRockit not to have the -client option in your java run.
Check my other post - Difference between -client and -server JVM command line options  for more.

Dig more:

12 July 2011

How to change Weblogic server's log file path

i.e. where WLS saves its log file.

Under Domain Structure
Go to Environment and
then Servers.
Click on the Server you want to change the log (that is DefaultServer(admin) in case of integrated WLS)
Click Logging tab

Set Log file name with desired path and name.

 You can tune other params there too.

06 July 2011

Setting up MS Active Directory as Authenticator Provider in Weblogic

How to do it
Got to:
Security Realms
My realm
Click: New
     Name eg. ADProvider,
     Type: ActiveDirectoryAuthenticator


All the Providers will be displayed.
Click Reorder
Set the new Provider to be on top

 Open again the newly created Provider

Enter: Control Flag: SUFFICIENT

Click Provider Specific and enter all the Active Directory related settings.
Check here for a how to.

Usually, depending on the priority that you want the providers to have, you may want to change the Control Flag from REQUIRED to SUFFICIENT in the DefaultAuthenticator of WLS, too.

To check if it works:
Restart Weblogic
Go to
Security Realms
My realm
Users and Groups
There you must be able to see the users and groups of your Active Directory.

Src/Dig more:

You might also like:

Related Posts Plugin for WordPress, Blogger...
There was an error in this gadget