Hunting Down an Apache PermGen Issue

imageWe have been experiencing a PermGen out-of-memory problem in an application we have been working on when redeploying our .war to a Tomcat 7.0.33 application server.  This would cause Tomcat to become unresponsive and crash after about 4 or 5 redeploys.

Our application is using Spring (Core, MVC, Security), Hibernate, and JPA.  Our database pool is being maintained by Tomcat and we get connections to this pool via JNDI.

When redeploying our .war by copying it to $CATALINA_HOME/webapps/ we would see a few lines in our logs that looked something like this:

SEVERE: The web application [/myApp] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@d73b31]) and a value of type [java.lang.Class] (value [class oracle.sql.TypeDescriptorFactory]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

So in our case it seemed to be an issue with the Oracle drivers.  Tomcat 7 does a bit better of a job determining whether there have been leaks and reporting the problem than 6 so there may or may not be useful info in the logs of an older Tomcat install.

Background

The PermGen memory space is typically a small memory space that the JVM uses to store class information and meta data.  It is different from the heap in that your application code has no real control over its contents.  When you start an application all the classes that are loaded will store information in the PermGen space.  When your application is undeployed this space will be reclaimed.

Cause

The cause of the problem of running out of PermGen space when re-deploying a web app in general is that some references to classes can’t be destroyed by the JVM when the application is unregistered.  This leaves all that classloader information sitting in PermGen since there are still live references.  This can be caused by threads started by the application that aren’t cleaned up, database connections that are not cleaned up, etc.  Sometimes this can be in libraries in use.

A good article on this can be found here:

http://plumbr.eu/blog/what-is-a-permgen-leak

HOWEVER, In our case this was caused by the way our application was deployed.  Maven was configured with ojdbc6.jar as a dependency thusly:

<dependency>
​    <groupId>oracle</groupId>
​    <artifactId>ojdbc6</artifactId>
​    <version>11.2.0.3</version>
</dependency>

This caused our war to include ojdb6.jar when deployed.  So in WEB-INF/lib we would see “ojdbc6.jar” in our deployed application.  This is where the problem starts.  When our code requests a database connection it is creating a DataSource object using our *local* ojdbc6.jar (actually our local ClassLoader which uses our local .jar) rather than the one installed in Tomcat’s shared library directory (via Tomcat’s own ClassLoader).  This now creates an object that Tomcat itself will hold on to  but references code in our application and thus our ClassLoader.  Now when our application is destroyed we get a leak since our ClassLoader can’t be removed.

Solution

In this scenario the solution is to tell Maven that the oracle library will be provided by our container.  It will be available at compile-time for our code but will NOT be included in our .war.

<dependency>
​    <groupId>oracle</groupId>
​    <artifactId>ojdbc6</artifactId>
​    <version>11.2.0.3</version>
​    <scope>provided</scope>
</dependency>

Now when our application requests a DataSource the server ojdbc6.jar will be used removing the dependency on our application.  This allows Tomcat to properly cleanup after “myApp.war” when it is removed.

6 ways to get your web presence and infrastructure in shape for 2010

In this lingering recession, everyone is looking for new ways to better position themselves to compete and grow revenue. A lower level of consumer and business spending will require efficiency, careful optimization and leverage of low cost assets and methods. It’s time to get into shape! Here are 6 ways to revamp and strengthen your web sites and infrastructure on a modest budget:

Revamp your web strategy for a web 2.0+ world.

The internet world has dramatically changed in the last 3-4 years. Social networks, user communities, user generated content, twitter, the iPhone and other mobile devices, GPS and location aware devices and the other components of Web 2.0 completely altered the way businesses and users communicate and transact online. Each of the Web 2.0 components come with their own set of opportunities and challenges. They provide new channels that enable communication at a fraction of the cost while demanding a new approach to openness, transparency and interactivity. Regulatory, security and governance concerns are not always easy to address. Chart a path in these new waters by rethinking your Web Strategy and redefine the role that the web and other digital channels will play in the company’s future and put a plan in place for its execution.  

Implement a social media strategy and measure its value

Social media tools are a great way to build honest online relationship with customers and other audiences. Doing it right is not always easy. A social media strategy will force you to think through and define where to be and what is to be communicated, set the tone and nature of interactions, set guidelines on how to respond to negative feedback, factor in legal and regulatory implications, address intellectual property and security issues and many other aspects need to be thought through. In addition, measuring the impact of these activities is not always easy. Building a model that can assess and provide value guidelines is very important. 

Reduce costs by Leveraging open source and Cloud web infrastructure components

We have a client who recently came to us asking advice after a planned $3M Oracle e-business implementation turned into a projected $15M 3 year project. We recommended they look at OfBiz and other open source e-commerce frameworks. Open source enterprise level software , SaaS and Cloud Computing have matured to the level that major organizations are leveraging these low cost scalable solutions to build a robust infrastructure that can replace big investments in hardware, software licenses and data centers.  

Take control of your content – Deploy a Content Management Solution

For many companies, fresh content is key to repeat visits. As sites scale, managing and maintaining them becomes an expensive and difficult task often dependent on IT or external resources. Content Management Systems (CMS) provide business users with the ability to modify and update sites and global structures that make graphical changes easy to implement. They also provide ability to segment users, add personalization and social features such as Blogs and community without the need for additional software and services.

User Experience Redesign

If your website has not gone through a redesign in the last 3 years, chances are that it looks dated. What looks fresh and relevant changes all the time and the key in the last few years has been incorporation of user engagement and interactivity, quality content that speaks more directly to the users, content targeting and using sites as relationship building tools rather than one way communication streams. Sites need to add rich content, video and mobile support as well as dynamic interfaces. All these changes contribute substantially to improved website ROI

Optimize sites for goals and conversion

It’s crucial that every marketing and search dollar is well spent. To do this, websites need strong web analytics so that sites can be continuously optimized to maximize conversion and be careful to avoid the main pitfalls. Web analytics capability allows businesses to test new ideas, layouts and promotions and to quickly refine them to drive sales and traffic as well as optimize search and marketing spend. With Google analytics and other low costs services, setting great analytics does not have to mean big bucks.

Web 2.0: Like Prego Spaghetti Sauce “It’s In There!”

It's in there!

It's in there!

Web 2.0 is giving me flashbacks to an old TV commercial for Prego spaghetti sauce; “Tomatoes, in there! Garlic, in there! Carrots, in there! Half of Italy, in there!…”  It seemed no matter what you asked for it was in that bottle of sauce.  Being a sauce, how could you really tell what was in there, or if it was really needed?  Plus, the tomatoes colored everything red so who knows?  Now we have another bottle of technical sauce here called Web 2.0; it’s in there!  It’s colored all Internet so how can you tell what is really in there, or if it is really needed?

Good question, seems like every vendor says they’re on the bottle of ingredients, in fact the most important one.  It would be funny if it was not so pathetic.  Unfortunately, the smell here is not a nice bubbling spaghetti sauce, closer to a warm crock of….., you get the concept.  Every vendor out there seems to believe companies will blindly buy anything labeled Web 2.0. Rather, the CIO’s are more apt to remember the Internet bubble and where that approach got them the last time.

What is required is more definition of what Web 2.0 is, and why we in IT need to move in that direction.  To get that basic understanding, we need to breakout that old spaghetti sauce pan again to boil out all the fancy analysis and obsequious technology.  Lo and behold! What remains is a simple concept: the inmates are now in control of the asylum.  Users of the Internet have turned the tables on the big players in the space, they are no longer happy being spoon fed from a portal. The denizens want to hunt it on their own terms, see it their own way, save it and dispose of it as they please.  If you stand in their way, this mob of Internet hunter-gatherers will crush you with the loss of their eyeballs (poor Yahoo, poor EBay, happy Facebook, happy iPhone).

If this basic principle is followed like a lode stone, much that is occurring in the Internet space is much more illuminating and the proper path forward (with supporting technology) is a great deal clearer to discern.  For example, the winning companies embrace openness and external developers.  There is no way their internal staff can create and the site push enough content and functionality to stay on top.  The Tao of a top site is to be one with the masses, following and attempting to push is uncool.  Allowing users to mash-up specialty widgets into cool personal discoveries is winning, monetization will ultimately follow.

By this point, you are thinking — how is all this ethereal philosophic spew helping me?  I need to get something together that can be called Web 2.0 or my IT existence is at risk!  Do not worry Grasshopper (I’m showing my ’70s again, rats!) I’ll put forward a corporate-friendly straw man.  If SharePoint is used to enable a project, process, or department; it is so Web 1.0 (boring!).  If we put the entire corporation up on SharePoint, acting like a corporate Facebook, we are getting there.  If we template it such that we now have ubiquitous collaboration; optimizing and moving our corporate intellectual property (IP) at light speed much nicer.  But for ultimate coolness, we need to commit heresy and wire a Google search appliance in, after adding all of our corporate content to the pile: documents, presentations, everything.  Then the cherry on top, flatten key data bases to HTML and toss them in.  Now, with proper organizational change management (Yes Billy! You can run with scissors, points down please), employees can use all of the power contained in Web 2.0 to maximize unstructured corporate data for speed and profit.  Mangiare! Mangiare!

Open Source Development: Top 5 Things I’ve Always Wanted to Know

I am primarily familiar with Microsoft products and languages, but am interested in learning more about open source.  To summarize, here are the top 5 things I’ve always wanted to know about open source development:

1.  Who do you talk to when you have a support issue regarding an open source product that you’re using?

2.  Are open source programs as secure as proprietary programs and are open source solutions embraced by large corporations even though they don’t have a name like Oracle, Microsoft etc. behind them for support?

3.  What are the most popular open source programs out there?

4.  What’s the best place on the web to learn about the open source solutions out there today?

5.  Are there any known drawbacks to open source versus proprietary software that people should know about before pursuing it just on a cost basis?

Granted, some of these I know the partial answers to, but as someone who doesn’t use open source very readily except for Log4Net, it’s fairly representative of the questions I’d have about open source technologies.