Hunting Down an Apache PermGen Issue

imageWe have been experiencing a PermGen out-of-memory problem in an application we have been working on when redeploying our .war to a Tomcat 7.0.33 application server.  This would cause Tomcat to become unresponsive and crash after about 4 or 5 redeploys.

Our application is using Spring (Core, MVC, Security), Hibernate, and JPA.  Our database pool is being maintained by Tomcat and we get connections to this pool via JNDI.

When redeploying our .war by copying it to $CATALINA_HOME/webapps/ we would see a few lines in our logs that looked something like this:

SEVERE: The web application [/myApp] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@d73b31]) and a value of type [java.lang.Class] (value [class oracle.sql.TypeDescriptorFactory]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

So in our case it seemed to be an issue with the Oracle drivers.  Tomcat 7 does a bit better of a job determining whether there have been leaks and reporting the problem than 6 so there may or may not be useful info in the logs of an older Tomcat install.

Background

The PermGen memory space is typically a small memory space that the JVM uses to store class information and meta data.  It is different from the heap in that your application code has no real control over its contents.  When you start an application all the classes that are loaded will store information in the PermGen space.  When your application is undeployed this space will be reclaimed.

Cause

The cause of the problem of running out of PermGen space when re-deploying a web app in general is that some references to classes can’t be destroyed by the JVM when the application is unregistered.  This leaves all that classloader information sitting in PermGen since there are still live references.  This can be caused by threads started by the application that aren’t cleaned up, database connections that are not cleaned up, etc.  Sometimes this can be in libraries in use.

A good article on this can be found here:

http://plumbr.eu/blog/what-is-a-permgen-leak

HOWEVER, In our case this was caused by the way our application was deployed.  Maven was configured with ojdbc6.jar as a dependency thusly:

<dependency>
​    <groupId>oracle</groupId>
​    <artifactId>ojdbc6</artifactId>
​    <version>11.2.0.3</version>
</dependency>

This caused our war to include ojdb6.jar when deployed.  So in WEB-INF/lib we would see “ojdbc6.jar” in our deployed application.  This is where the problem starts.  When our code requests a database connection it is creating a DataSource object using our *local* ojdbc6.jar (actually our local ClassLoader which uses our local .jar) rather than the one installed in Tomcat’s shared library directory (via Tomcat’s own ClassLoader).  This now creates an object that Tomcat itself will hold on to  but references code in our application and thus our ClassLoader.  Now when our application is destroyed we get a leak since our ClassLoader can’t be removed.

Solution

In this scenario the solution is to tell Maven that the oracle library will be provided by our container.  It will be available at compile-time for our code but will NOT be included in our .war.

<dependency>
​    <groupId>oracle</groupId>
​    <artifactId>ojdbc6</artifactId>
​    <version>11.2.0.3</version>
​    <scope>provided</scope>
</dependency>

Now when our application requests a DataSource the server ojdbc6.jar will be used removing the dependency on our application.  This allows Tomcat to properly cleanup after “myApp.war” when it is removed.

Is Custom Development Dead?

Is Custom Development dead? After the last two years of custom development’s nuclear winter, (following 2008s Financial Armageddon), one would think the the Grim Reaper did his best in the blast. I really hope not, designing and building strategic systems make the more mudane aspects of software engineering worth enduring the mind-numbing syntactical pain of creation. Nothing like the smell of napalming the competition with a totally new way of doing business in the morning (my view of “Apocalypse Now” with a business bent). Maybe, just maybe, I hope rumors of Custom Development’s death are greatly exaggerated.

Did Custom Development die of natural causes, maybe pulled off of life support by risk-hating Executive Management as a perverse form of parental control after the financial snafu (Custom Development moves from PG-13 to XXX)?  Off-the-Shelf software products and the ever increasing cost of continuing maintenance really hurt Custom Development as a viable systems choice, but is that enough to put it down? Cloud and “nouveau Cloud” technologies (read SaaS, SalesForce.com) may have provided the coup de grace.  I seriously doubt it, every time I look into the Cloud I get serious PTSD flashbacks to the 70s and 80s IBM Mainframe World Domination (OMG there is a 3270 in the corner!!).  At least there was alot less hype and easier choices back then (Nobody got fired picking IBM!).

It is possible Custom Development died offshore (simple Mickey Finn, bag over the head, Shanghaied and Held for Ransom!)?  While Business Processes and System Maintenance have done reasonably well offshore (Castor Oil of the Corporate world, let Mikey take it!), strategic custom development has had less success.  Quality innovation that can transform a corporation really requires a local team steeped both in the host company and surrounding culture.  Plus, Custom Development tends to have a high infant mortality rate so it is best attempted in short phases supported by an Agile Methodology, definitely not in Offshore’s financial model wheelhouse.  So I don’t think Offshore is implicated.

There is the theory that evolution has spoken and Product-based Solutions have succeeded Custom Development, just as mammals succeeded dinosaurs.  Product companies would like you to believe that, but does that seem plausable (Land of the Lost, Jurassic Park where are you?)?  While Product-based solutions have advantages in success rates and cost, they by their nature lack the true freedom that drives raw creativity and innovation.  Custom Development is that wellspring.

The only thing we have to fear, is fear itself!  Adversity to risk is curbing animal spirits, creativity, and innovation, ….for now.  Custom Development is not dead and will return from its vacation with Puff in the Cave by the Sea when Jackie Paper locks-and-loads and we begin some serious innovation and transformation with strategic custom software systems (BTW thats when the Jobs return too!).

6 ways to get your web presence and infrastructure in shape for 2010

In this lingering recession, everyone is looking for new ways to better position themselves to compete and grow revenue. A lower level of consumer and business spending will require efficiency, careful optimization and leverage of low cost assets and methods. It’s time to get into shape! Here are 6 ways to revamp and strengthen your web sites and infrastructure on a modest budget:

Revamp your web strategy for a web 2.0+ world.

The internet world has dramatically changed in the last 3-4 years. Social networks, user communities, user generated content, twitter, the iPhone and other mobile devices, GPS and location aware devices and the other components of Web 2.0 completely altered the way businesses and users communicate and transact online. Each of the Web 2.0 components come with their own set of opportunities and challenges. They provide new channels that enable communication at a fraction of the cost while demanding a new approach to openness, transparency and interactivity. Regulatory, security and governance concerns are not always easy to address. Chart a path in these new waters by rethinking your Web Strategy and redefine the role that the web and other digital channels will play in the company’s future and put a plan in place for its execution.  

Implement a social media strategy and measure its value

Social media tools are a great way to build honest online relationship with customers and other audiences. Doing it right is not always easy. A social media strategy will force you to think through and define where to be and what is to be communicated, set the tone and nature of interactions, set guidelines on how to respond to negative feedback, factor in legal and regulatory implications, address intellectual property and security issues and many other aspects need to be thought through. In addition, measuring the impact of these activities is not always easy. Building a model that can assess and provide value guidelines is very important. 

Reduce costs by Leveraging open source and Cloud web infrastructure components

We have a client who recently came to us asking advice after a planned $3M Oracle e-business implementation turned into a projected $15M 3 year project. We recommended they look at OfBiz and other open source e-commerce frameworks. Open source enterprise level software , SaaS and Cloud Computing have matured to the level that major organizations are leveraging these low cost scalable solutions to build a robust infrastructure that can replace big investments in hardware, software licenses and data centers.  

Take control of your content – Deploy a Content Management Solution

For many companies, fresh content is key to repeat visits. As sites scale, managing and maintaining them becomes an expensive and difficult task often dependent on IT or external resources. Content Management Systems (CMS) provide business users with the ability to modify and update sites and global structures that make graphical changes easy to implement. They also provide ability to segment users, add personalization and social features such as Blogs and community without the need for additional software and services.

User Experience Redesign

If your website has not gone through a redesign in the last 3 years, chances are that it looks dated. What looks fresh and relevant changes all the time and the key in the last few years has been incorporation of user engagement and interactivity, quality content that speaks more directly to the users, content targeting and using sites as relationship building tools rather than one way communication streams. Sites need to add rich content, video and mobile support as well as dynamic interfaces. All these changes contribute substantially to improved website ROI

Optimize sites for goals and conversion

It’s crucial that every marketing and search dollar is well spent. To do this, websites need strong web analytics so that sites can be continuously optimized to maximize conversion and be careful to avoid the main pitfalls. Web analytics capability allows businesses to test new ideas, layouts and promotions and to quickly refine them to drive sales and traffic as well as optimize search and marketing spend. With Google analytics and other low costs services, setting great analytics does not have to mean big bucks.

Enterprise e-Commerce on a Shoe String Budget?

e-commerce on a shoe string

Image courtesy of Flickr

While inexpensively built and operated mom and pop e-commerce websites are as common as snow in New England in January, is it possible to build and operate an enterprise grade e-commerce site on a shoe string budget? E-commerce at an enterprise level is not simply slapping a shopping cart to your website and calling it e-commerce enabled. The demands of an enterprise solution may require:

  • Integration with legacy systems
  • Integration with supply-chain systems
  • Support for multiple currencies and tax codes
  • Multiple store-fronts
  • Profile and history driven offer management
  • Integration with a content management system
  • Business user control over promotions and pricing
  • …and more

Challenges of integration with existing systems alone are daunting enough never mind the fancy e-commerce functionality that is often considered vital for competitive differentiation. No wonder why starting an e-commerce venture or an upgrade is considered a seven figure expense. The cost of an enterprise grade e-commerce product alone can easily account for twenty to forty percent of the budget. The other option is to go with a hosted or SaaS based approach and avoid capital expense for software and infrastructure – not a bad approach for testing the waters but in the long run, charges and fees can really add up.

A well executed e-commerce site can provide great returns on the investment by generating new revenue streams, enhancing existing ones, or reducing operational expenses – and that can’t be too bad for the budget or your career. However, in tough economic times the challenge becomes harder as getting approval for large complex projects becomes difficult and even the approved budgets can get slashed. If your budget gets cut, is there a way to still implement enterprise grade e-commerce? Can an open source e-commerce solution be the answer to the “do more with less” mantra? Is open source e-commerce ready to play with the big boys in the enterprise domain? Let’s explore these questions and the capabilities of the open source e-commerce solutions.

Let’s start with a common misconception that an open source e-commerce product requires significant customizations and the cost of customizations more than offsets any savings from not having to pay license fees. Implicit in this assumption is the notion that a commercial product requires little or no customizations. However, the real-world experience shows us that this is not the case. Even the best commercial products cannot be used out-of-the-box unless you decide to adopt their look and feel and their model of e-commerce. The cost of customizations can add up just as rapidly in a commercial product as they can in an open source one. Therefore a prudent approach would be to adhere to the industry standards and best practices and use out-of-the-box functionality in areas which are not competitive differentiators. Heavy customizations should be limited to the aspects of the website that are true differentiators and result in a unique user experience. This guiding principle applies regardless of the decision to use an open source or a commercial product.

There are a lot of inexpensive and open source e-commerce products out there; however, most of them are nothing more than a simple shopping cart. They are only suitable for the most basic needs of a simple web site. However, Apache OFBiz and Magento are two promising contenders that break from the pack and compete in the enterprise space. In this article we will primarily focus on OFBiz.

Apache OFBiz is actually an integrated suite of products that does not only include e-commerce capabilities but also provides support for accounting, order management, warehouse management, content management and more. An enterprise e-commerce implementation cannot exist as a point solution. It has to integrate and work well with other back office processes and applications. OFBiz’s integrated suite can be used to automate and integrate most back office functions. Even if you decide not to use the built-in functionality it can still be integrated with other existing systems albeit with more effort and cost. It provides enough e-commerce functionality out of the box to match most enterprise needs and the rest can be customized if needed. Here is a summary of our assessment of OFBiz:

Technical Capabilities

# Criteria Rating Comments
1. E-commerce capabilities B+ Provides Robust e-commerce capabilities OFBiz e-commerce capabilities include: catalog management, promotion & pricing management, order management, customer management, warehouse management, fulfillment, accounting, content management, and more.
2. Sign-on and Security B Granular and robust security framework The OFBiz security framework provides fine grain control of the security including multiple security roles and privileges. Roles can be used to control access to screens, business methods, web requests (URLs), and/or entire applications.
3. Technical flexibility & ease of use B Very flexible but complex  OFBiz is an application development platform that can be used to build applications and as such provides a tremendous amount of flexibility.  The use of the entire framework (which includes the database, an Object Relational Mapping (ORM) layer, business object layer, scripting support, and UI tools) is optional.
4. Integration with other apps and locations A Multiple integration methods  OFBiz business services can be exposed as services and accessed by multiple methods including Remote Method Invocation (RMI) and XML Web Services.  Integration directly with the OFBiz Relational Database is also possible.
5. Scalability A Highly Scalable  Java systems are highly scalable provided a production architecture that is designed to support heavy load.  A load balancing device and redundancy at the web, application and database servers can redundancy and scalability.
6. Relational database integration A Support for all major database platforms  The most popular OFBiz database platforms are PostgreSQL and MySQL (both of which are open source).  OFBiz has also been tested with Oracle, DB2, Sybase, and MS SQL Server.  The default installation uses an Apache Derby database which is not recommended for production use. Our research indicates some problems with MS SQL Server database – this should be investigated further prior to selecting that database platform.
7. Skill Set to support NA OFBiz framework and application are based in the following technology components:

  • XML
  • Web Development: HTML, CSS, AJAX/JavaScript, Apache
  • Java Development: Java, JSP, Freemarker, BeanShell, Tomcat application server (possibly)
  • Database Development and Administration: MS SQL Server (possibly), SQL, JDBC

Long term support of the application would require knowledge and familiarity in each of these technology sets.  While these technologies are mainstream and skills should be readily available in the future, skills and experience with the OFBiz framework that is built upon these technologies may not be.

Business Position

# Criteria Rating Comments
1. Financial stability B OFBiz is a “top level” project in the Apache Software Foundation.  The Apache Software Foundation provides support for the Apache community of open-source software projects. The Apache projects are characterized by a collaborative, consensus based development process, an open and pragmatic software license, and a desire to create high quality software that leads the way in its field.
2. Maturity of product suite B Open For Business (OFBiz) was initially launched in 2001.  In early 2006, the project went through the Apache Foundation’s “Incubation” process to review projects for quality and open source commitment.  OFBiz was promoted to a top level Apache project in December 2006.The community for OFBiz is very active.  The major web posting board receives between 20-40 postings per day relating to OFBiz.  The original contributors are very active in monitoring these sites and sharing knowledge.
3. Reference Accounts B- Total number of installations is unknown due to the nature of open source software. The OFBiz websites lists more than 70 companies that use their software. However, there are very few marquee names.

Implementing an enterprise e-commerce solution can be expensive and complex process that requires analysis and investment in people, processes, and technology. While it would be insincere to say that an enterprise e-commerce solution can be implemented on a budget in the ballpark of a mom and pop e-commerce store, the budget can be significantly reduced by:

  • Carefully crafting business requirements
  • Adapting the business model to match industry’s best practices
  • Reducing and carefully planning data migration and application integration
  • Keeping the customizations to a minimum
  • And using an open source e-commerce platform

OFBiz provides a viable open source e-commerce stack that can be used to implement enterprise grade e-commerce. When combined with good implementation practices and solid execution the combination can result in slashing costs by twenty to forty percent — which sometimes can make the difference between getting funded or getting shelved.

Top 5 Web Technology Trends for 2009

cball

The Holidays are here and with them all the yearly summaries and forecasts. It may be a good idea to go back a year and check how many oracles have seen this slowdown coming..

As I look ahead to 2009, I see a few clear trends for web technology areas that are providing value in these rough economic times and are maturing to the point that companies cannot ignore anymore.

Gartner published their list of strategic technology trends for 2009 a few weeks back. They highlight Cloud Computing, Web Oriented Architecture, Enterprise Mashups and Social Software and Social Networking.

Here are 5 additional web technology trends that will be important in 2009, in no particular order:

  1. Actionable Web Analytics as part of Enterprise BI and Dashboards.
    Web Analytics in many organizations is still an orphan with no real parents. Every department looks at its data but rarely does it get a strategic priority as an indicator of business trends and business intelligence asset. Investment in web analytics allows for customer insights, marketing spend ROI, conversion optimization and can impact the bottom line. As companies invest in sophisticated BI and analytical dashboards, web based data that is not transactional is usually not there. Integrating web traffic and user interest data into these systems can result in new insights and better actionable data.
  2. Phone Browser Compatibility
    Mobile computing is booming. About 13M iPhones were sold so far, and the support for location and browser that both the Android, Blackberry and all Microsoft based smartphones are offering, the percentage of traffic to web sites coming from phones is already in the 3%-10% range and will only increase. These are not the early 00’s WAP/WML jokes but full HTML browsers. Still, these special browsers are very different from the full version used on PC’s and Laptops. Bandwidth is still a challenge and their support for Rich Applications such as Flash and Silverlight is lacking. If you have a fancy Flash based site, your users will most likely not see a thing. Companies that have ignored it so far will have to adjust their sites or redirect mobile traffic to a mirror mobile optimized site.
  3. Location based services
    Continuing from the previous point, these mobile devices have GPS included and location based applications can drastically impact the user experience. Either as an iPhone/Android application or websites, the ability to share location information and get back location specific data about local services, other people, events, sales or anything else adds a new dimension to mobile applications.
  4. Increased reliance on open source infrastructure products and technologies
    Free is always a powerful word. Strong and reliable open source environments allow companies to create a robust e-commerce infrastructure with little or no proprietary platforms. The excellent Apache OFBiz for example, provides strong open source modules for e-commerce, ERP, CRM and many others. Alfresco offers a great content management solution and multiple open source development environments are available. The case for Enterprise Open Source web environment is getting stronger every day.
  5. Approaching Social Networking and Collaboration in a Strategic way
    Everyone now realizes the power of social networks and is rushing to get in, establish a FaceBook page, a Twitter account and get’s their PR to sprawl the web to “engage” people. Internally, companies are haphazardly trying various collaboration methods. We see a maturity process happening through 2009 that will force companies to look at all their collaboration points in a strategic way and tie them to business goals and processes. This new approach will transform them from toys to tools and will establish their place and value in the new order.

Web 2.0: Like Prego Spaghetti Sauce “It’s In There!”

It's in there!

It's in there!

Web 2.0 is giving me flashbacks to an old TV commercial for Prego spaghetti sauce; “Tomatoes, in there! Garlic, in there! Carrots, in there! Half of Italy, in there!…”  It seemed no matter what you asked for it was in that bottle of sauce.  Being a sauce, how could you really tell what was in there, or if it was really needed?  Plus, the tomatoes colored everything red so who knows?  Now we have another bottle of technical sauce here called Web 2.0; it’s in there!  It’s colored all Internet so how can you tell what is really in there, or if it is really needed?

Good question, seems like every vendor says they’re on the bottle of ingredients, in fact the most important one.  It would be funny if it was not so pathetic.  Unfortunately, the smell here is not a nice bubbling spaghetti sauce, closer to a warm crock of….., you get the concept.  Every vendor out there seems to believe companies will blindly buy anything labeled Web 2.0. Rather, the CIO’s are more apt to remember the Internet bubble and where that approach got them the last time.

What is required is more definition of what Web 2.0 is, and why we in IT need to move in that direction.  To get that basic understanding, we need to breakout that old spaghetti sauce pan again to boil out all the fancy analysis and obsequious technology.  Lo and behold! What remains is a simple concept: the inmates are now in control of the asylum.  Users of the Internet have turned the tables on the big players in the space, they are no longer happy being spoon fed from a portal. The denizens want to hunt it on their own terms, see it their own way, save it and dispose of it as they please.  If you stand in their way, this mob of Internet hunter-gatherers will crush you with the loss of their eyeballs (poor Yahoo, poor EBay, happy Facebook, happy iPhone).

If this basic principle is followed like a lode stone, much that is occurring in the Internet space is much more illuminating and the proper path forward (with supporting technology) is a great deal clearer to discern.  For example, the winning companies embrace openness and external developers.  There is no way their internal staff can create and the site push enough content and functionality to stay on top.  The Tao of a top site is to be one with the masses, following and attempting to push is uncool.  Allowing users to mash-up specialty widgets into cool personal discoveries is winning, monetization will ultimately follow.

By this point, you are thinking — how is all this ethereal philosophic spew helping me?  I need to get something together that can be called Web 2.0 or my IT existence is at risk!  Do not worry Grasshopper (I’m showing my ’70s again, rats!) I’ll put forward a corporate-friendly straw man.  If SharePoint is used to enable a project, process, or department; it is so Web 1.0 (boring!).  If we put the entire corporation up on SharePoint, acting like a corporate Facebook, we are getting there.  If we template it such that we now have ubiquitous collaboration; optimizing and moving our corporate intellectual property (IP) at light speed much nicer.  But for ultimate coolness, we need to commit heresy and wire a Google search appliance in, after adding all of our corporate content to the pile: documents, presentations, everything.  Then the cherry on top, flatten key data bases to HTML and toss them in.  Now, with proper organizational change management (Yes Billy! You can run with scissors, points down please), employees can use all of the power contained in Web 2.0 to maximize unstructured corporate data for speed and profit.  Mangiare! Mangiare!

Open Source Development: Top 5 Things I’ve Always Wanted to Know

I am primarily familiar with Microsoft products and languages, but am interested in learning more about open source.  To summarize, here are the top 5 things I’ve always wanted to know about open source development:

1.  Who do you talk to when you have a support issue regarding an open source product that you’re using?

2.  Are open source programs as secure as proprietary programs and are open source solutions embraced by large corporations even though they don’t have a name like Oracle, Microsoft etc. behind them for support?

3.  What are the most popular open source programs out there?

4.  What’s the best place on the web to learn about the open source solutions out there today?

5.  Are there any known drawbacks to open source versus proprietary software that people should know about before pursuing it just on a cost basis?

Granted, some of these I know the partial answers to, but as someone who doesn’t use open source very readily except for Log4Net, it’s fairly representative of the questions I’d have about open source technologies.