Java Platform, Enterprise Edition

Java EE Journal

Subscribe to Java EE Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Java EE Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


J2EE Journal Authors: Stackify Blog, Sumith Kumar Puri, Javier Paniza, Yakov Fain, Ken Fogel

Related Topics: Java EE Journal, Java Developer Magazine

J2EE Journal: Article

Benchmark Bust-Up in Javaland

Benchmark Bust-Up in Javaland

(November 8, 2002) - The J2EE community got a major call to arms this week when the J2EE experts, The Middleware Company, released a report on the performance of the J2EE PetStore application running on both .NET and J2EE.

The report seemed to suggest that Microsoft's .NET performed much better than its equivalent J2EE implementation, outperforming it by a huge margin as opposed to beating it by a whisker. A time for rejoicing at the Redmond campus?

Well not quite; some Java developers believe that .NET was given a unfair edge. So what are their arguments?

The Story So Far...

Here are the circumstances, as far as JDJ News Desk has been able to establish them. The Middleware Company (TMC), the owner-managers of TheServerSide.com, approached Microsoft to aid in the production of a "fair" benchmark suite for the implementation of the J2EE PetStore application for both .NET and J2EE. The motivation for this initiative dates back to the first time Microsoft published such a report, at which time they claimed that the PetStore ran faster on .NET than it did on a leading J2EE application server.

The major J2EE companies came out in protest to against Microsoft's "findings" at that time (Oracle, IBM, and SUN for example), saying the benchmark program neglected to utilize the overall J2EE platform.

On the face of it, the re-match was straightforward: TMC would optimize and tune the PetStore application for J2EE, and Microsoft would be given the opportunity to optimize and tune their implementation for .NET. However the scales began to, shall we say, to tilt when Microsoft invited The Middleware Company to their offices, paying for all their travel and subsidiary costs to spend time at Microsoft's main offices in Seattle.

When asked by JDJ News Desk if TMC had made a similar suggestion to them to garner help configuring and optimizing their respective application servers, Sun, IBM, and BEA Systems all said that no approach was ever made to them.

"Benchmarking of this nature," comments Eric Stahl of BEA," where an application is chosen and run by someone for performance metrics, is a highly flawed methodology."

"The very reason for TPC, SPEC and the prior ECperf organization," he continues, "is to create a fair environment for running and reporting of these performance metrics in a way that properly compares the products in question. And, even with all of the overhead of a committee and its rules, the numbers can still be controversial."

"PetStore Not Designed for Performance Analysis," Says Sun

So was Sun's PetStore application the best application to use for this test?

Sun has always been quick to point to the fact that the PetStore application was never designed for performance analysis. As Glen Martin, Sun Microsystems' Lead Product Marketing Manager for J2EE, notes, "In the PetStore, given the primary goal of teaching, lucidity wins any contest. This is entirely different from the way the decision is made in real application development. People should use the PetStore to learn techniques and the application of patterns. Perhaps we haven't been clear enough on this point."

Many Java supporters have come out against this obvious oversight on TMC's part. JDJ's very own J2EE editor, Ajit Sagar, sums it up very nicely: "Comparisons are done for a purpose. If the purpose of this report was to give a fair analysis of how J2EE performs as compared to .NET, Sun as well as Microsoft should have been involved in deciding what the guidelines for the test should have been. This seems more like a "fixed" match. Was Sun approached for time, space, and resources as Microsoft was? Did they agree that Pet Store was the right application to test for performance (obviously that has not been its purpose)? Was the J2EE application deployed in Sun's facility (as the .NET one was in Redmond)? If the answer to these questions is NO, then the report is definitely one-sided and benefits Microsoft. It should be ignored by the J2EE community and treated as another marketing gimmick."

Not everyone believes, though, that TMC did a disservice to the community. Greg Leake, Lead Product Manager for .NET at Microsoft, notes: "TMC did choose a J2EE architecture that represents Sun's recommended approach using Entity Beans."

That may be very well be, but as any architect will tell you, the "recommended" approach is not always all the best approach in the real world. The Middleware Company are reputed custodians of this area, publishing well-known best practices for J2EE applications. But here lies the mystery: as Eric Stahl at BEA comments, "...the rewritten PetStore does not follow *any* of TMC's best practices. It's a joke."

TMC chose to put not one, but two leading J2EE vendors against Microsoft's .NET implementation. The two servers were not officially named, but it was obvious from the configuration files published which two servers were eventually used.

"Seriously Let Down the Java Community"

A startling fact, that both JDJ's editor-in-chief Alan Williamson and Glen Martin from Sun find hard to believe, is that "AppServer #B" failed to respond to any further requests after just 4 hours. Alan Williamson comments, "For this very reason alone, TMC had a duty to call in that particular J2EE vendor to make sure they exhausted all configuration options. This was a negligent error on a massive scale. TMC have seriously let down the Java community." Such issues wouldn't have affected the .NET implementation, as Microsoft had their own highly skilled .NET engineers poring over every single line and configuration option to ensure the best possible performance was squeezed out of it.

The question remains: why? Why would a highly respected J2EE authority be so careless? Rickard Öberg believes there is more politics to this debacle, "TMC really disqualified itself as they were recently bought by a company who has Microsoft as a strategic partner," he says - a reference to Precise Software Solutions, Inc.

"Whoever writes such a report needs to make sure that the results of it aren't released to any of the related parties before official publication," Öberg continues. He has published evidence of Microsoft having had access to the report before publication.

Leaving the potential skullduggery issues aside for the minute, from a technical level did TMC do the best possible job for the J2EE community? Rickard Öberg believes not, and has produced a rather detailed technical analysis on the PetStore implementation that the TMC used. It would appear TMC didn't even use the latest PetStore application. As Öberg points out: "The 'original PetStore' they used was 1.1.2, which is over two years old. The most recent PetStore (1.3.1) already have many of the optimizations I outline in the report."

Nigel Thomas of SpiritSoft agrees. "The comparison deals with just one possible (EJB -centric) application architecture; taking a straightforward tightly coupled/synchronous approach to application development. That's pretty much an 'entry level' approach," he continues, "that true application scalability comes from building loosely coupled applications, with non-urgent processing pushed into the background using (JMS) queues. That gives virtually unlimited scalability."

But can a benchmark be developed that would allow us to compare "apples with apples" as opposed to oranges? Many believe it can, if all parties can be brought together, and as Glen Martin says, "...a benchmark needs to be developed."

TMC has listened to the community and as a result will be re-running the tests, this time giving the J2EE application vendors the same opportunity afforded to the Microsoft team. Says Greg Leake from Microsoft, "We welcome this set of tests, and will eagerly participate with a .NET implementation of the benchmark application to showcase our technologies."

Looks like this passionate issue will go to Round #3, but this time, all parties will be participating, so the results should be far more meaningful.

JDJ News Desk looks look forward to that report.

External Resources:

The Middleware Company Report

Rickard Öberg Analysis

More Stories By Java News Desk

JDJ News Desk monitors the world of Java to present IT professionals with updates on technology advances, business trends, new products and standards in the Java and i-technology space.

Comments (51) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
David McElhaney 04/28/03 08:15:00 AM EDT

I'm not interested in the Sun/MS evil empire duality. "The right tool for the job" said it all. I've worked with Java for a while and it used to save my but when complexity ran up against the cross-browser compatibility constraint. I'm working with .NET now, and I find it to be very nice too. But I can't reconcile this vehement partisanship I am reading here with my own primary loyalty--to my clients.

Bob 01/18/03 04:41:00 PM EST

Those who lose always have an excuse

Mark Nuttall 11/20/02 05:22:00 PM EST

Why do java if one is developing on Windows? Because one might not be deploying there or doesn't want to be limited to just Windows. Look at what Fed Ex just did. BTW if performance is an issue, you can still use Java. If the person has no clue how to build good software and properly deal with performance, may the solution is to let someone else do it. They are going to have problems no matter what the software.

JMA 11/18/02 06:24:00 PM EST

A language that tries to be everything to all, bloats itself with APIs, and is controlled by a conglomerate of corporations with no sound strategic direction is doomed.

I like Java. Always have. But it's days are numbered.

If you are developing on Windows and performance is an issue (perhaps even time-to-market), why the heck would you even *consider* Java?

Rick 11/18/02 05:40:00 AM EST

This very JDJ article smacks of Microsoft FUD (fear, uncertanty, doubt). I liked the previous post pointing out that benchmarks are irrelevant, marketing hype. Focusing on customer needs and delivering solutions is what is really important. Each system has it's strengths and weaknesses, so choosing the right one for the job is the key. Pounding a square peg into a round hole takes more time, looks silly, and frustrates everyone involved. I once heard a wise co-worker once say that systems are like golf clubs, just take the time to pull the right club from the bag. It's extremely rare that any company can create the super widget that does everything for everyone. One size fits none.

Ted Nanayakkara 11/17/02 12:26:00 AM EST

If you need to compare J2EE and .Net, you have to rewrite petstore using only Stateless EJBs. Microsoft only has stateless components. Stateless components are the fastest. Write a stateless version in J2EE and send it to Redmond. This is all they understand.

Those J2EE developers out there, follow the J2EE blueprints and use all types of beans available in J2EE. Bill Gates will have somthing comparable to Entity Beans in about 5 years. Then we can have a one to one comparision.

In the meantime, we can laugh at the Windows developers who lack these sophisticated services we have in J2EE. Can you imagine coding a real world application without stateful, entity, MDB, etc?

Reasonable 11/16/02 04:20:00 AM EST

Java is slow. That's only a big shocking suprise to retards. Get over it.

"I have found that performance is not the #1 priority"

Yeah, because otherwise, Java would not be their first choice. Java has many great benefits. Why can't you admit it's shortcommings? If we can all agree that Java wasn't designed for performance, then why are you denying that it's been outperformed? Get a grip.

Roland Van Liew 11/12/02 01:34:00 PM EST

If this airhead Ed Roman is considered a heavy hitter in JavaLand, then Java is in serious, serious Trouble (yes, with a capital

Only Rational Guy in a Group of Zealots 11/12/02 01:09:00 PM EST

Whine, whine, whine. Losing to .NET is all this magazine ever talks about. I'm sick of hearing it, folks. Why don't you just admit .NET is better and close the doors on Java? You could rename your magazine to the .NET Journal or whatever.

Might as well be called that now.

Brian Kiser 11/12/02 10:36:00 AM EST

The light of reason shines! :) Some people get too attached to Java just because it's a good tool and is not the evil and nasty Microsoft Corporation.

Java is actually owned by the evil and nasty Sun Corporation. But at least it's not Larry Ellison. :)

Rich Katz 11/12/02 04:19:00 AM EST

I've read a number of accounts of the benchmark fiasco and there's something missing for me. Why don't people simply suggest what makes a benchmark fair to begin with? Or is everyone afraid to?

First, both J2EE and .Net should use THE SAME INSTANCE of THE SAME DATABASE and call the same views or stored procedures on it. There is no reason this can't be done somehow.

A) Out of the myriad of JDBC drivers for MS SQL Server surely AT LEAST ONE OF THEM will work well enough to be run on a J2EE system - especially since one (DataDirect) is recommended by Microsoft and another (Kona) is manufactured by the same people who make J2EE "Server A." There are FIVE DRIVERS for MS SQLlisted on JavaSkyline forpeetsake.

B) And if not MS SQL, run Oracle on both systems since Oracle makes an ADO provider. Either way, you have to get them to use the same instance of a database - so you aren't comparing dollars and doughnuts and you begin to eliminate network configuration problems.

It's fine to say that the benchmark isn't "database" limited. But that's no excuse and when the difference is less than an order of magnitude, the DB can make a big difference.

Second, EJB entity beans have got to go. I know it takes a long time to reprogram, but it just adds too many differences when A) there are no managed objects in the .Net version and second, and B) even if there are, we have no idea of the comparitive functionality of EBs vs. managed objects. Get rid of them - both of them. Just use stateless sessions.

Third, when do "Server C" and "Server D" and "Server E" get a chance? By that I mean C) Oracle 9i AS and D) Mort Bay with some unnamed open source EJB server - where you get to use Mort Bay's slick JDK 1.4 NIO enabled servlet engine - and E) Caucho Resin and F) Macromedia JRun Server G) Novell and H) SunONE.

You can't really say and believe that J2EE is slower than .Net unless you at least let the Hot Rods onto the track.

Fourth, let the App Server manufacturer say whether they run best on some version of Unix or some version of Windows. "Server C" might like Linux while "Server B" likes AS/400... If you pay the same amount for the Hardware, why does it matter which OS is used?

Just some thoughts.

Folks, start your engines!

Regards and the best to all

Rich Katz

David Lanouette 11/11/02 09:56:00 PM EST

I think most readers agree that the Java solution could have been made faster.

But, the test does demonstrate that the .NET solution is fast/scalable enough for 98% of all applications out there (on a single 8cpu machine at 1/2 the cost).

Also, I was impressed with the almost linear salability. Intel is FINALLY starting to make some fast processors, so the suitability for a given high end task will only get better.

My only serious question would be about reliability. MS has historically come up short in this measure.

Tim Nance 11/11/02 09:19:00 PM EST

Here is the point if you run the test in Linux Java beats .Not by 100 to 1. Oh wait .Not keeps on failing in Linux from my tests so what does this say for .Not. All .Not is just a M$ sham and it has back fired badly. Message to Microsoft you can bribe the justice department, TMC and middleware, Alan but the consumer knows the truths we had enough of VB crap and windows crashing.

Philippe Leothaud 11/10/02 08:58:00 PM EST

Mr Evident, if only you had taken time to read the benchmark report, you'll know that Microsoft .Net PetShop has run only twice as fast as this very very badly architected version of Java Petstore.

Furthermore, this pityful version designed (?) by TMC used very old components (EJB1.1 were released in 2000), furthermore EJBs are distributed objects, with all that this implies in terms of lack of performances, while you won't even find an object using .Net Remoting in Micrsoft PetShop!!!

All in all, I'me quite sure that some well designed Java version of petstore, like JPetstore 2.0, which will be released soon, should perform a couple of times better than PetShop on Windows, and infinitely better on any other operating system, like Unix or Linux or *Bsd, or IBMs Mainframes, which represents something like 80% of companies servers (including Microsoft, as I was told recently, which amused me a lot...)

Just one last point to end this post. Microsoft has never been a lion, but they sure are monkeys since the very beginning : they invented very few (I say thanks for SOAP), bought some stuff(remember DOS : as I told, since the beginning...) and copied much, like monkeys (OS with windows, language running in a VM, strongly typed, using a Garbage Collector --Java ? no, C#)

Monkeys, it's Evident

J 11/10/02 07:03:00 PM EST

It's funny how I can bring up this article from Outlook. It's even funnier that I can go to the sys-con.com website from the URL location box. I wonder if Microsoft purposely put Java-related filters in IE and Outlook??? I'm totally frustrated, and planning to switch over to Mozilla!

Charlie 11/10/02 04:11:00 PM EST

This whole spectacle is really embarrassing reading. The only true value that has come out of this odyssey is really Rickards Analysis. The part with reflections regarding CMP, Cashing ... that is. A pitty really, handled properly a benchmark initiative could really had made an honest footprint (at least from an "archtectural comparison" point of view).

Dion Muddle 11/10/02 02:27:00 PM EST

Question: did TMC approachthe J2EE vendors for participation in the production of the report?

November 8 TMC finally answers: "We did not approach the J2EE vendors."

Why wasnt this information in the report?
Why wasnt this information in the FAQ?
http://www.sys-con.com/java/article.cfm?id=1739

Nick Riordan 11/10/02 04:51:00 AM EST

I develop high transaction volume applications for a large investment bank with one of the J2EE products benchmarked in the test - and can testify to the (truely awful) quality of these tools. They are very expensive, the performance is average, and the support is average to poor. How these products were ever shipped I do not know - the number of bugs in the most basic of functionality is just astounding. I wish people would not keep on bashing Microsoft as "the evil empire" - they have produced a great set of tools and technologies. They are also stable, well designed and properly tested and integrated - something that is not true of J2EE and Java.

I am totally unsurprised by the benchmark results - any application that tries to do OR mapping with entity beans will have a lower performance than one that is using straight optimized SQL. To make the Petstore perform, re-code it to use the same design patterns as Microsoft - junk the entities, use stateless sessions wrapping stored procedures. BUT - it is not good enough to say that "Petstore was never designed to be a high performance app" - if this is the reference example, then this is what developers will copy - this test is therefore a fair comparison.

Having said that I am less interested in the absolute performance figures than the following facts that were highlighted in the benchmarks - number of lines in the implementation (2000 vs 14000), problems in tuning the J2EE application servers, cost of the infrastructure, resiliancy, stability and consistency of performance. These are the important issues in the real world - absolute performance is rarely the deciding factor. In the review this is where J2EE really falls flat.

No review is perfect - but this one was not as biased as many people are making out. I really want to like J2EE, but having written applications for it for the last two years I am totally fed up with it - I would rather use .NET - it's more likely to let me get on with the job of delivering solutions to my business.

Robert McIntosh 11/09/02 09:05:00 PM EST

TMC used to be regarded as a leading educator for J2EE and now it seems they have done a major disservice to the Java community. As a previous instructor for the company I can say that at the time (before they were bought/merged/whatever with someone else) that all of the instructors were top notch. Now I am almost shamed to say I worked with them.

Perhaps times got tough and they sold out, who knows.

Mark Grant 11/09/02 05:37:00 AM EST

Sigh,

Once again we enter the 'evil MS' versus the 'good Java' wars (of which
MS are partly to blame here). The (original) plan here was to provide and
compare the 2 architectures using a reference e-commerce application.
The J2EE version concentrates on what it
is good at - OO and design patterns; the .net one on what it's framework wants to provide - concentrating on streamlining (maybe at the expense of
design pattern re-use - maybe not an issue?).
Unfortunately (?) MS then started to push the performance issue on what was not intended as a 'performant architecture' and so we now enter the 'us versus them' wars again.
Let's get back to basics - use the Java version as a reference when designing J2EE applications, and the .net version when designing .net applications.

Mark

Tina 11/08/02 08:22:00 PM EST

Microsoft is a evil empire. They know that they cannot beat java even though they poorly copied c# but failed miserably. Now they are bribing companies like serverside and TMC to show Java runs slow. I think once for all we should ban Microsoft from our work and homewe should be using Linux and opens source tools and java. This will be the only way to kill this evil company.

Michael 11/08/02 07:43:00 PM EST

This benchmarking may be dubious, but let's be accurate about the pros and cons of .NET as well as Java. To any external web service client, there is no difference between a web service from a .NET server and one from a J2EE server. I don't know what Vince Marco read that claimed that but is plain wrong. I work on a system that uses both .NET servers and Weblogic servers, and they communicate perfectly through web services. I've spent the last three years developing J2EE apps running on Weblogic and Oracle, and the past year writing C# code on .NET servers, and Java's advantages are not in performance. If Sun built a JVM right into Solaris, then , maybe it could be outperform .NET. Java gives you a price option against Microsoft. If you want to go cheaper than .NET, go Java on Linux and you won't lose a whole lot in performance. If you can afford to go with Java on Solaris, then you can blow away .NET. So if you aren't tied into the world of Microsoft already, there's no way you're going to go with .NET and if you are, then you'll only go with Java if you outgrow x86 processors.

Dennis Miller 11/08/02 07:36:00 PM EST

I would question MS's credibility and business ethics based on their past history.

Vince Marco 11/08/02 05:59:00 PM EST

I don't understand why everyone is surprised. Do you not understand that Microsoft has lots of resources and that their ultimate objective is to own every computing market?

Having looked at their PetShop code, I can't believe any company would build enterprise software using .NET. While it has some very nice tools for rapid development....those tools very quickly lock the solution into the .NET platform, as well as sacrifice maintainability. As an example I read recently that the .NET web services are not entirely functional outside of .NET as their is some dependency on C# in the SOAP messages. That definately breaks the intent and much of the motivation for even building web services.

Building J2EE solutions can be nearly as rapid, and an order of magnitude less expensive and more rubust, both in the short-term and the long-run. It's just that any organization doing J2EE needs to focus on which frameworks and designs to use and create a productive environment suited to their objectives....since there is an entire market of tools and frameworks to choose from.

But we live in a world in which people have a need to be marketed. I wonder how long it will take before the Microsoft hype fades away leaving many enterprise projects wanting more than the one solution they chose.

Mark 11/08/02 05:51:00 PM EST

I think Rick said it best, but to paraphrase Larry Ellison (of Oracle)...."Java's enemy is not that company in Redmond, but performance" Get over and make it better.

Dion Ashamed 11/08/02 05:45:00 PM EST

Some have noted that older J2EE App Servers were used against Beta .Net.
You can not even download the .NET runtime they used:
http://www.oreillynet.com/pub/wlg/2240
(Dion Almaer a Middleware Company employee)

Using BMP instead of CMP2/Local Interfaces resulted in GC thrashing
and terible performance.The TMC should be
ashamed of themselves:
http://dreambean.com/petstore.html#optimizations

Others have asked if Microsoft wrote a specialized version of the runtime:

http://www.oreillynet.com/cs/user/view/cs_msg/11132

Dan 11/08/02 05:41:00 PM EST

In Strike #2 for the Java World
Posted by Rick Gipson on Nov 8
" You guys are as bad as Al Gore...demanding recount after recount...thinking the result will eventually be in your favor.
Face it Microsoft has spent billions on .NET and is now light years ahead. "

The truth was AL Gore acturally won and G. W Bush lost.

Personally I don't care who win, but I want to know the truth.
GW Bush spend more money then AL Gore and who go the Chair, but he was loser.

Jeff C. 11/08/02 05:41:00 PM EST

For years I have been strictly using ColdFusion. I have found it to be the fastest tool to deploy a website (my opinion of course) and "in some cases" run faster and more efficient that other servers.

Note I said "In some cases". As 'iz' stated above use "the right tool for the right job."

Recently I had to code a site in ASP and later .NET. I found them to be great products. I decided to dabble a little in PHP and PERL. Although I found PERL to be a little more difficult (learning curve) I felt that .NET and PHP were both great products to use. And in some cases certain sites would run faster using different coding techniques provided by different features in the products.

I have decided to stick with CF as my primary coding for websites, but thats just because I am most familiar with the product. Just recently I started learning Java (basics). I figure "If that many people claim its a great product, why not learn that as well."

Each product has its "UPs and DOWNs." So Java got a bad 'rap' do to a poorly run benchmark. Micro$oft is famous for things like this. It just helps us to realize who is the better man here (I couldn't think of a good expression to use :).

Dan Albarran 11/08/02 05:05:00 PM EST

Java will always be slower then .NET on Intel but performance is only one consideration in designing an application. The true strength of Java is vendor choice, code manageability, platform independence, and design possibilities. I

Rick Gipson 11/08/02 04:30:00 PM EST

I find it hysterical as your own article freely admits that TMC is a leader in J2EE development and writes all the industry best practices, yet they couldn't properly develop the PetStore for the test. You go on to say that the PetStore is not a valid application for such a test, so are you saying the Java cannot perform adequately in ecommerce market? You guys are as bad as Al Gore...demanding recount after recount...thinking the result will eventually be in your favor. Face it Microsoft has spent billions on .NET and is now light years ahead.

Gennady Kostinsky 11/08/02 04:17:00 PM EST

Java may win in so called "code portability", but even with fair amount of tweaking it is still going to be slower than .NET, which is more native to Windows.
Java's strengths are in the the other aspects than performance. Can you deploy .NET on Linux?
Sun opened the stupid debate by exholting the virtues of PetShop app and now again is paying for its stupidity.
Sun's leadership in Java field drags language down. With no decent IDE or app server Sun cannot even make money on language they developed.
Face it, in performance .NET will beat Java on Windows platform. Question is, will it beat it on Linux in the future?

Jon Weathers 11/08/02 03:33:00 PM EST

Yet more politics, crying, etc... sheez. Go ahead and use J2EE. I don't care. I'll be out on the golf course.

Jack 11/08/02 03:01:00 PM EST

It's all about credibility. Speaking of which - I'd believe this JDJ article a lot more if u lurned how 2 tipe beter!!

Ryan Smith 11/08/02 02:55:00 PM EST

The Petstore, as well as all the J2EE Blueprints, are always going to render poor performance. The truth is that the design patterns professed and used are marketecture (marketing + architecture). Any J2EE application that uses entity beans is going to be a dog.

Steve Hatfield 11/08/02 02:45:00 PM EST

Greg- You made a very valid point about cost. However, benchmarking has little if anything to do with how much it costs to develop an application. Now, the object oriented paradigm, and the best implementation of that paradigm to date, Java, does affect developments costs, especially if one considers total cost of ownership. The vast majority of money spent on software application development is spent on maintenance and enhancements after the application is in production. This is where OO really shines. But again, I feel this has nothing to do with performance benchmarking.

todd 11/08/02 02:28:00 PM EST

Ok, the subject line of The Middlware Company's post pretty much says it all! ;)

Though, I think it's spelled with an "e" and not an "a" - making it Impotent Announcement

Too funny!

Iz 11/08/02 02:22:00 PM EST

You all take this as a personal attack. I find that amusing.

Remember this, if you are really in the IT field:

"The right tool for the right job."

If you are so hard up on Java, .NET, or ANY platform that you exclude all others from the list of options for a project then you aren't really a professional.

Attacking each others platform does nothing for either community. I suggest you try them both out and use them where they fit and stop wasting time with supposedly 3rd party benchmarks.

Thats just my two cents....

Greg Garvey 11/08/02 02:15:00 PM EST

'What users/customers care about is does the application they are using meet their needs and does it do so at a performance level they will accept'

This statement also has to be qualified with 'a cost level they will accept'.

Therein lies the rub, one of the goals for any dev platform is to minimize development costs and maximize the output devlivered. Those costs are justified by benchmarking against the competition, so while THIS benchmark may be meaningless, they can be meaning full.

I believe the benchmark's intent was less about defining a winner and more about a marketing effort designed to quell the belief that M$ does not scale as well as J2EE. The intended effect being to alter perception... There are various proc/cons for .NET vs J2EE, now scalability has to be taken off the table as a differentior.

Tim Nance 11/30/02 02:28:00 PM EST

http://www.vnunet.com/News/1137229

The core Windows-based banking application has been rewritten in Java to make it available on any system, including over the web. One iSeries 820 server now runs the entire European operation from London, substantially reducing administration and licence costs.

"Linux runs Java much quicker than Windows. It's the natural operating system to run Java," said Evans, who added that other applications are now gradually being ported to Linux.

The Middleware Company 11/08/02 01:51:00 PM EST

The Middleware Company does not believe that it was fairly represented in this article. We encourage JDJ readers to look towards TheServerSide.com announcement that we are about to make discussing this benchmark as well as a round 2 benchmark, and make their own determinations, rather than taking the opinion of the editor of this article, as well as JDJ's advertising partners, as fact.

whoa 11/08/02 01:46:00 PM EST

you've lost all credibility with your incredibly poor communication and equally weak spelling. learn the english language before you learn the java language

Johnny Bravo 11/08/02 01:44:00 PM EST

To me, this isn't about what is better. I'd say let the better technology win. But to create a biased benchmark and claim that it "proves" that .NET is superior is sad. As a performance engineer, I will be the first to admit that benchmarking is a glorified form of advertising. But it is meant to be fair and somewhat meaningful. No, it does not tell you precisely how well the technology will perform in real life, but it does give you somewhere to start so that you can make a better informed choice. .NET beating J2EE in one benchmark shows that it can outperform in one particular set-up. Now MS and TMC should ask themselves if the bad publicity and a scar in their reputation was worth that ONE test.

MS should grow up and learn to play by the rules. If they really do have a better technology, then prove it in a fair way.

Dave Clegg 11/08/02 01:34:00 PM EST

The ECPerf benchmark is the standard one for measuring performance of J2EE application servers.
That specification would make much more sense as the basis for a J2EE/.NET competitive analysis.

Mr Evident 11/08/02 01:24:00 PM EST

Suppose you run the test again and do all the fixes to the application as per the Monkey (Java fans) require.
It may be that the final result is that the NET version runs 100 times faster, instead of 200 times.
This is like the story of the monkey fighting the lion, and the monkey has both hands tied.

n n 11/08/02 01:16:00 PM EST

TheServerSide.com should NOT be any part of the "New" benchmark. Their actions speak for themselves.

Steve Hatfield 11/08/02 12:55:00 PM EST

Benchmarks are meaningless, IMHO. I do not care one iota what one app dev platform does compared to another in any contrived test. What users/customers care about is does the application they are using meet their needs and does it do so at a performance level they will accept. Another question that often gets overlooked is what type of application is required to solve the problem. Not all problems require full-blown J2EE or .NET solutions. There are hundreds of applications out there far less complicated (say servlets and JSPs) than full-blown J2EE applications that are doing the job quite well. People who worry about benchmarks should spend that time worrying about meeting their user/customers needs. In the end, that is what will make or break an application, not how some contrived application performed in a contrived test.

Clinton Begin 11/08/02 12:41:00 PM EST

JPetStore 2.0 will be out soon and will expose the reality behind this marketing circus.

In the last round we destroyed Microsoft's .Net Pet Shop 1.5.2 in:

* Architecture
* Design
* Cost
* Productivity
* Lines of code (NEW! read the site)

How hard can it be to beat them in performance? We can do it!

Join the rebellion at http://www.ibatis.com

Colin Saxton 11/08/02 11:38:00 AM EST

I went over the test spec myself and found a number of key points that where not outlined or explained in the spec (See below...)

I am always scepticle about tests that are performed by one company?

I think it would be better to have an intermediary company who then invites individuals to come in with their Application servers set them up and then run them.

This report is extremely biased...Even though they have tried to flatten it with plenty of figures it falls over in key areas...

I am not surprised that .Net performed well on Windows but can you run .Net on a ZSeries? Java on ZSeries machines would easily out perform any of the .Net implementations in the world, at the time of writing this.

There are a number of key areas that where not explained correctly so the report is largely useless...

1.

JDBC drivers can play a leading part on the performance of load tests. I have tested numerous JDBC drivers and find the Oracle drivers to be quite poor.

2.

Why was not the same database used through out the tests. Its obvious that .Net is going to perform well with MS SQL Server since it is basically tied in with it (It is a .Net component). Try changing .Net to run with Oracle...what different would that make?

3.

Windows was chosen over Linux because "it was found that J2EE application servers performed better on Windows". What was there setup criteria for this conclusion?

a. Did they run Linux as standalone fronted by IIS on a windows machine or did they setup clustered Apache Web servers on Linux with clustered Linux Applicaton servers behind??

b. Did they slim down the Linux kernel so that it was only setup to run the application server services.

They don't state their results on why J2EE ran quicker on Windows?

4.

Partly linked with point (3). Application server clustering...What happened to this? It does mention clustering (in two places) but they do not mention how many application servers existing in the cluster, if a cluster existed at all??!!

5.

Cost of ownership?? Tut, tut middleware ""you know what you did""...JBoss is free which can run on Linux which is free which can also connect to Postresql/MySQL database servers, which are free and also can run on Linux. Total software cost = ZERO

My conclusion here is that if we had a proper test environment setup inviting each company to come in and tune their own application servers and also choose what operating system and database is used (as long as they keep to the same hardware spec!) Then I can guarantee that we will see more of a less bias result...what I would expect is with a single high spec intel architecture machine that .Net would probably just pip java but in a clustered environment using linux on the same spec machines linked in with apache up front then I think we would see a different picture...

My last point is that remember that you can run Java on Sun servers and ZSeries IBM machines...These will always kill off .Net, until MS allow the .Net framework and windows for that matter to run on different architectures...

Another interesting point to note is that the ammount of desperation that M$ attached to disproving J2EE technology...could this mean that .NET is just not selling as good as they thought?

Fred Grott 11/08/02 11:11:00 AM EST

With the move on Sun's part to include AspectJ as a plugin in SunOne and recommedning to use in getting better performance rather than entity beans

adn teh total rewrite that a specifc site did to PetStore for perofrmance and even then .NET did not beat out J2EE..

We all know this was nothing more than PR FUD to satisfy TMc owner's partner...

If MS wants this bad to look stupid I recommend that BIll Gates should buy TMC because right now TMC has lost all creditability with developers..

Aleksandar Susnjar 11/08/02 11:06:00 AM EST

Java and .NET were originally made for entirely different purposes. Yes, you will tell me that they do the same thing... now. Well... that is where the problem is.

Let me say that I am one of many people who, in fact, do not like Microsoft for their business policies and the way they develop code. However, from this message you mat (wrongly) get a different idea.

Java was originally designed and optimized for efficient platform-independent client-side usage in an interpretive fashion. It had to be easy to transfer and easy to run. Then we all started using it on the server... in more or less unaltered form (we've got more libraries and JIT compilers trying to make best of the code made for interpreters).

.NET technology started as the exact opposite of this. The optimizing compilation is assumed, not interpreting, and it is intended for server-side execution, not client.

Yes, currently .NET runs only on M$ platforms. But, even if it showed to be slower on these tests, purely because of its design it has got more potential for speed than Java (notice that I mentioned 'speed', not performance in general). Them think what company has most experience making highly optimizing compilers for today's most widely used hardware platform (do not think about the big guys as there aren't as many of them). Like it or not - it is M$. And ... they bet all their Visual Studio .NET languages on this.

Java was never and never will be a choice for someone who needs performance. Heck, Java still doesn't do real time HDTV video processing... and you can do it in C and assembly.

Java is not only a language or a platform. It is both (Java is not multiplatform, it only has efficient virtual platform simulators on many hardware platforms). If that is what you need, then the choice is clear - Java.

If you are targeting companies without IT departments, then why bother them with maintaining Java platform in addition to that Java platform's host platform (unless they already do and need this). In most cases such companies use M$ server and have .NET available. In this case the choice is clearly not Java.

In between there is a gray area ... and the choice depends on other factors (possibly) known only to those who need to make a decision.

Regards,

Aleks