This past week, I have been struggling with a strange problem on a client’s ASP.NET 2.0 website. The server, a 64-bit Windows Server 2003 with 4 Gb RAM, used up almost all available memory within a few seconds of starting the w3wp.exe (the IIS 6.0 worker process). During this excessive consumption, the CPU usage was 100%, and then dropped to 1-2% during normal operations. Then every half hour or so, another wave of 100% CPU cycles and a jump in memory consumption, then after a few minutes more CPU cycles and a steep fall in memory consumption. This would go on, wave after wave, hour after hour.

The problem seemed unrelated to the number of concurrent user sessions (approx. 500 at peak), and most of the time the wildly excessive memory consumption didn’t even seem to cause any problems. Yet some times - a couple of times a week - the server response times would drop dramatically or even grind to a halt, yet no OutOfMemory exceptions were ever thrown.

I was very determined to fix the issue, because our client expects the website, which is currently in its infancy, to scale to almost a million registered users within the next couple of years.

As a short term fix, the server was scaled up to 16 Gb RAM, based on the old “when in trouble, double” advice that everyone always gives without solving the underlying problem.  Well, surprise surprise, the problem scaled up as well - now the w3wp.exe could consume 7-8 Gb RAM almost right after IIS was started. In waves, just like before.

Using perfmon, I was able to determine that the 7-8 Gb of RAM was mostly to be found in the regular heap in generations 1 or 2. In other words, it seemed that a lot of small objects were being created (since the memory was not used by the large object heap). These objects were being used for a little while, then moved through the garbage collection system. That explained why there were no OutOfMemory exceptions - the memory was actually ready for garbage collection, there were just too many objects to handle it efficiently, so things got slowed down.

Next on the list was to get a fresh copy of one of the world’s most complicated debugging tools, WinDbg. Along with “sos.dll” (found in your .NET framework folder), it can be used to debug .NET memory usage, among other things. Using ADPlus, I created a memory dump of the w3wp.exe process (all 7 Gb of it), and proceeded to attempt to decipher what it all meant.

I should say right now that if you came here looking for advise on how to use WinDbg, you’ve come to the wrong place. The right place is http://blogs.msdn.com/tess/ There’s really no point going anywhere else - Tess is who you want. Either her or call Microsoft Premier Support, which is what I ended up doing.

Back to the matter at hand. Using WinDbg and with the help of a very nice guy from MPS, I was able to determine that my memory dump showed an abnormal amount of one specific type of object being created - SD.LLBLGen.Pro.ORMSupportClasses.EntityField. More than 7 million of these objects were lying around in memory.

For those who don’t know, LLBLGenPro (www.llblgen.com) is one of the world’s most productive Object-Relations Mapping tools for .NET development. I love it. It has literally saved me hundreds of hours of work on previous projects. It allows you to write very elegant and strongly typed code against any database engine (DB2, Oracle, MySql, MSSQL, etc), and supports lazy loading, which was used on this particular project.

This time, however, it seemed that I got bitten in the ass by LLBLGenPro. Somewhere, something was really malfunctioning, but based on previous experience I was pretty sure that LLBLGenPro was not directly to blame. Rather, the problem was probably caused by bad website code.

What to do? Well, WinDbg allows you to look at any object in the heap you like, using obscure commands such as “!dumpheap -stat” and “!dumpobject <address>”. However, I had 7+ million of these objects, and in all likelyhood some of those were bound to be fine/expected, while others were not. So looking at each individual object was a no-go.

Enter my new best friend, CLR Profiler. It’s a world class tool for debugging .NET applications that almost nobody uses, except those who are into .NET games progamming with XNA (like me). But I really think that every .NET developer should grab a copy of CLR Profiler here:

http://www.microsoft.com/downloads/details.aspx?FamilyId=A362781C-3870-43BE-8926-862B40AA0CD0&displaylang=en

Don’t be scared by the interface or the docs, which are a little hard to comprehend. Just fire up CLRProfiler.exe and hook it up to the .NET application you want to profile. In my case, I hooked it up to the problematic ASP.NET website on a dev machine.

After starting up the tool, I hit the front page of the website a few times and then took a look at the CLR Profiler’s summary page. It looked like this:

The first thing to notice is the amount of garbage collections that had taken place. I had only hit the website a few times, yet generation 0 had already been collected 156 times, and generation 1 more than 50. Far too much, considering that the machine I tested this on had 1 Gb of RAM and the website is fairly simple.

Remember that garbage collection only takes place when it is needed. It’s not like it is scheduled to run every 2 seconds. When writing XNA games, you strive to do all your allocations at the start of the game, then never allocate another object again while the game is being played. That way, you don’t trigger any garbage collection, which causes “hiccups” because the CPU works hard during GC.

When looking at that summary, you may feel like you are miles away from a solution. Yet, the solution is only about 5 clicks away..!

Clicking on Timeline presented me with a simple (ok, insanely complex) looking overview of the garbage collection that had taken place. It looked like this:

The trick here is to select as much of the timeline as you’d like to investigate. The list on the right side of the window will then be sorted according to which type has been allocated the most. In my case, it showed SD.LLBLGen.Pro.ORMSupportClasses.EntityField, just like what I found with WinDbg.

Now here’s why CLR Profiler is my new best friend. You can simply right-click a type and select “Show Who Allocated”. After a few seconds of work, you get the following display:

How cool is that? It’s a walkthrough of the .NET application, which shows clearly how the data got to be allocated. Scrolling a bit, I found my culprit:

One simple little method, AccountManager.GetAvailableCredits() was causing all this fuss. Nothing left to do except to jump into the code, find the mentioned method and see what it did.

It turned out that to calculate the sum of available credits, LLBLGenPro’s fancy lazy loading technique was being used to read all account lines for the user, inflate each one and sum up the amounts to get a total. It looks really neat in C# code, but is a huge performance killer. LLBLGenPro offers many other ways to solve this problem, or a stored procedure can be used. Just never use lazy loading for aggregating data.

Did any of this make sense? If not, I apologize. The main lesson here is to grab a copy of CLR Profiler, read the docs, view the webcast and start optimizing your code!

Ok, a bit of a personal post this time.. Whistling “The Grand March” from Aïda is clearly not enough for our extremely talented daughter. Today, at the age of almost 13 months old, she took her first real steps:

Cute, right?

Chris Carper, a member of the Silverlight business development team at Microsoft, has started a blog about the business impact that Silverlight is expected to have on the industry. It’s brand new, so no real gems there yet, but we’re all looking forward to hear what the Silverlight team has to say about the business side of Rich Internet Applications..

Check it out here: http://silverlightbiz.blogspot.com/

I really hate Javascript. It’s so inconsistent, loosely typed and annoying to debug. But there are some things that you just can’t do without Javascript. For instance, allowing an HTML page to communicate back and forth with an embedded Flex application.

The Flex documentation gives a very good example of how to call an AS3 method from Javascript. It’s simple, it’s easy to understand and it doesn’t work properly in a real life scenario. But let’s start with the AS3 code example from the documentation:

function callMe(name:String):String
{
    return "busy signal";
}
ExternalInterface.addCallback("myFunction", callMe);

And here’s the corresponding Javascript and object/embed tags for use in an HTML page:

<script language="JavaScript">
    // result gets the value "busy signal"
    var result = flashObject.myFunction("my name");
</script>
...
<object id="flashObject"...>
    ...
    <embed name="flashObject".../>
</object>

All well and good. And it works, too. That is, except when you are dynamically adding object/embed tags with Javascript’s document.write(). Which is what you do, if you don’t want that annoying security frame around your Flex application in the browser. If you use document.write(), the above method only works in Internet Explorer. Not FireFox. Whaaa? I could have sort-of understood it, if it were the other way around.

Anyways. The solution, which I was unable to find in the Flex docs, is the call the flashObject’s method using the following Javascript syntax:

    var result = document["flashObject"].myFunction("my name");

Sheez. Only took me an hour to figure out. Hope someone else finds this post if they are in a similar situation.

I really hate Javascript.

Here’s a little trick in C#.

If you ever need to distribute and use resources such as text or xml files in a class library, you can use embedded resources. It’s very simple: You add a .txt file to your class library, right click it, choose Properties and set Build Action to “Embedded Resource”. Then when you need to read the file during runtime, you use Reflection:

Assembly assembly = Assembly.GetExecutingAssembly();
Stream stream = assembly.GetManifestResourceStream(“MyClassLib.MyResourceFolder.MyTextFile.txt”);
StreamReader streamReader = new StreamReader(stream);
String myText = streamReader.ReadToEnd();

That’s all. How clever is that?

Obviously, you would use XmlTextReader instead, if the resource is an Xml file, etc.

If you don’t want to use reflection every time you need that chunk of text, just store it in a variable after the first read. Or you can cache it with HttpRuntime.Cache or HttpContext.Current.Cache, depending on whether you are working on a Windows/console or an ASP.NET application.

Edit: Fixed a small bug in the above code

Unfortunately I’m knee-deep in diapers and other parental leave oriented tasks, so I haven’t been able to play around with it much, but you can read more here:

 http://www.microsoft.com/presspass/press/2008/mar08/03-05MIX08PR.mspx

Edit: I have now had a bit of time to try out some things in Silverlight 2 beta 1, and my conclusions are:

- Lots of new stuff has been added, such as the much anticipated networking features (Sockets, POX, etc.) as well as many new standard controls, and that’s great news!

- There’s even a “Go-Live” licence, which, for those unfamiliar with Microsoft terminology just means that you are welcome to build something on Silverlight 2 beta 1 and actually deploy it to the public. Nothing business-critical, mind you.

- The environment, toolset and plugin is still far from being finished. I’ve had several crashes already, and I haven’t been using it for more than a few hours. Also, sometimes my Silverlight plugin is not recognized by the sites I visit.

- The documentation is generally speaking much better than before, but still a long way from being finished. For instance, there is no explanation on how to load on-demand a classlibrary user control, except for an MSBuild walkthrough. No VS.NET example, and we are left guessing.

- And yes, the ball bounces just as bad in Silverlight as in Flash.

So.. Back to Flex 3 / Flash for now. But considering the progress that Microsoft made between the Silverlight 1.1 alpha and the 2 beta 1 in about six months time, things are really moving in the right direction.

It’s going to be an interesting spring this year; Microsoft has recently revealed some of the core functionality in the upcoming Silverlight 2.0 beta 1, which is expected to be released in the next month or so.

With Adobe Flex 3 just out, Microsoft is keen to show that their offering will be the better product, and they have certainly listened to the feedback from the  1.0 and 1.1 CTP releases, both of which were limited featurewise, frankly to the point of being useless.

Among the features included in Silverlight 2.0 beta 1 are:

• - WPF-style layout using XAML, deeply integrated with Expression Blend (the design tool)
• - A wide array of built-in controls (you know, TextBox, CheckBox, RadioButton, Panels, etc.)
• - Support for REST, WS*/SOAP, POX, RSS, and plain old HTTP and even Sockets - yay
• - Extensive .NET base class library support, including collections, IO, generics, threading, globalization, XML and local storage
• - HTML/Javascript/DOM interaction with .NET code
• - LINQ support

There is more, but they are not telling about it yet.

So what about install size, prerequisites and browser and OS support? I’m just going to quote Scott Guthrie, who knows everything there is to know about all things .NET and Silverlight, because I can’t word this any better than him:

“Silverlight 2 does not require the .NET Framework to be installed on a computer in order to run.  The Silverlight setup download includes everything necessary to enable all the above features (and more we’ll be talking about shortly) on a vanilla Mac OSX or Windows machine. 

The Beta1 release of Silverlight 2 is 4.3MB in size, and takes 4-10 seconds to install on a machine that doesn’t already have it.  Once Silverlight 2 is installed you can browse the Web and automatically run rich Silverlight applications within your browser of choice (IE, FireFox, Safari, etc).”

Collections, threading, globalization, Sockets and local storage .. It’s going to be great! As much as I love Flex by now, I can’t wait to get my hands on Silverlight 2.0 beta 1…

Read Microsoft’s accouncement on Scott Guthrie’s blog here, which includes a nice tutorial.

The long wait is over! Well, ok, the wait really wasn’t that long, because there have been three betas out of which the last two were feature complete and well functioning.

But anyways, grab it from here: www.adobe.com/flex.

I’ve previously written about the wonders of Rich Internet Applications (RIAs), and I’ve have been looking at both Silverlight and Adobe Flex. For now, at least, I go with Flex, because it’s the mature and complete of the two.

After being sidetracked for a while with my love for casinos and games, I thought it would be appropriate to return to something more business-friendly. So, for my next project, I have chosen to implement a chat client/server with Flex on the frontend and C# on the backend. It could be used for customer service or be a community feature on a website.

I was thinking about doing the whole thing with Web Services, but that has a couple of shortcomings.

First of all, Web Services are stateless; Consuming a Web Service means sending one request from a client to a server and receiving one response. Each request/response pair is separate from all others - stateless. A chat application needs to be more “connected” the entire time, so that it’s possible to push things from either end to the other without the client having to continuously poll the server.

Secondly, Web Services include a bit of overhead in terms of SOAP envelopes, and requires code-generation (in Flex), etc., and I wanted something clean and fast. Ok, sure, my chat application probably won’t be used by thousands of simultaneous users, but why go for the Kia when you can have the Ferrari at more or less the same implementation cost?

The fastest ride in town is “asynchronous sockets”, and the easiest and cleanest implementation of that is using TCP. And luckily, it’s completely supported by both Flex and C#, so that’s what I went with.

http://www.topholt.com/ChatClient/bin-debug/ChatClient.html

A Few Implementation Details 

The chat client above is setup to use port 8000 for communications, so your firewall should allow you to make TCP connections to that port if you want to try it out. It shouldn’t be a big deal. If you download the code, you can modify it to use whatever port you like.

All communications are currently unsecured, which means no encryption, no authentication, no authorization and no data checks of any kind. The downside is that anyone can mess with the chat, but the upside is that you, dear readers, can grab the code and easily modify it for your own needs.

The application’s communications architecture is simple: Each time someone says something, a 256 byte UTF-8 encoded datagram is sent to the server and then immediately distributed to all connected clients, including the sender. That way, everyone sees what everyone is writing.

Edit: View Source on the Flash/Flex file to get the client code or download the zip. And get the C# server source code here. It’s built with VS 2008 and compiled for .NET 3.5, but should work just as fine with 2.0.

Edit 2: I’m currently having server hosting problems, so I am getting ready to move my stuff to a new provider. Therefore, the chat server may be offline at times. The code is still available for download, though.

Right! I’m getting seriously side-tracked here. I was planning on using a bit of my spare time to investigate news on the Rich Internet Applications (RIA) front, but I found myself trying to create an Arkanoid clone in Flex.

It didn’t go too well, because Flex/Flash doesn’t update the visuals fast enough. I think it has to do with the fact that Flash rendering uses a software device, instead of direct hardware access, such as through DirectX or OpenGL.

At any rate, I just couldn’t let the thought of creating an Arkanoid game be, so I browsed around and found GarageGames’ excellent product, Torque Game Builder. Using the product’s out-of-the-box features, some sample graphics and a bit of TorqueScript, I have put together the following:

To see it in action, just download the following file, un-zip it and run the .exe file: www.topholt.com/arkanoid/arkanoidtest1.zip

I’m actually pretty happy with it, considering that it only took a few hours to put together. It uses simple rigid body physics (!) to bounce the ball, and just for fun I added rotation to the paddle.

There are tons of things still missing or not working correctly. Some of the collision detections are a bit off, and the physics are not completely correct - but that’s my fault, not the product’s. I think I’ll keep playing with this and see if I could make it into a real “casual” game.

Comments are always welcome!