Beware Of Deploying Debug Code In Production

Posted on May 05, 2004

44 comments

You have spent several months developing a killer web application (a web site, perhaps) and the long anticipated release day has come. You deploy the application and take it for a test drive. As you navigate from page to page you notice that each page “thinks” before rendering. What’s going on? Didn’t Microsoft folks promise code compilation and ultra fast execution? Isn’t it why we beat ASP, Java and PHP by such a wide margin?

Take a peek inside your web.config. Are you telling the framework to compile debug or release code? Does it have to stumble across and compile one page at a time, or compile them all in one pass? It's all in the <compilation> tag. Let's take a closer look at it.

The compilation element has many attributes. The one we are going to examine is debug. The attribute may take one of the two values:

  • true to have the framework produce debug binaries
  • false to produce release (retail) binaries

Let's dig in. ASP.NET maintains a directory where it stores compiled code, be it debug or release code. The debug switch determines the manner in which it compiles code. In the .NET 1.1 installation you can find this folder in \Windows\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files. ASP.NET creates a folder for each web application root in there.

A tree of temporary foldersFor example, I created a sample project for this article and named it DebugTest. As promised ASP.NET created a folder called debugtest for me.

The sample project, TestDebug, has two pages named test1.aspx and test2.aspx. Note that every page is actually a class. The project itself compiles executable code into a dll, TestDebug.dll. Quite naturally, we expect to see both test1 and test2 classes inside this DLL.

Let's expand the tree of folders with cryptic names and locate our assembly. The last folder in this food chain has two files: TestDebug.dll and __AssemblyInfo__.ini. The ini file is a simple one:

[AssemblyInfo]
MVID=69266042ccf63445be7d9c996d9658aa
URL=file:///c:/inetpub/wwwroot/DebugTest/bin/DebugTest.DLL
DisplayName=DebugTest, Version=1.0.1584.40802, Culture=neutral, «
     PublicKeyToken=null

Compiled assemblyNow let's peek inside the DLL itself. If you own a copy of a disassembler, such as ILDASM or Reflector, fire it up and open the assembly.

Bingo! Test1 and test2 are there just like we anticipated! Also, it's interesting to see that globals.asax got compiled into its own class! Makes you wonder if you really need it and whether you have use for all those event hooks to the Session and Application objects.

When you create a brand new web project in Visual Studio.NET your web.config contains the following compilation tag by default:

<compilation defaultLanguage="c#" debug="true" />

Let's see now how the settings of the debug attribute affect output.

debug="false"

When the debug attribute is set to false ASP.NET conducts a batch compilation, i.e. all files in the current folder are compiled into one DLL.

Now, pay close attention to this: when an ASP.NET file is requested for the first time the framework conducts a batch compilation on that directory. It compiles files in that directory only into a DLL. It does not traverse subfolders. You need to hit a page in every folder to have each folder batch compiled. This is an important point.

For example, these are the files I end up with under my debugtest directory:

Tree of temporary folders

The xml files correspond to pages in the project. Each xml file indicates where a page was compiled and lists where the original page resides to track changes:

<preserve assem="qu_mr0sg" type="_ASP.test1_aspx" «
hash="ff19a32b35478349" batch="true">
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\test1.aspx" />
</preserve>

You will have as many xml files as there are pages in your project. Files with the .web extension are used by ASP.NET to manage file updates (this is to say I don't know how exactly they are used but ASP.NET keeps tabs on them for its own reasons).

You might've noticed there are two DLLs in this folder. Why so? Remember my statement about batch compilations? Well, both test1 and test2 were compiled into one assembly while globals.asax got its own assembly. I could have removed it from the project to keep this exercise cleaner. I decided to keep it and emphasize that globals.asax is an important part of every project and you should think twice if you really need it. So far it's been getting in the way a lot.

So now we know this much: all pages from the current folder are compiled into one DLL and each and every page has an xml file "assigned" to it. Let's make a simple change to, say, test1.aspx. Anything. Now run the page again and look what happens:

Tree of temporary folders

We have a new DLL! And... the xml file for test1.aspx now points to it:

<preserve assem="ugtscdas" type="ASP.test1_aspx" «
hash="e65cd41d34f3d4">
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\test1.aspx" />
</preserve>

Wait a second! The xml file for test2.aspx still points to the old DLL:

<preserve assem="qu_mr0sg" type="_ASP.test2_aspx" « 
hash="ff19a36e8f347176" batch="true">
   <filedep name="c:\inetpub\wwwroot\DebugTest\test2.aspx" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
</preserve>

Compiled assemblyWe simply changed test1.aspx and now we have two assemblies? That's right! Fire up your favorite disassembler and load both the old DLL and the new one.

The DLL produced as the result of a minor change to test1.aspx led to compilation of that one page only into a separate DLL (it's the second one down, called ugtscdas.dll in this particular case)! The old DLL will be used to service current requests only.

Let's make another change to test1.aspx. Anything at all. Now run it. Yet another assembly! I think you get the point by now‚you keep changing a page and ASP.NET will keep recompiling it to have the latest code running. It won't bother recompiling everything. What does this tell you? If certain pages change a lot, move them out to a separate folder if possible to spare the rest of the code from constant recompilations!

If you keep changing test1.aspx and requesting it in your browser again and again you will see an interesting thing: in the temporary directories you'll have files with the .delete extension. These are the outdated DLLs, the ones replaced upon recompilations.

Tree of temporary folders

Basically, these DLLs are doomed to be purged. Go ahead and try deleting one. You can't because it's locked. The DLLs are loaded into an AppDomain. To evict a DLL from an AppDomain either change something in web.config or global.asax, or restart the application. Then start your application by requesting a page and take a look at your temporary directory. All .delete files should be gone.

Maximum Number Of Recompiles

We've talked about changing a page and causing the framework to recompile it. How much of this abuse can ASP.NET take before restarting the whole application and starting from scratch? The threshold is configured in machine.config. Go to your \Windows\Microsoft.NET\Framework\v1.1.4322\Config folder, load machine.config and find the <compilation> tag.

Do not change anything in machine.config. Changes to this file will affect all web applications on your computer. Do so only if you know what you are doing.

The attribute we're after is numRecompilesBeforeApprestart. MSDN states its purpose as follows:

Indicates the number of dynamic recompiles of resources that can occur before the application restarts. This attribute is supported at the global and application level but not at the directory level.

The default for this attribute is 15 recompilations. Once the limit is reached the AppDomain unloads (more on this in a minute), the application restarts upon a subsequent page request and old DLLs are finally deleted.

So there you have it. Excessive recompilations will cause your web application to restart often.

Maximum Number Of Pages Per Batch Compilation

Along the same lines, the maximum number of files from the same folder that can be compiled into a DLL during batch compilation is 1000. This default is configured right there, in the maxBatchFileSize attribute of the <compilation> element, too. According to MSDN maxBatchFileSize specifies the maximum number of pages per batched compilation.

Why set this limit? Batch compilation happens once you reach a web application for the first time. Since having a user wait is not a good idea there needs to be a sane limit to how long we can have the user's attention before they decide to bail. Also, the size of the DLL might become unacceptably large. Therefore the limit is imposed.

Unloading AppDomain To Release DLLs

In .NET you can load a DLL but can't unload. I don't know if it's a good thing or a bad thing. In this case it's a bad thing. Here's why: every time a page is recompiled a new DLL is loaded into the AppDomain. The smallest unit that you can unload in .NET is an AppDomain. This means we have to unload the entire AppDomian to release old DLLs and have them scavenged. I can think of a couple of options to make this happen:

  1. Modify globals.asax or web.config. This causes a batch compilation. A fresh start, so to say.
  2. Wait until numRecompilesBeforeApprestart compilations happen (15 by default).
  3. Run iisreset or reboot.
  4. ASP.NET worker process takes up too much memory and the framework restarts it. This is pretty much outside of your control.

debug="true"

All this time we were assuming the debug attribute of the <compilation> element was set to false, i.e. no debug information was being generated. Now, on with the fun part—set it to "true" and observe what happens.

The temporary folder contains a lot more files this time around. Batch compilation doesn't take place! To the contrary, every page is compiled into its own assembly! This is a very important point. More so, every page comes with a bunch of other files: debug symbol file, compiler command line file, compiler output file, etc. I'll reiterate: every page will have to be compiled individually into a separate assembly. Do I have to say this alone may kill performance of your web app when deployed live?

Note that you also get source code for free! This is what the compiler produces with the help of CodeDOM and compiles as DLLs. Except that in this case it leaves the source code and other byproducts of compilation on the disk.

These days you can't even watch on TV without being warned of yet another computer worm that is spreading fast. "Security" has become a very popular buzzword. In light of this I will just point out that exposing your source code in this manner is not a good idea. Nobody will pad you on the back for this. Just the opposite—you may get in trouble. Not that you exposed the source code in plain and clear as there is a lot of vagueness in the produced code, but it comes close.

Before we wrap up I wanted to share insight into a couple of problems you might encounter when dealing with page compilations, batch compiles, etc. I assume you've installed .NET Framework 1.1. A lot of bugs have been fixed since 1.0 and I won't address them here.

BUG: XCOPY Deployment Enlarges Temporary ASP.NET Folder Size (Q310450)

I've already mentioned that in the latest version of .NET, which is ver 1.1 as of now, you cannot unload individual DLLs from an AppDomain. As you modify pages and cause their recompilation new DLLs stay loaded until you do one of the following:

  • Modify web.config or machine.cofig.
  • Modify global.asax.
  • Modify any file in the bin folder.

Any of these steps trigger unloading of the AppDomain and deletion of older DLLs. Otherwise old DLLs pile up and eat up your disk space. In an extreme case you'll run out of space on your hard drive.

PRB: CS0013 or CS0016 Compilation Errors in ASP.NET Web Applications (Q825791)

This one is easy. If you receive the error

CS0016: Could not write to output 
file 'c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary «
ASP.NET Files\xxxxx'. The directory name is invalid.

check your TEMP and TMP system variables. Chances are they point to a non-existing folder. Knowledge Base article Q825791 tells you how to fix it.

PRB: Access Denied Error When You Make Code Modifications with Index Services Running (Q329065)

This one is vicious! I see it come up in newsgroups all the time. Here's the scoop of the problem: the Indexing Service, if enabled, may engage in rescanning the Temporary ASP.NET Files folder for a couple of minutes. As the Knowledge Base article Q329065 states:

The length of time of the lock depends on the size of the directory that causes the Aspnet_wp.exeprocess (or W3wp.exe process for applications that run on Microsoft Internet Information Services [IIS] 6.0) to not load the particular DLL.

There are two ways to address this—either disable the Indexing Service altogether, or if you absolutely need it add the Temporary ASP.NET Files folder to the service for exclusion.

Insufficient User Rights To Run The ASP.NET Worker Process

You see different flavors of this predicament in the newsgroups all the time, too. Check out Q833444 and Q811320 on your own to see if they apply to you.

Keep The Code Warm

Microsoft published a great whitepaper titled Deploying .NET Framework-based Applications at Patterns & Practices. This prescriptive guide advocates "warming up" a web application by triggering a batch compilation. How?

  • Manually. Navigate to each folder to have it batch-compiled. Next time the application restarts for whatever reason you have to do this again.
  • Automatically. The said whitepaper suggests writing a script that simply places page requests. You can have a scheduler run this script every once in a while.

A Windows service might be even a more powerful option here. A Windows service may monitor the status of IIS as a whole or a particular web app and run the script automatically each time it restarts. WMI can lend a helping hand to monitor the state of processes.

The whitepaper also lists JIT compilation, but I won't go there. It's a questionable approach which is outside of the scope of this article. Please, refer to Chapter 4 of the whitepaper, "Deployment Issues for .NET Applications", to read more about NGEN and JIT compilation.

Conclusion

It's interesting how a little obscure switch somewhere in a configuration file can make so much difference. It helps to understand the underpinnings of ASP.NET and how it handles page compilations. Be careful when deploying code in production and make sure you set the <compilation> tag properly as outlined here.

44 comments

SomeNewKid
on May 15, 2004

Excellent article! Thanks for investigating the issue, and writing about it.


Sam Amer
on May 17, 2004

Very informative stuff. Great work. Where on earth do you get these kinds of details that are really useful. Thanks for sharing.


Milan Negovan
on May 18, 2004

Well, I gather as much information as I can and dig as deep as I can. I don't really have any fancy titles of MCP, MCSD, MVP, "insider", etc (neighter am I going after them). Well, I guess my only title is CTO, but that's it. :)


Watcher
on May 31, 2004

Excellent article! Your site is one of the best out there. Keep up the good work. :)


Amit
on June 11, 2004

Interesting article. As I understand it, the gist of the article is the line :"If certain pages change a lot, move them out to a separate folder if possible to spare the rest of the code from constant recompilations!". The thing I'm not able to understand is, on the production server, the pages are not going to be changed once they're put there. So that means they're not going to be recompiled constantly. When a release is done, the recompilation will be a one-time thing and once the dlls are there they stay there. Have I got this right? Thanks for sharing!


Milan Negovan
on June 11, 2004

Well, sometimes you have to update a web site with minor changes. A page here and there. There's no need to redeploy everything-everything. Or maybe some pages are "assembled" by server-side code. If it's done aggressively I can imagine there will be quite a bit of recompilation going on.


Robert Merriott
on July 07, 2004

Good article. However, I have one main question.
Is this also true for projects compiled in VS.Net or only pages within inline code? I have always assumed that this setting was ignored in projects compiled in VS.Net.


Milan Negovan
on July 07, 2004

Robert, this is about any change to an aspx page, i.e. a change to the markup, not code-behind.

VS.NET doesn't bother with the debug setting from web.config. Once you build your project in VS.NET and deploy it, compiled code-behind DLLs don't get recompiled no matter what.


Netski
on August 09, 2004

Useful information, presented in a very confusing format. You're on to something here, but the way it is written does not do it justice at all. At some points it even seems like you contradict yourself.


Ben Henderson
on August 10, 2004

Milan,

Thanks for taking the time to put this information together. A couple of observations:

1. Under the heading [debug="false"], the first sentence says:

“When the debug attribute is set to TRUE ASP.NET conducts a batch compilation, i.e. all files in the current folder are compiled into one DLL.”

It sounds like you're describing behavior when debug is set to FALSE. Did you make a typo or have I missed something?

2. You conclude:

“If certain pages change a lot, move them out to a separate folder if possible to spare the rest of the code from constant recompilations!”

You explained that when an ASPX file is modified, a new assembly containing ONE class that represents the modified ASPX file, is created. Given this fact, why do you assert that rest of the code in the folder while be recompiled?

Again, I really enjoyed the material. Not many people are talking about this topic.


Milan Negovan
on August 10, 2004

Ben, great catch! Thank you. It is supposed to say that when debug="false" batch compilation happens.

To answer your second question let me quote the "Coding Strategies With The Microsoft ASP.NET Team" book:

"The batch compilation has directory granularity, so pages that are updated can cause an assembly to be invalidated and cause additional compilations. But once an assembly is loaded, it can't be unloaded until the application is unloaded. Instead, new assemblies are created for the updated content. Plan accordingly on sites with a high rate of churn. Pages that are updated frequently should be separated from relatively stable pages so that the number of recompilations is limited."


nitin chittal
on September 01, 2004

the article is excellent and well presented. i actually serached for the article and read it becuase i had deployed a web application with the debug flag as 'true' and i felt the site is a bit lethargic. by chance i discovered this tag and changed it... i am not sure about the performance after the change... but a part of the website, which deals with several xml creations/upadations on submit bombed. the message that the sql database returned is reporduced below. after i changed it back, it worked fine. am still trying to find out whether it was related to the debug tag. has anybody encountered such an issue.

the error msg:

2004-09-01 10:52:06,843 [3112] ERROR ErrorLogger [] -
-------------------------------------
NAVManager : Void UpdateStatus(Int64, Int64, Int64, System.String)
Exception: System.InvalidOperationException
Message: This SqlTransaction has completed; it is no longer usable.
Source: System.Data
at System.Data.SqlClient.SqlTransaction.Rollback()
2004-09-01 10:52:06,843 [3112] ERROR ErrorLogger [] - at System.Data.SqlClient.SqlTransaction.Rollback()
Exception: System.InvalidOperationException
Message: This SqlTransaction has completed; it is no longer usable.
Source: System.Data
at System.Data.SqlClient.SqlTransaction.Rollback()


Milan Negovan
on September 03, 2004

Wow, that's an interesting error message. I can't think of a reason why the debug tag would affect your data access logic...


Manoj Kholia
on September 14, 2004

I have found this problem .. and here debug = true, will change it to False and get back to u if realy works fine...

This SqlTransaction has completed; it is no longer usable.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:


[InvalidOperationException: This SqlTransaction has completed; it is no longer usable.]
System.Data.SqlClient.SqlTransaction.Rollback() +115
QTicket.TicketQueue.btnAssign_Click(Object sender, EventArgs e)
System.Web.UI.WebControls.Button.OnClick(EventArgs e) +108
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler. RaisePostBackEvent(String eventArgument) +57
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +18
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +33
System.Web.UI.Page.ProcessRequestMain() +1292


nitin chittal
on December 13, 2004

hi manoj,
any luck on this one... got a bit side tracked on another project... i have still kept the debug tag = true.
rgds,
nitin


Lars Kjeldsen
on January 05, 2005

Excellent article!

Can anyone explain to me how the Debug setting in Web.config differs from the Debug/Release dropdown in VS.NET?

Thanks


Milan Negovan
on January 05, 2005

It's pretty confusing that there's a Release/Debug mode in VS.NET and a debug option in web.config.

The one in VS.NET controls what kind of DLLs are compiled. Debug ones cotain symbol information in pdb files. These DLLs are much slower, although their benefit is that symbol information allows to build a meaningful trace stack when an exception is thrown.

Once DLLs are compiled and deployed, the debug option in web.config determines how they are handled (whether batch compilation happens and so forth).


charlie arehart
on January 29, 2005

Here's some info that may be valuable to some of your readers (and you may want to add to your list of how to manually stop the web app if desired).

You can manually stop the web app by calling the UnloadAppDomain method of the System.Web.HttpRuntime class. No argument required. Just call it and the app is unloaded. As with your other approaches, the next request will start up the app domain.


Patrick Tangelder
on July 12, 2005

Excellent article!

Indeed the ASPX pages will be compiled, regardless the build is made with Debug or Release option in VS.Net.

Thanks


Phani
on July 22, 2005

Well analyzed and beautifully written.


Balasaheb
on August 26, 2005

I like this document on web.config's < compilation >tag.
But I need more information on web.config.
I want to know more about web.config
Please guide me on remaining tag


Milan Negovan
on August 27, 2005

I think MSDN itself is a good enough resource to learn about the rest of the tags.


Prashant
on September 26, 2005

Awesome article, thanks a lot for sharing the information.


Rutger van Hagen
on November 03, 2005

Very good article. BUT I've got a problem around this subject. When i set my asp.net webservice to DEBUG=FALSE a method in my webservice hangs ather a minute. This webservice is proccessing a lot of data. With DEBUG=TRUE everything is going fine. Someone got an idea?? Thanx!

Rutger van Hagen


Rens
on November 10, 2005

Excellent article and well presented....one I was searching for a long time..


Garth Hinman
on January 11, 2006

Does anybody know where to find the whitepaper, "Deployment Issues for .NET Applications", to read more about NGEN and JIT compilation as mentioned at the end of the article. Google, mdsn and msdn archive serches produce no results.


Milan Negovan
on January 12, 2006

This is strange. I looked at the whitepaper with the exact same title at the Patterns and Practices, but its content is different. I guess they edited it heavily since then.

I do happen to have an older copy which you are welcome to download. The mentioned Chapter 4 is there. ;)


Tony
on January 30, 2006

Well, let me first say that its an excellent article. What I could conclude from the article is that really, the debug attrubute will only matter if any files are changed after deployment. If none of that file are changed after depolyment, setting this attribute to true or false shouldnt really matter.Please correct me if I am wrong.

Also when they say that setting this attribute to false will improve performance, are the factors that you described in the article the only ones affecting performance, or there any others?


Dave
on February 01, 2006

Great article! I'm off to change my Debug attribute straight away :-)


Ritesh Uniyal
on July 04, 2006

This is an excellent article with excellent presentation. It is upto the point and very clearly explains each and every thing that is being shown


Jyotin
on August 04, 2006

Exellent Article.... :o)
Short and to the point...


Angel
on August 30, 2006

V useful article. Here's a weird behaviour that I'm hoping someone will have an opinion on. On our system some directories of aspx pages can compile to show text updates no problem BUT some directories can't and we continue to see old versions. Eventually the application restarts and all the updates show and directories that didn't work before do update again. At some point the server decides it can't update different directories whilst going on updating others.

What is most likely to be going on in this case? You can always add new pages to even a directory that won't compile and it will compile the new pages. It feels like .NET does compile the page but it's still linking to the old DLL so the changes aren't showing... But only in some random directories! Does this make any sense to anyone????


Nawaz Ali
on January 06, 2008

Thank you for a great post and thanks for sharing it, i have learnt alot. Great post.


KBN Sarma
on January 21, 2008

Excellent. This is the problem I am facing now.


yashpal sharma
on April 14, 2008

Hi Milan,
Such a great article i was looking for it from long time becoze there is not so many post those tell us about the hiden things as u told in article but u have written a excellent article.
thank you :)


Rukhsar Ahmad
on July 22, 2008

Thanks for this goodarticle.


Priya
on October 21, 2008

Excellent Article, Thanks


luciano
on November 18, 2008

Excellent Article, Thanks


brendo
on January 31, 2009

I see a few guys have had issues with things breaking when DEBUG=false is set...setting it back to TRUE seems to fix it...did you consider timeouts as a possible issue - particularly around SQL transactions...I'm not 100% sure , but I think that setting DEBUG=true increases the request timeouts, or even removes the timeout altogether.


Milan Negovan
on February 04, 2009

brendo, your observation is correct. The debug mode is a whole different ball game.


Monty
on August 07, 2009

I have a question regarding how the compilation element is used. I'm not certain if this is the correct place to post this question, but if you can offer any more information, I would greatly appreciate it.

I have several web services that were built in Release mode with the compilation debug="true".
Recently we went and set that value to false by manually editing the web.config. The web services seem to mostly work fine, but on certain calls we're getting a System.Threading.ThreadAbortException
It appears to be on calls that take an excessive amount of time to do their work and return. I have verified that changing the value back to true fixes the problem.

So my question boils down to this:
Does the comiplation debug setting have any effect at the time of compiling the web service, and could changing it manually like I've done cause some sort of mis-match?
Or is there any other pertenant information that would help me understand the use of the compilation tag?


Milan Negovan
on August 13, 2009

If my memory serves me right, pages don't time out in debug mode. They do in release. Maybe you're seeing a long-running operation which eventually times out.


Vikas
on September 01, 2009

Very informative material. Great task done. I was not aware of these things. Thanks.


anonymous
on November 25, 2009

Thank you!!! Solved frustrating restart issue while developing.