HTTP error 503 Service Unavailable, MVC, IIS 7.5 - Troubleshooting

This article relates to MVC 3/Razor, IIS 7.5, .Net Framework 4.0 and Server 2008 R2. It may be useful for other configurations.

I have recently deployed an MVC application to a new SQL/IIS server. While I knew it wouldn't be as simple as copy files, set as web app and go, it was more challenging than anticipated. Hopefully the learnings here will speed up the resolution for others with the same problem.

Of course, being an MVC app, you will need to install MVC 3 on the IIS server.

I used the Visual Studio 2010 "Publish" functionality to get the release on the server. This will be the subject of another article, as it took some setting up, but it is very convenient and supports web.config transformations per environment [debug, release, etc], which is a bonus.

Having got the files on the server, and appearing as an Application in IIS, I fired up IE and hit the URL - HTTP error 503 - Service Unavailable. Ok, first principles, what does this really mean? It means the web application has failed to start, or perhaps started and immediately failed.

So what happens when we hit a .Net MVC web app URL for the first time? The app needs to run as a user/identity in order to "do things" under Windows. The app user then has to read the web application directory/files, including the root web.config and continue from there. The web.config specifies the .Net runtime to use, the assemblies to use, both system and custom, and usually one or more connection strings to data resources [to which the app user must have access]. If all of these are working, your web application should work. Any errors encountered along the way will be in the IIS logs and Windows Event Logs [Application and System] as appropriate.

In IIS, the user/identity for the application to run as is configured under the Application Pool settings. Upon inspection of the app pools, there were just two, each configured for the .Net 2.0 Framework. This was puzzling, as the server had .Net 4.0 installed before it was given the IIS role.

The two highlighted app pools were not present initially.

To get the new ones to appear, I had to register the .Net 4.0 Framework with IIS:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319 > aspnet_regiis -ir

This gave me two new App Pools, labelled as .Net 4.0 Framework version. Unfortunately on closer inspection, one of them was actually pointing at the .Net 2.0 runtime; it always pays to check the default configuration of new objects!

For my scenario, using the new App Pool was appropriate, as there will only be one application on the server. For a server with multiple sites/applications consider creating more app pool identities in order to increase your security. It goes in to a lot more detail here.

Having introduced a new app pool identity, we need to set our site/app to use it via Advanced Settings:

Set the Application Pool from the available list.

 

Next up, does the app pool identity have access to the web application folder and files?

On first glance, it looks like our permissions are correct.

While you're doing this, take the opportunity to review the file/folder permissions for the group. In our case we have "Read and execute", "Read" and "List folder contents", which is all we need. Avoid the temptation when things aren't working to simply click Full Control - bad idea. If your application identity is compromised, it could be used to delete your entire site.

Drilling down into the IIS_IUSRS group, our app pool identity was not present.
Using Server Manager, I browsed to Local Users and Groups and looked at the properties for the IIS_IUSRS group:

Our app pool identity is missing and needs adding.

I added the missing identity via the Add button.

Now we add the app pool identity to SQL Server to provide enough access to service the application's needs. We should only select the db_datareader and the db_datawriter on the target database, rather than just ticking the db_owner role. Again, that is a bad idea. If your app pool identity is compromised, it could be used to amend the schema and/or programmability [tables, SPs, functions, views, etc]. By limiting the user to only what is needed you are significantly adding to the security of your application.

Fingers crossed, and hit the URL once more - success! Only from this point forward would we expect meaningful application error messages in the IIS logs and Windows Event Logs, as we now have a running application.

For me, this was the end of the problem, but reading up, others have experienced application errors, or indeed still got a 503 at this point. The application errors should now be manifesting as meaningful error messages in the IIS Logs and Windows Event Logs [Application and System].

For some, the 503 remained due to them inadvertently using reserved URLs, or in one rare case, having previously reserved port 80 globally, such that it could not be re-used. To check this aspect, run the following command to interrogate the http.sys settings:

C:\> netsh http show urlacl

If the output shows a clash with your application's URL, you will need to change one or the other.
For one user, they were trying to use /Reports, which is reserved by SSRS.

Hope that's been useful.

References/further reading:
Troubleshooting 503 Service Unavailable errors
Troubleshoot IIS 7 errors like a pro
The IIS 7 team rocks (Error 503 resolution)

Built-in Quality

In the world of software development, quality means different things to different people. While there are recognised standards for software quality, this article is concerned with quality as part of your culture, applied pragmatically, rather than adherence to a fixed standard.

Quality seems to be something of a marmite topic, splitting people firmly into one of two camps, those that recognise quality does not happen by chance and is essential to success, and those for whom the word "quality" is used rarely, if at all, beyond the conversations that engage the paying client.

I have worked for organisations where "quality" is assumed to be present if the customer signs off the deliverables and the project was delivered on/under budget. By inference, it is clear such cultures do not value, nurture or assess quality.

I do not believe that quality costs more than it returns; on the contrary, a quality approach will always deliver solid results. With a quality approach, the biggest savings are realised on the post-release issues; that is to say having the least issues post-release means that the project is most likely to come in on budget. All too often the overrun bleeds into the support budget without question.

Of course, a caveat here is how post-release issues are tracked. If they are tracked under some separate catch-all "support" [or indeed "other"] budget, then you could fool yourself into thinking you are more successful than you are, as an organisation. For me, all project issues should be accounted to the project and the budget closed only once the client has formally signed off the project as delivered, and the project formally moves into support. Only if the project is accurately tracked will the financials accurately model the real world, thus permitting informed reflection.

No hard and fast rules

Each quality measure should be applied pragmatically, rather than assuming it is a one-size-fits-all solution. Specific examples are discussed following. These are not exhaustive, but are meant to inspire the reader to review their own approach to quality [or lack of] and to come up with quality measures that fit your organisation's unique needs.

Design

Do not produce design documentation purely as a paper-generation exercise to satisfy a project deliverable. A design document is always useful and can be pitched at different levels for the intended audience. High-level wording, including industry buzz-words, for the client and their management are appropriate for a less-technical business audience. Low-level technical wording, with discussion of specific objects and their implementation [versus other approaches] are appropriate for atomic elements of the solution and the technical audience that will require the information as input. Do not feel the need to make the document wordy; the shorter the better. Pictorial information is always welcomed in such documents.

Development

Most organisations have coding standards. Adhering to these alone will not generate quality software. They are more concerned with style and syntax than semantics. Implementing design patterns correctly, for the appropriate use, will improve the quality of your code base. Each pattern is looking to provide something different: increased testability, increased performance, increased security, increased resilience, increased interoperability, decreased coupling, and so on.
One thing I am sure of is that code should be well-documented. Ask yourself, "could the most junior member of the team understand this code in my absence?" I always aim for comments for entry-level staff. That is not to say the code has been simplified at all, but it is laid out clearly and documented appropriately.

Pair programming or parallel programming?

In software, there exists the concept of pair programming. This is not something I have tried, and so I cannot pass any judgement. On the face of it, it seems counter-intuitive having two developers working on the same code at the same time, but this is purported to be negated by the improved output. One approach I have used however is parallel programming, or "two heads are better than one". Two developers are intentionally separated to work alone on the same piece of the solution, working to a prototype-level of functionality [i.e. the code works but lacks production-level error-handling, logging, UI, etc]. Once they have both got their prototypes working, the approaches are reviewed [by the same or other developer(s)] and the best of each is taken as the way forward. You will be looking to trade-off certain aspects against one-another: performance, robustness, clarity, maintainability, extensibility, etc. This approach should be used for the critical elements of the system, but as the quality increases as a result, can be used wider without fear of wasting time. Of course, a dependency of this approach is that the developers need to be on-board, and certainly should not have ego issues; it requires people to accept that they do not have the definitive "best" approach to any given situation and that collaborative working will always deliver the best results.

Waterfall versus Agile methodologies

Nowadays Agile methodologies are commonplace. Some customers are sold an Agile approach, fully aware of the benefits and implications, others are sold a traditional fixed-price solution that is then implemented internally as an Agile project of fixed duration with the Project Manager representing the Customer.
The benefits of an Agile approach is that it is iterative over short periods [say, two weeks] and so is responsive to change. Each iteration delivers production-quality functionality. Large projects benefit the most, as over time the needs of the business may change and the project may inspire the business to change the requirements as they see the possibilities unfold.
Waterfall by comparison is rigid and resistant to change; the cost of which can be very high for the paying customer. Organisations like fixed price solutions. If they buy a building, or a new printer or car, the cost is fixed. They desire the same predictable cost from their IT solutions.
When I am working fixed-price I tend to work in a hybrid manner. Rather than the top-down approach prescribed by waterfall, I pick a use case/business function and implement it fully end to end right through the application stack. This takes longest for the first but along the way you will have had to decide on your architectural approach as best suited to the business requirements. You should evaluate your options with prototype code, discuss the pros and cons with your peers, manager and customer to ensure all needs are best met. As you implement more and more business functionality you will write less and less code, all of which is of production quality. I find this approach delivers the solution the fastest, with quality in mind, and so if it is possible to deliver the solution within the fixed window, this will deliver it.

Test Driven Development or Tested Development?

I have worked on TDD projects. The TDD element thereof has been implemented with varying levels of understanding and success, as measured by the reduction in bugs [or lack of reduction!].
TDD is a sound approach in my opinion, but not for the entire system.
Before TDD came along I always had a separate Test Project in my Visual Studio solution. By comparison to the very atomic testing TDD delivers, some of my tests were quite high-level, perhaps testing an entire business case in one go. This is clearly better than no testing at all, and if I encountered a bug, I would craft something more specific to test that bug and others "nearby".
Under TDD you write the test first, which will fail due to a lack of logic/implementation, and then you write/correct the code to pass the test. After a while [of writing good tests] you get into the habit of writing simpler code that performs more atomic tasks. Once this habit develops your code will be easier to understand and inherently more maintainable.
Now, left to my own free will, I tend to use a hybrid approach, developing the most critical and most used parts of the solution much closer to TDD, and then easing off and writing higher-level testing for the remainder of the system. The approach is a trade-off of time-used versus real quality achieved that I am happy with and delivers results.

Review failure and success

I am puzzled by organisations that only "review" a project, or sub-part thereof, when things go wrong, to whatever degree. Surely all projects of a non-repetitive nature should be reviewed, overall and more atomically, to agree what worked and what could have been improved. This should then lead into your "best practice" knowledge base, which may include document templates, documented procedures, coding design patterns, code examples, third-party utilities, all organised in such a way as to be found easily. All too often organisations reinvent the wheel without knowing it. Having a knowledge base intranet site with document libraries, images and wikis should be the norm, and the collaborative contribution and consumption of it by the whole team should be second nature.

Hopefully there are some new ideas here to inspire you to improve the quality within your organisation!

How to reuse an old PC as a Hyper-V Server

I recently decided to build a Hyper-V server at home. As a developer/techie, this is possibly long overdue, after all, who wouldn't want loads of virtual machines running a variety of OS' and configurations! I'm probably also long overdue an introduction to modern unix flavours, and a VM is a great way to dip your toe with minimal investment/risk.

I figured I'd need 8GB RAM and a reasonable processor/motherboard chipset to make it a reality.
Looking around for re-usable hardware, I found my previous gaming rig. Clearly not an ideal candidate, but is the effort to make it into a Hyper-V server justified and cost-effective? Read on.

On with the hardware downgrade

So the hardware needed calming down somewhat before it could be power-efficient as a server. It had four SATA drives in it, connected to the on board RAID controller, configured as two stripes. This wasn't a suitable configuration for a server. I read the documentation and hoped that it had RAID 1+0, but it doesn't. However, it does have RAID 0+1. Not quite the same, but it will suffice.
The RAM had to go - high speed gaming stuff with heat sinks - 4GB in total. I checked the motherboard documentation and it can take up to 8GB, as 4 x 2GB. These are surprisingly cheap as a standard configuration (rather than performance/gaming)
Last up, the graphics card - no point having a GTX 550 Ti sitting in a server. A quick look online reveals an nVidia 520 card for under £30 that only draws 30 Watts (as opposed to 100+ W from the outgoing 550 Ti).
So hardware changed to 0+1 RAID, 8GB standard RAM and an entry-level, low-power graphics card, we can proceed with the OS install.

There is no discussion here on Hyper-V versus VMWare; I am a Microsoft consultant/developer with an MSDN subscription. I have used VMWare commercially, small-scale, and have had no more issues than I've had with Hyper-V. I opted for Hyper-V 2008 R2 as I use Windows 7 clients mainly.

Let's try Hyper-V Free (Windows Server Core edition)

I started out with the free version of Hyper-V, based on Windows Server Core. The Core edition lacks the UI components to manage the server via local log on; leaving you the command line locally and full GUI remote administration from a client machine, such as Windows 7. The Server Core flavour was introduced to reduce the attack surface, along with a potential performance increase from the server not spending time on the UI. On the face of it, its a perfect candidate for a Hyper-V server.

The install went without issue and on first proper log on (after setting passwords and such) you are presented with a standard command line window and a PowerShell-looking window, showing the textual menu of the sconfig script. Running down the menu in order you can quickly get the server ready for remote administration. Also, check out the additional HVRemote script for more help and tools for checking the config remotely. Remote desktop is not so useful at this point, as all you get is the same two command windows and no UI. Remote administration tools include the Hyper-V manager, the standard Server Manager , and of course more basic access such as the admin file shares, C$ and so on.

Some basic checks and usage revealed that both disk access and VM operational speeds were below expectations. Local network file copy was under 1 MBps, which on a gigabit network is terrible. Device Manager is available when running remotely [once configured], but due to the low-level nature of hardware drivers, it is read-only. I noticed a few question marks, which were soon resolved with a mix of turning off devices in the BIOS and installing Windows 7/Vista drivers, such as a ACPI driver.

So, no question marks over devices in Device Manager - all sorted, ...yes? The lack of question marks over devices simply means compatible drivers are installed. These are unlikely to provide the best performance on domestic/custom hardware. Local network file copy was still under 1 MBps, so things were not right. To cut a very long story short, I managed to replace most of the compatible drivers with manufacturer-specific drivers via the server command line and managed to get network file copy up to speed on the host server, but not on any of the running VMs. The learning here is that network access for VMs is a crucial element for a Hyper-V server, and my motherboard, with its nVidia chipset, was not optimised for use as the physical network adapter for the virtual network switch/adapters surfaced within Hyper-V and the VMs.

During the previous investigation, I learned more about networking. I had only recently upgraded the home network to gigabit. Having bought a reel of Cat 5e network cable years ago and made my own cables previously, I had assumed these were working and gigabit compatible. After all, they were allowing remote control, internet browsing, etc. What didn't add up was the speed of large file copy. Long story short, the cables weren't wired for gigabit, so I chopped the ends off and rewired - performance now up to expectations, at the physical level, as indicated by the correct LED colour next to the network cable socket.

Two NICs, or not two NICS?

The Microsoft Best Practices Analyzer for Hyper-V had already alerted me to not having two NICs, as recommended. Some serious web trawling of people having general Hyper-V/network issues revealed that by far, the best candidate for the VMs networking requirement was an Intel PRO 1000 based PCIe card. This was under thirty pounds on Amazon uk. So now we had the motherboard nVidia NIC for the Hyper-V server's communication, not shared with the VMs, and the Intel PRO NIC was used exclusively for the virtual switch for the virtual adapters for the VMs.

Unfortunately the addition/reconfiguration of a new network adapter confused the server no end, so I started again re-installing the Core edition of Hyper-V, not a long process, but one I'd rather have avoided. So now I had a two-NIC server installed with Hyper-V 2008 R2 Server Core edition and device drivers patched as far as the command line would allow. Time to review performance again. Unfortunately things were barely improved from before: network file copy speed around 10 Mbps and acceptable VM performance and network throughput without actually being impressive.

I was convinced I could get more from the server, after all the hardware was reasonably impressive for domestic use. I had to bite the bullet and re-install the Hyper-V OS again, this time with a Server 2008 R2 Full Installation. This version is not free, but I have it as part of my MSDN subscription.

Hyper-V 2008 R2 Full Installation

This time the rebuild took a little longer, having to add the Hyper-V role and so on from a basic full installation. Using the Device Manager natively and having the UI environment to run normal install executables provided a full set of updated, manufacturer-optimised drivers for all key components - motherboard chipset including NIC and RAID controllers, graphics card, and Intel PRO software for the NIC. Time to test performance again.

This time things were very impressive. Network file copy was in the region of 100-150 MBps copying from Hyper-V to my Windows 7 desktop, and about 80-100 going in the other direction. The reason being that the Windows 7 desktop is my latest gaming rig with much more performant hardware. The VMs were as fast as any corporate physical or virtual machine I had used, internet access was the full speed of my 30 Mbps internet (~4 MBps). At last, a clean sheet on all fronts!

Was it worth it?

Yes, I think so, I now have a low-power server running VMs on-demand. The standard benefits of virtualisation apply - adjustable RAM/CPU per VM and checkpoints to allow the testing of changes/deployments with painless rollback. An upcoming article will detail the physical-to-virtual [P2V] conversion of an old Windows XP machine; suffice to say, with more RAM and a modern processor at it's disposal it was surprisingly nippy.

Summary of changes

Hardware downgraded to a standard graphics card, memory changed from 4GB gaming heatsinked RAM to 8GB standard RAM and RAID reconfigured to use four drives in a 0+1 configuration.
One native nVidia NIC used exclusively for server access, and an additional PCIe Intel NIC card for VM network access.
Full Windows Server 2008 R2 Hyper-V install
Full manufacturer device driver installs for motherboard chipset [including nVidia NIC and nVidia MediaShield RAID controller], nVidia graphics card, Intel PRO 1000 PCIe network card.

Notes

The standard pre-requisites for Hyper-V apply, including having a CPU that supports virtualisation natively. Check out the official download page for a full list of pre-requisites.

This article is about repurposing domestic hardware for use as a Hyper-V server. Such machines are generally not "standard corporate hardware". Standard corporate hardware, i.e. racked servers from the main suppliers, will almost certainly run at [near] full performance with the standard drivers installed and so the Server Core edition of Hyper-V will probably suit such hardware and provide the benefits of a reduced attack surface and a leaner OS install/runtime leading to increased throughput.

Tricks and tips

This article will be updated ad infinitum so bookmark it, revisit, etc.
All entries are alphabetical under whatever word I thought most appropriate.

Creating Documentation

Screen/Image capture and adding highlights/captions

I have found FastStone Capture to be the most flexible and cost effective tool for taking screen/window image grabs and then offering useful annotations from within its own editor.
It has some nice touches like grabbing a scrolling window, including the mouse pointer in the grab and delayed grab so that you can get your mouse pointer/menus as you need them before the image is grabbed. The standard download allows a 30 day free trial. After that, I'm sure you'll be parting with your cash as it is a massive time saver.
The images used on this blog were obtained using the software.

Virtualisation

 

Copying large files to/from a Hyper-V Virtual Machine

When copying large files to/from a VM, it can be time-consuming and inefficient to have to boot the OS before then copying large files. A better way is to shut down the host OS and then using the host server, mount the VHD using Disk Management:


Once mounted, the logical disk(s) will appear with the next available drive letters, and these in turn are available over the network as the default admin shares, i.e. \\servername\G$. This should provide the simplest access and the fastest file copy.

Visual Studio

GhostDoc documentation helper

I have used GhostDoc for many years and versions of Visual Studio. It auto-xml-documents your object properties from the variable names and types [which encourages better naming] which can then be used to create a CHM help file. The Pro version does that automatically; the free version requires a little more manual effort. Check it out.