Setup for Remote Debugging with Symbol Server and Source Indexing

Troubleshooting production issues of an older release cause headache to developers. It's due to non availability of symbols and source code. Sometimes it'll be also hard to create these issues in development environment.

Some definitions:
PDB

• Program database aids debugger to debug applications
• A PDB file is generated as a result of building a dll or exe.
• It contains information about variables, functions etc 
• It also contains list of all the source files used for build
• A PDB has one to one relationship with its exe or dll 
• When a dll or exe is loaded, the debugger loads its PDB only when its matching PDB is found.

Symbol server

• A database of symbols (PDBs) generated from the build. 
• It contains the PDBs  corresponding to the various versions of the dll or .exe
• It enables the debugger to automatically map and load symbols of a particular version of dll or exe

Source Indexing

• It’s the process of injecting a command in the PDB to retrieve the source file from source control archive.
• Typically it contains a command to the archive to retrieve the file of a particular revision. 
• Therefore, it enables the debugger to automatically load the correct revision of the source code directly from the archive

Remote debugging

• Enables debugging of applications remotely in production like environments where debuggers, symbols, and source code are unavailable.
• With the aid of symbol server and source indexing, it’ll be possible to debug from developer box*any* build *without* source code or symbols on the developer box.

Setup

As shown above a typical setup consists of Build PC., Source Code archive PC. The developer PC should have access these two at the time of debugging. Alternatively, a mini symbol server and source code associated with it can be deployed onto   the developer PC eliminating dependency.
The VS Remote debugger uses DCOM to communicate with the production box.

Source Code Archive PC

Note it's assumed that the source code archive is svn. In this case it's https://riouxsvn.com/. It's also possible to use any other source code control. For example using RTC  server is discussed here.

Build PC Setup

• Install Debugging Tools for Windows which can be downloaded separately from Microsoft
• Install Active state Perl
• Create a directory to store symbols and share it. This will be referred as the symbol server.
• Edit svn.pm and srcsrv.ini files under C:\Program Files (x86)\Debugging Tools for Windows (x86)\srcsrv folder for source indexing. These typically involves providing the repository URL and credentials to access it,

Build PC Operation

After successful build, Execute following command to inject commands to open source files in PDBs by the debugger.

srcsrv\svnindex /debug /source=<path containing source files>  /symbols=< location of the pdbs>   /rev=<revision number>

Example:

srcsrv\svnindex /debug /source="D:\Github\TechBlog\COM+\APITester\Source" /symbols="D:\Github\TechBlog\COM+\APITester\bin" /rev=2

Execute following command to add the pdbs to symbol store

symstore add /r /f  <location of binaries and pdbs> /t <name of the product> /z pri /S <symbol server path>

Example:

 symstore add /r /f "D:\Github\TechBlog\COM+\APITester\bin" /t APITester  /z pri /S "D:\Github\TechBlog\COM+\APITesterSS"

The screen output is as shown below:

Share the symbol server 

Production PC Setup

The production PC should be setup for remote debugging and DCOM as discussed below.

• Install Remote Debugging component which can be downloaded separately from Microsoft
• Setup for DCOM
• Local Security Policy

• Network Adapter

• Disable Firewall or configure it to allow DCOM traffic for logged on user

Production PC  Operation

Run MSVS Monitor

• Create or update system environment variable _NT_SYMBOL_PATH and map it to the symbol store directory.

NOTE: For native code, symbols are first searched and loaded from the Developer pc where the debugger is running.

For managed code, the symbols are first searched in the remote machine where the debugged application is running.

Start the Application to debug

Developer PC Setup

The developer  PC should be setup for remote debugging and DCOM as discussed below.

• Local Security Policy

• Network Adapter

• In case Production PC belongs to a workgroup, in order to make DCOM to work, the developer PC should create the same user account with identical password as in production box and add it administrators group. In addition it should also use this account to remote debug. In case Production PC belongs to a domain in order to make DCOM to work, the developer PC should use this account to remote debug.
• Disable Firewall or configure it to allow DCOM traffic for debugging user

Developer PC Operation

Launch VS under the same  account (using Runas command) a  discussed above
Check “Enable Source server support” by selecting Debugging->General

Add a new entry pointing to symbol server location by selecting Debugging->Symbols

From Tools->Attach menu option, launch “Options” dialog

From Tools->Attach menu option, launch “Attach to Process” dialog
Enter IP address of the Production PC in the “Qualifier” edit and click Refresh button
Select the process to Debug

• The symbols associated with the modules of the debugged application will be from the symbol server. This can be confirmed from the “Modules” window. 
• To debug, in the debugger, select break all. In the Threads window, navigate thru threads and their call stacks. Double click on the interested call stack frame to download the code.
• Debugger will automatically download appropriate source files from archive as shown below and load it in the debugger.
• Breakpoints can be set in the already opened source files.
•  When an exception occurs, the debugger will catch the first chance exception and break. The correct source file will be downloaded from the archive and loaded into the debugger.

The file gets loaded in VS editor. Note the path of the downloaded file.

SVN is internally supported by Microsoft. See this  post for supporting another source archive such as IBM's RTC, not supported by Microsoft.