Truly Portable Patch Tool and file comparer

In the diversified development environment of today, multiple code repositories are used by cross development teams. As each have an indigenous way to deal with patches, a generic method of creating and applying patches is much desired along with recursive folder comparison and viewing the file differences and other operations. The portable patch tool attempts to address these.

Portable patch tool has following features.

Create Patch

1. Create patches between two folders
2. Filter out special folders and files
3. Include additional files
4. Create patches from a file list
5. Include source files in the patch
6. Work with any file type – text or binary

Apply Patch

1. Preview changed files hierarchically before applying the patch
2. Apply changed files in patch selectively
3. Files that are changed are flagged

Compare Folders

1. Recursively compare folders such that files are flagged as unique left or right and different
2. Options to view files and sort based on directory name, extension etc
3. Options to delete and copy files
4. Export file list
5. Options to  view only left , right or diff files
6. Options to filter files including  and excluding patterns
7. Export/Import/Extract    

Export/Import/Extract

Export patch files to a .ZIP file

Import patch files from zip file

Extract contents of a patch file

Create patch

A Patch contains differences between two folders. It can also contain additional files.

“Source Folder” contains the code downloaded from the source repository.

“Updated folder” contains developer changes that needs to be reviewed

“Patch Archive” is a temporary folder hosting the patches

Options

The “Options” dialog gives additional options as discussed below:

Typically source control repositories use a hidden folder or file name such as .svn or .copyareadb or .git etc. These need not be included in a patch. Users can specify all such file extensions or folder names separated by a semicolon in “Exclude All”.

As mentioned earlier, the patch contains only files that are different. Additional files from “updated folder” can be added to or removed from patch using + or – buttons. The list of additional files are stored as additional_input.txt in the patch folder. In addition, a list of files can be inputted via text file using … button. This file can be exported in “Compare Folders” tab discussed below.

Typically the patch contains the files from “Updated Folder”. The “Include Source” option enables including corresponding files from “Source Folder” as well.

By default, every file that differs is included in the patch. Instead users can also provide a list of files to be included in a patch in the “Import File List”. Again this file can be exported in “Compare Folders” tab discussed below.

The differences of the files can be viewed by specifying the application in “File difference viewer”. The Arguments <File 1> and <File 2> must be specified along with others if any. They are replaced by actual file names when passed as command line arguments to the application.

Finally a patch is created by clicking the “Create” button. A patch consist of three files – patch_meta.csv containing the metdata of the files in the patch. Patch_data.dat containing the raw patch files and Patch_source.dat containing the files from the |source folder” 

Apply patch

Patches created in “Patch Archive” folder can be applied to the contents of the “Target Folder”.

Before applying the patch, changes can be previewed by clicking “Preview” button. This brings up GUI as shown below:

Preview

Patches are applied by clicking “Apply” button. This brings up GUI as shown below.Files are displayed hierarchically.  

The "Diff” branch contains files that are different. “New” contains files that are new.

If “Winmerge” is installed, changes of any file under “Diff” can be viewed by double clicking on it.  Typing “Ctrl+C” copies the file name to clapboard.

Clicking on any file under “New” loads it in “Notepad”.

Users can filter a specific file extension in “Filter Extensions”. Multiple file extensions can be separated by a semicolon.  All possible file extensions available in the file set are also listed,

+ / _ closes or expand tree.

“Merged View” combines “Diff” and “New” branches to present a unified view.

Apply

This displays a flattened view of the files “AutoMerge” column indicates weather Patch needs to be manually merged or not. True – means no and false means yes. In this case file can be manually merged using tool such as Winmerge.

The “Remark” column has three values – “Diff” files are different. “Same” files are same. “New” the file is new.

The files can be sorted by clicking on any column header. This is listed in “Sort Columns”. The sorting can be cleared by clicking “Clear” button.

“Select All” selects all files.

Filter

If the file set is huge, filtering of the files can be done by clicking the “Filter” button which brings up following GUI.

“Exclude All” option excludes all file containing the file pattern. And “Include Only” includes only files containing the file pattern.

Predefined patterns “Reset” removes any filtering. “Diff”  ,“New” and “Same”  shows files as in the “Remark” column discussed above.

Finally files can be marked for applying. If the file list is huge, files can be selected based on the filtering. It’s possible that a certain filtering may not show previously selected files. In such cases “Show selected” options lists only the selected files overriding filtering.

“Apply Patch” button applies the patches. If “automerge” for a file is True, then the file in the “Target Folder” is overwritten. If the “Remark” is New, the file is copied into the “Target Folder”.

If the “automerge” is False and Remark is Diff, a new file is created appending “_new” to the file name. Sometimes this is undesirable so in such cases, those files must be manually merged and selected out from patching. In other words, patching should be done to “automerge”  True or New files only.

Compare Folders

Two folders can be compared for differences in “Compare Folders”.

The “Left” and “Right” point to two folders with similar file structure. After comparison, a file can be categorized as “Diff” - file that exists in both folders but different, “Left” file only in Left folder. “Right” only in right folder.

Options

The “Options” dialog gives additional options as discussed below:

Typically source control repositories use a hidden folder or file name such as .svn or .copyareadb  or .git etc. These need not be included.

Users can specify all such file extensions or folder names separated by a semicolon in “Exclude All”.

The results of comparison are as shown below.

The “Remark” column has three values – “Diff” files are different. “Left” unique files. “Right” unique files. 

The files can be sorted by clicking on any column header. This is listed in “Sort Columns”. The sorting can be cleared by clicking “Clear” button.

“Select All” selects all files.

Filter

If the file set is huge, filtering of the files can be done by clicking the “Filter” button which brings up following GUI.

“Exclude All” option excludes all file containing the file pattern. And “Include Only” includes only files containing the file pattern.

Predefined patterns “Reset” removes any filtering. “Diff”, “Left” and “Right” files as in the “Remark” column discussed above.

Before applying the patch, changes can be previewed by clicking “Preview” button. This brings up GUI as shown below:

Preview

If the file list is huge, files can be selected based on the filtering. It’s possible that a certain filtering may not show previously selected files. In such cases “Show selected” options lists only the selected files overriding filtering.

The selected files can be exported using “Export” option to a txt file. This file can be later used to specify an additional file or input file during patch creation discussed earlier.

The “Delete” option deletes selected files.

The “Copy Left” option copies the selected files to the “Right Folder”. Similarly “Copy Right” option copies the selected files to the “Left Folder”.

The read only attribute of file can be also removed before delete or copy operation.

Export / Import / Extract

As a Patch contain multiple files, it  can be exported to a .zip file which can be imported.

 

“Patch archive” indicates folder that is used for export or import.

Export

Brings up a Save as dialog to select the name of the zip file to save the contents of the patch.

Import

Brings up a Select Files dialog to select the name of the zip file and extract its contents to the patch archive folder.

Extract

The contents of patch can be extracted using “Extract” option, The differences are written to “left” folder the archive and if the source files are included in the patch, they are written to “right folder also.

Examples

The following discusses some of the examples.

Deploy Patches

1. At the source the patch can be created as discussed in “Create Patch”
2. The patch can be exported as discussed in “Export/Import/Extract”
3. At the destination the patch can be imported as discussed in “Export/Import/Extract”
4. The patch can be applied as discussed in “Apply Patch”.

Code Review

1. At the source the patch can be created as discussed in “Create Patch”
2. The patch can be exported as discussed in “Export/Import/Extract”
3. At the destination the patch can be imported and extracted as discussed in “Export/Import/Extract”
4. The reviewer views the changes and even adds review comments
5. Creates a patch and exports the patch back to the sender

Code Rebase

During development of a feature, changes are not checked in to the repository  for some time while the repository gets updated by others developers. In such cases the feature developer can stay updated as below.

1. Download latest code to repository workspace
2. Compare it with developer workspace as discussed in “Compare Folder”
3. Manually merge changes if necessary else overwrite.
4. Add in coming files and remove deleted files at source

Source and Binaries can be found here.

Visual Studio Migration Tool

During a project's life cycle, moving the code base to the next version of visual studio becomes necessary to take advantage of newer features, This can be daunting task if there are a large number of projects.

The "Migration Helper" utility makes it a breeze as described below.

• Updates project settings ((PlatfromToolSet and Framework version) ) and solution settings (Visual studio version) to a newer version of visual studio 
• Generates a report containing settings of all the projects and solutions to analyze the results of migration. 
• Generates a report describing the CLR and Version of all the DLLs generated after successful build. 
• Generates a report analyzing each managed executable 's references to detect potential "mixed mode" situations.

The Migration Helper tool is divided into three tabs as shown below
The "Source Folder" specifies the root of the project folder. All the actions recursively  scans directories looking for projects and solutions. The "Output Folder" houses  reports generated.

List Tab

List

This recursively scans  for projects and solutions listing the project and solution settings as shown below.

Runtimes

This action should be invoked after the  build,, This recursively scans  through  all the DLLs and lists the DLL version and CLR version (if managed) as shown below. The  "ildasmpath" setting of of MigrationHelper.exe.config should point to the ildasm.exe. Similarly "dumpbinpath" should point to dumpbin.exe.

Dependencies

During migration, when some of the managed executables move to a newer CLR (> 4.xxx),, they might still reference assemblies from an older CLR (< 4.xxx). This leads to "Mixed mode" and the exe's app.config should address this.
Identifying such executables can be daunting as the references must be recursively scanned for dependencies. This is made simple by Dependencies option. This scans through all the executables and their references recursively and tries to load. Any errors related to "Mixed mode" are captured and reported as shown below. A detailed map of the Executables and their references is also generated,.

D:\xxx\yyyyy.vcxproj,exe
Mixed mode assembly is built against version 'v1.1.4322' of the runtime and cannot be loaded in the 4.0 runtime without additional configuration information.

VS Upgrade Tab

The Visual Studio provides an option to upgrade projects and solutions by invoking devenv.exe with upgrade option. The path to the devenv.exe pointed by "devenvpath" setting of MigrationHelper.exe.config. Be warned that migration may not succeed in some cases. 

Run

Based on the selection - solutions or projects, the source folder is scanned recursively for .slns or .*Proj files.  Upgrade is attempted for selected files and reports are generated  If the migration was successful,  a detailed information of the migration can be obtained from  upgradelog.htm  generated by dev env for for each project or solution.

VS Update Tab

This is similar to VS Upgrade except the solution and project files are updated. 

Run

The source folder is scanned recursively for .slns or .*Proj files.  Upgrade is attempted for selected files as described below.
To update a solution, the settings  "FormatVersion", "VSName", "MinimumVisualStudioVersion" , and "VisualStudioVersion" from MigrationHelper.exe.config are used.
To update a project, the settings "ToolsVersion","TargetFrameworkVersion" and "PlatformToolset" from MigrationHelper.exe.config are used.

Reports are generated  as shown below.

The following displays the sample settings.

       
<appSettings>
    <add key="devenvpath" value="C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe"/>
    <add key="ildasmpath" value="C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\ildasm.exe"/>
    <add key="dumpbinpath" value="C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\dumpbin.exe"/>
    <add key="TargetFrameworkVersion" value="v4.5.2"/>
    <add key="PlatformToolset" value="v140"/>
    <add key="ToolsVersion" value="14.0"/>
    <add key="FormatVersion" value="12.00"/>
    <add key="VSName" value="14"/>
    <add key="VisualStudioVersion" value="14.0.24720.0"/>
    <add key="MinimumVisualStudioVersion" value="10.0.40219.1"/>
</appSettings>
       
 

Source and Binaries can be found here.

Source Indexer for RTC repository

As discussed in "Remote Debugging with Symbol Server and Source Indexing" post, Symbol Server along with Source Indexing greatly helps in debugging production issues independent of the installed software version. The source server was configured to work with svn repository.
The current Source Indexing setup from MS doesn't support RTC repository. Writing the same in Perl seems complicated for those unfamiliar with Perl.

Repository agnostic Recipe for Source Indexer Tool Development

Prerequisites

Install Debugging Tools for Windows or just copy files from an existing installation

Steps
1. Gather all the PDBs in a Symbol Server to avoid duplication.
2. Extract source files from PDBs using srctool -r command.
3. Create a source Server stream containing the command to extract the source file from the repository for the PDB.
4. Inject source stream into the PDB

Working with RTC Repository

It's assumed that the the system used for debugging is installed with RTC client. lcsm batch file is added to PATH environment variable.
A file can be extracted using get file command by invoking lscm.bat as below

lscm  get file -r <repository url> -s <snapshot name> -c <component name> -f <file path in the stream> -o <file path on the disk>

Important things to keep in mind are :

1. The user id and password needs to be cached using following command

lscm login -r <repository url> -c

2. It's assumed that a unique snapshot is created containing baselines after successful build
2. If possible use the unique identifier for components instead of the display text
3. The repository path is case sensitive and should include  / for separator
4. The get file command doesn't create the destination folder automatically, therefore it should be created before downloading
5. It's preferable to use component ids over components when querying RTC since the same component name can be assigned to multiple projects

In case working offline, the entire archive can be downloaded using following commands. Here a temporary workspace is created from the snapshot and later deleted after use.

lscm create workspace --snapshot <snapshot name> -r <repository url> mmmm

lscm load -r <repository url> -d <path on disk> mmmm <component name>

lscm delete workspace -r <repositiry url> mmmm

SourceIndexer Tool
Source Indexer source indexes PDBs for RTC repository.
In order to source index, following inputs are needed. These are obtained from the SourceIndexer.config

sourcefileslocation  ->contains root of the source files

pdblocation -> contains symbol store root of the build

lscmlocation -> path to lscm.bat

repository -> url of the repositiry

snapshot -> name of the snapshot

components   ->components display name separated by comma

componentids ->components unique ids separated by comma

Usage Example.

Configuration

       
    <appSettings>
        <add key="sourcefileslocation" value="D:\Github\TechBlog\COM+\APITester - Test any COM component using its typelibrary" />
        <add key="pdblocation" value="D:\Github\TechBlog\COM+\APITester - Test any COM component using its typelibrary\bin" />
        <add key="lscmlocation" value="C:\Program Files (x86)\IBM\TeamConcert\scmtools\eclipse\lscm.bat" />
        <add key="repository" value="https://rtc.vrrao.com:9443/ccm/" />
        <add key="snapshot" value="APITESTER_1969" />
        <add key="components" value="Source" />
        <add key="componentids" value="Khri$ha" />
    </appSettings>
       
 

After executing SourceInder.exe, following will be output:

The output folder under SourceIndexer has following files

files.txt -  contains of output of srctool command querying for source files from APITeser.pdb

APITester.pdb.ssidx - contains the stream that will be injected into the APITeser.pdb

Details

a. runsrctool command returns 380 files 

b. 31 files belonging to APITester project are selected. stream is created in APITester.pdb.ssidx file.

This file contains lscm get  command  for each of the selected 31 files of APITeter project.

example,

d:\github\techblog\com+\apitester - test any com component using its typelibrary\source\superedit.cpp*"https://rtc.vrrao.com:9443/ccm/"*"APITESTER_1969"*"Khri$ha"*SuperEdit.cpp*Source*

c. runpdbstr command  injects  APITester.pdb.ssidx  into  APITeser.pdb

d. This also can be verified by running command pdbstr.exe  -r -p:APITester.pdb -s:srcsrv

The output will be as shown below

Source and Binaries can be found here.

Setup for Remote Debugging with Symbol Server and Source Indexing

Troubleshooting production issues of an older release cause headache to developers. It's due to non availability of symbols and source code. Sometimes it'll be also hard to create these issues in development environment.

Some definitions:
PDB

• Program database aids debugger to debug applications
• A PDB file is generated as a result of building a dll or exe.
• It contains information about variables, functions etc 
• It also contains list of all the source files used for build
• A PDB has one to one relationship with its exe or dll 
• When a dll or exe is loaded, the debugger loads its PDB only when its matching PDB is found.

Symbol server

• A database of symbols (PDBs) generated from the build. 
• It contains the PDBs  corresponding to the various versions of the dll or .exe
• It enables the debugger to automatically map and load symbols of a particular version of dll or exe

Source Indexing

• It’s the process of injecting a command in the PDB to retrieve the source file from source control archive.
• Typically it contains a command to the archive to retrieve the file of a particular revision. 
• Therefore, it enables the debugger to automatically load the correct revision of the source code directly from the archive

Remote debugging

• Enables debugging of applications remotely in production like environments where debuggers, symbols, and source code are unavailable.
• With the aid of symbol server and source indexing, it’ll be possible to debug from developer box*any* build *without* source code or symbols on the developer box.

Setup

As shown above a typical setup consists of Build PC., Source Code archive PC. The developer PC should have access these two at the time of debugging. Alternatively, a mini symbol server and source code associated with it can be deployed onto   the developer PC eliminating dependency.
The VS Remote debugger uses DCOM to communicate with the production box.

Source Code Archive PC

Note it's assumed that the source code archive is svn. In this case it's https://riouxsvn.com/. It's also possible to use any other source code control. For example using RTC  server is discussed here.

Build PC Setup

• Install Debugging Tools for Windows which can be downloaded separately from Microsoft
• Install Active state Perl
• Create a directory to store symbols and share it. This will be referred as the symbol server.
• Edit svn.pm and srcsrv.ini files under C:\Program Files (x86)\Debugging Tools for Windows (x86)\srcsrv folder for source indexing. These typically involves providing the repository URL and credentials to access it,

Build PC Operation

After successful build, Execute following command to inject commands to open source files in PDBs by the debugger.

srcsrv\svnindex /debug /source=<path containing source files>  /symbols=< location of the pdbs>   /rev=<revision number>

Example:

srcsrv\svnindex /debug /source="D:\Github\TechBlog\COM+\APITester\Source" /symbols="D:\Github\TechBlog\COM+\APITester\bin" /rev=2

Execute following command to add the pdbs to symbol store

symstore add /r /f  <location of binaries and pdbs> /t <name of the product> /z pri /S <symbol server path>

Example:

 symstore add /r /f "D:\Github\TechBlog\COM+\APITester\bin" /t APITester  /z pri /S "D:\Github\TechBlog\COM+\APITesterSS"

The screen output is as shown below:

Share the symbol server 

Production PC Setup

The production PC should be setup for remote debugging and DCOM as discussed below.

• Install Remote Debugging component which can be downloaded separately from Microsoft
• Setup for DCOM
• Local Security Policy

• Network Adapter

• Disable Firewall or configure it to allow DCOM traffic for logged on user

Production PC  Operation

Run MSVS Monitor

• Create or update system environment variable _NT_SYMBOL_PATH and map it to the symbol store directory.

NOTE: For native code, symbols are first searched and loaded from the Developer pc where the debugger is running.

For managed code, the symbols are first searched in the remote machine where the debugged application is running.

Start the Application to debug

Developer PC Setup

The developer  PC should be setup for remote debugging and DCOM as discussed below.

• Local Security Policy

• Network Adapter

• In case Production PC belongs to a workgroup, in order to make DCOM to work, the developer PC should create the same user account with identical password as in production box and add it administrators group. In addition it should also use this account to remote debug. In case Production PC belongs to a domain in order to make DCOM to work, the developer PC should use this account to remote debug.
• Disable Firewall or configure it to allow DCOM traffic for debugging user

Developer PC Operation

Launch VS under the same  account (using Runas command) a  discussed above
Check “Enable Source server support” by selecting Debugging->General

Add a new entry pointing to symbol server location by selecting Debugging->Symbols

From Tools->Attach menu option, launch “Options” dialog

From Tools->Attach menu option, launch “Attach to Process” dialog
Enter IP address of the Production PC in the “Qualifier” edit and click Refresh button
Select the process to Debug

• The symbols associated with the modules of the debugged application will be from the symbol server. This can be confirmed from the “Modules” window. 
• To debug, in the debugger, select break all. In the Threads window, navigate thru threads and their call stacks. Double click on the interested call stack frame to download the code.
• Debugger will automatically download appropriate source files from archive as shown below and load it in the debugger.
• Breakpoints can be set in the already opened source files.
•  When an exception occurs, the debugger will catch the first chance exception and break. The correct source file will be downloaded from the archive and loaded into the debugger.

The file gets loaded in VS editor. Note the path of the downloaded file.

SVN is internally supported by Microsoft. See this  post for supporting another source archive such as IBM's RTC, not supported by Microsoft.