Running Custom applications using task scheduler when the system is idle

Many times  it is necessary to run background tasks when the system is idle. 

For example, perform background encryption / decryption  using manage-bde tool on volumes containing data which could not be completed during installation.

In such cases the task scheduler can be used to schedule a task under the system account, upon OnIdle condition (when the system becomes idle with no user inter action or no background process running), after a delay say 1 minute. This task may launch a batch script start_idle_task.cmd.  As this runs under the system account, it'll run even none is logged in.  

When the system goes out of idle say due to an user action, this task is immediately terminated if it's still running. 

In the some cases, when the on idle task is terminated, it's desirable to take an action such as run another task. As windows OS provides no direct way to achieve this, it can be implemented by tracking the process termination of start_idle_task.cmd using WMI class Win32_ProcessStopTrace in a different process and launch the new task.

Example:

Create a batch file start_idle_task.cmd as below. This batch file will be started when system becomes idle and trigger ProcessWatcher.exe when it's terminated by user action.

rem processwatcher will trigger  when idle_task.cmd is terminated due to user action
start D:\Github\TechBlog\IdleTask\Scripts\ProcessWatcher.exe
echo started %date% %time% >> c:\temp\idletask.log
echo started %date% %time% >> c:\temp\idletask.log
rem run any command to start when system is idle
rem start manage-bde -resume d:
rem wait for the task scheduler to kill
pause       

Create a task scheduler task that will be launched when system becomes idle. This task will run under system account and run start_idle_task.cmd.

schtasks /create /f /sc onidle /i 1 /tn idletaskstart /tr "D:\Github\TechBlog\IdleTask\Scripts\start_idle_task.cmd" /ru system

ProcessWatcher.exe

Create a winform project. Remove Form1 and add following lines in Program.cs. This will wait for termination of start_idle_task.cmd by the system.  Then execute customaction() and terminate.

       
    static class Program
    {
        static ManagementEventWatcher processStopEvent;
        static void customaction()
        {
            System.IO.File.AppendAllText(@"c:\temp\processwatcher.txt", "ended " + System.DateTime.Now.ToString() + "\n");
        }
        static void processStopEvent_EventArrived(object sender, EventArrivedEventArgs e)
        {
            try
            {
                customaction();
                processStopEvent.Stop();
                processStopEvent.EventArrived -= processStopEvent_EventArrived;
                Process.GetCurrentProcess().Kill();
            }
            catch
            {
            }
        }

        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            int parentpid = 0;
            using (var query = new ManagementObjectSearcher(
                "SELECT * " +
                "FROM Win32_Process " +
                "WHERE ProcessId=" + System.Diagnostics.Process.GetCurrentProcess().Id))
            {
                parentpid = query
                    .Get()
                    .OfType<ManagementObject>()
                    .Select(p => System.Diagnostics.Process.GetProcessById((int)(uint)p["ParentProcessId"]))
                    .FirstOrDefault().Id;
            };


            processStopEvent = new ManagementEventWatcher("SELECT * FROM Win32_ProcessStopTrace where ProcessID="+ parentpid);
            processStopEvent.EventArrived += processStopEvent_EventArrived;
            processStopEvent.Start();

            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            Application.Run();
        }
    }
}
       
 

Output

c:\temp\idletask.log

started 07-06-2022  4:19:21.85 

ended 07-06-2022 11:59:01

Source and Binaries can be found here.

Share large allocated memory across applications without duplication

In some client - server applications running on same box,  a large amount of data may need to be shared. For example, an image acquisition application sharing the image with its clients. This typically involves using file Mapping objects to share memory across processes. However sometimes this can be overbearing if only two processes are involved.  Instead ReadProcessMemory and WriteProcessMemory APIs can be used.

What this means is a Process A can share its array with Process B without doing  IPC except sharing its PID and address of the array. Debuggers exploit this..

In the demo below, Writer process fills an array and shares it with Reader process. The array is read by the reader process using  ReadProcessMemory api . Also the array is repopulated using WriteProcessMemory api and is available to Writer process.

Sequence

Reader process starts

Writer process starts

Writer Process: 

creates an array of 100000  in writer process

Writes 100000 bytes containing 1

signals Reader with process handle and array memory address

sleeps for 5 seconds

Reader Process: 

Opens array using writer process handle and array memory address

Reads 100000 bytes containing 1

Writes 100000 bytes containing 2

Writer Process: 

Reads 100000 bytes containing 2

Source and Binaries can be found here.

Use System Cache memory for your applications

Windows OS uses Cache memory set aside for faster IO with disk and disk. For example when an user launches an application  for the first time, it's loaded into cache memory. Next time when user launches the same application it loads faster since it's loaded from the cache memory instead from disk.

The size of cache is determined during system startup.

The red box in the screenshot below shows size of the cached files on the system.

The system cache is highly scalable and memory management is done by the cache manager in conjunction with memory manager. The following video describes internals for system cache.

User applications can also take advantage of this such that data can be directly written to cache memory and read from there.    The trick is to set FILE_ATTRIBUTE_TEMPORARY option during file creation. This will instruct cache manager to keep data in cache memory rather than write to disk.

It's also important the file readers should not try to open the file in anyway. This will make the file
contents to be written to disk. The alternative is to create a duplicate handle and use it to access the file.
Using overlapped IO for read and write ensures proper management of the file pointer by readers and writers.

The list of files in the screenshot below are the actual cached files of  the data files from an acquisition.  The red box indicates that 0 bytes are committed to the disk for the all the cached data files. Similarly the cyan colored rectangle indicates the actual sizes of the selected data file.

Example:

In this example, 100000 bytes are written by the writer process to the system cache via temporary file  as in the first screenshot and the same is read from the system cache by the reader process as in the second screenshot. From the third screenshot we can notice that all it stays in system cache memory and nothing is committed to disk.

Sequence

Reader process starts

Writer process starts

Writer Process: 

creates test1.dat on system cache

Writes 100000 bytes containing 1

signals Reader with duplicate file handle

sleeps for 5 seconds

Reader Process: 

Opens test1.dat on system cache using duplicate file handle

Reads 100000 bytes containing 1

Writes 100000 bytes containing 2

Writer Process:

Reads 100000 bytes containing 2

Source and Binaries can be found here.

Simple Managed Inter Process Communication(IPC) Framework

If the need is just to do a cross process communication within the box, it'd be bit heavy to use frameworks such as WCF as it comes with a learning curve and complex setup. However  home grown solutions such as messaged based implementation suffer from flexibility and heavy maintenance.

 The simpleIPC framework tries to strike right balance with the interface based programming coupled with zero setup.

IPC Overview

A lightweight Inter Process Communication (IPC) across process within the same PC in .Net can be implemented using named event object or Window kernel object and shared memory.

• Interface based programming
• Duplex communication support
• Servers are identified by a unique string to which clients can generate a proxy to communicate. Some of the features:
• Servers/ Clients generate Stubs/ Proxy based on an interface for communication using reflection.
• As the proxy is based on the RealProxy object, intellisense is also supported in the Visual Studio IDE editor. Easy to debug, just a single file of code.
• To access a managed server from an unmanaged client, reversePinvoke can be used.

As shown in  the diagram above. IPC consists of a Client, Proxy, Stub and Server as described below

• Client uses proxy based on RealProxy to make API call
• Proxy serializes the input data using binary formatter to shared memory and signals stub
• Stub deserializes the input data using binary formatter from shared memory and makes the call on the server using reflection
• Stub serializes the results to shared memory and signals Proxy
• Proxy deserializes the results from shared memory and returns it to the client

Implementation

Comes in two flavors windows kernel object based or named event kernel object based.

Windows based uses a window kernel object for implementing the server

As shown above multiple Windows are hosted in a thread. Each thread hosts a server. 

• A Window based container can supports multiple servers per thread
• A window based server cannot work across window stations

Named object based uses a named event object for implementing the server.

• A Named event object based container supports only one server
• A named event object based server can work across window stations

Example

The interface  ICallInterface shown below is implemented by the server and ICallbackInterface is implemented by the client.

namespace Example
{
    [Serializable]
    public class regdata
    {
        public string name;
        public string regid;
    }

    public interface ICallInterface
    {
        string current { get; }
        string register(string name, string cbservername, out int ticket);
    }

    public interface ICallbackInterface
    {
        void update(int ticket, regdata data);
    }

}

Windows Simple IPC  Demo

Server Process

Simple IPC Server name for ICallback : winserver

Client Process

Simple IPC server name for ICallbackInterface : winclient

Named Object Demo

Server Process

Simple IPC Server name for ICallback : namederver

Client Process

Simple IPC server name for ICallbackInterface :  namedclient

Operation

Server process starts

client process starts

Client:

creates proxy for server

calls register() on server using proxy

Server:

executes method register() 

server returns success with Ticket #100

Client:

results are received from proxy : success, Ticket:100

Server:

creates proxy for client implementing callback interface

calls update() method on the proxy

. 

Client:

executes update() method

Source and Binaries can be found here.

Unattended System monitoring with ProcessMonitor and UI automation

ProcessMonitor from Microsoft is hugely popular and widely used to monitor events such as registry or file related updates. 

If Process monitor is used to track an event  (e.g., a registry change) system wide, in an unattended scenario, some customization will be needed since ProcessMonitor  creates a large amount of log files in a short span of time i.e., around 5gb in 30 minutes. 

The plan is to check Process monitor log files every 30 minutes for the event, take a snapshot and clear the logs. Using a task scheduler task and UIA this can be accomplished as discussed below.

Background

It was noticed that  on a particular windows 10 machine, the windows scaling factor changed from 100% to 150% randomly over a period of 1 week. The plan was to track changes made to the registry entry hkcu\control panel\desktop\logpixels.

Implementation

The solution should be generic so that it can be deployed seamlessly on different PCs.

When done manually, following steps are executed.

Steps

1. First time, folders for the pm logs and captured events are created. This is done by importing  the configurations from ProcMonConfiguration.pmc.
2. After that, when a specific user logs in, the unprocessed  logs are scanned for the event and cleared
3. Every 30 mins logs are scanned for the event, logs are saved if found and then logs are cleared

Step 1

 Init.cmd will do the initial setup.

Add a filter  for registry entry as shown below:

Change file backing as shown below.  A dedicated folder in required.

A scheduler task startpm is created that runs on logon of specific user to  run startpm.cmd described below.

A scheduler task savepmlog  is created that runs startpm.cmd for a specific user every 30 minutes to check for the event , take a screenshot and  clear the logs.

Step 2

startpm task is executed on logon of a specific user. This tasks runs savepmlog.cmd.

Step 3

savepmlog  task is executed every 30 mins after logon of a specific user. This tasks runs savepmlog.cmd.

Init.cmd

This script will do initial setup as discussed above.

       
@echo on
rd /s /q c:\temp\procmon
md c:\temp\procmon\capture
md c:\temp\procmon\pmlogs
schtasks /create  /f /sc onlogon /tn startpm /it /ru rvvya /tr "%~dp0\savepmlog.cmd"
schtasks /create  /f /sc minute /mo 30  /tn savepmlog /it /ru rvvya /tr "%~dp0\savepmlog.cmd"
start "" "%~dp0\..\bin\Procmon64.exe" /accepteula /terminate
start "" "%~dp0\..\bin\Procmon64.exe" /quiet /loadconfig "%~dp0\ProcmonConfiguration.pmc"
       
 

savepm.cmd

This script will check logs for the event and saves it along with a screenshot if found and then clears all logs.

       
@echo on
setlocal enabledelayedexpansion
del C:\temp\procmon\capture\snapshot.bmp>nul
del C:\temp\procmon\capture\logfile.csv>nul

tasklist /fi "imagename eq Procmon64.exe" | find /i "Procmon64.exe">nul
if !errorlevel! equ 0 start "" /wait "%~dp0\..\bin\Procmon64.exe" /Terminate
start "" "%~dp0\..\bin\Procmon64.exe" /quiet /openlog C:\temp\procmon\pmlogs\uiatest.PML
echo saving....
start "" /min /wait "%~dp0\..\bin\savelog.exe"
call :rename_logfile
goto :eof

:rename_logfile
set logfile=logfile_%date%_%time%
set logfile=%logfile::=_%
set logfile=%logfile:/=_%
find /c /v ""  C:\temp\procmon\capture\logfile.CSV | find /i ".CSV: 1">nul
if !errorlevel! equ 0 (
del C:\temp\procmon\capture\snapshot.bmp>nul
del C:\temp\procmon\capture\logfile.csv>nul
echo %date% %time% not found >> C:\temp\procmon\capture\results.log
) else (
move C:\temp\procmon\capture\snapshot.bmp "C:\temp\procmon\capture\%logfile%".bmp
move C:\temp\procmon\capture\logfile.csv "C:\temp\procmon\capture\%logfile%".csv
echo %date% %time% found >> C:\temp\procmon\capture\results.log
)
exit /b
      
 

Savelog.exe

This is driven by UIA. This executable is launched by savepmlog.cmd. It takes a snapshot, saves pmlog files and then clears them. 

Deployment

The solution is deployed as shown below:

\bin

ui automation executables and the libraries and processminitor executable.

\scripts

contains task scheduler scripts described below and setup script

Demo

The following movie shows actual operation

1. Regedit is started to add an event

2. savepmlog task is triggered to check and record.

Output 

The  output files are available in C:\temp\procmon\capture folder.

snapshot (logfile_11-06-2022_15_26_36.34.bmp)

saved log file (logfile_11-06-2022_15_26_36.34.csv)

"Time of Day","Process Name","PID","Operation","Path","Result","Detail"

"15:26:06.7849475","regedit.exe","10344","RegQueryValue","HKCU\Control Panel\Desktop\LogPixels","SUCCESS","Type: REG_DWORD, Length: 4, Data: 150"

results.log

11-06-2022 14:32:30.01 not found 

11-06-2022 15:00:22.52 not found 

11-06-2022 15:26:36.36 found 

Source and Binaries can be found here.

User Interface Automation Framework

The Microsoft UI Automation APIs enable navigation of user interfaces programmatically.

Some of the areas of application are UI Automation testing,  manufacturing Quality checks, Accessibility,

The UIAFramework  is a light weight framework that can be used to achieve UI Automation.either by Scripting languages or C#. It has two components ControlDBTool and UIADriver. The UIAFramework is based on .net CLR 3.5 and works with Windows XP and above. The process involves two steps.

1. Identify controls participating in UI automation in an user interface and generate a control database using ControlDBTool.
2. Using the UIADriver component, manipulate the control by invoking a method  or property from the supported control patterns of the control. For e.g., Invoke method of Button control.

ControlDBTool

ControlDBTool is a GUI based tool that aids identification of UI elements that participates in UI automation. The users can basically select UI elements from an application and save them into a file based repository. ControlDBTool also supports record and play feature where code can be generated from user actions.

UIADriver

UIADriver is an easy to use managed class library that provides an interface to simulate user interactions on various UI elements such as Buttons, Grids, combo boxes etc. This is exposed to script based and unmanaged clients via COM.
UIADriver supports both UIA (user interface automation) and MSAA (Microsoft Accessibility Api) technologies. These basically provide the foundation for UI automation and are part of the OS. UIA is the successor of MSAA and can be used for both WPF and Win32/Winform based UI controls.  However, it exhibits anomalies with certain winform based controls. In such cases, MSAA can be used to fill the gap.
The code generator of the ControlDBTool discussed earlier uses the UIADriver library to perform UI operations.

Example

In this example, calucalator.exe is launched and  multiplication calculations are performed as discussed below.

1. Launch calucalator 

2. Launch ControlDBtool and set toolbar only mode

.

3. Click on record button and perform following actions on the calculator using mouse

4. click digits 8 7 8 4 x 1 6 x 1 0 4 0 as shown in the below

5. stop recording the controldb tool 

6. save the recording

7. recorded actions are listed as below. if there is any unmapped event, it needs to be mapped to the action.

8. set  controldb tool to regular mode

9. save the recording

10. click on generate code and save the selection

11. select all the actions

12. select C# code from the dropdown

13. Save the code to calc.cs

14. edit calc.cs file and add highlighted code in main() and save.

	   
public static void Main()
{
	string brs = null;
	System.Diagnostics.Process.Start("calc.exe");
	System.Threading.Thread.Sleep(2000);

	UIAAutomationElement.UIADriver.SetAutomationElement (objsample.button_Clear_clearButton_1294_432);15. open a command window from the saved location

16. compile the code to generate exe using following command

17. execute runcalc.exe. the playback looks as in the video below

Source and Binaries can be found here.

File Splitter and Merger Tool for Transmission

What's File Splitter and Merger Tool?

It's an command line utility to truncate a big file into small pieces, later to be assembled.
Useful for sending big attachment files in mails, copying a big file into floppies etc.

How does it work?

During Disassemble, input for the source file, destination folder and truncation size are accepted. Later the file is split into multiple files with the name filrname.nn where n is the file number in the destination folder.  e.g., Test.data.00,  Test.data.01 etc

During Assemble, input for the destination file that will be assembled, source folder that contains truncated files are  accepted. Later the destination file that will be assembled back from the multiple  truncated files from source folder .

How to use it?
let's say if we want to break a big file d:\data\data.mdb of 10MB into eleven(11) 950K files:
open a Command window and type the command below.

FileSplitterMerger D d:\data\data.mdb  950000

you can see that 11 files d:\data\data.mdb.00 ..... d:\data\data.mdb.10  are created.

To assemble back, open a Command window and type 
FileSplitterMerger A  d:\data\data.mdb  
you can see the d:\data\data.mdb created after the merge

Source and Binaries can be found here.

Script to Zip and Unzip Files using Window shell

Often times applications require files to be zipped to a .zip file and extract files from the same. makezip_unzip.vbs  uses Windows shell do this easily.

Create Zip File Example

1. Arrange all folders, sub folders and files matching zip file structure inside a "stage" folder. Example, d:\temp\reports
2. Execute  in command window

           makezip_unzip.vbs   "d:\temp\reports"   "d:\reports.zip"

if not exist, d:\reports.zip will be created.

Extract from Zip file Example

Execute in command window

           makezip_unzip.vbs    "d:\reports.zip"   "d:\temp\reports"

If not exist, d:\temp\reports folder will be created to receive files from the .zip file. 

Note that full path must be supplied to the arguments.

Source and Binaries can be found here.

Innovative File organizer with Duplicates finder and Hierarchical viewer

No matter how big the hard drive's capacity gets, the prevalent issue of lack of storage remains.

When there is an urgent need to free up space, getting a fair idea of  the storage distribution becomes paramount.

FileOrganiser tool attempts to fill the gap by providing a hierarchical  view of the storage distribution along with options to sort data in multiple ways.

File organizer with Duplicate file finder has following features.

Folder View

Lists folders and files in a single screen hierarchal order along with their disk sizes and file count.

• The files can be viewed hierarchically  and can be sorted by File count, File Size and File Names.
• The file size can be viewed in different disk size units such as KB, MB and GB
• Selected Files can be deleted or path can be exported to file or copied to another folder.
• Files can be selected individually or entire folder can be selected using toggle selection
• Width of file display can be changed

Duplicates View

Multiple copies of the same file such as pictures gets spread across the disk. It also becomes important to keep track from security Point of View.

The user interface is same as compare folder view discussed below except duplicates are searched in the same folder.

Compare Folders View

Compares two different folders and lists duplicate files in hierarchical order  along with individual files.

Duplicate Hierarchical View

Duplicate Flat View

Non Duplicate Hierarchical view

• The duplicates files can be viewed hierarchically  or flat and can be sorted by File count, File Size and File Names.
• The file size can be viewed in different disk size units such as KB, MB and GB
• Selected Files can be deleted or path can be exported to file or copied to another folder.
• Files can be selected individually or entire folder can be selected using toggle selection
• Width of file display can be changed

Export

Exports files along with their unique MD2 values. Useful for large directories where faster processing can be done using SQL based queries.

Individual entries are separated by |

Source and Binaries can be found here.

A Backup, Restore, and Update Tool with a refined approach

Let's face it, over the period number of files grow, change, removed, copied and moved etc. Keeping track of these will be a challenge unless they are backed up and tracked to restore when needed.

The Backup and Restore Tool discussed here has following features:

• Create Backup collections and add / remove folders that will be backed up.
• Add exclude folder list such as system folders. e.g., c:\window
• Track files by CRC so that only a single copy of that file is stored.
• Files are backed up to a password protected archive.
• Each archive's metadata is also password protected.
• During backup, an integrated view is presented where each folder is indicated as archived and partial , modified or changed.
• During backup, an integrated view is presented where each file is indicated as archived and modified 
• During restore, an integrated view is presented where each folder is indicated as same, deleted and changed 
• During restore, an integrated view is presented where each file is indicated as same, deleted and changed 

Main

The following diagram illustrates the backup and Restore tool.

New

Creates new archive. The GUI is shown below.

Operation

1. Enter a name for the archive. e.g., TechBlog
2. select a location  for the archive. A new folder will be created under this folder to store contents of the archive. c:\archive\TechBlog.
3. Enter a description for the archive
4. Select folders to backup. e.g., d:\techblog. The checkbox of the folder to be backed up should be checked to take the backup.
5. Folders that needs to be excluded from backup such as system folder can be specified in exclude.txt under the collection folder.
6. After Clicking OK, user is asked to enter a password for the archive. This password is used for storing metadata of the archive.

Edit

Once the archive is created, edit can be used to add additional folders to backup or select deselect folders for taking backup.

Backup Tool

A collection is a set of folders that needs to be archived in multiple backups. Each backup contains a snapshot of the archived folders at the time backup was taken. 

Operation

1. First select folders to backup as discussed above and Click Backup button
2. An integrated view is presented to select files/folders and status of the file and folder items. 

The status of a file item can be as below.

• New - New file. Never backed up.
• Changed - Existing file changed after last backup. 
• Archived - Existing file that didn't change since last backup.

Similarly the status of a folder item can be as below.

• New New Folder. Never backed up.
• Partial Existing folder structure changed since last backup. 
• Archived  No changes since last backup
• 
  

3. Finally, Check the checkboxes to choose the files that needs to be backed up. Click Backup Now button to start taking backups. The backups will be saved into a internally generated password protected 7zip file. 

 Restore Tool

The restore tool can be used to restore files from previously archived backup.

Operation

Select a backup from the dropdown list to restore.

Files are synched with disk if "sync with folders" is checked. This will compare the crc of the existing files with the previous backups and determine which files needed to be restored. 

Each file can have three states -  Same, Changed or Deleted.

• Same means the file has not changed hence no restoration is needed.
• Changed means the file has changed hence restoration is needed
• Deleted means the file is deleted  hence restoration is needed

In the same way, A folder can have one of the three states -  Same, Changed or Deleted.

• Same means the folder has not changed.
• Changed means one or more files in the folder has changed hence restoration is needed.
• Deleted means the entire folder is deleted  hence restoration is needed.

By default all the files from the previous backups are listed. If  changes only is checked, only files from the selected backup are listed.

If Hide Same is checked, all the unchanged items are removed from the display.

Export exports file  items listed to a csv file along with the crc.

Refresh Resyncs with disk for latest changes.

Finally, Check the checkboxes to choose the files that needs to be restored. Click Restore button to restore. As shown below, the files can be saved to a different folder or to the same original folder where they existed.

 Update Tool

It's a hybrid of Backup Tool and Restore Tool. The Update tool can be used to update previously archived backup. Folders and Files can be added and removed from the archive without having to use Backup Tool. This is helpful incase the original backup is not available.

Operation

Select a backup

Expand the tree and select the folder that receives files

Drag and drop files. after acknowledgement  files will be added automatically.

Source and Binaries can be found here.