Use System Cache memory for your applications

Windows OS uses Cache memory set aside for faster IO with disk and disk. For example when an user launches an application  for the first time, it's loaded into cache memory. Next time when user launches the same application it loads faster since it's loaded from the cache memory instead from disk.

The size of cache is determined during system startup.

The red box in the screenshot below shows size of the cached files on the system.

The system cache is highly scalable and memory management is done by the cache manager in conjunction with memory manager. The following video describes internals for system cache.

User applications can also take advantage of this such that data can be directly written to cache memory and read from there.    The trick is to set FILE_ATTRIBUTE_TEMPORARY option during file creation. This will instruct cache manager to keep data in cache memory rather than write to disk.

It's also important the file readers should not try to open the file in anyway. This will make the file
contents to be written to disk. The alternative is to create a duplicate handle and use it to access the file.
Using overlapped IO for read and write ensures proper management of the file pointer by readers and writers.

The list of files in the screenshot below are the actual cached files of  the data files from an acquisition.  The red box indicates that 0 bytes are committed to the disk for the all the cached data files. Similarly the cyan colored rectangle indicates the actual sizes of the selected data file.

Example:

In this example, 100000 bytes are written by the writer process to the system cache via temporary file  as in the first screenshot and the same is read from the system cache by the reader process as in the second screenshot. From the third screenshot we can notice that all it stays in system cache memory and nothing is committed to disk.

Sequence

Reader process starts

Writer process starts

Writer Process: 

creates test1.dat on system cache

Writes 100000 bytes containing 1

signals Reader with duplicate file handle

sleeps for 5 seconds

Reader Process: 

Opens test1.dat on system cache using duplicate file handle

Reads 100000 bytes containing 1

Writes 100000 bytes containing 2

Writer Process:

Reads 100000 bytes containing 2

Source and Binaries can be found here.