QStar provides a set of software products which let you monitor HPC file systems and archive only those files which have changed during the latest monitoring period. Selection of the files to be processed is driven by flexible policies based on standard file attributes such as modification and access times, file size, content, ownership, names and extensions. Obviously the first time the system is installed it takes a while to build a base copy of the file system, but in subsequent periods only files that satisfy the policy are processed.
The above approach provides huge advantages when compared to a full backup in terms of archive capacity and performance.
Archiving performance in many respects depends on the type of HPC file system because it affects how the monitoring can be achieved. Scanning the whole file system for changes is highly inefficient, but if the file system does not provide any better tools that is the only route possible. Some advanced file systems have special interfaces to monitor file system status such as the DMAPI interface (IBM Spectrum Scale and HyperFs), although such architecture is not widely supported and leads to degradation of system performance.
The latest trend in file system design is to provide a change journal which can be used to record information about file system changes without involving user level DMAPI processing. Microsoft NTFS, Lustre and some other file systems offer this feature enabling others to create efficient file system monitoring software.
QStar QNM supports many HPC file systems and performs monitoring in the most efficient way afforded by the file system.