Help Memory Error

Processing large data sets can lead to the error message "HELP MEMORY". For users of Matlab on a Windows 32 bit PC this error occurs if the if the size of an individual array exceeds the size of the largest contiguous free block of Memory. In practice this limits the size of data sets that can be held in memory to 1.5 GB.

When confronted with the error "HELP MEMORY" one can do a number of things:

  • choose a more efficient storage of data, for example int16 instead of double;
  • use sparse matrices instead of full matrices;
  • change the representation of data so that you do not need contiguous memory;
  • save part of the data on disk instead of memory;
  • switch to the 64 bit Matlab version.

Deflate Utility

The Modelit deflate utility compresses a matrix in memory. The contents of a compressed matrix can be accessed as any other matrix. Typically, compression reduces the contents with 85%. But matrices that contain highly repetitive data can be compressed to much smaller sizes.

Example

Verify Modelit deflate utility

Note: in this example the Matrix L compresses to about 12.5%. This is somewhat of a worst case because L is populated with fully random logicals. If instead L is populated with more repetitive data, the compression rate will increase to over 97%:

Main application

The deflate utility is used in TRIP. This package is used to prepare a historical database of traffic data for the on-line route planner/travel time predictor TripCast. Thanks to in memory compression, all traffic detector data for a full year can be kept in memory simultaneously. This greatly facilitates the necessary computations to check, supplement and aggregate these data.

Download source code

The source code of the deflated matrix utility is free to use and included in a zip archive:

modelit/ToolboxModelit/Modelit_20160406_deflate.zip

When publishing work involving the software, please cite as follows:

Zijpp, N.J. van der (2016), Modelit Deflated Matrix utility for Matlab (Version yyyymmdd) [Software]. Available from www.modelit.nl

(with yyyymmdd the creation date of the version you have been using)