Data Compression
Monday, February 20th, 2012The size of the FITS files generated by the mcrx component of Sunrise is daunting. With my current choice of settings, each mcrx.fits file weighs in at over 40GB. The files are so large because they contain a spectrum in each pixel of six separate images of the simulated galaxy. After doing more research, I may eventually be able to get away with storing fewer wavelengths, reducing the resolution of the images and/or reducing the number of images per file, but the files are still likely to be rather large. The mcrx component of Sunrise includes an option for using file compression, but the compression method is lossy and it’s not recommended for use. Fortunately, my FITS files are generated synthetically; unlike images from a real telescope, the dark parts of the images are noise-free! Therefore, the areas that appear blank are essentially all identical. The presence of many identical pixels makes the files potentially highly compressible via lossless compression algorithms. (more…)