Searching PDF files with pdfgrep

March 20th, 2013

Most people who are familiar with Unix-like systems are aware that the grep utility can search many text files for a particular string using regular expressions. For instance, the command

$ grep function_name *.cc *.h
searches all of the C++ source (.cc) and header (.h) files in the current directory for the string "function_name,"

$ grep -E '\\todo' *.cc *.h
searches the same files for "\todo," and

$ grep -E '^class' *.cc *.h
searches the same files for lines that begin with the keyword, "class."

Suppose you are writing a research paper or a proposal. In the process of preparing to write, you read and skimmed many articles in PDF format. Unless you kept very careful notes while you were reading—or you have an exceptionally detailed memory—you will probably have trouble finding certain details from the papers that you read. Google is often not terribly helpful in this situation and skimming through the articles again can be very time-consuming.  Luckily, if you have saved the relevant PDFs to your computer's hard drive, you can search through the files with a single command using pdfgrep. Unlike regular grep, pdfgrep is typically not installed by default. To install on a Debian-based GNU/Linux distribution, simply use

$ sudo apt-get install pdfgrep
Then search your PDFs. See the manual page for details.


$ pdfgrep 'tidal dwarf' *.pdf
searches all of the PDF files in the current directory for the string "tidal dwarf," while

$ pdfgrep 'clumpy dust' */*.pdf
searches all of the PDF files contained in sub-directories of the current directory for "clumpy dust."

Yes, there are other tools that can be used to organize PDFs, but pdfgrep is very convenient. You don't even have to rename your PDFs when you download them from a website. You can even search your web browser's cache directory or your download folder.

Preparing Paper #2 & GSnap Updates

February 7th, 2013

During the recent AAS meeting in Long Beach, CA, I gave a dissertation talk describing the current state of my project and the planned work. After the talk, Jong-Hak Woo,  and others asked some questions that motivated me to perform a bit more analysis. Over the next few days, I added several new features to GSnap in order to answer these questions quantitatively. In summary:
1) What is the probability of measuring \sigma_{*} above a given value when the two nuclei pass through one another?

In order to investigate this, I wrote a histogram routine that allows me to plot the probability of measuring \sigma_{*} in several velocity bins. In the upcoming paper,  I will present the probability distribution data computed by measuring \sigma_{*} along 10,000 random directions.
2) How does \sigma_{*} vary with viewing direction (i.e., viewing angle)?

I added the ability to measure velocity dispersion along any specific direction. This allowed GSnap to be scripted so that \sigma_{*} can be plotted as a function of the polar angle and azimuthal angle.
3) Measurements of velocity dispersion include both streaming, bulk flow motion and true velocity dispersion; how significant is the contribution of the streaming motion in the measurements of velocity dispersion shown my time series plots?

I derived an expression for the apparent velocity dispersion in terms of intrinsic velocity dispersion, relative mass, and relative (streaming) velocity. This allows me to separate the streaming and random components of the motion in special cases.

In addition to adding more velocity dispersion analysis features, many other new features have been added to GSnap. Most significantly:

  • The pixel brightness scaling curve is now more complicated than a simple gamma correction. This results in a much smother transition to black and therefore fewer ragged edges in the images (see images below).
  • Renderings of the gas component are no longer monochromatic. The user can specify an arbitrary color map. If a gradient is used for the color map, warm and cool gas can be distinguished from their colors (see images below).
  • A basic scripting language has been defined. GSnap can now read scripts and perform automated tasks while loading the snapshot file only once. Previously, the snapshot needed to be re-loaded for each different task.

NUMA Control and LibNUMA

December 6th, 2012

In order to take full advantage of the performance of a shared memory machine that uses Non-Uniform Memory Access (NUMA), you need to minimize communication between NUMA nodes; Threads on one NUMA node should refrain from accessing memory that belongs to another NUMA node unless it is absolutely necessary. In order to do this on a Linux system, there are two primary options:

1.  If you do not have access to the source code of the software that you are running or you don't have the time or the expertise necessary to modify the source code to improve its performance, the NUMA Control tool (numactrl) may be useful. With numactrl, you can specify which NUMA nodes a particular program should use and which specific CPU cores should be used. Example: Suppose a particular system has 6 CPUs per NUMA node, the command
 numactl --physcpubind=0-5 --membind=0 YourProgram options 
runs the program named YourProgram on CPUs 0-5 and instructs the operating system to allocate memory on NUMA node 0 (this assumes that CPUs 0-5 belong to node 0).

When is this useful?  If YourProgram was not optimized for NUMA machines, it may scale very poorly when more than one NUMA node is used. If, for example, YourProgram performs the same type of computations on individual files (analysis, for example), then you could simultaneously launch YourProgram on NUMA node 0 using CPUs 0-5, on node 1 using CPUs 6-11, and so on until you have used all of the nodes available in the system. Each instance of YourProgram would perform computations on a separate file, of course. The several separate instances of YourProgram running on separate NUMA nodes will be more efficient than one instance of the program that uses all of the nodes simultaneously in an inefficient manner.

2. If you are a developer of the code in question or you possess the source code and sufficient desire and programming skill, you can use the NUMA library (libnuma) to make the code NUMA-aware. Among other things, this allows you to perform memory allocations that are local to specific NUMA nodes. In addition to the man page linked above, there are several guides available online to get you started with libnuma; just do a Google search for "libnuma examples."  


Screen Brightness / Black Level

November 11th, 2012

Displaying images with a black background in a consistent way is difficult. Very few people bother to calibrate their monitors, so their screen brightness is usually too low or too high. As a result, the black levels of any two randomly selected display devices are typically quite different. Parts of an image that should be black appear grey when the brightness is too high, while parts of the image that contain dim pixels appear black if the brightness is too low. I have tried to find a set of image parameters in GSnap's rendering scheme that will display reasonably well on a wide variety of display devices, but I think I may have to try using a slightly different pixel value scaling rule eventually (at the moment, I just use a power law that is clipped above a certain maximum value). Here are three images that use the best combination of parameters that I've found thus far. I have not added noise to these images (as mentioned in the previous post)
Read the rest of this entry »

Adding Noise

September 24th, 2012

After viewing the images in the previous post on several computer monitors with varying brightness and contrast levels, I found that the "edges" of the galaxies were clearly visible on some displays. Adding noise to the images seems to help fix the issue. Tweaking the brightness and gamma settings in GSnap's parameter file can produce nice images for a specific display, but without noise, the image quality is less consistent on other displays.

Here are some "noise-enhanced" images. The noise was added using GIMP's RGB noise filter. I tried using ImageMagick's noise generators, but they didn't produce the desired result. As an alternative to adding noise, I could also use a background image that already includes some noise. It would be straightforward for GSnap to load a background image and paint on top of it instead of starting with a blank canvas.

Read the rest of this entry »

Color Images

September 24th, 2012

GSnap can now create images of the starlight in a galaxy using the same volume rendering technique that is used for creating images of the gas. For added realism, the starlight is attenuated by dust in the galaxy and very young stars are drawn bluer and brighter than their older counterparts. The resulting images are already looking pretty nice after just one day of working on this new viewing technique. On my quad core Phenom II processor, the total rendering time for a 3 million particle snapshot at 1280x720 resolution is about 3 minutes. That's a long time to wait if you are making a thousand frames for video, but it is considerably faster than using Sunrise to create similar images. Furthermore, creating images using GSnap only requires one command, whereas the Sunrise workflow requires four steps. The primary reason for the performance difference is that GSnap is just trying to make nice images, while Sunrise tries to get all of the physics correct. The rendering time will likely drop a bit after I optimize the code.

Here are the first example images. As usual, click for the full-size versions:

Read the rest of this entry »

Image Quality & Rectangular Images

September 16th, 2012

For the past week, I've been running a set of simulations on the Gordon supercomputer at the San Diego Supercomputer Center. While I waited for the simulations to complete, I improved many internal features of GSnap and made plans for much more significant improvements. Those plans can be found on the GSnap development wiki, here. I've also registered a domain name that will point to GSnap's official web site. Currently, the page is a very rough draft. I'll add more CSS styling and get rid of the frameset when GSnap is launched. The domain is

On the path to implementing the planned improvements to the code, I have allowed GSnap to render general rectangular images. Previously, only square images could be rendered because I only set a single resolution that acted as both the x and the y resolution. It sounds simple to generalize a square image to a rectangular image, but in reality it was a lot of work.  Hundreds of lines of code spanning 10 source files needed to be edited. The entire process took nearly 4 hours to complete and test. Beyond just creating rectangular images, the volume renderer now uses a volume of arbitrary depth; the volume was previously a cube, now it is a rectangular prism.

In addition to the image and volume generalizations, a series of incremental improvements have been made to the image quality. The image below shows a 720p HD resolution frame from a 2:1 mass ratio merger simulation that recently finished running on the supercomputer.


And here is a frame from a 1:1 merger


September 2nd, 2012

GSnap's volume renderer now includes an opacity effect so that foreground objects obsure the view of background objects. The opacity leads to a much clearer sense of depth—especially when combined with the perspective projection. Here's a demonstration:

Before & After

August 29th, 2012

Before and after images illustrating the latest improvement to GSnap. Click:

GSnap Improvements

August 25th, 2012

I have been working on two things lately: 1) Setting up the final set of simulations that will be used for the remainder of my thesis project. 2) Making Improvements to GSnap.  This post deals with GSnap...

  • Many small fixes and improvements have been made, notably:
    • The GADGET-3, type 1 snapshot format is now supported.
    • GADGET-3 files can be converted to the GADGET-2 format.
    • The distance between black holes can be reported.
    • The "make install" target has been set up; on Unix systems, GSnap can now be easily installed at the system level.
  • I began using the Zim wiki software to create online help pages.
    •  The interlinked Zim notes can be exported to create an HTML website.
    • A new help browser window (a QtWebkit browser) is integrated into the GSnap GUI, so that users have easy access to the HTML help.
    • A large subset of MathJax is now included with GSnap; the LaTeX syntax in the help is typeset using MathJax rather than TeX/LaTeX.
  • The GSnap web presence is beginning to take shape...
    • The main GSnap website contains:
      • the help pages 
      • a link to the issue tracker / reporter on Bitbucket.
      • a link to the GSnap development Wiki on Bitbucket.
    • The webpage will contain:
      • screenshots.
      • sample renderings.
      • sample videos (animations).
      • sample snapshots.
      • the code's Doxygen documentation.
      • A download link.
In addition to all of that, I just finished implementing an improved version of the volume rendering scheme. The renderer now properly handless regions of low particle density. Examples follow. The last two images illustrate the difference between orthographic projection using particles and perspective projection using volume rendering. Notice how sparsely the particles are distributed in the "bridge" between the two galaxies. The improved volume renderer smooths these particles quite nicely.

 Note: the images may appear bright and jagged in some versions of Firefox.