You might remember that we were planning on releasing HiRISE images to the public on a monthly basis. That plan was delayed by issues with our processing software, hardware and other events. A productive summer working on these issues culminated last week with one of our larger releases of Mars images!  Here are some statistics about our September 2009 release, which includes the images the HiRISE camera took of the Martian surface between Mars Reconnaissance Orbiter (MRO) orbits 12,600 to 14,199, or roughly April 4 through August 6, 2009:

• 2,996 RDRs, 1 TB
• 42,370 EDRs, 1 TB
• 34,481 RDR Extras, 1.6 TB
• 83,784 EDR Extras, 0.02 TB
• 636 Anaglyphs, 0.01 TB

Totals for this release: 163,631 image products, 3.6 TB

This brings our total released product numbers and data volume to:

• 22,676 RDRs, 12 TB
• 317,120 EDRs, 10.4 TB
• 192,270 RDR Extras, 15.3 TB
• 612,769 EDR Extras, 0.1 TB
• 2,892 Anaglyphs, 0.5 TB

Total: 1,148,363 images, 37.5 TB

In summary, we released nearly 1500 observations, most of those with both black & white and color RDR products. Several newer observations matched up with older observations from a slightly different angle of the same location on the surface, resulting in 636 awesome new anaglyphs. The RDRs are the fully processed, geometrically projected products best for scientific inquiry. If you really want to, though, anyone can download and process HiRISE data from scratch. You can do this using ISIS software, which is publicly available for free download. See the ISIS Web site for download information, processing instructions, and tutorials.

Starting this week, I will be looking over the observations taken August 6 through August 26 before MRO went into safe mode and make sure they are ready for release. We plan to release these images in early October. We are also in the process of reprocessing those Extended Science Phase mission images prior to all the latest processing pipeline fixes and updates.  Once we are satisfied with that data set, we will release them to the public and then start reprocessing the images from the Primary Science Phase…a major project that should keep me and the rest of Downlink busy for several months!

Starting with the 10/10 release, color images are included for the first time. We’ll describe how we process these in the days and weeks to come. But what I’d like to do first is give a brief description of all our product types as they currently are available. You’ve no doubt noticed a mind-boggling array of new options on our product pages. They now include what we call our “NOMAP” products; NOMAP means that they are not map-projected. In other words, not rotated to the direction of north, not mapped to a coordinate system, and not scaled to any particular geometric resolution.

I’ve prepared this ugly table that outlines each of the products now available (excluding the raw EDRs). So reading the columns from left to right: there are three types of “NOMAP” products, two types of lossy “QLOOK” (Quicklook) RDRs, and two types of lossless RDRs.

With that as a reference, now I’ll try to define everything more precisely.

“NOMAP”

Non map-projected product. Always lossy compressed for smaller size and quicker viewing. These are not formal Planetary Data System products; they’re “special”, meaning there is no PDS label and no Software Interface Specification describing them. Available for IRB, RGB and RED.

RDR

Reduced Data Record: reduced in the sense of refined or processed, not raw data. Formal PDS products with accompanying labels and a detailed SIS document describing their format and processing steps. Available both in lossless and quicklook formats for both RED & COLOR.

“QLOOK”

Quicklook: a special product that is a lossy compressed version of the RDR. In a normal RDR, all of the original data is retained. But with a quicklook, some of the highest resolution detail is discarded to make for quicker viewing.

RED

The image obtained by the red-filtered CCDs. It will be over the full swath width, typically data from all ten red CCDs. Covers the visible wavelength band from 550 to 850 nanometers.

IR

Infrared. Covers the near-IR wavelengths from 800-1000 nanometers.

BG

Blue-Green, visible wavelengths from 400-600 nm.

COLOR

A color RDR. It contains data from the IR, BG and center RED ccds. Typically this will be a skinny strip (”center swath”) inside a skinny strip, or as I like to say, the bacon-strip effect.

IRB

An enhanced color NOMAP. It has the same color bands as the RDR: IR, RED and BG.

RGB

An enhanced color NOMAP. It contains only data from the RED and BG. The blue is derived from the difference between the RED and BG. The color bands are RED, BG and the synthetic blue.

EDR

Experiment Data Record, a formal PDS product that is raw uncompressed data with a label header.

Note: we will be working towards making all of these products available for all prior releases.

Spacecraft missions are complicated endeavors that result in a wealth of scientific and engineering data. Long after the mission has ended, these data can be extremely useful for later study and discovery. With so many missions over so many years, how can later generations find and make use of these data?

The solution for many NASA missions has been the development of the centralized Planetary Data System (PDS). The PDS is several things: a collection of websites, a search capability, an archive, a database, a learning tool, etc. The PDS Imaging Node is located at http://pds-imaging.jpl.nasa.gov/ and acts as “the curator of NASA’s primary digital image collections from past, present and future planetary missions.” These missions include Voyager, Galileo, Cassini, and many more. Now the Mars Reconnaissance Orbiter (MRO) has been added to the list, with the HiRISE team releasing our first several months of image data.

• MRO PDS page: http://pds-imaging.jpl.nasa.gov/Missions/MRO_mission.html
• MRO Product Search page: http://pds-imaging.jpl.nasa.gov/search/index.jsp
• HiRISE Volume: http://hirise-pds.lpl.arizona.edu/PDS/

What we have released is an archive of the HiRISE Experiment Data Records (EDRs) and Reduced Data Records (RDRs). EDRs are in the *.IMG file format and represent individual CCD channels (remember, there are 14 CCDs in the HiRISE camera and two channels per CCD, for a total of 28 channels). These EDRs are cleaned up, calibrated, stitched together, and mapped to Mars’ geometry, resulting in the RDR products. RDRs are in the *.JP2 and *.LBL formats. JPEG2000 is the technology that enables us to offer our gigantic images to the scientific community and the public in a timely and efficient manner. An observation’s image data are in the *.JP2 file and its meta data are in the detached *.LBL files. To view these products, JPEG2000 compatible software is required (see our site for a list of offerings).

While we have been trying to release up to five captioned images a week for the past few months, the PDS release represents several hundred images, most of them without captions. You can find them using the PDS search capabilities, and you can also find them on the new HiRISE site, unveiled today to coincide with this first PDS release. The redesigned site focuses on the images while providing, hopefully, a more user-friendly interface:

• HiRISE Site: http://hirise.lpl.arizona.edu/
• “About Our Redesign”: http://hirise.lpl.arizona.edu/profil.php
• Images released to the PDS: http://hirise.lpl.arizona.edu/pds_release.php

As word gets out about the new site and the PDS release, you may experience some site slowness. Please be patient, and thank you for your interest!

I thought I’d offer a few more words as to what is done with images at HiROC. Validation has been mentioned in the blog, and I’d like to explain a bit more about that. I’ve been involved in writing the primary validation tool, HiVali, and I will be the primary student validator for the next month. (The regular student validators are from out of state, and are going home for the Christmas holidays. I’m from around here, and offered my services to look at pretty pictures from Mars all day;-))

(more…)

After a channel of raw data has been downloaded and converted into an *.IMG file, we need one more conversion before cleanup of the image can begin.

The EDR_Stats pipeline creates a *.cub file from the *.IMG file. These cube files are the file type used in ISIS 3.0, an image processing software package provided for planetary science missions by the United States Geological Survey (USGS). This package contains an entire suite of useful tools, many of which are used by our pipelines.

During the creation of a cube, a variety of statistics are gathered. For example, the number of gaps, saturated pixels, calibration pixels, and other pixels are counted. Image mean, standard deviation, and other statistics are also calculated. EDR_Stats takes these results and uploads them to our database. The resulting cube is archived in our storage directory.

The final EDR_Stats pipeline step lets the next pipeline – HiCal – know that an image channel cube file is ready for calibration processing. Let the cleanup of image data begin!

The Planetary Society has an excellent article entitled “HiRISE Image Processing” based on Tuvas’ HiBlog post “Processing images at HiROC“. Both articles explain the EDRgen pipeline very well.

It is important to note that while there are a multitude of image formats available, Experimental Data Records (EDRs) are a standardized way of packaging planetary science data sets for release to the world while ensuring future access to said data. In the case of HiRISE images, there are two components to an EDR product: (1) the image data and (2) the label.

The EDRgen pipeline uses a program called HiRISE_Observation to create an EDR from the original channel raw data. The image data is converted into a file type with the extension *.IMG and important information about the observation is attached to this *.IMG file in the form of a text label. This label includes information about this mission; the observation name, commanding, time, and temperatures parameters; and other useful information.

After the EDR is created, it is archived in our storage directory hierarchy (we follow a hierarchy that includes mission phase, orbit range, and observation ID). Finally, the database sources table for the next pipeline – EDR_Stats – is updated with the location of the new EDR. Further processing of this EDR, in a different format, is necessary to start cleaning up the image.

How long do each of these pipelines take? HiDog generally downloads a new channel file in a few minutes or less. EDRgen can create a new *.IMG file in a few minutes or less, and we have a few EDRgen pipelines working in parallel. The fact is, most of the pipelines are incredibly fast on our processing cluster. Later pipelines that stitch and mosaic take significantly longer, but rapid progress in computer technology have blown away early conservative estimates of how long HiRISE image processing would take.

Some of you out there may be asking: what happens to a HiRISE image between the time that it is taken and the time that it is released to the public? Well, I’d like to give a summary here.

(more…)