The Peer Review Process
All data formally incorporated into the PDS archives must undergo a peer review. The purpose of the review is to determine that:
- The data are complete (e.g., no missing calibration files)
- The data are suitable for archiving (i.e., of sufficient quality and with enough documentation to be useful and intelligible in the distant future)
- The PDS standards have been followed
Any problems identified during the review must be resolved before the data are accepted into the PDS archives.
Before the Review
PDS will organize peer reviews and these will involve both technical specialists on archiving and scientific peers who are not connected with either PDS or the mission (much like referees of journal articles).
The Personnel
The peer review panel will typically consist of the following people:
- One or more representatives from the discipline node.
The node manager usually chairs the meeting. Other node personnel may be there to address questions about PDS standards, take notes or record liens. - The data preparer.
A representative for the group responsible for preparing the data for ingestion is also present to answer questions about the formatting and content of the data. - The peer reviewers.
Generally 2-3 people knowledgeable in the type of data being reviewed, but not directly connected with the project which produced it.
At the Review
For small data sets, it is possible to conduct peer reviews by phone conference or even email. Larger data sets, though, tend to be best dealt with in face-to-face meetings. Prior to the review meeting, all people involved should have received copies of the data sets and labels, as well as any documentation and catalog object files being included. Distributing this information is the responsibility of the discipline node.
During a typical review each data set is considered in turn:
- Any problems or difficulties encountered by the reviewers in the mechanical process of accessing the data are noted.
- Reviewers are asked to comment on the archival quality of the data.
- Specific problems and deficiencies are noted on a list of liens, which is collected by one of the discipline node representatives.
- A decision is made based on the recommendations of the reviewers to either:
- Accept the data as is
- Accept the data conditional on the resolution of identified liens
- Reject the data
- The results of the review, together with any liens placed on files during the process, are compiled, filed with Central Node, and distributed to interested parties.
After the Review
All recorded liens must be resolved before the data are incorporated into the PDS archives. The mission is responsible for addressing all liens identified in the peer reviews, just as an author is responsible for addressing all comments of a journal referee. As with journal refereeing, this does not always mean doing everything suggested by the review panel, but it does mean doing many of the things and explaining why other requests for change were not accepted. The scientific peers are often the most severe critics of an archive. The final data submission should occur prior to the end of the mission.