LCLTOOLS(L) LCLTOOLS(L) NAME lcltools - Local tools developed at the SBN DESCRIPTION A number of tools and utilities have been developed here at the SBN for working with files and labels of various types in our archiving activi- ties. A full man page is provided for each utility. Source code is available on request for either single utilities or entire suites. Brief descriptions of the utilities follow. FILE UTILITIES These routines manipulate file names, file lengths and records. addcol add blank columns (i.e., bytes) to ASCII file records addcr add a carriage-return (\r) character before all linefeed (\n) characters. addlf add a linefeed (\n) character after all carriage-return (\r) characters. chkascii check for non-printing ASCII characters (7-bit or ISO 8859-1 8-bit). chkdos check file records for DOS (\r\n) record delimiters chkfix check that a file has fixed-length records delcol delete columns (i.e., bytes) from ASCII file records findbl print a table showing the number of blanks in each byte column of an ASCII file findtab find tab and other non-printing characters in ASCII text files fixfn replace file names with inconvenient characters fixlen blank-pad (or truncate) variable-length ASCII records to a given length halve split an ASCII file by records into two new files inscol insert a (possibly non-blank) column (i.e., byte) into ASCII file records joinfile merge two files by concatenating corresponding records. Oppo- site of splitfile. lblcol add bye-count labels to an ASCII file lcfn force file names to all lower-case lcount count executable lines in C (or Perl) source code linnum add line number tags to the beginning of ASCII file records null2blank replace null characters with ASCII blank characters rmVMS remove VMS file version numbers rmcr remove carriage-return (\r) characters rmlf remove linefeed (\n) characters sample create a sample file listing of an ASCII file showing selected records, with byte count labels scanlen scan an ASCII file for maximum and minimum record length splitfile split a file into two new files by dividing each record. Oppo- site of joinfile. swap swap bytes in a binary file. Useful for converting binary data from byte-swapped systems (like VMS). sz Quick & dirty file size estimator for Unix systems ucfn force file names to all upper-case varlen strip trailing blanks from the end of linefeed-delimited ASCII records to produce variable-length records FITS FILE UTILITIES These routines are useful for manipulating FITS files. break_fitstab split a FITS ASCII table into separate header and data files in preparation for running the f2pconverter. brkstrm break a stream text file into fixed length records concat concatenate two files into a third (C program) fits_datachk check the elements in the primary data unit for extrema and proper padding fitsidx generate an index table by extracting keyword values from FITS headers fitspad pad an otherwise FITS-formatted file out to a multiple of 2880 bytes fixaz fixes faulty FITS files produced in AZ during Hale-Bopp observa- tions; removes non-printing characters from header blocks, prop- erly pads data blocks fixfits reformat incorrectly formatted FITS ASCII tables in which each block has been blank-padded to 2880 bytes, rather than just the last one flt2int convert IEEE FITS file to long integer FITS file make_fitstab combine a delimited PDS ASCII table and corresponding FITS header produced by the p2futility into a standard FITS ASCII ta- ble mkstrm remove record delimiters from an ASCII file to create a stream file primfits extract the primary FITS header and corresponding data from a FITS file which may have extensions sepfits separate FITS primary header/data and extension header/data into separate files splitfits split a combined FITS file into separate files for header and data, adding carriage-control if desired PDS FILE UTILITIES These routines are useful for handling PDS labels and catalog files and for working with the PDS Data Dictionary (PSDD). addcolhdr Add a column label header to a PDS ASCII table data file and adjust the label accordingly addcolnum Adds COLUMN_NUMBER elements to PDS labels containing COLUMN def- initions addquotes Forces quotes around all string values in a PDS ODL file change_dsid Systematically changes DATA_SET_ID values in a directory tree. checkids Search label and catalog files for undefined _NAME and _ID fields and unreferenced catalog objects checkpdstext Check PDS text files (labels and catalogs) for invalid charac- ters, record lengths and line delimiters chkpdssum Check the contents of a PDS volume against the checksum.tab file generated by mkpdssum. chkisonames Check a directory tree for file and directory identifiers that don’t meet ISO 9660 (and PDS) requirements chunkdir Divide a directory into chunks of a specified size (generates a listing to be used for dividing up the directory) detachpds split PDS files with attached labels into separate detached label and data files. dsids generates a list of unique DATA_SET_ID values found in a set of PDS detached labels dstree create PDS volume OBJECTs to describe the data sets in a direc- tory tree fixdt fix DATA_TYPE fields in a PDS label describing TABLE data (ASCII or binary) to conform to current standards version fixqt Wrapper to run "addquotes" on a set of PDS ODL files, forcing quotes around string values and overwriting the input fixztimes Removes the trailing "Z" from PDS/ODL file keywords ending with the TIME classword fmt_allrefs format the "allrefs.out" file as downloaded from PDS CN into an ODL-formatted reference catalog file gather_ref extract references from "allrefs.cat" corresponding to input list of REFERENCE_KEY_ID values gettargets generate a DATA_SET_TARGET list by checking PDS data file labels for all unique TARGET_NAME values grepdd Search for matching strings and patterns in the DESCRIPTION text of the PDS data dictionary file index generate an index list of file sizes under a given root direc- tory labeltext create PDS TEXT labels for unlabeled text files lblsumm show a summary of names, sizes and data types for OBJECTs in a PDS data file label lvtool wrapper for the PDS label verifier tool (also called ’lvtool’) to simplify local use make_lcldd Utility for merging locally-defined keywords into the full PDS data dictionary file mkpdssum Generate a table of MD5 checksums for a PDS volume (or any) directory tree mkvolidx create the "index.tab" and "index.lbl" files for a PDS volume mkcumidx combine volume indices created by mkvolidx into a cumulative index mklcldd extract definitions for keywords and object appearing in data labels from main Data Dictionary to create a local dictionary for the file set nullval replace blank fields with null values in a PDS ASCII TABLE pdef return complete object definition from PSDD given an alias pdsalias return full name from PSDD given an alias pdsdef return full definition from pdsdd.full file given full ele- ment/object name; search NAME fields for partial string matches pdsidx generate an index by extracting values from keywords in PDS labels pdslinkchk check links in HREF fields of HTML files to ensure all links are relative and internal pdstv PDS/SBN label and data verifier for TABLE-type objects ppodl pretty-printer for PDS ODL (labels and catalog files) rmcolumn remove a COLUMN definition from a PDS ASCII TABLE label (and optionally, the data file), and adjust COLUMN_NUMBER and START_BYTE values accordingly sbnlblchk checker for required keywords and valid DATA_TYPES in all PDS detached label files in a directory structure. Runs sbnlc on all "*.lbl" files. sbnlc check PDS label for required PDS and SBN keywords; also check for valid INTERCHANGE_FORMAT and DATA_TYPEs (as used at the SBN) sbnlv local label verifier for PDS labels and catalog files showrefkeys list REFERENCE_KEY_ID values for listed files and directories sortref sort a file containing REFERENCE object by REFERENCE_KEY_ID val- ues split_ref split a file of REFERENCE objects into separate files for each REFERENCE srchpds search the PSDD for a partial match of the input keyword name srchref search the master reference catalog file for matching REFERENCE objects tbldelimit add field delimiters to PDS ASCII TABLEs and adjust labels accordingly tbltmpl create a template for an ASCII TABLE label, with COLUMN charac- teristics as determined from reading the data file tblundel remove field delimiters from PDS ASCII TABLEs and adjust labels accordingly vc_files check a PDS volume for required files and subdirectories check a PDS volume index for label validity and table completeness vc_labels check a PDS volume to ensure that every file is labelled vc_refint check a PDS volume for referential integrity check a PDS voluem to ensure that all text files meet PDS format and character restrictions NODE MAINTENANCE UTILITIES These routines are specific to the comet node of the SBN. archlist generate an HTML-formatted list of data set names, IDs, file sizes and links from the information in the (detached) PDS label and catalog files cometid return cross-identifications for comet names or designations ftpstats process an FTP xferlog file to produce usage statistics on the number of data file requests processed getman remove annoying control characters from man page output and save into a flat ASCII file lblsumm display a summary of PDS label contents, including COLUMN description for TABLE objects (also callable as a CGI script) linkchk Check SBN HTML files for valid local references make_dsc create data set contents (.dsc) files for PDS data sets (local SBN) make_dschtml create HTML files for posting on the WWW from data set contents (.dsc) files (local SBN) mkexport Create Unix/Linux versions of local tool source and build files mklclwhatis Create a local ’whatis’ database for the local tools man pages pdsmet_ftp generate required PDS metrics from a monthly FTP log file pdsmet_www generate required PDS metrics from a monthly HTTPd access_log file WWWraw scan the WWW archive log and report on raw usage WWWstats process a WWW archive access log to extract numbers on real data transfers PROBLEMS Problems, questions or requests should be directed to Anne Raugh, raugh@astro.umd.edu. 30 Nov 2004 LCLTOOLS(L)