Data Definition Specification (DDS)

 

SOLAR2000/E10.7 SET/SEC Hourly SWDC-Format Archival Data Files

 

Version f, 17 Apr 03

Prepared by Dave Bouwer, SET/SpaceWx

 

 

(1) The s2kops nowcast data file (also referred to by the “hourly s2kops SWDS file”), is created 24 times a day (current actual I/O data values change only about two-four times a day based on Mg II and F10.7 inputs; the particular time may occur at any time during the day).  Typically, the last data input (currently F10) to SOLAR2000 occurs at about 21 UT.  Assume an hourly cadence, but software or system failures may prevent a data file being created, in which case a “default” set of fields might be substituted with missing values in the database. These data need to be archived for at least 90 days at NOAA/SEC.  SET/SpaceWx maintains triple-redundant online and off-line archives.

 

(2) All the default (i.e., missing) min/max values are given below in the next file description. “data time” refers to the time the SOLAR2000/s2kops associates with data, e.g., F10.7 is defined at Penticton as Local noon, and it has been converted to the associated UT for the F10.7 “data time”.  “File Time” refers to the computer system time associated with the files (e.g., the time one would see from a “ls –alt” directory listing). The time stamp in the filename is in “file time” – which should correspond, within one hour, to “data time” if all is well. All times have a SQL-formatted date/time string and a double-precision Julian Day. The distinction between “file time” and “data time” is particularly important for validation software, and for other maintenance and analytical uses.  Generally, end-users will be primarily interested with the data times.

 

(3) In the Java code that creates the data file, a Hashtable class is used, converting it to a String with name/value pairs as a single line (see sample below).  The fields shown below do not follow the same order as the name/value pairs in the data file (an artifact of the Hashtable.toString() method in Java).  It is important that software that retrieves the data file do so on the basis of the name = value fields, not on the particular order (thus new fields can easily be added later).

 

(4) All date/time strings are in SQL format, e.g., yyyy-mm-dd hh:mm:ss.mmm, in UTC. Nearly all times are retrieved from the computer system time, which must be in (Greenwich noon) UTC!

 

(5) The default – or missing – values  in all dates and times are 1970-01-01 00:00:00.000, JD = 2440587.5.

 

(6) All float values should have –1.0 as a missing-value flag.

 

(7) Integer values should have –1 as a missing value flag.

 

(8) String values, other than date/time, should have “-1” as a missing value flag.

 

(9) The units of E10, F10, Ly-a are:

a.     F10, F81 are daily and 81-day solar 10.7 cm radio flux in x10-22 Watts per meter squared per Hertz

b.     Lya, L81 are daily and 81-day Lyman-alpha in x10+09 photons per centimeter squared per second

c.     E10, E81, are in x10-22 Watts per meter squared per Hertz. E10.7 is observed, integrated 1-105 nm EUV in F10.7 units

 

(10) The hourly “nowcast” data file will written to a single file on the NOAA/SEC operational computer “ion” (and most other systems) in the directory/filename:

/spacewx/s2kops/sftp/swds/2k-SWDSnowcast-YYYY_MM_DD_HH_MM.txt

The date/time fields in the filename will correspond to file time.

 

Table1: Hourly s2k-nowcast/SWDS Record Field Definitions

1

Created

String * 23

The time this file was created

2

F10inp

Float

The current F10.7 (uncorrected) value

3

Mg2inp

Float

The current NOAA/SBUV Mg II c/w value

4

F10

Float

The F10.7 daily value used in SOLAR2000 (extrapolated)

5

F10_81

Float

The F10.7 81-day average calculated by SOLAR2000

6

Lya

Float

The Ly-a value calculated by SOLAR2000 from Mg II

7

Lya_81

Float

The Ly-a 81-day average calculated by SOLAR2000

8

E10

Float

The E10.7 index  calculated by SOLAR2000

9

E10_81

Float

The E10.7 81-day average calculated by SOLAR2000

10

RunStatusCode

Int

A flag (0-9) indicating data quality; 0 means all is OK

11

Mg2source

String * 6

The satellite (e.g. NOAA16) used to measure the MgII c/w ratio

12

S2Kvers

String *40

A version ID for the s2kops and SOLAR2000 software

13

UTCfileJD

Double Float

The Julian Day the SOLAR2000 output file was last modified (created)

14

InpFileJD

Double Float

The Julian Day the input data file to SOLAR2000 was last modified

15

UTCdataJD

Double Float

The Julian Day associated with the SOLAR2000 data output values

16

FinpJD

Double Float

The Julian Day associated with the current F10.7 data value

17

MinpJD

Double Float

The Julian Day associated with the current  MgII c/w data value

18

UTCfileSQLtime

String*23

The SQL Date/Time the SOLAR2000 output file was last modified

19

InpFileSQLtime

String*23

The SQL Date/Time input data file to SOLAR2000 was last modified

20

UTCdataSQLtime

String*23

The SQL Date/Time associated with SOLAR2000 data output values

21

MinpSQLtime

String*23

The SQL Date/Time associated with the current  MgII c/w data value

22

FinpSQLtime

String*23

The SQL Date/Time associated with the current  F10.7 data value

*Note: in most instances, the file modification time is the file creation time.

 

(11) A sample  s2k-nowcast-yyyy_mm_dd_hh_mm.txt file, using the Hashtable.toString() method:

 

MinpSQLtime=2003-03-23 15:22:17.000, MinpJD=2452722.140486111, Lya=4.32, Lya_81=4.71,InpFileJD=2452722.1405092594, FinpSQLtime=2003-03-23 15:22:20.000, F10_81=116.4,UTCfileSQLtime=2003-03-23 15:22:17.000, F10=87.8, Mg2inp=0.26985, F10inp=89.0,RunStatusCode=0, S2Kvers=s2kopsV02_S2K_V1.24/256C78rumba,UTCdataSQLtime=2003-03-23 15:22:17.000, InpFileSQLtime=2003-03-23 15:22:20.000,UTCdataJD=2452722.140486111, FinpJD=2452721.0, E10=95.4, E10_81=116.4,Created=2003-03-23 15:22:17.000, Mg2source=noaa16, UTCfileJD=2452722.140486111}

 

(12) The data file will written to a single file in /spacewx/s2kops/sftp/s2k-SWDSnowcast-YYYY_MM_DD_HH_MM.txt. In other applications, files are overwritten with a single filename swdsNowcast.txt or in a similar  convention, e.g.,

 

A sample /spacewx/s2kops/sftp/swds/s2kSWDShourlyTable.txt file (it wraps around in this document, but there are two long lines):

 

UTCfileJD InpFileJD UTCdataJD FinpJD MinpJD F10inp Mg2inp F10 F10_81 Lya Lya_81 E10 E10_81 RunStatusCode UTCfileSQLtime UTCdataSQLtime InpFileSQLtime MinpSQLtime FinpSQLtime Mg2source S2Kvers

2003-04-17 18:24:01.000 2452747.2569444445 2452747.2430439815 2452748.2569444445 2452746.0 2452746.5055555557 99.0 0.27124 103.0 117.0 4.5 4.67 111.9 117.0 0 2003-04-17 18:10:00.000 2003-04-18 18:10:00.000 2003-04-17 17:49:59.000 2003-04-17 00:08:00.000 2003-04-16 12:00:00.000 noaa16 s2kopsV02_S2K_V1.24/256C78quickstep

 

The min/max values are:

 

  1. Created, String*23
  2. UTCsqlData (seconds and milliseconds are always 0), String*23

-        Year: min = 1947, max = 2147, plotmin = 1970, plotmax = 2047

-        Month: min = 1, max = 12, plotmin = 1, plotmax = 12

-        Day: min = 1, max = 31, plotmin = 1, plotmax = 31

-        Hour: min = 0, max = 60, plotmin = 0, plotmax = 59

-        min = 0, max = 60, plotmin = 0, plotmax = 59

  1. DataJulianDate, Double Float

-        min = 2405600.0, max = 2505600.0, plotmin = 2440587.0, plotmax = 2468712.0

  1. F10, Float

-        min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0

  1. F81, Float

-        min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0

  1. Mg2
  2. -           min = 0.20, max = 0.35, plotmin = 0.260, plotmax = 0.295
  3. Mg81
  4. -           min = 0.20, max = 0.35, plotmin = 0.260, plotmax = 0.295

-         

  1. Ly, Float

-        min = 1.0x10^11, max = 9.0x10^11, plotmin = 3.0x10^11, plotmax = 8.0x10^11

  1. Lya81, Float

-        min = 1x10^11, max = 9x10^11, plotmin = 3x10^11, plotmax = 8x10^11

  1. E10, Float

-        min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0

  1. E81, Float

-        min = 0.0, max = 400.0, plotmin = 25.0, plotmax = 350.0

  1. RunStatusCode, Int

-        min = 0, max = 9, plotmin = 0, plotmax = 9

 

(13) Notes:

    1. An open issue is how to index records in a SQL database.  Data record times should monotonically increase, and only 1-minute resolution is required. There must be only one record per time. One suggestion is to use a large integer based on the YYYYMMDDHHMM.
    2. If the RunStatusCode is greater than 3, the data should be considered unreliable. When it is 2-3, it might be suspicious, but probably is adequate.  If the RunStatusCode is 0-1, no errors have been detected in the validation software: Extensive exceptions, bounds checking, and error-handling  is performed automatically. For verification, the daily data values have had an SEC operator verify the SOLAR2000 E10.7 after 1-3 days, but only post-analysis with corrected input values (e.g. the corrected Penticton F10.7 daily values) and auxiliary measurements can fully verify SOLAR2000 data.
    3. Even if current data input is unavailable, s2kops will “gracefully fail”, i.e., it will produce the last observed input data and predicted SOLAR2000 values.  The s2kops software runs in crons, and automatically restarts and recovers from most error conditions.
    4. An open issue is that in the event of fatal system failures (e.g., prolonged downtime due to system crashes, network outages, etc.), how data records with missing data flags will be back-filled. SET/SpaceWx maintains triple-redundant archives.
    5. At some time in the future, it should be anticipated that other I/O data (e.g., NOA17 MgII, background X-rays) may be added.
    6. On rare occasion, date/time values may be zero or otherwise improbable due to operating system malfunctions: this is unavoidable.  In such instances the data must be ignored and the problem promptly corrected.
    7. F10.7 is that used at the end of the day, and may be corrected later for final archival the the NOAA World Data Center.  The corrected data typically occurs prior to the final archival of a months data at the WDC.  These corrections may be due to correction of the measurement to remove solar radio bursts effects.

 

 

The default (missing) data values are:

 

HourlySWDSrecord test-> Verifying with Public attributes, default missing values:

HourlySWDSrecord test->1  Created, 1970-01-01 00:00:00.000 =>String, The time (SQL format) this hashtable (file) was created

HourlySWDSrecord test->2  UTCfileJD, 2440587.5 =>Double, The Julian Day the SOLAR2000 output file was last modified (created)

HourlySWDSrecord test->3  UTCfileJD, 2440587.5 =>Double, The Julian Day the SOLAR2000 output file was last modified (created)

HourlySWDSrecord test->4  UTCdataJD, 2440587.5 =>Double, The Julian Day associated with the SOLAR2000 data output values

HourlySWDSrecord test->5  FinpJD, 2440587.5 =>Double, The Julian Day (12 UT) associated with the current F10.7 data value

HourlySWDSrecord test->6  MinpJD, 2440587.5 =>Double, The Julian Day associated with the current  MgII c/w data value

HourlySWDSrecord test->7  F10inp, -1.0 =>Float, The current F10.7 (uncorrected) value

HourlySWDSrecord test->8  F10, -1.0 =>Float, The F10.7 daily value used in SOLAR2000 (extrapolated)

HourlySWDSrecord test->9  F10_81, -1.0 =>Float, The F10.7 81-day average calculated by SOLAR2000

HourlySWDSrecord test->10 Lya, -1.0 =>Float, The Ly-a value calculated by SOLAR2000 from Mg II

HourlySWDSrecord test->12 Lya_81, -1.0 =>Float, The Ly-a 81-day average calculated by SOLAR2000

HourlySWDSrecord test->13 E10, -1.0 =>Float, The E10.7 index  calculated by SOLAR2000

HourlySWDSrecord test->14 E10_81, -1.0 =>Float, The Ly-a 81-day average calculated by SOLAR2000

HourlySWDSrecord test->15 RunStatusCode, -1 =>Integer, A flag (0-9) indicating data quality; 0 means all is nominal(OK)

HourlySWDSrecord test->16 UTCfileSQLtime, 1970-01-01 00:00:00.000 =>String, The SQL Date/Time the SOLAR2000 output file was last modified

HourlySWDSrecord test->17 UTCdataSQLtime, 1970-01-01 00:00:00.000 =>String, The SQL Date/Time associated with SOLAR2000 data output values

HourlySWDSrecord test->18 InpFileSQLtime, 1970-01-01 00:00:00.000 =>String, The SQL Date/Time input data file to SOLAR2000 was last modified

HourlySWDSrecord test->19 MinpSQLtime, 1970-01-01 00:00:00.000 =>String, The SQL Date/Time associated with the current MgII c/w data value

HourlySWDSrecord test->20 FinpSQLtime, 1970-01-01 00:00:00.000 =>String, The SQL Date/Time associated with the current F10.7 data value

HourlySWDSrecord test->21 Mg2source, noaaXY =>String, The satellite (e.g. NOAA16) used to measure the MgII c/w ratio

HourlySWDSrecord test->22 S2Kvers, -1 =>String, A version ID for the s2kops and SOLAR2000 software