SOHO
Experiments reiewed during this 3rd HAUS group meeting: CELIAS, UVCS  
Experiments reiewed during the 2nd HAUS group meeting: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO
Remaining experiment to review: SWAN

ULYSSES (complete)
Experiments reiewed during this 3rd HAUS group meeting: EPAC, URAP    
Experiments reiewed during this 2nd HAUS group meeting: COSPIN, DUST, FGM/VHM, GAS, GRB, SWICS, SWOOPS 

Curation/Completeness review guidelines

For each experiment 

List of data products available on the archive per instrument
Which datasets are available?
What is not calibrated? 
Can it be calibrated? 
How?  through community software package for instance?
If not, who/which group could do it? with eventually an estimate of the manpower needed when applicable.
What is missing? Please compare to what is available on PI or PI team member website(s)?
What would you recommend to archive in the long term?
Any ground-based data essential to support data analysis that shall be stored as well?
Are datasets stored in format compatible with community (visualization/data analysis) tools?

Be aware, that for instance on SOHO, there are various (PI) websites containing up-to-date derived products that are not stored in the archives yet. Which one of those, if any, shall be stored in ESA archives?

FP7/H2020 EU funded projects

EU funded projects related to Heliophysics 
-----------------------------------------------------
H2020    HESPERIA   
H2020    Flarecast
FP7       SEPServer       

Review guidelines: shall we archive anything of the value-added datasets produced by the various Heliophysics archives related FP7/H2020 projects listed below? If yes, which one(s)? 

General HAUS recommendation 

For long term preservation, the HAUS group recommends that the best practice is to store a calibrated version of each dataset, together with raw data. Clear guidelines on how these calibrated datasets have been generated shall of course be made available. 

Agenda

HAUS group Recommendations

Major
HAUS 03, MAR-1 about UFA support: The HAUS group acknowledged the unique scientific value of the data available through the ESAC Science Data Center Ulysses Final Archive (UFA). It recommends to provide ASAP the ESDC heliophysics archives’ group enough manpower not only to maintain UFA online, but also ingest new (best calibrated) data and develop up-to-date command line tools/APIs to improve UFA's interoperability.  These tools are key to support the scientific community not only to access the Ulysses data but also compare them with data collected by the Parker Solar Probe and upcoming Solar Orbiter missions. Moreover, these data could be used by Solar Orbiter PI teams in their science activity plan, in particular for out-of-the ecliptic science plan. 

HAUS 03, MAR-2 about Calibrated EIT files, EIT PI team to deliver ASAP to SOHO archives at ESAC and Goddard EIT level 2 calibrated files to avoid Solarsoft dependence for data calibration, wider data exploitation and long term preservation..

Minor
HAUS 03, MIR-1 on UFA interoperability: SAMP is implemented on the Ulysses server but a contribution by the ESDC for the most popular Heliophysics Python 
libraries to be able to easily download Ulysses data from these libraries would be an asset. A TAP server may also be considered.

Actions

1. Ulysses archive at ESDC: experiment reviews and recommendations

See Annexes 1, 2 and 3

1.1    URAP review

See Annex 1 

Short experiment description
The Unified Radio And Plasma wave experiment (URAP) measures the electric field from DC to 1 MHz and the magnetic field from 0.22 to 450 Hz. It is composed of a 72.5 dipoloe antenna in the spin plane, a 7.5 m monopole along the spin axis and 2 orthogonal search coil magnetometers. The PI is Bob MacDowall (GSFC) and data contact point is Roger Hess (GSFC). For more details on instrument charcteristics and main scientific objectives and first results, please see Annex 2.

PI website
https://urap.gsfc.nasa.gov

Data content
URAP at the Ulysses Final Archive at ESDC contains science data (in physical units)and quicklook plots for the whole mission. Science data are composed of the following folders

– df: direction finding results, monthly .zip files, within each is daily ascii .txt file, 10 min resolution, there is good documentation file for this product
– pfra/pfrp/rara/rarp/wfba/wfbp/wfea/wfep: frequency spectra in different ranges, yearly .zip files, within each is daily ascii .ulyfile, 10 min resolution, good documentation
– rara144: same as raraexcept 144 second resolution, and uses .ravfile extension

Some additional data sets and, documentation and software.

HAUS 03, AI-2 on ESDC, URAP user notes say sounder data is for calibration and will be published in the literature FES data will be archived at a later date. Hence, ESDC to check if URAP Sounder data and/or Fast Envelope Sampler data exist and could eventually be stored in the ESA Archive
HAUS 03, AI-3 on ESDC, to check with the PI team if there is any URAP higher resolution data that is not on the archive, or better calibrated. Contact URAP US website to be linked to UFA (https://urap.gsfc.nasa.gov/links.html)
HAUS 03, AI-4 on ESDC, to estimate the manpower necessary to implement the following recommendation of the HAUS group reviewer on the Ulysses URAP experiment

• CDF files (ISTP compliant meta-data) preferred, + ascii files are helpful too
• Directory structure could be made clearer
–no need to split into prime resolution and additional data
–logical file structure, labelled in consistent way
–documentation files in one place
• Make available by machine interface, e.g. ftp
• Prefer unzipped files

1.2    EPAC review

See Annex 2

Short experiment description
The Energetic PArticle Composition (EPAC) experiment measures the energetic particle fluxes (electrons, protons, heavuy ions), PI: N. Krupp (MPS, Göttingen, Germany). In more details, EPAC measures the flux anisotropy and chemical compostion so particles with energy between 300 keV and 25 MeV per nucleus. It is composed of four identical telescopes with different inclination angles with respect to the spin axis of Ulysses.

PI website
https://www2.mps.mpg.de/de/projekte/ulysses/epac/ ; no data provided at the PI website.

HAUS 03, AI-5 on ESDC, To fix typo and broken links on the EPAC instrument description on the Ulysses Final Archive (Title to be corrected to Energetic PArticle Composition instead of The Ulysses Particle Investigation) and in the Data available section: Full resolution of Electron and heavy ions fluxes instead of Full resolution Electron and Proton intensities) + fix broken link. Moreover, in the documentation (UDSepacdata_EPAC_Doc.pdf, update the link http://www.mps.mpg.de/en/projekte/ulysses to https://www2.mps.mpg.de/de/projekte/ulysses/epac/

Data content
Prime resolution PHA: 1990 292 - 1998, 291 (binary files), calibrated, documented

Prime resolution ulykep-xx: whole mission (1990-2007), calibrated, documented
No high resolution data

HAUS 03, AI-6 on ESDC, To double check with PI and/or Ulysses expert if the data content of the EPAC PHA product in the UFA is complete wrt to mission operations. At the moment, it only covers from 1990 DOY 292 to 1998 DOY 291.

HAUS 03, AI-7 on ESDC, To recover a number of "Related Documents mentionned in the Ulysses EPAC documentation (UDSepacdata_EPAC_Doc.pdf)

1.3   New AIs and recommendations

Ulysses Final Archive (UFA) is a legacy archive hence any work is done on best effort basis. The most urgent action at the moment is to upgrade of hardware, DB, java version, OS (planned around Jan. 2019). New data (e.g. DUST) will be ingested afterwards (2019). We are also happy to report that Cecil Tranquille, Ulysses expert, recently joined the Cluster SOC team at ESAC and is willing to support the HAUS group on a best effort basis. All AIs related to Ulysses are reviewed in Annex 3.

New Action Items (AIs)
HAUS 03, AI-1 on ESDC, implement clear on-line documentation on how to use the command line on the Ulysses archive

HAUS 03, AI-2 on ESDC, URAP user notes say sounder data is for calibration and will be published in the literature FES data will be archived at a later date. Hence, ESDC to check if URAP Sounder data and/or Fast Envelope Sampler data exist and could eventually be stored in the ESA Archive

HAUS 03, AI-3 on ESDC, to check with the PI team if there is any URAP higher resolution data that is not on the archive, or better calibrated. Contact URAP US website to be linked to UFA (https://urap.gsfc.nasa.gov/links.html)

HAUS 03, AI-4 on ESDC, to estimate the manpower necessary to implement the following recommendation of the HAUS group reviewer on the Ulysses URAP experiment

• CDF files (ISTP compliant meta-data) preferred, + ascii files are helpful too
• Directory structure could be made clearer
–no need to split into prime resolution and additional data
–logical file structure, labelled in consistent way
–documentation files in one place
• Make available by machine interface, e.g. ftp
• Prefer unzipped files

HAUS 03, AI-5 on ESDC, To fix typo and broken links on the EPAC instrument description on the Ulysses Final Archive (Title to be corrected to Energetic PArticle Composition instead of The Ulysses Particle Investigation) and in the Data available section: Full resolution of Electron and heavy ions fluxes instead of Full resolution Electron and Proton intensities) + fix broken link. Moreover, in the documentation (UDSepacdata_EPAC_Doc.pdf, update the link http://www.mps.mpg.de/en/projekte/ulysses to https://www2.mps.mpg.de/de/projekte/ulysses/epac/

HAUS 03, AI-6 on ESDC, To double check with PI and/or Ulysses expert if the data content of the EPAC PHA product in the UFA is complete wrt to mission operations. At the moment, it only covers from 1990 DOY 292 to 1998 DOY 291.

HAUS 03, AI-7 on ESDC, To recover a number of "Related Documents mentionned in the Ulysses EPAC documentation (UDSepacdata_EPAC_Doc.pdf)

Recommendations
HAUS 03, MIR-1 on UFA interoperability: SAMP is implemented on the Ulysses server but a contribution by the ESDC for the most popular Heliophysics Python libraries to be able to easily download Ulysses data from these libraries would be an asset. A TAP server may also be considered.

HAUS 03, MAR-1 on UFA support: The HAUS group acknowledged the unique scientific value of the data available through the ESDC Ulysses Final Archive. It recommends to provide ASAP the ESDC heliophysicsarchives’ group enough manpower to maintain, ingest new (best calibrated) data and develop up-to-date command line tools/APIs.  These tools are key to support the scientific community not only to access the Ulysses data but also compare them with data collected by the PSP and upcoming Solar Orbiter missions.

Please note that all action items related to Ulysses at the last meeting have been either closed or commented in the Actions section.

2. SOHO archive at ESDC: experiment reviews and recommendations

Reviewers
Louis Harra (UVCS), see annex 4
Nina Dresing (CELIAS), see annex 5

2.1   UVCS

See Annex 4

Short experiment description
UVCS (UltraViolet Coronograph Spectrometer) made measurements of the solar corona between 1.3 and 12 solar radii from sun center with high spectral and spatial resolution. The instrument creates an artificial solar eclipse in ultraviolet light, and allowed observation of the less intense light from the extended corona.The corona emits most of its ultraviolet intensity in spectral emission lines, containing information of the highly-ionized coronal plasma. At UV wavelengths:

• proton velocity distribution
• proton outflow velocity
• electron temperature
• O VI, Si XII, and Mg X outflow velocities and densities
• O VI, Si XII, and Mg X velocity distributions
• electron density and minor ion densities

UVCS has observed about 37 spectral lines in the extended solar corona for H I and ions of C, N, O, Mg, Al, Si, S, Fe, and Ni.

The purpose of the UVCS is to provide a detailed description of the extended solar corona that can be used to address a broad range of scientific questions regarding the nature of the solar corona and the generation of the solar wind.

PI team information
John L. Kohl: Principal Investigator (Harvard, USA)
Giancarlo Noci: Co-Principal Investigator (Universita di Firenze, Italy)
Website: https://www.cfa.harvard.edu/uvcs/

The UVCS PI team has defined the levels of its processed data as follows. As described on the PI website, the output from the UVCS instrument and delivered to the UVCS project by NASA, is called level 0 data. Level 0 data is not useful for any purpose except processing into level-1 data. 

Level-1 data is processed raw data which has been decompressed and repacked into data files in a logical form representative of the data products as delivered by the UVCS instrument. Level-1 data is stored in a standard external storage protocol (FITS files). Types of Level-1 data are the following: 

1. Spectral Data Files: These are FITS files containing uncalibrated detector count data (counts per pixel), as well as instrument configuration and exposure timing data. Each Exposure Sequence produces one Spectral Data File per detector. Each file includes a header that contains a brief, standard description for use by investigators.  
   
   
2. Visible Light Data Files: These are FITS files containing the time series of count rates from the White Light Channel, along with pointing information and information on the polarizer and neutral density filter. Each visible light data (VLD) file contains the data corresponding to one exposure sequence. Each file includes a header that contains a brief, standard description for use by investigators. 

Level-2 data are calibrated level-1 data which has been decompressed and are stored in a standard external storage protocol (FITS files). Types of Level-2 data are the following: 

1. Calibrated Spectral Data Files: These are FITS files containing calibrated detector intensities (photons cm ^-2 s^-1 sr ^-1), as well as instrument configuration and exposure timing data. All sequential exposures in a given file for a given height, grating position and common instrument configuration are added together except in the case of Star and CME observations. Each Exposure Sequence produces one Spectral Data File per detector. Each file includes a header that contains a brief, standard description for use by investigators.  
   
   
2. Calibrated Visible Light Data Files: These are FITS files containing polarized brightness (pB) values from the White Light Channel, along with pointing information and information on the polarizer and neutral density filter. Each VL Data File contains the data corresponding to one exposure sequence. Each file includes a header that contains a brief, standard description for use by investigators. 

Calibration Data Files contain the following: a) Parameter files (ascii) for pointing, spatial, wavelength and radiometric calibrations, and b) Flat field and dark count data (FITS). 

The UVCS Data Analysis Software (DAS version 5.1) allows the user to perform calibrations and obtain spectra that are wavelength calibrated and converted to photons cm ^-2 s^-1 sr ^-1 nm^-1. Calibrated pointing information can also be derived using this software. The software allows the user to write out processed spectral data files and processed visible light data files. Temporary processed spectral and visible light data files are produced at the EOF to be used as Quick Look data for observation planning. Processed data files are examples of level-2 data.

In addition, the UVCS program produces level-3 data which has undergone a higher level of processing. For example, image data files, including synoptic maps, are FITS files such as HI Ly-alpha maps of the corona that are provided as part of the Summary Data. There are also Data Catalogs to permit users to select observations based on pointing, observation date, type of feature observed, or other criteria.  They also include references to Mission Log Files, which record the observing sequences and other commands sent to the instrument. Most level-3 data will be produced by scientists in the course of analyzing the data using software and data that is publicly available. 

The UVCS/SOHO Data Analysis Software (DAS) and the calibration and help files are included in the SolarSoft package. 

Data content at ESDC
Only Level 1 data are available at ESDC SOHO archive. SOHO data stored at Goddard, in the Solar Data Analysis Center (SDAC) contain the level 2 files (final calibration).
However, level 2 or calibration status never appear in the result page when requesting UVCS data from SDAC, through the Virtual Solar Observatory. Moreover, the file naming convention of these FITS files are not standard.

HAUS 03, AI-11 on B. Fleck/UVCS PI team/ESDC, SOHO PS to contact the UVCS PI team to clarify which of the files delivered to SDAC are calibrated and transfer those files to ESDC for their ingestion and availability.

2.2   CELIAS

See Annex 5

Actions related to CELIAS (see HAUS 03, AI-12 on ESDC/B. Fleck)

Short experiment description
CELIAS (Charge, ELement, and Isotope Analysis System) measures the mass, ionic charge and energy of the solar wind, supra-thermal ions, and low energy energetic particles. The CELIAS instrument consists of three different sensor units (CTOF, MTOF, and STOF) coupled to a Digital Processing Unit (DPU).

Most urgent action related to CELIAS data

ingest on the SSA mission long file bundles available for (STOF) Solar EUV Monitor and (MTOF) Proton Montor

Proton Monitor data
Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html: 

• • CELIAS_Proton_Monitor_5min.tar.gz: bundle of year long files containing measurement at 5 minutes time resolution, from 1996 to 2016, please note that data for the year 2016 only, data are available up to 28 Sep. 
  • CELIAS_Proton_Monitor_30s.tar.gz: bundle of year long files containing measurement at 30s minutes time resolution, from 1996 to 2015.
  • These two files shall be ingested on the SSA with 
    • a processing level set as "Calibrated (L1)" (internal SSA metadata)
    • a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
    • an observation type set to "Solar wind param. 30s" or "Solar wind param. 5mn" instead of Proton
    • a Temp Res internal metadata parameter set to either 30s or 5mn

 Solar EUV Monitor

Two mission long tar files are available on the Virtual Solar Observatory for the calibrated CELIAS Proton Monitor data https://vso.nascom.nasa.gov/new/web_patch/SOHO_ML.html: 

CELIAS_SEM_15sec_avg.tar.gz, data from 1996 to 2017-05-11 stored in either monthly or daily ascii files per year, 15s resolution
CELIAS_SEM_daily_avg.tar.gz, data  from 1996 to 2017-05-11 stored in yearly files

 For these two files shall be ingested on the SSA with 

• a processing level set as "Calibrated (L1)" (internal SSA metadata)
• a calibration status set as true (internal SSA metadata), essential for command line access and GUI usability
• a detector set to Solar EUV Monitor and not STOF
• an observation type set to "EUV Irradiance 15s avg." or "EUV Irradiance 1 day avg." instead of Solar EUV Monitor
• a Temp Res internal metadata parameter set to either 15s or daily

Point of discussions with the SOHO PS

• • The two PM tar files superseed all the historical 1304 daily CDF files from 21/01/1996 to 24/02/2000, 5 min time resolution, available now at the SSA. These CDF files shall not be downloadable anymore. To be double checked with SOHO PS.
    
  • The two SEM files superseed the 49 monthly SEM CDF files of type SO_CEL_STOF_SEM_L1_V01-YYYYMMDDHHmm.cdf of 5 mn time resolution, covering the period from 1996 to 1999. These files shall not be available anymore for download.
  • Is there any way to get more up-to-date data bundles?
  • SOHO PS to confirm if SEM files can be associated with the SEM and not STOF as detector, even if SEM is a subunit physically attached on top of the STOF sensor; at the moment all the SEM files available are associated with STOF hence no data are available if SEM is selected; which is confusing. If STOF is the sensor decided, all these files shall be re-ingested accordingly.
  • Possible ingestion of all (up-to-date) PM calibrated files per Carrington Rotation at 1h time resolution available at the University of Maryland: http://umtof.umd.edu/pm/crn/; please note that these files do not content any (predicted not measured) Helium speed/velocity.
  • Up-to-date SEM files 15s and daily averages are available here: https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ ; shall SSA ingest them on a daily basis?

Processing level (general remark)

There are two processing levels for all these detectors: Level 1 Science data and Level ASCII, both are unclear and shall be renamed, for instance by

Calibrated (L1) instead of « Level 1 Science Data »
Raw data instead of « Level ASCII »

Calibrated files shall be the ones selected by default when searching any data from the CELIAS sensors and their subunits.
To be discussed confirmed with the SOHO PS. 

CTOF 

Status of CTOF data content at ESDC
CTOF sensor has produced 221 calibrated data during around 4 months (April 1 to August 16 1996) including the physical parameters
O densities
Fe densities
Heavy ion freeze-in temperatures 

Their processing level is Level 1 Science data. A mention of the time resolution (5 minutes) would help (see HAUS 03, AI12 (i))

The other processing level is called Level ASCII (unclear name, see HAUS3-AI12(ii)). It is related the rest of the (raw) data products: Pulse Height Data called Pulse Height Analyzer, Spectral Matrix Rates called Rates and Matrix Elements called ME. Additionally, Housekeeping data files are available.

For CELIAS/CTOF

(i) CTOF 221 CDF « Level 1 Science Data » calibrated files at ESDC/SSA need to be reingested as follows

• Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
• Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
• Temp Res keyword set to 5 mn
• Modify Observation type metadata as follows

(ii) Re-ingest all 15,616 raw data files with a Processing level set to « Raw data » instead of « Level ASCII » which is meaningless.

(iii) The availibility of raw data files up to 7 Sep. 2018 is confusing while only 4 months of science data available (April 1 to August 16 1996); documentation would help. PI team may provide it, to be discussed between SOHO PS and PI team.

(iv) New high level data (velocity spectra of heavy ions+n,T) could be provided by the PI group (at some point), as mentioned by the reviewer. Hence, a discussion between the SOHO PS and CELIAS PI team of data delivery timeline shall be held.

MTOF 

Status of raw data content: up to 7 September 2018 (25,146 files on February 20, 2019), to be re-ingested with Processing level set to « Raw data »
Status of science data content: 1 file, SO_CEL_MTOF_CME_L1_V01-199701082357.cdf, to be re-ingested with

• • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
  • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
  • Temp Res keyword set as 5 minutes
  • Modify Observation type set to Heavy ion densties and SW param. instead of "coronal mass ejection"

Content of MTOF science data for the whole mission is 1 file (except Proton Monitor data), covering a week in 1997 (08th to 16th January), a CME event subject of several scientific publication! This file contains calibrated measurements of densities (Ne, Np, O. Mg, Si, S, Ca, Fe, Na, Al, Ar, He, C, N) and velocities (Vsw, Vkinetic). The observation type is "coronal mass ejection", which has no obvious scientific meaning in this context. According to the PI team, there does not exist any level 1 data due to instrumental problems (but counts yes with many papers on the discovery of rare elements and isotopes in the solar wind). To be further discussed with SOHO PS and PI team if any other calibrated file of this type is available.

STOF

• Status of raw data content: up to 7 September 2018 (29,767 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
• Status of science data content: from 1996 to 1999, 49 monthly SEM files, 5mn average; most likely superseeded by the SDAC mission long data bundles and up-to-date data available at https://dornsifecms.usc.edu/space-sciences-center/download-sem-data/ 

Measures ions in the range from 20 keV to 3 MeV per charge through two separate sections of the electrostatic deflection system, one with curved (STOF) and one with flat deflection plates at higher energies (HSTOF). Energy measurement is done via a solid state detector. The Solar EUV Monitor (SEM) is physically mounted on top of the STOF detector. it measures the full solar disk absolute photon flux for the He II 30.4 nm line, as well as the absolute integral flux between 17 and 70 nm. The detector associated with the SEM data is STOF, hence no calibrated SEM data appear with SEM is selected as detector

HSTOF

• Status of raw data content: up to 7 September 2018 (9,341 files on February 26, 2019), to be re-ingested with Processing level set to « Raw data »
• Status of science data content: 1 file, SO_CEL_HSTOF_EHA_L1_V01-199602290000.cdf, to be re-ingested with
  • • Processing level renamed as « Calibrated (L1) » instead of « Level 1 Science Data » 
    • Calibrated metadata set to true, essential to improve usability and accuracy of the GUI and the SAIO, also called API or command line. 
    • Modify Observation type set to Hydrogen ENA flux instead of "Energetic Neutral Hydrogen Atom"

Content of HSTOF (high energy part of STOF) science data for the whole mission is 1 CDF file, covering period from 29/02/1996 to 23/12/1997. HSTOF has been working for years and is presently the only ENA instrument with energy range overlapping with Voyager Low Energy Charged Particle measurements of the energetic ion fluxes in the heliosheath, according to this article https://www.aanda.org/articles/aa/pdf/2018/10/aa32432-17.pdf. Until 2003, the whole range of the ecliptic longitude was scanned yearly. After 2005, only the flank sectors are accessible to observations. Hence, it is not understandable that only 2 years of calibrated data only are available on the SSA. To be further investigated.

Data Completeness wrt to GSFC

Here is the status on 26 February 2019

- CTOF: Goddard 20360 files (max (imported_date) 2019-01-11 07:58:54) /  SSA 15837 files
- MTOF: Goddard 33823 files (max (imported_data) 2019-01-11 07:59:13  /  SSA 21147 files
- STOF: Goddard 20463 files (max (imported_data) 2019-01-11 07:59:31)  /  SSA 29816 files
- HSTOF: Goddard 6768 files (max (imported_data) 2019-01-11 07:59:00)  /  SSA 9342 files

Discrepencies shall be investigated

Detectors Ordering on the SSA GUI

The ordering of the CELIAS detectors on the SSA is in alphabetical order and could be improved by listing them by sensors as suggested by the reviewer from high to low energy detectors, to be discussed with the SOHO PS. At the moment there are listed alphabetically.

Alternatively it could be listed as follows

Proton Monitor
Solar EUV Monitor
CTOF
MTOF
STOF
HSTOF
DPU 

To be discussed.

2.3   New AIs and recommendations

Here is the list of actions related to the review of the ESDC SOHO archive related to the following SOHO instruments: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO

HAUS 03, AI-11 on B. Fleck/UVCS PI team/ESDC, SOHO PS to contact the UVCS PI team to clarify which of the files delivered to SDAC are calibrated and transfer those files to ESDC for their ingestion and availability.

HAUS 03, AI-12 on B. Fleck/UVCS PI team/ESDC, for CELIAS/CTOF,

(i) All CTOF files of processing level "Level 1 Science data" are calibrated. Hence, these files shall be re-ingested to eventually be selectable as calibrated products. These files shall be the ones selected by default.
(ii) Observation types of the calibrated CTOF files shall better reflect their file content, including their time resolution
(iii) The processing level entitled Level ASCII is unclear and must to be corrected, even though extracted automatically from the metadata of these (raw data) files.
(iv) The availibility of raw data files after August 15, 1996 is confusing and not in line with the 4 months of science data available (April 1 to August 15 1996) .
(v) New high level data (velocity spectra of heavy ions+n,T) could be provided by the PI group (at some point). Hence a discussion of data delivery timeline shall be considered.

[1]CDS calibration: assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps

3. SOHO archive at GSFC and long term archives of HINODE and IRIS

See Annex 6

GOLF

• Most relevant scientific products at GSFC: Level 2, 3 mission-long files (11 April 1996 – 10 April 2018), newly calibrated line-of-sight Doppler velocities files, hence 3 files; GOLF_22y_PM1 (from photmultiplier 1), GOLF_22y_PM2, GOLF_22y_MEAN with timing as good as ±5s, as sdescribed in Appourchaux et al. 2018A&A...617A.108A. Old (deprecated) data have been moved to a private area on the GSFC server and will be available upon request (but not searchable through the GUI/VSO).
• Other products: daily photomultiplier data (not really useful for community, since raw counts)
• Coming up: New series (with improved FITS header info) expected later this year (2019)

VIRGO

• Most important scientific products at GSFC: Level 2, mission-long, calibrated Total Solar Irradiance (TSI) time series (daily + hourly averages)
• Other products: daily raw data (named level 0) for TSI (PMO6 and DIARAD), SPM, and LOI
• Coming up:Mission long, detrended SPM files (blue, red and green channel) expected as soon as FITS headers are fixed, VIRGO team also working on calibration of LOI files

MDI

• Most important scientific products at GSFC: all MDI magnetograms, in particular the 96-min full disk synoptic magnetograms. However, no Dopplergram series are available, because these files were transferred directly to Stanford and the size of the MDI helioseismology data was considered too big at that time (MDI high-rate telemetry 160 kb/s vs 40 kb/s for rest of SOHO). MDI helioseismology series currently served by JSOC at Stanford Univ., as part of the SDO archive 
• Coming up: to comply with long-term archiving NASA requirements, the complete SDO archive is being copied to the SDAC at GSFC, i.e. in the future the SOHO archive at GSFC will include all MDI data.

SUMER

• Coming up: full reprocessing of SUMER data, both level-0 and level-1 (calibrated) completed by the PI team. New organization of SUMER data sets, now bundled by “studies” and provided as tar balls of individual FITS files for each exposure, and a FITS header describing the tar ball. SUMER team is currently working on some minor consistency issues and a description of the new data sets. Ingestion of new data expected very soon. Old (deprecated) data will be moved to private area on server and will be available upon request (but not searchable through GUI). SUMER off since 4 April 2017.

CDS

• Coming up: Complete reprocessing of CDS/NIS data, both level-0 and level-1 (calibrated) completed by the PI team. However, inconsistencies detected in pointing info in headers during final quality check. Last telecon with CDS team to resolve these issues on 20 Dec 2018. Hopefully, these problems can be fixed quickly and we have the final, complete and consistent level-0 and level-1 data in the archive before summer 2019. Old (deprecated) data will be moved to private area on server and will be available upon request (but not searchable through GUI). CDS hibernated since 5 September 2014

EIT

• Most important scientific products at GSFC: all raw data that can be calibrated with SolarSoft.
• PI still working on “final” calibration of level-2 files, see recommendation HAUS 03, MAR-2

UVCS

• Most important scientific products at GSFC: final calibrated data
• No changes to UVCS data expected
• 23 Jan 2013 end of UVCS operations
  
  

LASCO

• No new data product

SWAN

• No new data product

CELIAS

• Most important scientific products at GSFC: calibrated mission-long 
  • solar wind Proton Monitor (PM) data (proton speed, density, thermal speed, direction), 5-min and 30s averages
  • SEM EUV flux (daily and 15 s averages)
• No new data expected

COSTEP

• Most important scientific products at GSFC: since early October 2018, new EPHIN Level-3 data product available including

- Mission-long, GEANT4 calibrated proton and helium fluxes for all four energy channels
- Daily, hourly, 30 min, 10 min, 5 min, and 1 min averages
- Served as mission-long tar balls

• Coming up: level-3 electron data – much more difficult to calibrate, but COSTEP/EPHIN team at Kiel working on that

ERNE

• Most important scientific products at GSFC: On 30 August 2018, the entire ERNE data set has been replaced with new, corrected data.ERNE team had noticed some errors in part of the ERNE helium data collected before 18 April 2000. For completeness and consistency, ERNE team re-processed entire ERNE data set.To avoid confusion and duplication, all old ERNE records were deleted (moved to private area of server) and the complete, new ERNE data ingested (both at GSFC and ESAC).

Long-term plans for Hinode and IRIS archive

Hinode and IRIS data currently served from Hinode SDC Europe at Univ. of Oslo
Hinode and IRIS data also served from several other places, e.g. LMSAL
Archive structure rather simple, just a directory tree with FITS files
After operational phase of Hinode, complete set of FITS files will be transferred to ESAC and ingested into heliophysics archive for long-term preservation

Summary

SOHO archive at GSFC in good shape overall

GOLF, LASCO, and SWAN teams a bit late in delivering latest data
LASCO quicklook data served to community by FTP
UVCS finalized
SUMER and CDS being finalized
Complete MDI data sets being copied to SDAC
Access to SOHO archive at GSFC through VSO and IDL/Python APIs
GUI to GSFC archive had to be taken down because of security issues in the 20+ year old Perl code we were using
Same Perl scripts also used for real-time images, movie theater, bibliographies
Some services not up yet due to limited manpower to rewrite software

4. FP7 projects related to ESA heliophysics missions

See Annex 7

Reviewer: V. Génot

FP7 projects reviewed at this meeting: HESPERIA, FLARECAST, SEPSERVER
FP7 projects previously: IMPEX, HELIO, HELCATS, CASSIS, Spaceinn

4.1 HESPERIA

High Energy Solar Particle Events foRecastIng and Analysis (HESPERIA), https://www.hesperia.astro.noa.gr/; H2020 project from May 2015 to April 2017. HESPERIA exploits available large datasets stored into databases including the Neutron Monitor Database (NMDB) and SEPServer that have been developed under FP7 projects from 2008 to 2013

Model 1: Release 

• Uses the fact that electrons arrive 30 to 90 minutes faster at L1 than the protons to predict proton flux by utilizing the actual electron flux and the increase of the electron flux in the last 60 minutes
• Based on Posner, A., Up to 1-hour forecasting of radiation hazards from solar energetic ion events with relativistic electrons, Space Weather, Vol. 5, S05001, doi:10.1029/2006SW000268, 2007
• The model uses SOHO/EPHIN (0.25 to 1.0 MeV ) and ACE/EPAM (0.175 to 0.315 MeV) electron fluxes to forecast proton fluxes in 30, 60 and 90 minutes
• The same model runs at NASA/GSFC with SOHO/EPHIN only (https://costep2.nascom.nasa.gov/). An alerts system is in place; registration by email

Model 2: UMASEP

• A predictor of >500 MeV SEP proton events at geostationary orbit
• >500 MeV solar protons are so energetic that they usually have effects on the ground, producing what is called a Ground Level Event (GLE).
• Based on [Núñez, 2011, 2015]
• Did not work at the time of the review

Catalogue

• Catalogue of events to investigate SEP spectrum prediction using microwave observation.
• Investigates if the hardness or softness of the proton spectrum in interplanetary space can be predicted from the shape of the microwave spectrum (see report)
• Catalogue with 64 events between Nov 1997 and Jan 2016
• GLE inversion software
• Relativistic (>500 MeV) Solar Energetic Particles (SEPs) enter the Earth’s atmosphere sporadically leading to an increase of the intensities recorded by the worldwide network of neutron monitors (NMs) on the ground. These events are known as ground level events or ground level enhancements (GLEs).
• HESPERIA developed the first inversion software for NM observations that infers the release timescales of relativistic SEPs at or near the Sun and the characteristics of their transport in the interplanetary space. 
• The Python code can be used to estimate the particle fluxes at 1 AU to assess e.g. radiation dose rates at flight altitudes

Potential value added product based on SOHO data

The result of REleASE code for the (30-90 min) forecasts based on electron measurements by SOHO/EPHIN. However, no obvious scripting way to access the results/plots

4.2 FLARECAST

Flare Likelihood And Region Eruption foreCASTing (FLARECAST) EU programme ran from January2015 to December2017; http://flarecast.eu/
Flare prediction API works for 2013-2017, computes predictions of flares of different GOES class (X, M, C) in different forecast windows with several algorithm used
Results are fully integrated in theAPI: https://api.flarecast.eu/
A REST API is available and can be included in (Python, IDL, JAVA) codes to embed the prediction algorithm
Can certainly be used as a prototype for future automatic detection of flares but not directly for SOHO and Ulysses post-event analysis

4.3 SEPSERVER

FP7; finished since a few years, some links already do not work
SEPServer is a tool developed for the analysis of solar energetic particle (SEP) events.
http://server.sepserver.eu/
SEPServer provides access to SEP datasets, related electromagnetic observations and browsing, plotting and analysis tools facilitating the use of these data.
Access to an “inversion program” (linux on IDL, last version from 2013) which enables to infer SEP origins by inverting processes due to detection and interplanetary transport

SEP inversion: IDL Software for Analyzing Solar Energetic Particle Events

http://server.sepserver.eu/inc/SEPinversion/index.htm
Running the program for one event produces 4 PDF (~100p) of plots
Here: 24/9/2001 for Ulysses

Certainly usefull but difficult to use without reading all the documentation and references given on the server
Edit after N. Dresing’s intervention: it would be beneficial to, at least, store the code (+Green’s functions and other useful data) as the code is unique and well documented

SEPServer value added products of interest

SOHO and ULYSSES proton event lists (2)
Associated plots (rich list)
Important note: it is necessary to register to access some of these resources; otherwise only the (not rich) catalogues are available

4.4   New AIs and recommendations

5.Interconnecting ESA helio archives with Ground-Based data

The goal here is to ease the life of scientific for her/him to download together GB data related to ESA helio datasets as an option. ESA helio archives and heliophysics ground-based data can indeed be scientifically linked (SuparDarn and EISCAT network with Cluster, Solar Orbiter and DKIST, SOHO and GONG etc...). M. Roth presented his vision on how these archives could be linked in the future without GB data being stored at ESAC. As reminder, the SOHO HMI datasets alone are 10 times bigger in volume than the entire SOHO database for all the other SOHO experiments.

6. ESA's Long Term Strategy for science archives discussion

Guido De Marchi illustrated the long-term strategy for the space science archives that was adopted by ESA in December 2018. He addressed how the previous strategy has evolved in order to reflect the many suggestions and comments submitted by internal stakeholders and by members of HAUS as well as of the Astronomy Archives Users Group and Planetary Science Archives Users Group, i.e. the three users group. The main pillars of the long-term strategy continue to be

1. enable maximum scientific exploitation of the data;
2. preserve data and information (including software and documentation) in the long term;
3. whenever possible merge the archives into the operations of the missions in order to facilitate the long-term preservation.

The new strategy stresses the importance of legacy missions, whose data and associated software must be dutifully identified and preserved, and of the role of the scientific community in general and the users group in particular in advising ESA on the priorities.

The presentation also covered new activities that will be implemented by the ESDC to support archives users, including a Science Exploitation and Preservations Platform (SEPP) and an Archival Research Programme (ARP) that had been already discussed with HAUS members at the previous meeting. By integrating information and processing assets from multiple domains into a single environment, SEPP will deliver advanced analysis and collaboration services. This will allow the ESDC to host high-level science products based on ESA science data and associated software and other legacy products generated by the community, for instance as part of EU funded programmes such as Horizon 2020 and Framework Programme 7. Similarly, by allowing science users to bring their code to the archives, SEPP will permit quick-look and analysis software to be run on the data before downloading them. The ARP is meant to help archive users, and in particular early career scientists, to learn how to make the best use of the archival data to pursue their research interests. Besides summer schools, the programme includes opportunities to visit ESAC or ESTEC to explore the data. More specific details of the ARP will be discussed with all Users Groups in the coming months.