Meeting Date20180604 - 20180605RefHELIO-MOM-0001
Meeting PlaceKIS, Freiburg University, GermanyChairmanHarra, L, MSSL, UK
Minute's Date20180625Participants

Chen, C., QMW college, UK
De Marchi, ESTEC/ESA (WebEx)
Dressing, N., FMI, Finland  (WebEx)
Génot, V., IRAP/CNRS, France
Harra, L., MSSL, UK 
Janvier, M., IAS, France (WebEx)
Martinez, B., ESAC/ESA
Masson, A., TpZ for ESA 
Merin, B., ESAC/ESA (Webex)
Osuna, P., ESAC/ESA (WebEx)
Roth, M., Freiburg University, Germany 
Taylor, M., ESTEC/ESA (reviewer)


SubjectMinutes of the second Heliophysics archives USer group meetingCopy

Scope of this meeting

Storing raw data files is of course not enough, ESDC needs to make sure that the heliophysics archives contain the most up-to-date calibrated files. If not calibrated, clear instructions on how to calibrate them shall be made available with the calibration software (at least a valid hyperlink to a community software). 

  The general focus of this meeting was on the completeness and possible curation of some of the datasets contained in the Ulysses and SOHO Heliophysics archives (see detailed list below). Additionally, a number of value added datasets generated by EU funded FP7/H2020 projects have been reviewed. The central question being: if based on ESA heliophysics archives, would the HAUS group recommend to store any of these value-added datasets together with their related archives? As reminder, this has been done for the MAARBLE and ECLAT FP7 projects within the ESA Cluster science archive. 

 Each HAUS group member was invited to review at least two experiments from either Ulysses or SOHO, and/or two FP7/H2020 EU funded projects related to Heliophysics. During this meeting the following experiments and EU projects (in dark bold) have been reviewed. The ones highlighted in red bold have not and will be reviewed during upcoming meetings. 

SOHO
Experiments reiewed during this 2nd HAUS group meeting: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO 
Remaining experiments to review: CELIAS, SWAN, UVCS  

ULYSSES
Experiments reiewed during this 2nd HAUS group meeting: COSPIN, DUST, FGM/VHM, GAS, GRB, SWICS, SWOOPS
Remaining experiments to review: EPAC, URAP 


Curation/Completeness review guidelines

For each experiment 

List of data products available on the archive per instrument
Which datasets are available?
What is not calibrated? 
Can it be calibrated? 
How?  through community software package for instance?
If not, who/which group could do it? with eventually an estimate of the manpower needed when applicable.
What is missing? Please compare to what is available on PI or PI team member website(s)?
What would you recommend to archive in the long term?
Any ground-based data essential to support data analysis that shall be stored as well?
Are datasets stored in format compatible with community (visualization/data analysis) tools?

Be aware, that for instance on SOHO, there are various (PI) websites containing up-to-date derived products that are not stored in the archives yet. Which one of those, if any, shall be stored in ESA archives?

  

FP7/H2020 EU funded projects
Review guidelines: shall we archive anything of the value-added datasets produced by the various Heliophysics archives related FP7/H2020 projects listed below? If yes, which one(s)? 

EU funded projects related to Heliophysics 
-----------------------------------------------------
H2020    PROGRESS         
H2020    HESPERIA   

H2020    Flarecast

FP7      SPACECAST        
FP7      SPACESTORM       
FP7      ESPAS            
FP7      COMESEP          
FP7      SWIFF            
FP7      EURISGIC         
FP7      ATMOP  
               
FP7      IMPEx           
FP7      HELCATS         
FP7      SOTERIA                
FP7      SEPServer
       

FP7   HELIO

FP7  CASSIS 

General recommendation 

For long term preservation, the HAUS group recommends that the best practice is to store a calibrated version of each dataset, together with raw data. Clear guidelines on how these calibrated datasets have been generated shall of course be made available. 

Action items of the meeting

HAUS 02, AI-1 on M. Roth/A. Masson, to a review the content of VIRGO datasets, software and value-added products at PMOD/WRC and the scientific interest of their possible inclusion in the SSA at ESDC. 

Answer from SOHO PS: Good idea. Note that we already include the daily and hourly TSI values as single ASCII files in the Goddard archive, as well as the photospheric solar activity index S_Ph (from CEA Saclay).

HAUS 02, AI-2 on B. Fleck, to check if any of the dozens MDI datasets available at Stanford, http://soi.stanford.edubut not at ESAC SSA shall be included in the SSA for long term preservation. A detailed list of datasets is available in Annex B.

HAUS 02, AI-3 on ESDC, to ingest a single FITS file of mission-long calibrated GOLF velocity series, available at GSFC SOHO data repository 

HAUS 02, AI-4 on A. Masson, to check with the SOHO PS if the following value added datasets suggested by M. Roth, highly complementary to SOHO data, shall be archived within the ESDC SSA

  • ingestion of HMI datasets during the overlapping period when both instruments were operating,
  • "Mark-I" solar spectrophotometer database (IAC, Spain)

This point was raised in a telecon (June 25, 2018) and those data are not considered to be included in the SOHO archive for the following reasons: HMI data itself is orders of magnitude  bigger than the SOHO archive. And the Mark-I instrument is only one instrument of the global BISON network. More generally, the SOHO PS does not think it is 
necessary for ESAC to replicate services offered by the Virtual Solar Observatory (VSO), which provides access to data from all these instruments.

HAUS 02, AI-5 on A. Masson/B. Fleck, to check if a calibrated version of all SUMER data exists at Max Planck MPS and shall be ingested within the ESDC SSA, together with the raw data presently available. Thumbnails of the SUMER data only available at MPS may also be considered to be ingested in order to browse these data.

SOHO PS: They do exist, but as IDL save files. We are in contact with the PI to provide these as FITS files to the Goddard archive, that will then be transferred to the SOHO archive at ESAC. We are 99% there, and simply need a last "push" to complete this action. Key personnel at MPS has been busy with Solar Orbiter in recent years.

HAUS 02, AI-6 on A. Masson, to discuss with SOHO PS on possible ingestion of LASCO level 1 data, a calibrated version of all LASCO data and CME catalogs (e.g. CACTUS, ARTEMUS, CORIMP, CDAW, NRL LASCO CME list incomplete, COR1 CME catalog, HI1, catalogs generated by the HELCATS project). 

SOHO PS: Yes, if and when available, we should include Level 1 data. As for the CME catalogs, as long as the mission is active and those catalogs evolve, we should provide pointers. Once the mission is in post-ops, we should include the final catalogs.

HAUS 02, AI-7 on ESDC, to ingest new level 3 COSTEP/EPHIN data set under development at Kiel university (protons, helium) and documentation when received, expected end of August 2018. Possibility of visualizing the data shall be considered 

HAUS 02, AI-8 on ESDC, to ingest new ERNE .SL2 data set with H and He intensities in preparation by Turku group and documentation when received. The delivery of the same data with a finer energy resolution is also envisaged by this group. Furthermore, the ERNE PI group also want to provide heavy ion data with channels. Possibility of visualizing the data shall be considered.

HAUS 02, AI-9 on ESDC,Basic visualization is very useful for spectroscopic data. An 'atlas' is included at the RAL CDS website (http://solar.bnsc.rl.ac.uk/atlas/atlas.shtml). Manpower needed to be investigated.

HAUS 02, AI-10 on ESDC,Access to calibrated CDS data is a priority. ESDC SSA shall store calibrated Level 2 data. At bare minimum clear information about the calibration procedure with solarsoft shall be made available. A simple calibration version of the data assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps.

SOHO PS: The CDS team is going through a final quality check before submitting the final calibrated Level 2 data. This is expected very soon

HAUS 02, AI-11 on ESDC,EIT Calibration info on https://umbra.nascom.nasa.gov/eit/eit_guide/gui de.html shall at least linked, best transferred.

SOHO PS: Yes, and we will also have calibrated Level 2 EIT data in the archive, so that the users won't have to run eit_prep.pro anymore. This is the top priority as regards EIT. Note that all instruments have "resource pages" describing the calibration of their data:
https://soho.nascom.nasa.gov/data/archive/instruments.html

HAUS 02, AI-12 on ESDC,EIT movies from CME catalogues shall be included, this is a useful resource.

SOHO PS: "I wonder how useful those are. I very much prefer a flexible tool such as the SOHO Movie Theater, which allows users to set the time frame and control the speed of the movies."

HAUS 02, AI-13 on ESDC,For imaging data, the EIT daily movies are critical.  They can be used by users from any community easily. And can give basic info (e.g. is there a coronal hole, is there a flare, are there active regions?). Movies available at IAS shall be ingested in the ESDC SOHO science archive.

https://www.ias.u-psud.fr/eit/movies/

SOHO PS: see above. And note also other tools such as Helioviewer and JHelioviewer.

HAUS 02, AI-14 on ESDC,for the Ulysses COSPIN experiments, update all hyperlinks to instrument team pages.

HAUS 02, AI-15 on N. Dresing/A. Masson,to follow up on the delivery of Ulysses KET and HET intensity files to ESDC.

HAUS 02, AI-16 on N. Dresing/A. Masson,to check any data is missing from Ulysses HET for high resolution data (until 2004 173), prime resolution HET data provides yearly zip files until mid 2009, daily files

HAUS 02, AI-17 on ESDC,for Ulysses DUST, Ingest V3.1 PDS3 DUST dataset available at the NASA PDS small bodies sub node

HAUS 02, AI-18 on ESDC,to interact with the Ulysses FGM/VHM PI team for a possible delivery of a CDF ISTP compliant version of the files, together with a more logical dataset tree (all data from one instrument in one place) with unzipped files  labelled in clear consistent way, use 4-digit years in filenames, use of consistent time stamps in files.

HAUS 02, AI-19 on ESDC,for the Ulysses GAS experiment, fix the broken link to the PI team at Max Planck MPS; ingest newest calibrated version dataset, recalibrated in 2013. 

HAUS 02, AI-20 on ESDC,for the Ulysses GRB experiment, to correct the data description (wrong labelling of the schematic view of the detectors); compile C and Fortran programs to generate the solar flares (prime resolution) and GRB datasets (high time resolution) in a readable format; ingest them in the UFA; Data visualization with meaningful time resolution. 

HAUS 02, AI-21 on ESDC,for the Ulysses GWE experiment, to investigate why no data is stored in the archive.

HAUS 02, AI-22 on ESDC,for the Ulysses SWICS experiment, to make available the latest version of the latest calibration software (version 2.3), available on the Ulysses cosmos page. Detailed test of this software shall be performed. 

 HAUS 02, AI-23 on ESDC,for the Ulysses SWOOPS experiment, to interact with Bruce Goldstein and Anna Tenerani to ingest the following missing datasets in the UFA 

  • missing ion distributions: important to include (Bruce Goldstein)
  • updated ion moments: moments with anisotropy/drift at NASA, should be included 
  • electron distributions  (Bruce Goldstein)
  • full electron moments, including heat flux? (Goldstein and Anna Tenerani)
  • If possible: DSN +operational reports
  • spacecraft pointing for 3D distributions

Discuss with PI and NASA their conversion to CDF

HAUS 02, AI-24 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogue developed through the HELIO FP7 project after the end of operations, for long term data preservation.

HAUS 02, AI-25 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogues developed through the HELCATS project, all catalogues are now closed, for long term data preservation.

Outline

Agenda

04 June, 14h-18h

Review of SOHO and Ulysses ESDC archives and FP7/H2020 Helio projects: rationale (A.Masson), 25 mn

ReviewIMpex, HelCATS, HELIO, CASSIS FP7 projects(V. Génot), 45 mn

Review of Ulysses VHM, FGM, SWOOPS experiments(C. Chen), 45 mn

Coffee break

Science Archives Long-term Strategy (G. De March), 45 min

Review of Ulysses SWICS (M. Taylor), 20 mn


05 June, 09h-13h

Review of SOHO CDS and EIT instruments, report about FP7 HELCATS (L. Harra), 45mn 

Review of ULYSSES COSPIN and SOHO COSTEP and ERNE experiments (N. Dressing), 45mn

Coffee break

Review of SOHO MDI, VIRGO and GOLF; EU Helioseismology program (M. Roth), 45mn

Review of Ulysses DUST, GAS, GRB and GWE experiments (A. Masson), 45 mn

Review of SOHO LASCO and SUMER, remarks on EIT and GOLF (M. Janvier), 45mn


Review of SOHO and Ulysses ESDC archives and FP7/H2020 Helio projects: rationale (A.Masson), 25 mn

ReviewIMpex, HelCATS, HELIO, CASSIS FP7 projects(V. Génot), 45 mn

Review of Ulysses VHM, FGM, SWOOPS experiments(C. Chen), 45 mn

Coffee break

Science Archives Long-term Strategy (G. De March), 45 min

Review of Ulysses SWICS (M. Taylor), 20 mn


05 June, 09h-13h

Review of SOHO CDS and EIT instruments, report about FP7 HELCATS (L. Harra), 45mn 

Review of ULYSSES COSPIN and SOHO COSTEP and ERNE experiments (N. Dressing), 45mn

Coffee break

Review of SOHO MDI, VIRGO and GOLF; EU Helioseismology program (M. Roth), 45mn

Review of Ulysses DUST, GAS, GRB and GWE experiments (A. Masson), 45 mn

Review of SOHO LASCO and SUMER, remarks on EIT and GOLF (M. Janvier), 45mn

1. Rationale of this review

See Annex 2

1.1   Curation and completeness of the ESDC helio archives

  The terms of reference of the HAUS group can be found in the Annex A of this document.

It consists of four points, one of them being: "Address shortcomings or missing functionality in individual archives (e.g. Ulysses data visualization; ISS-SOLACES value added products; visualization of SOHO in-situ instruments data...)".

 The rationale behind this sentence is the following. To best serve the scientific community and eventually design an overarching Heliophysics multi-missions archives, a complete review of the status of the ESDC helio archives needs to be performed. This is to make sure they contain the most up-to-date calibrated datasets and whenever possible a calibration version of the files. Otherwise, its use may be very limited.

 Recently, a significant increase in the usage of the Ulysses archive has been observed. This increase may be attributed to the preparation by the scientific community of the upcoming Parker solar probe and Solar Orbiter. This reinforces the need of legacy archives and the importance of keeping it up to date. As we will see, once the operations of a mission are over.  some of the PI teams have been working on generating better calibrated data which are not all included in the Ulysses archive. In other words, data from these Ulysses are not frozen once the operations are over and regular updates would enable to keep these archives up-to-date and eventually considered by the community as the central data/metadata/software/documentation repository. 

1.2   Rationale of the review of Ulysses and SOHO archives

SOHO/Ulysses data are either stored as fully calibrated files, not fully calibrated (e.g. count rates instead of flux) or raw files+calibration software. 

Goal of this exercise is to get a clear detailed picture within ESDC SOHO/Ulysses archive of their 

  • data availability, 
  • calibration status, 
  • what should be better calibrated, by who?
  • value added datasets that could/should be included(e.g. PI websites, FP7/H2020)

1.3   Status of the SOHO archives

SOHO (1995-)

Status of SOHO archives 

NASA: prime archive of access up to early 2018

ESDC:since 2011, mostlyup to date, some minor identified ingestion issues

IAS, MEDOC: historical, incomplete

Various PI sites: some up-to-date and some with value added products

Status of SOHO archive at NASA

Offline for months due to security issues identified, which requires recoding of old Perl scripts; this is affecting several components of the archive system. This may take a while to fix it => reinforces the importance of the SOHO archive at ESAC.

The most up to date datasets are available through the VSO.

1.4   Status of the Ulysses archives

ESDC: modern GUI with some visualization capabilities 

NASA: historical ftp server

Review of ESDC SOHO and Ulysses ESDC archives and FP7/H2020 Heliophysicsprojects: rationale

2. SOHO archive at ESDC: review and recommendations

The following SOHO experiments have been reviewed.

CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO 

Reviewers

Nina Dresing (COSTEP, ERNE)

Miho Janvier (LASCO and SUMER)

Louise Harra (CDS, EIT)

Markus Roth (MDI, VIRGO and GOLF)

2.1   Helioseismology

See Annex 3

2.1.1      VIRGO

VIRGO: Variability of solar IRradiance and Gravity Oscillations, consists of several instruments 

Very low resolution of the sun, basically „Sun as a star“ 

Data from the instruments seems to be complete on the archive

PMOD

DIARAD

LOI

SPM (all three channels)

+ housekeeping data


Other sources of SOHO VIRGO data

CEA Saclay: SPM data working; FITS data 

IAS Orsay: Some documentation not sufficiently maintained 

WRC/PMOD, Davos: offline at the time of the review

NASA Archive: Points to Davos, which was offline at the time of the review


Conclusion: VIRGO LEVEL 1 datasets at ESDC SSA are complete and calibrated, pending a review of the content of VIRGO datasets, software and value-added products at PMOD/WRC.


HAUS 02, AI-1 on M. Roth/A. Masson, to a review the content of VIRGO datasets, software and value-added products at PMOD/WRC and the scientific interest of their possible inclusion in the SSA at ESDC. 

2.1.2      MDI

MDI: Michelson Doppler Imager

Telescope opening: 12.5 cm 

Camera: 1024 x 1024 CCD maximum harmonic degree: l ~ 1000

Field of view: 

–  Whole Sun (34‘x34‘) with lower resolution (20‘‘) 

–  Whole Sun (34‘x34‘) with low resolution (4‘‘) 

–  Sections (10,5‘ x 10,5‘) with high resolution (1‘‘) (only 1-2 months per year) 


Spectral range: 676.7 ±19 nm

Target: photosphere

Sampled at four wavelength points 

Doppler shift is determined from intensity profile


Cadence: 1 minute

Operation May 1996- April 2011 (recently reactivated for a short time: J. Schou priv. comm.)


SOHO science archive (SSA) at ESAC MDI data content

Only 96m magnetograms, 60s dopplergrams and continuum images 

No raw data, filtergrams, line depth, various magnetograms and dopplergrams

No derived products

Data format: FITS


But a lot of MDI datasets available at Stanford, http://soi.stanford.edu

are not available at ESAC and need to be included eventually to make this part of the SSA archive complete and an equivalent source of MDI data.


Conclusion: For MDI, SSA holds only a limited number of datasets. Dozens of MDI datasets available at Stanford, http://soi.stanford.eduare not available in the SSA archive.  The full list can be found in Annex B. A discussion with the SOHO PS is needed to check if any or all of these datasets shall be included in the archive for long term preservation.


Overlapping data period of a few months when HMI and MDI where both operating is a must. Requires the ingestion of a few months of HMI data (Level-0 and Level-1). See AI-4.


HAUS 02, AI-2 on A. Masson, to check with the SOHO PS if any of the dozens MDI datasets available at Stanford, http://soi.stanford.edubut not at ESAC SSA shall be included in the SSA for long term preservation. A detailed list is available in Annex B.


2.1.3      GOLF

GOLF: Michelson Doppler Imager

Sun as a star observations in Doppler (integrated over solar disk)

Two line positions at Na D

SOHO Science Archive v2.7.1 looks complete for calibrated line-of-sight velocity until September 2017. 

PI data at CEA Saclay with own calibration: email to Rafa García rafael.garcia@cea.fr



Remark from M. Janvier (see Annex 4)

GOLF: 3 main calibration teams:

•        Nice (Obs. Lagrange): Calibration by G. Grec https://www-n.oca.eu/renaud/golf.html

•        US team (R. Ulrich, previously Tucson then Boulder?)

•        IAS – CEA (P. Boumier + R. Garcia): same calibration, using magnetic modulations. 


Both data should be the same at the moment: https://www.ias.u-psud.fr/golf/templates/index.html  but P. Boumier has started working on full calibration of data. Hopes to have full database, newly calibrated by spring 2019 (new calibration includes instrumental correction).


HAUS 02, AI-3 on A. Masson, to check with the SOHO PS if the calibrated version of GOLF data from the PI data at CEA Saclay and Nice Observatory shall be eventually ingested in the ESAC SSA for long term preservation. 


2.1.4      Related Helioseismology measurements

SDO HMIsince May 2010

         Successor of MDI, HMI data when operated in parallel with MDI could be considered to be archived, at least linked


GONGsince 1995

         Parallel observations (Dopplergrams, Intensity, Magnetograms) to MDI, GOLF, VIRGO from the ground


BiSON

         Ground-based network operated by University of Birmingham since the 1980s

         Sun-as-a-star observations in velocity

         Parallel observations to MDI, GOLF, VIRGO from the ground

         Potassium line


Possibility to download GONG and Bison data together with MDI, GOLF and VIRGO would be very useful.


Sunrise/IMAX

         Imager on a balloon flight 


HAUS 02, AI-4 on A. Masson, to check with the SOHO PS if the following value added datasets suggested by M. Roth, highly complementary to SOHO data, shall be archived within the ESDC SSA

  • ingestion of HMI datasets during the overlapping period when both instruments were operating,
  • "Mark-I" solar spectrophotometer database (IAC, Spain)



2.2   Chromosphere and Corona

Where to find the data?

From the SOHO website, pointing to the ESDC Java web start archive

ESDC archives website, pointing to the ESDC Java web start archive

Studies in JWS is useful and not availabl on the VSO

VSO

MEDOC

SUMER and LASCO instrument webpages

2.2.1      SUMER

SUMER: Solar Ultraviolet Measurements of Emitted Radiation

Current PI: Werner Curdt, MPS

PI website: https://www2.mps.mpg.de/projects/soho/sumer/

In hibernation since August 2014

The SUMER Data in the SOHO Archive, described in the Solar Physics paper linked below 

https://link.springer.com/article/10.1007/s11207-013-0449-7

Types of original telemetry: FITS  and IDL-restore file (at Max Planck MPS)

Both types of data, FITS files and restore files, are not calibrated or converted to physical units nor are they corrected for shortcomings of the hardware of the SUMER instrument, but correction and calibration procedures are available. 

Corrections to do: 

  • dead-time correction 
  • local gain correction 
  • flat-field correction 
  • geometrical distortion correction 
  • radiometric calibration 
  • Some other corrections can also be important in particular cases 

According to IAS colleagues of M. Janvier: it seems like someone at MPS has calibrated all SUMER data, but nobody knows whether this has been made accessible (maybe contact W. Curdt?). At IAS, P. Lemaire (retired) still works on calibration. 

Thumbnails are SUMER data only exist at MPS and could be added.

HAUS 02, AI-5 on A. Masson/B. Fleck, to check if a calibrated version of all SUMER data exists at Max Planck MPS and shall be ingested within the ESDC SSA, together with the raw data presently available. Thumbnails of the SUMER data only available at MPS may also be considered to be ingested in order to browse these data.

2.2.2      LASCO

LASCO: Large Angle and Spectrometric Coronagraph experiment

Coronagraph instruments in 3 parts (C1, C2, C3) 

C1 1.1 to 3 solar radii 

C2 1.5 to 6 solar radii 

C3 3.7 to 30 solar radii 

First year-and-a-half of the SOHO mission, all three instruments worked perfectly. However, C1 is not functioning since 1998.


PIs

C1: Rainer Schwenn
C2: Philippe Lamy (prev. LAM, now LATMOS)
C3: Russ Howard 


PI website: https://lasco-www.nrl.navy.mil

Website from NRL is well maintained:
-calibration explanation -data products 

-retrieval of data 


Data stored in ESAC SSA: raw data Level 0.5


Calibration procedure for C2 and C3 through SolarSoft explained here

https://lasco-www.nrl.navy.mil/index.php?p=content//level_1/lascocal_index


Retrieval of data also possible from PI website where Level 1.0 available => shall this be stored as well?

https://sharpp.nrl.navy.mil/cgi-bin/swdbi/lasco/img_short/form

Some of the CME catalogs are linked on the SOHO website

CDAW 

CACTUS

SEEDS


Other CME catalogs exist (see below), at least links shall be made updated, and a possible inclusion of these catalogues in the archive shall be envisaged

CACTUS - generated by the software package "Computer Aided CME Tracking" (Automated).  

ARTEMIS - generated by "Automatic Recognition of Transient Events and Marseille Inventory from Synoptic maps" (Automated)  

CORIMP - generated by automatic detection of SOHO LASCO images (Automated).  

CDAW Catalog - generated by visual manual methods (Manual).  

NRL LASCO CME List - generated by operators (Manual).  

COR1 CME Catalog - generated by the STEREO COR1 team (Manual) 

HI1 Event List - generated by the STEREO HI1 UK team (Manual)

+HELCATS


Value added products at French MEDOC archive

Data products available (synoptic maps from SOHO/EIT, to be included in JHelioviewer, DEM maps, simulations), to be investigated

New interface available soon

http://idoc-medoc.ias.u-psud.fr/sitools/client-user/index.html?project=Medoc-Solar-Portal

 Please note that new calibrated data for C2 are being prepared by P. Lamy


HAUS 02, AI-6 on A. Masson, to discuss with SOHO PS on possible ingestion of LASCO level 1 data, a calibrated version of all LASCO data and CME catalogs (e.g. CACTUS, ARTEMUS, CORIMP, CDAW, NRL LASCO CME list, COR1 CME catalog, HI1, catalogs genertaed by the HELCATS project). 

AOB

Broken links found (see below)

 2.3   Energetic particles

All these instruments are measuring energetic particles (electrons, protons and heavier ions) (up to) relativistic energies. 

Review of data access 

  • Java application :( 
  • Registration needed to download data :(  
  • not clear which time resolution 
  • only one resolution available? 
  • No plotting tool for SOHO data 
  • Data description is "strange"

 Command line broken link found (see below)

2.3.1      COSTEP (EPHIN)

EPHIN (Energetic Proton Helium Instrument) 

  • Protons, Helium, electrons at MeV energies 
  • Three file types: PL2 (pulse height data), RL2 (intensities), KOR (correction file to determine correct ion intensities) 
  • daily files 
  • readme missing (file structure, how to apply correction, instrument description)
  • new level 3 data set under development in Kiel (protons, helium) 
  • continuous data set for the whole mission (the same energy channels) 
  • reason: failure of certain detectors 
  • new data based on PHA data
  • no correction file needed anymore 
  • electron data not changed (use old data  set without KOR-files) 

Kiel will provide new data and readme by end of August 2018

(email 14 June 2018) According to N. Dresing, regarding the update of SOHO/EPHIN data P. Kühl and B. Heber offer that the new data product will be ready by the end of August. Apart from a comprehensive documentation which also explains the procedure of determining the new data product, no other data product will be provided to the ESA archive.


HAUS 02, AI-7 on ESDC, to ingest new level 3 data set under development in Kiel (protons, helium) adn related documentation when received, expected end of August 2018

2.3.2      ERNE

ERNE: Energetic and Relativistic Nuclei and Electron

ERNE has two sensors: LED & HED (ions and electrons > 1 MeV/nuc, 

elemental and isotopic abundance H to Fe)
• 3 file types: science level2, pulse height (each for LED & HED), status 

• readme missing (file structure, instrument description) 


Contact to PI group (Turku, Finland) 

  • Documentation can be provided
  • Development of new data set with some instrumental effects corrected  and possibly finer energy resolution (from certain time on) so that one can determine combined energy channels 


After the meeting, N. Dresing has contacted the PI group of ERNE in Turku (Eino Valtonen and Rami Vainio). This PI team confirmed they can updated ERNE .SL2 data (some data issues solved) which is H and He intensities. Additionally, they would like to also provide the same data with a finer energy resolution.


HAUS 02, AI-8 on ESDC, to ingest new ERNE .SL2 data set with H and He intensities in preparation by Turku group and documentation when received, expected end of August 2018. The delivery of the same data with a finer energy resolution is also envisaged by this group. Possibility of visualizing the data shall be considered.

2.4   EUV imaging and spectroscopy

2.4.1      CDS

From the ESDC SSA Java web start

Can download the file
Can check details of each study 

Cannot have quicklook to the file

File not calibrated (requires solarsoft) 

All datasets available 

No ground based data is essential 

Datasets available in FITS – but  seems to need CDS solarsoft to read in (I tried just with readfits). 


HAUS 02, AI-9 on ESDC,Basic visualization is very useful for spectroscopic data. An 'atlas' is included at the RAL CDS website (http://solar.bnsc.rl.ac.uk/atlas/atlas.shtml). To be investigated.


HAUS 02, AI-10 on ESDC,Access to calibrated CDS data is a priority. ESDC SSA shall store calibrated Level 2 data. At bare minimum clear information about the calibration procedure with solarsoft shall be made available. A simple calibration version of the data assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps

2.4.2      EIT

ESAC SSA has quicklook plots available (which is a +), while VSO does not seem to.


HAUS 02, AI-11 on ESDC,EIT Calibration info on https://umbra.nascom.nasa.gov/eit/eit_guide/gui de.html shall at least linked, best transferred.


HAUS 02, AI-12 on ESDC,EIT movies from CME catalogues shall be included, this is a useful resource.


HAUS 02, AI-13 on ESDC,For imaging data, the EIT daily movies are critical.  They can be used by users from any community easily. And can give basic info (e.g. is there a coronal hole, is there a flare, are there active regions?). Movies available at IAS shall be ingested in the archive.

https://www.ias.u-psud.fr/eit/movies/

2.5   Recommendations

Here is the list of actions related to the review of the ESDC SOHO archive related to the following SOHO instruments: CDS, COSTEP, EIT, ERNE, GOLF, LASCO, MDI, SUMER, VIRGO

HAUS 02, AI-1 on M. Roth/A. Masson, to a review the content of VIRGO datasets, software and value-added products at PMOD/WRC and the scientific interest of their possible inclusion in the SSA at ESDC. 

HAUS 02, AI-2 on A. Masson, to check with the SOHO PS if any of the dozens MDI datasets available at Stanford, http://soi.stanford.edubut not at ESAC SSA shall be included in the SSA for long term preservation. A detailed list of datasets is available in Annex B.

HAUS 02, AI-3 on A. Masson, to check with the SOHO PS if the calibrated version of GOLF data from the PI data at CEA Saclay and Nice observatory shall be eventually ingested in the ESAC SSA for long term preservation.

HAUS 02, AI-4 on A. Masson, to check with the SOHO PS if the following value added datasets suggested by M. Roth, highly complementary to SOHO data, shall be archived within the ESDC SSA

  • ingestion of HMI datasets during the overlapping period when both instruments were operating,
  • "Mark-I" solar spectrophotometer database (IAC, Spain)

HAUS 02, AI-5 on A. Masson/B. Fleck, to check if a calibrated version of all SUMER data exists at Max Planck MPS and shall be ingested within the ESDC SSA, together with the raw data presently available. Thumbnails of the SUMER data only available at MPS may also be considered to be ingested in order to browse these data.

HAUS 02, AI-6 on A. Masson, to discuss with SOHO PS on possible ingestion of LASCO level 1 data, a calibrated version of all LASCO data and CME catalogs (e.g. CACTUSARTEMUS, CORIMP, CDAW, NRL LASCO CME list incomplete, COR1 CME catalog, HI1, catalogs genertaed by the HELCATS project). 

HAUS 02, AI-7 on ESDC, to ingest new level 3 COSTEP/EPHIN data set under development at Kiel university (protons, helium) and documentation when received, expected end of August 2018. Possibility of visualizing the data shall be considered.

HAUS 02, AI-8 on ESDC, to ingest new ERNE .SL2 data set with H and He intensities in preparation by Turku group and documentation when received, expected end of August 2018. The delivery of the same data with a finer energy resolution is also envisaged by this group. Possibility of visualizing the data shall be considered.

HAUS 02, AI-9 on ESDC,Basic visualization is very useful for spectroscopic data. An 'atlas' is included at the RAL CDS website (http://solar.bnsc.rl.ac.uk/atlas/atlas.shtml). To be investigated.

HAUS 02, AI-10 on ESDC,Access to calibrated CDS data is a priority. ESDC SSA shall store calibrated Level 2 data. At bare minimum clear information about the calibration procedure with solarsoft shall be made available. A simple calibration version of the data assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps

HAUS 02, AI-11 on ESDC,EIT Calibration info on https://umbra.nascom.nasa.gov/eit/eit_guide/gui de.html shall at least linked, best transferred.

HAUS 02, AI-12 on ESDC,EIT movies from CME catalogues shall be included, this is a useful resource.

HAUS 02, AI-13 on ESDC,For imaging data, the EIT daily movies are critical.  They can be used by users from any community easily. And can give basic info (e.g. is there a coronal hole, is there a flare, are there active regions?). Movies available at IAS shall be ingested in the ESDC SOHO science archive. https://www.ias.u-psud.fr/eit/movies/



SOHO

experiment

Ingestion status

Ok means

same data content at ESAC and GSFC

Additional datasets 

Complementary datasets

Thumbnails/QL

Visualization needed?

Visualization available?

CDS

Latest data at 

SSA/ESAC: 2013-03-18

GSFC: 2013-05-02

Generate and ingest calibrated version (Level 2)[1]

None

None, possible ingestion of the "Atlas" catalogue (CDS website at RAL)

Yes

No

COSTEP

Ok

Ingest new level 3 COSTEP/EPHIN (protons,helium) and documentation when available (08/2018) 

None

None

Yes

No

EIT

Ok

Calibrated version of the images shall be produced and ingested

Calibration information with Solarsoft

Thumbnails available through time animator; possible ingestion of daily EIT movies and from CME catalogues 

Yes

Yes, time animator

ERNE

Ok

Ingest new ERNE .SL2 data set with H and He intensities once available

None

None

Yes

No

GOLF

Ok

Ingestion of calibrated version from CEA Saclay or Nice observatory

None

None

Yes

No

LASCO

Ok

Level 1 data and Calibrated version of the images to produce & ingest

Calibration info with Solarsoft 

CME catalogues

Thumbnails available through time animator

Yes

Yes, time animator

MDI

Ok

Possible ingestion of MDI datasets available at Stanford university

HMI data when both instruments were operated

Thumbnails available through time animator

Yes

Yes, time animator

SUMER

SSA/ESAC: 2009-11-10

GSFC: 2014-10-24

Ingest calibrated version of the data available at MPS

None

Ingest thumbnails available at MPS

Yes

No

VIRGO

Ok

Complete

PMOD/WRC datasets to investigate

N/A

Yes

No

[1]CDS calibration: assuming single component shift for each spectral line enables to produce a calibrated version of  velocity and intensity maps


3. Ulysses archive at ESDC: review and recommendations

The following Ulysses experiments have been reviewed.

COSPIN, DUST, FGM/VHM, GAS, GRB, GWE, SWICS, SWOOPS 


Reviewers

Chris Chen (FGM/VHM, SWOOPS)

Nina Dresing (COSPIN)

Arnaud Masson (DUST, GAS, GRB, GWE)

Matt Taylor (SWICS)


3.1   COSPIN

See Annex 5

COSPIN is an energetic particle suite containing five instruments: HET, LET, KET, AT, HFT 

Ulysses archive in pretty good shape 

However, in the instruments tab, some links are either not working or not available


There are two types of time resolution 

Prime resolution data (10 mn) and high resolution (1 mn)

In red, possible issues found

3.1.1      Anisotropy Telescope (AT)

Protons 700 keV to 7.6 MeV in 16 different viewing directions

Yearly Zip files, 1990-2008

Intensities (calibrated)

ReadMe exists

email sent to S .Dalla

3.1.2      High Energy Telescope (HET)

Electrons, protons and heavy ions in the MeV range

Spin averaged rates not calibrated 

PHA data (intensities of protons and He nuclei stopping in the detectors K1-K6) in only one file

yearly zip files until mid 2009, daily files

readme exists

email sent to B. McKibben

3.1.3      Kiel Electron Telescope (KET)

Electrons, protons and helium up to GeV (SEPs and cosmic rays)

10 min rates and single detector count rates not calibrated

Intensity data shall be provided by PI team at Kiel University

PHA data in just two files covering the whole mission

yearly zip files in daily files until mid 2009

readme exists (must be updated when new data are provided)


3.1.4      Low Energy Telescope (LET)

Protons and heavy ions in the lower MeV range

10 mn proton, ion an electron intensities calibrated

PHA data in just two files for the whole missions

yearly zip files until mid-2009 in daily files

readme exists, 176 pages with data file structure included

3.1.5      High Flux Telescope (HFT)

Protons, Helium and Heavier ions in the keV range

10mn proton and ion intensities

yearly zip files, daily files until 2008 244, check if any data is missing

readme exists

3.1.6      High resolution data

LET: housekeeping, PHA (yearly files), full resolution (daily files, 1min intensities), same documentation as for prime data
HET: readme exists; house, sect1+sect2, PHA, omni1+omni2 (whole mission folder, daily files, until 2004 173) 

KET: readme + intensities, sectors, PHA. Yearly folder until 2009 181, with extra readme
HFT: yearly folders until 2007 (daily files, however no readme), software folder (c-files etc.) 


HAUS 02, AI-14 on ESDC,for the Ulysses COSPIN experiments, update all hyperlinks to instrument team pages.


HAUS 02, AI-15 on N. Dresing/A. Masson,to follow up on the delivery of Ulysses KET and HET intensity files to ESDC.


HAUS 02, AI-16 on N. Dresing/A. Masson,to check any data is missing from Ulysses HFT (until 2008 244) and AT (until end 2008) for prime resolution and HET for high resolution data (until 2004 173)


3.2   DUST

See Annex 7

Experiment description

Direct observations of dust grains 

with masses from 10-16g to 10-6g (a million times better than previous detectors)

Characterize their speed 

Direction of arrival

Electric charge

As a function of heliocentric distance, ecliptic latitude, solar activity

Key question: what portion of (small dust particles) dust is provided by

Comets

Asteroids

and Interstellar particles?


DUST experiment data consists of a single file listing all dust particles detected with their mass, speed, arrival direction and related mass error and speed error.


It was found that the latest version of this dataset is not contained in the UFA but available on NASA PDS. The key physical parameters are the same but the fill values are in line with the header description in this newer version, contrary to the dataset version available at UFA


HAUS 02, AI-17 on ESDC,for Ulysses DUST, Ingest V3.1 PDS3 DUST dataset available at the NASA PDS small bodies sub node


3.3   FGM/VHM

See Annex 8

Magnetic field investigation with 2 sensors at the tip of the Ulysses boom

  • Vector helium magnetometer 
  • Fluxgate magnetometer 

Data produced: B vector and magnitude, 1hr, 1mn, 1sec resolution from VHM 


People 

PI: Andre Balogh (Imperial) 

Data: Bob Forsyth (Imperial) 


Data content

VHM/FGM: 

  • 1 hour: zipped ascii, 1 file per year, in main directory 
  • 1 minute: zipped ascii, 1 file per 10 days, in sub-directory 
  • 1 second: zipped ascii, 1 file per day, in sub-directory 
  • cdf directory: zipped ascii, 1 file per month, no documentation 
  • umag directory: zipped ascii, 1 file per day, no documentation 
  • “High Resolution Data”: unzipped ascii, 1 file per day, same as 1s? 
  • Not all formats are consistent / clear 

VHM/FGM 

  • A quick look suggests nothing major missing 
  • Better documentation for some folders, which sensor, calibration info? 
  • Data in spacecraft/sensor coordinates? (maybe not essential) 

VHM/FGM 

Calibration Status 


  • in physical units, RTN coordinates 
  • may be some spin tones (not sure of extent) 
  • previously were time tag issues (not sure if fixed) 
  • data gaps are missing rather than flagged 
  • calibration status needs further investigation 

Concluding remarks

  • Need CDF files (ISTP compliant meta-data), + ascii files are helpful too 
  • Directory structure should be made clearer
  • put all files in logical structure (all data from one instrument in one place), labelled in clear consistent way, use 4-digit years in filenames, use consistent time stamps in files 
  • Make available by machine interface, e.g. ftp 
  • Prefer unzipped files 
  • Prefer to flag data gaps rather than remove data points 
  • Quicklook plots – combined instruments 
  • Have informal community questionnaire at some point? 
  • Ask PIs if they want to update anything? 

HAUS 02, AI-18 on ESDC,to interact with the Ulysses FGM/VHM PI team for a possible delivery of a CDF ISTP compliant version of the files, together with a more logical dataset tree (all data from one instrument in one place) with unzipped files labelled in clear consistent way, use 4-digit years in filenames, use of consistent time stamps in files.


3.4   GAS

See Annex 9

Interstellar medium:  matter and radiation in between  stars in a galaxy. This matter includes gas in ionic, atomic, and molecular form, as well as dust and cosmic rays.

Key questions

Origin

Remnant of extinguished stars?
contributes to galaxy mass?
Potential building material of new stars?
Local Interstellar Medium (100 pc or 326 light years)


Nearby Interstellar Medium (10 pc or 326 light years), key questions

Sun Motion in its galaxy
Local Interstellar cloud motion
Composition of the Interstellar medium (ISM)
Interaction with the solar wind


Solar Interstellar Medium

Ions

Neutrals (mainly H, He)

Dust

Heliosphere is transparent to neutrals

Indirect measurements

ISM near nearby stars < 5 pc         

radio observations and absorption lines 

(e.g. Lallement et al., 1990)

ISM neutral H and He using solar EUV/UV back-scattering methods 

(e.g. Chassefiere et al., 1986)

Local ISM neutrals from interstellar pickup ions in the solar wind 

(Moebius et al., 1995)

First direct measurements of interstellar He

Ulysses GAS (e.g. Witte et al., 1993)


The Ulysses GAS dataset consists of pixel map data called skymap derived from the measurement of UV photons and the first direct estimates of neutral Helium particles. The Ulysses Final archive for GAS include

A text file describing the data format

A brief user guide

Skymaps daily files from the UV count rates, available in the folder full_sky

Skymaps daily files based on neutral Helium particles, available in the folder is_helium

Skymaps daily files measured during Jupiter flybys, available in the folder Jupiter

Skymaps daily files at least a star in its field of view, available in the folder stars

Data coverage: full mission (1990-2007)


Review

1. Data description clear? Yes        

2. Data description PI hyperlink? Broken link  

3. Data coverage? Mission full length

4. Data content: daily files when measurements

5. Are data calibrated? No, count rates /s

6. Does the ESDC archives contains the latest/Best calibrated data? No       


Reprocessed data from 2013 exist at Max Planck PI team, available online and shall be ingested


HAUS 02, AI-19 on ESDC,for the Ulysses GAS experiment, fix the broken link to the PI team at Max Planck MPS; ingest newest calibrated version dataset, recalibrated in 2013. 

                                                        


3.5   GRB

GRB: Gamma-Rays Burst (GRB) experiment

See Annex 10


The three main scientific objectives of the GRB experiment are

  • The study and monitoring of solar flare X-ray emission 
  • The detection and localization of cosmic gamma-ray bursts  
  • The in-situ detection of Jovian auroral X-ray radiation


GRB is composed of two detectors: a soft X-ray photons detector (5-20 keV) w and a hard X-ray photons detector (25-150 keV). Gamma rays correspond to photons of energy higher than 100 keV.


In the experiment description, the drawings of the experiment shall be re-labelled as the schematic view of the hard X-ray detector is labelled soft X-ray and the drawing of the soft-Xray detector is labelled hard Xray hard detector.


In flight, hardware performance was as follows

  • Hard X-ray detectors continuous operations for 13 years, then power sharing
  • Soft X-ray detectors never worked 

=> Both information shall be mentioned in the experiment description



GRB science objective I

Solar hard X-rays (flares)

Result from the interaction of flare produced energetic electrons (> 20 keV) with ambient ions in the solar atmosphere

How are the e- accelerated in the first place?

From where hard X-radiation is coming from?

Chromospheric footprints of coronal magnetic loops? Elsewhere?

Comparison with GOES at high latitude



“Prime resolution” datasets

Count rates /s with time resolution 0.25–2 s

         depending on telemetry rate of the downlink

Energy spectra at time resolution of 8-64 s


Datasets are non-standard binary files that require a C program to read them. The compilation is not straightforward and clear explanation shall be added.


Value added datasets related to GRB that could be included on the UFA

  • Catalog of all 1556 Solar X-ray flares detected by Ulysses GRB (1990–2003)



GRB science objective II

Gamma Ray Bursts and SGR

Photons with energy > 100 keV up to MeV

GRB related to supernova / hypernova 

Magnetars

Neutron star binary system

GRB measured by at least 3 spacecraft separated by several AU is needed to precisely triangulate the location of the GRB to find its location and possibly counter parts like optical, X-ray, UV etc...

Ulysses GRB data are pioneer data


“High time resolution” datasets

Automatic trigger when count rates exceeds 8 sigma Count rates /s with time resolution 8 ms during 16 s

Count rates /s with time resolution 32 ms during 64 s

Energy spectra at time resolution of 1-32 s.

Data are ASCII files but usable. A data pipeline shall be applied. This s pipeline is detailed in document available on the archive but the reviewer was not able to compile the Fortran program called by a shell script to get accurate timing and spectral information 

=> uncompressed ASCII files are meaningless

To be further investigated


GRB and SGR

Catalog of all GRB and SGR detected by all satellites (Inc. all from Ulysses) is published on a monthly basis by Prof. Hurley, PI of GRB

This one is included in the UFA (1990-2010).




Visualization of the GRB data

The time resolution of the GRB data at the UFA is 5 minutes. This is clearly not adapted to the science objectives of this experiment. Time resolution shall be set to the highest resolution or not made available.



Figure 1. Left panel: GRB measurments of a solar flares, right panel: same time interval visualized on the UFA with a time resolution only available at 5 minutes. It does not allow to observe the physical phemomena measured by GRB.


Conclusion

1. Data description clear? No         

2. Data description PI hyperlink working? yes

3. Data coverage? Mission full length

4. Data content: daily binary files (X-flares) or compressed ASCII files (GRB/SGR) +software

                   Unreadable

5. Are data calibrated? Count rates

6. Does the ESDC archives contains 

         the latest/Best calibrated data? Yes 

Conclusion

  • Compile C and Fortran programs
  • Generate datasets in a readable format
  • Ingest them in the UFA
  • Data visualization with meaningful time resolution

 

HAUS 02, AI-20 on ESDC,for the Ulysses GRB experiment, to correct the data description (wrong labelling of the schematic view of the detectors); compile C and Fortran programs to generate the solar flares (prime resolution) and GRB datasets (high time resolution) in a readable format; ingest them in the UFA; Data visualization with meaningful time resolution. 

 

3.6   GWE

GWE: Gravitational Waves Experiment 

See Annex 11


No data available; to be further investigated.


HAUS 02, AI-21 on ESDC,for the Ulysses GWE experiment, to investigate why no data is stored in the archive.


3.7   SWICS

See Annex 12

  • List of data products available on the archive per instrument
  • Which datasets are available?

2 products available (~ level 4 data in PSA terminology – derived data from which the original data they come from cannot be reconstructed)

a) Heavy ion main charge state

b) Selected Charge state and abundance ratio            

In addition, you can access lower level products in the software area. 

There is a link to the “SWICS archive” which provides level 0 CDF’s and software from which to access lower level data (count rates etc.). It would be preferable to have this data available under the same area as the higher level products, but at least it is available.

The software can produce count rates which can then be utilized to calculate the density, speed and thermal speed of selected solar wind ions. As such the archive provides access to level 1 , 2 and higher products. BUT they are not in the same location.

The software is not available for Mac, so the reviewer was unable to look at anything.


  • What is not calibrated? 
  • Can it be calibrated? 
  • How?  through community software package for instance?

Data available via the SWICS archive are lower level data (the basic instrument measurements of count rates as a function of energy/charge.) The software can convert these count rates to physical units (differential flux, phase space density), to correct for spillover between different MRs, and to obtain kinetic parameters (density, speed, thermal speed) of selected ions.

This software is available via the archive, but not in MAC format.

  • If not, who/which group could do it? with eventually an estimate of the manpower needed when applicable.

Only ‘need’ the reviewer could identify was that to provide a mac version. Not sure what the cost is. To be further investigated.


What is missing? Please compare to what is available on PI or PI team member website(s)?

This looks complete. One issue is that the links to the UDS view in the software page has LOTS of broken links due to IDL ownership changing. I would suggest avoiding pointing archive pages to external sites. 

  • What would you recommend to archive in the long term?

It is not clear to me whether the SWICS archive is to be supported within the ESDC architecture ? This needs to be checked.

  • Any ground-based data essential to support data analysis that shall be stored as well?

Not that I am aware of

  • Are datasets stored in format compatible with community (visualization/data analysis) tools?

Standard archive format (cdf) and also simply tables


HAUS 02, AI-22 on ESDC,for the Ulysses SWICS experiment, to make available the latest version of the latest calibration software (version 2.3) available on the Ulysses cosmos page. Detailed test of this software shall be performed. 


3.8   SWOOPS

See Annex 8

Data content

–  electrons: ascii zipped, 1 file per month, n&T for core&halo, some info 

–  ions: ascii zipped, np/na/Tplarge/Tpsmall/v, hr(1file)&hires(1/month) 

–  “Additional Data Sets”: electron PAdata(ascii)/plots + documentation 


Datasets completeness

  • missing ion distributions: important to include, can get from Bruce Goldstein 
  • updated ion moments: Goldstein and Anna Tenerani are currently archiving moments with anisotropy/drift at NASA, should be included 
  • electron distributions: important to include, from Goldstein or? 
  • full electron moments, including heat flux?... 
  • DSN +operational reports
  • spacecraft pointing for 3D distributions


Calibration status

–  (some) moments only in RTN 

–  not sure accuracy, need more investigation 

–  some bad data points 

–  data gaps are missing rather than flagged (can be quite sporadic) 


Data format

Non-standard data format, CDF ISTP compliant advised


HAUS 02, AI-23 on ESDC,for the Ulysses SWOOPS experiment, to interact with Bruce Goldstein and Anna Tenerani to ingest the following missing datasets in the UFA 

  • missing ion distributions: important to include (Bruce Goldstein)
  • updated ion moments: moments with anisotropy/drift at NASA, should be included 
  • electron distributions  (Bruce Goldstein)
  • full electron moments, including heat flux? (Goldstein and Anna Tenerani)
  • If possible: DSN +operational reports
  • spacecraft pointing for 3D distributions

Discuss with PI and NASA their conversion to CDF


3.9   Recommendations

HAUS 02, AI-14 on ESDC,for the Ulysses COSPIN experiments, update all hyperlinks to instrument team pages.

HAUS 02, AI-15 on N. Dresing/A. Masson,to follow up on the delivery of Ulysses KET and HET intensity files to ESDC.

HAUS 02, AI-16 on N. Dresing/A. Masson,to check any data is missing from Ulysses HFT (until 2008 244) and AT (until end 2008) for prime resolution and HET for high resolution data (until 2004 173)

HAUS 02, AI-17 on ESDC,for Ulysses DUST, Ingest V3.1 PDS3 DUST dataset available at the NASA PDS small bodies sub node

HAUS 02, AI-18 on ESDC,to interact with the Ulysses FGM/VHM PI team for a possible delivery of a CDF ISTP compliant version of the files, together with a more logical dataset tree (all data from one instrument in one place) with unzipped files  labelled in clear consistent way, use 4-digit years in filenames, use of consistent time stamps in files.

HAUS 02, AI-19 on ESDC,for the Ulysses GAS experiment, fix the broken link to the PI team at Max Planck MPS; ingest newest calibrated version dataset, recalibrated in 2013. 

HAUS 02, AI-20 on ESDC,for the Ulysses GRB experiment, to correct the data description (wrong labelling of the schematic view of the detectors); compile C and Fortran programs to generate the solar flares (prime resolution) and GRB datasets (high time resolution) in a readable format; ingest them in the UFA; Data visualization with meaningful time resolution. 

HAUS 02, AI-21 on ESDC,for the Ulysses GWE experiment, to investigate why no data is stored in the archive.

HAUS 02, AI-22 on ESDC,for the Ulysses SWICS experiment, to make available the latest version of the latest calibration software (version 2.3), available on the Ulysses cosmos page. Detailed test of this software shall be performed. 

HAUS 02, AI-23 on ESDC,for the Ulysses SWOOPS experiment, to interact with Bruce Goldstein and Anna Tenerani to ingest the following missing datasets in the UFA 

  • missing ion distributions: important to include (Bruce Goldstein)
  • updated ion moments: moments with anisotropy/drift at NASA, should be included 
  • electron distributions  (Bruce Goldstein)
  • full electron moments, including heat flux? (Goldstein and Anna Tenerani)
  • If possible: DSN +operational reports
  • spacecraft pointing for 3D distributions

Discuss with PI and NASA their conversion to CDF





Ulysses

experiment

Ingestion status

Ok = latest version at ESAC 

Additional datasets / software

Complementary datasets

Thumbnails/QL plots

Visualization needed?

Visualization available?

COSPIN

Double check completeness of HFT, AT and HET datasets

First calibrated data from HET and KET are expected

None

Yes

Yes

Moments only

DUST

Newer data at NASA PDS small bodies subnode (V3.1 PDS3)

None

None

None

No

No

FGM/VHM

Ok

CDF ISTP version would help

None

Yes

Yes

Better resolution neeeded

GAS

Recalibrated data at MPS shall be ingested 

Other extra datasets available at MPS

None

Yes

Yes

QL plot viewer

GRB

Ok

Compile C and Fortran 77 routines to provide calibrated data

Solar X-flares catalogue over the Ulysses mission

Yes but time resolution meaningless for Solar flares and GRB

Yes

Yes but time resolution => meaningless

GWE

No data

None

None

No

No

No

SWICS

Ok

Update SWICS software (v.2.3)


QL plots available

Yes

Moments only

SWOOPS

Missing files available at NASA

Interact with Bruce Goldstein and Anna Tenerani for double checking data completeness

None

QL plots available

Yes

Moments only

4. FP7 projects related to ESA heliophysics missions

See Annexes 3 and 13

Reviewers

V. Génot

M. Roth

4.1   IMPEx

Integrated Medium for Planetary Exploration

  • http://impex-fp7.oeaw.ac.at/
  • 2011-2015 (48 months)
  • “An infrastructureto bridge the gap between observational data bases and scientific modelling tools, enabling their joint interconnected operation for the better understanding of related physical phenomena.”
  • Coordinator : Maxim Khodachenko (Graz, Austria)
  • Distributed architecture
  • A general portal + simulation databases + observation databases + analysis tools
  • Simulations and models remain at the modeler sites and are accessed on demand with web-services
  • Simulations (MHD & hybrid) and analytic models (magnetic field) concern only planetary environments
  • Analysis tools are multi-missions and multi-instruments


Conclusion: IMPEx results are not relevant for the ESDC Heliophysics Archives


4.2   HELIO

http://www.helio-vo.eu/

  • 2009-2012 (42 months),  Coordinator : Bob Bentley
  • The Heliophysics Integrated ObservatoryHELIO, is a Research Infrastructurethat addresses the needsof a broad community of researchers in Heliophysics; it has deployed a distributed network of servicesthat provides a rich search environment and the most comprehensive integrated information system in this domain. The architecture of HELIO, based around multiple instances of services, has resulted in an infrastructure that is resilient, extensible and sustainable.


  • After reviewing services and talking with some persons in charge at the time (J. Aboudarham, C. Jacquey) it appears that the most prominent / relevant service is the HFC = Helio Feature Catalogue


Full list of services by HELIO


Status of the service

  • Some catalogues were stopped at the end of the project, other continued at Meudon Observatory then were stopped. Recently the catalogue concerning Active Regions was restarted at BASS2000 (http://bass2000.obspm.fr/)
  • The task is to maintain the codes and run them automatically
  • Catalogues (Active Regions and Radio Bursts) are also available on VESPA http://vespa.obspm.fr/
  • To access the catalogues the most complete interface is the copy of the HELIO HFC GUI at http://voparis-helio.obspm.fr/hfc-gui-dev/


Conclusion: the Heliophysics Feature Catalogue appears very relevant to the ESDC heliophysics archives


HAUS 02, AI-24 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogue developed through the HELIO FP7 project after the end of operations, for long term data preservation.


4.3   HELCATS

https://www.helcats-fp7.eu/

2014-2017 (36 months) ; Coordinator : R. Harrisson (RAL)

  • The aim of HELCATS was to provide catalogues of heliophysics structures during the STEREO era
  • Contrary to HELIO, all catalogues are now closed
  • They’re all available in different formats (txt, json, votable)
  • These catalogues represent a small dataset of high scientific value
  • CDPP has already a local copy of these catalogues and CME/CIR observed at Earth are propagated at planets and probes in the CDPP Propagation Tool
  • Not derived from SOHO and Ulysses but useful in a larger heliophysics context


Conclusion: HELCATS results are very relevant for the Heliophysics Archive


HAUS 02, AI-25 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogues developed through the HELCATS project, all catalogues are now closed, for long term data preservation.


4.4   CASSIS

  • http://cassis-vo.eu/
  • 2010-2013 (36 months)
  • The aim ofCASSIS, the Coordination Action for the integration of Solar System Infrastructures and Science, is to create an integrated environment necessary to span the inter-disciplinary boundaries in Heliophysics. 
  • Coordinator : Bob Bentley
  • The main achievements of CASSISare contained in a set of reports


Conclusion: CASSIS results are not relevant for the Heliophysics Archive


4.5   SpaceInn

Spaceinn FP7 program(http://www.spaceinn.eu

Exploitation of Space Data for Innovative Helio-and Asteroseismology

Creation of the SEISMIC plus portal

  • Virtual Observatory to combine ground- and space data
  • Asteroseismic parameters
  • Seismic proxies
  • Theoretical models
  • Development of new tools


Contains

  • CoRoT archive (UPS, France)
  • KASOC, Kepler archive (AU, Denmark)
  • HMI/SDO European data centre (MPG, Germany)
  • BASS2000/TBL legacy (archive of the spectropolarimetric data from Narval, UPST, France)

After end of project: Operation by Paris Observatory with best efforts Thanks to E. Michel

Results relevant for PLATO


Additional resources 

(in red, value added products relevant to SOHO datasets)

  • Ground-based CoRoT complementary archive (INAF, Italy, HARPS data)
  • Stellar/Solar Models: (CSIC, Spain/CAUP, Portugal)
  • Seismic Indices data base
  • (OBSPARIS, France)
  • "Mark-I" solar spectrophotometer data base (IAC, Spain)

Global Helioseismology

(in red, value added products relevant to SOHO datasets)

  • Study the changing Sun as it evolves in many different time scales: new magnetic activity index
  • New techniques to extract low-amplitude signals (p and g) modes by combining information provided by all the instruments.
  • Tools to combine Sun-as-a-star observations (e.g. GOLF, BISON and VIRGO) with the imaged ones (e.g. GONG, MDI or HMI)
  • Improve the realism of solar 3D models in which the convective envelope is non-linearly coupled to the deep solar radiative interior



Local Helioseismology

  • Explore systematic effects present in the analysis of space observations
  • Make available and exploit numerical simulations of seismic waves and their interactions with internal heterogeneities and magnetic regions.
  • Preparation of the Solar Orbiter mission to be launched in 2020
  • Implement Fourier-Hankel/Legendre analysis (FH) to study structures like sunspots
  • Develop fast, automated data processing by implementing efficient workflows to process historical data sets (MDI and GONG) and for the continued analysis of SDO-HMI data


General conclusion

•        Higher-level data products are relevant should be archived

•        PI-own-calibration + complementary data exists and could be considered

•        Documentation, codes, scripts, systematics (leakage) to be added


4.6   Recommendations

HAUS 02, AI-4 on A. Masson, to check with the SOHO PS if the following value added datasets suggested by M. Roth, highly complementary to SOHO data, shall be archived within the ESDC SSA

  • ingestion of HMI datasets during the overlapping period when both instruments were operating,
  • "Mark-I" solar spectrophotometer database (IAC, Spain)

HAUS 02, AI-24 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogue developed through the HELIO FP7 project after the end of operations, for long term data preservation.

HAUS 02, AI-25 on ESDC/SOHO PS,to consider the inclusion of the Heliophysics Feature Catalogues developed through the HELCATS project, all catalogues are now closed, for long term data preservation.

5. ESA's Long Term Strategy for science archives discussion

Guido De Marchi illustrated the current on-going process within ESA to define a long-term strategy for the ESDC archives. The HAUS group provided feedback that is captured by Guido De Marchi in the revision of the Archives Long-Term Strategy document. There was in particular some dicussion on how to connect ground-based data with solar space observations. To get feedback from the community on long-term strategy, HAUS group members proposed to advertise a survey through Space Weather European Network (SWEN) newsletter, SPA, Solar Physics newletters, Solar Orbiter SOWG/SWT. There was also a discussion about ESA sponsoring young researchers to attend conferences when they have results to present based on data from the ESDC archives. Suggestions were: data usage should not be restricted to ESA archives alone; papers should be at least submitted/accepted to qualify for a grant, and not just a proposal.

Annex A. ToR

The scope of the HAUS group is defined below. This is an excerpt of the terms of reference (see Annex 1)

Identify the most important functionalities of a future overarching multi-missions Solar Archive (e.g. combining data from SOHO, PROBA-2, ISS-SOLACES and the upcoming Proba-3 and SOLO missions).

Advise on the main functionalities of a future web service linking the Heliophysics archives and the Planetary Science Archive. One key aspect offered by this service would be the ability to identify and download data of a solar transient observed on the Sun and to compare its impact on the various planets/solar system objects captured by missions throughout the solar system (e.g. VEX, Cluster, MEX, TGO and Rosetta). This service may include online modelling (e.g. ENLIL, EUFHORIA) and/or the Heliophysics Events Knowledgebase (HEK) with a propagation model.

Address shortcomings or missing functionality in individual archives (e.g. Ulysses data visualization; ISS-SOLACES value added products; visualization of SOHO in-situ instruments data...).

Review the development of the overarching Heliophysics web services activities to ensure they best meet the needs of the user community within the available resources. 

AnnexB. : MDI datasets at Stanford University

Dozens of MDI datasets are available at Stanford university but not at the ESAC SOHO archive. Here is a detailed list.

 1.    Dopplergrams

On the SOHO archive at ESAC

  • Full-Disk Dopplergrams 60s Cadence: mdi.fd_V


At Stanford and not on the ESA SSA archive at ESAC

  • Full-Disk Dopplergrams binned (Linearly polarized only)          mdi.fd_bin_2x2
  • Full-Disk Dopplergrams at 30s cadence                                        mdi.fd_V_30s
  • Full-Disk Dopplergrams binned at 30s cadence                            mdi.fd_V_bin2x2_30s
  • Full-Disk Dopplergrams Extract(Circularly polarized only)        mdi.fd_V_extract
  • High Res Dopplergrams                                                                mdi.hr_V
  • High Res Dopplergrams high cadence (linearly polarized only)  mdi.hr_V_12s
  • High Res Dopplergrams binned                                                    mdi.hr_V_bin2x2
  • Gaussian Smoothed Dopplergram                                                 mdi.loi_V
  • LOI Dopplergram                                                                          mdi.loi_V


2.    Filtergrams

At Stanford and not on the ESA SSA archive at ESAC

  • Full-Disk Filtergrams                                                                    mdi.fd_I0
  • Full-Disk Filtergrams extract                                                        mdi.fd_I0_extract
  • High Resolution Filtergrams                                                         mdi.hr_I0



3.    Magnetograms

On the SOHO archive at ESAC

Full-Disk Magnetograms at 96m cadence                                         mdi.fd_M_96m_lev182*


At Stanford and not on the ESA SSA archive at ESAC

  • Full-Disk Magnetograms                                                               mdi.fd_M_lev182*
  • Full-Disk Magnetograms Extract                                                  mdi.fd_M_extract
  • High Resolution Magnetograms                                                    mdi.hr_M
  • High Res Magnetograms binned                                                   mdi.hr_M_bin2x2


4.    Continuum intensity

On the SOHO archive at ESAC

Full-Disk continuum 6h average                                                       


At Stanford and not on the ESA SSA archive at ESAC

  • Full-Disk Continuum(Linearly Polarized)                                    mdi.fd_Ic
  • Full-Disk Continuum High Cadence: (Linearly Polarized)          mdi.fd_Ic_30s
  • Full-Disk Continuum high cadence binned (Linearly Polarized)mdi.fd_Ic_bin2x2_30s
  • High Resolution Continuum (Linearly Polarized)                        mdi.hr_Ic
  • Rebinned Continuum                                                                     mdi.rwbin_Ic
  • LOI Continuum                                                                             mdi.loi_Ic

5.    Line depth

At Stanford and not on the ESA SSA archive at ESAC

  • Full-Disk Line depth                                                                     mdi.fd_Ld
  • High Resolution Line depth                                                          mdi.hr_Ld
  • High Resolution Line depth Binned                                              mdi.hr_Ld_bin
  • Rebinned Line depth Binned                                                         mdi.rwbin_Ld


6.     Calibration data


Not in ESA SOHO Archive at ESAC

  • MDI calibration tables for Dopplergrams                                     mdi.caltables_doppler
  • MDI calibration tables for (original) Dopplergrams                     mdi.caltables_doppler_orig
  • MDI calibration tables for Intensitygrams                                    mdi.caltables_intensity
  • MDI Roll table                                                                               mdi.roll_table
  • MDI Scale corrections Table                                                         mdi.scale_corrections


7.     Other relevant data products for helioseismology 

28 MDI data products available available at Stanford, not at ESAC SSA

have been identified and listed below 

  • mdi.vw_V_sht_72d (archived) - raw timeseries retiled to a length of 72 days with one l per record.
  • mdi.vw_V_sht_gaps_72d (archived) - window functions of the above timeseries.
  • mdi.vw_V_sht_secs_72d (keywords only) - sections of continuous data within each timeseries, used for subsequent detrending
  • mdi.vw_V_sht_gf_72d (archived, permanent online) - detrended and ga pfilled timeseries.
  • mdi.vw_V_sht_gf_gaps_72d (archived, permanent online) - window functions of the above timeseries.
  • mdi.vw_V_sht_modes (archived, permanent online) - ascii tables containing mode parameters fit using symmetric lorentzians.
  • mdi.vw_V_sht_modes_asym (archived, permanent online) - ascii tables containing mode parameters fit using asymmetric profiles
  • mdi.vw_V_sht_modes_archive (archived) - full results from all iterations of symmetric peak bagging corresponding to mdi.vw_V_sht_modes.
  • mdi.vw_V_sht_modes_asym_archive (archived) - full results from all iterations of asymmetric peak bagging corresponding to mdi.vw_V_sht_modes_asym.
  • mdi.vw_V_sht_2drls (archived, permanent online) - ascii tables of internal rotation, its errors, and resulting a-coefficients
  • mdi.vw_V_sht_2drls_asym (archived, permanent online) - ascii tables of internal rotation, its errors, and resulting a-coefficients.
  • mdi.vw_V_sht_gf_retile (unarchived) - timeseries of various lengths retiled from mdi.vw_V_sht_gf_72d
  • mdi.vw_V_sht_gf_gaps_retile (unarchived) - window functions of the above timeseries.
  • mdi.vw_V_sht_pow (unarchived) - power spectra of detrended and gap filled timeseries.
  • mdi.fd_V_sht (archived) - raw timeseries retiled one l per record (various lengths in time).
  • mdi.fd_V_sht_gaps (archived) - window functions of the above timeseries.
  • mdi.fd_V_sht_secs (keywords only) - sections of continuous data within each timeseries, used for subsequent detrending
  • mdi.fd_V_sht_gf (archived, permanent online) - detrended and gap filled timeseries.
  • mdi.fd_V_sht_gf_gaps (archived, permanent online) - window functions of the above timeseries.
  • mdi.fd_V_sht_modes (archived, permanent online) - ascii tables containing mode parameters fit using symmetric lorentzians.
  • mdi.fd_V_sht_modes_asym (archived, permanent online) - ascii tables containing mode parameters fit using asymmetric profiles.
  • mdi.fd_V_sht_modes_archive (archived) - full results from all iterations of symmetric peak bagging.
  • mdi.fd_V_sht_modes_asym_archive (archived) - full results from all iterations of asymmetric peak bagging.
  • mdi.fd_V_sht_2drls (archived, permanent online) - ascii tables of internal rotation, its errors, and resulting a-coefficients
  • mdi.fd_V_sht_pow (unarchived) - power spectra of detrended and gap filled timeseries.
  • mdi.leakage (archived, permanent online) - leakage matrices. Two string prime keys, the first of which should always be omitted. For fd_V, the second prime key is "fdreference". For vw_V, it is "vwreference".
  • mdi.eigenfunctions (archived, permanent online) - oscillation eigenfunctions used for inversions. One string prime key, so far only "default".


8.     Available modules from data pipelines

Modules (Fortran & C)

The global pipeline consists of five modules and one standalone executable. Their source code can be found 

at jsoc.stanford.edu/cvs/JSOC/proj/globalhs/apps/

and the corresponding "libraries" can be found 

at jsoc.stanford.edu/cvs/JSOC/proj/globalhs/libs/


  • jv2ts- takes images (usually dopplergrams) as input and outputs timeseries chunked in l. It provides the combined functionality of three historic modules: v2helio, helio2mlat, and qdotprod. It can optionally provide the output of the first two, namely longitude-sin(latitude) maps and these Fourier-transformed in longitude and transposed.
  • jretile- input and output are timeseries, changes tiling in time and chunking in l.
  • jtsfiddle- detrends and gapfills timeseries. Output can simultaneously be timeseries, Fourier transforms, power spectra, etc.
  • jtsslice- performs Fourier transforms on a section of a timeseries to save on I/O. Output can be these and/or power spectra.
  • jpkbgn- extracts mode parameters from Fourier transforms. Currently input are timeseries, output is ascii tables.
  • inv2d.x - standalone program to perform two dimensional RLS inversions for internal rotation.
  • jrebinsmooth- performs binning, gaussian smoothing, and subsampling of images. 


9. tcsh scripts

The pipeline is almost always run by the following scripts. They can be found at jsoc.stanford.edu/cvs/JSOC/proj/globalhs/scripts/


  • doglobalhs - sets up working directory, environment variables, and runs and checks all subsequent scripts. This is typically the only script that is used at the command line.
  • dosht - performs spherical harmonic transforms. Sets up and submits cluster scripts to execute jv2ts.
  • doshtcheck - checks that all the output expected from dosht is present (uses IDL).
  • doretilen - retiles input to multiple output time intervals. Sets up and submits cluster scripts to execute jretile. Jobs are divided in time, each one retiles all l.
  • dodscopyn (optional) - used to copy output of doretilen to another dataseries using dscp. Only needed for certain combinations of write slices and compression tiles, usually not used
  • doretile1 - retile input to a single output time interval. Sets up and submits cluster scripts to execute jretile. Jobs are divided in l, each one retiles the same interval in time. There is some overlap of functionality with doretilen.
  • dodscopy1 (optional) - used to copy output of doretile1 to another dataseries using dscp. Only needed for certain combinations of write slices and compression tiles, usually not used
  • domkgaps - examines timeseries for l=0,1,2,5,10,20, and 50 to construct window functions for raw timeseries (uses IDL).
  • dogapfill - performs detrending and gap filling. Sets up and submits cluster scripts to execute jtsfiddle.
  • dogfgaps - examines one low-l timeseries to construct window functions for gap filled timeseries (uses IDL).
  • dopow (optional) - performs Fourier transforms and/or constructs power spectra. Sets up and submits cluster scripts to execute jtsfiddle. Only needed when output cover a different interval in time than the input.
  • dopowslice - performs Fourier transforms and/or constructs power spectra for consecutive slices of the input timeseries. Sets up and submits cluster scripts to execute jtsslice
  • dopkbgn - sets up and runs all iterations of the peak bagging, utilizing the scripts found in jsoc.stanford.edu/cvs/JSOC/proj/globalhs/scripts/pkbgn/. It calls the script doiter to set up and submit cluster scripts to execute jpkbgn.
  • doinvert - performs 2d RLS inversions. Does not use cluster, but rather executes inv2d.x on the local host.


The following script is not called by doglobalhs.

  • dorebinsmooth - sets up and submits cluster scripts to execute jrebinsmooth. Used to recreate vw_V from full-disk dopplergrams.
  • dorepeatpow - used to regenerate Fourier transforms and/or power spectra originally created using dogapfill and that have aged off disk. dogapfill itself should not be repeated since the gap filled timeseries are archived. However, data that were created using dopow or dopowslice may be recreated by simply rerunning those scripts.

10.  Other mode frequencies

Jesper Schou:

http://quake.stanford.edu/~schou/anavw72z/Readme

  • Additional frequency tables + own calibrated SHT time series
  • Used in many analyses


Sylvain Korzennik by e-mail: skorzennik@cfa.harvard.edu

  • Additional frequencies
  • Leakage matrices (best available; only from his code)

See Oral Preseentations (PDF) 

Next meeting: Next HAUS group meeting will be held at ESAC, 15-16 January 2019

  • No labels
Write a comment...