ices-tools-dev / eggsandlarvae

Development of tools for the eggs and larvae datasets

EggsAndLarvae

Development of tools for the eggs and larvae datasets

Visit the Website for more information

Unified portal to access data from ichthyoplankton surveys carried out in ICES areas

Fish egg and fish larvae data have been collected in the ICES area for a long time for use in stock assessments and fisheries management. The collection of the data is usually organized by international survey expert groups. The Eggs and Larvae database aims to store, and make available, data collected by ichthyoplankton surveys for use by ICES and the wider marine community. It provides an overview of available fish egg and larvae survey data, and a unified portal for access to the ichthyoplankton survey data.

eggsandlarvae's Issues

Add MIK NL 2020 data again to the database.

I have re-upload the MIK data for NL in 2020.
For some reason the first upload only uploaded the data until Sample. did not upload the measurements.
We need to try to upload the file again to see if we can find what was the issue.

ELDB: review disclaimers text

Check email from Malin, resolve and communicate.

Add the MIK historical data to the database

Upload

I have uploaded data of our September 2019 survey. They are imported into the database, but they do not appear on the map or download data. Any reason for that? Must uploaded data be released?

Update the survey data in the database.

Update the survey data in the database

Download of Data in the new format

When users download the format in the new data, if there was two hauls with the same ID then it was not possible to link the EH file with the EM file.

Uploaded numbers with decimals are rounded in the download, the decimals should be available

• FlowMeter Int. Calibration is a rounded number in the down load, it is quite important that the uploaded decimals are also available in the down load
• VolumeFiltInt is a rounded number, it would be good to have some decimals for this number as well

FEAT: Add option to upload large images in the webserver

Requested by WGMEGS, to have an option to upload large fecundity images.

One option is for this to be done through the webserver together with the data, and they will be publicly available to download. They can be linked to gonadID, if it is written in the picture name.

ELDB: move from EL_Survey to Survey vocab

Work in progress:

• Identify Survey codes to match EL_Survey codes
• Map existing data to new Survey codes
• Implement change in DATSU
• Internally transform EL_Survey to Survey

EH fields are empty in the downloaded file, despite being uploaded and upload fields are linked to the wrong download field

• Country, Haul Duration, FlowMeter Ext. Calibration, Max Sampling Depth, Bottom salinity are empty
• Institute, GearDeployment, MeshType, CodendMesh, ELHaulFlag, NetopeningArea, NetClogging, FlowmeterType, FlowmeterBrand, ELVolFlag, DepthUpper are missing in the download
• FlowEfficiency is missing, there is a field ‘Gear Efficiency’ in the download, but this is empty.
• Volume Filter External is not provided in the upload
• Water Depth is empty, which is actually BottomDepth in the upload
• Haul Depth is all ‘0’. This field seems to be the uploaded DepthUpper, but should actually be the uploaded DepthLower
• Which uploaded field becomes Temp max sampler depth? It is either the uploaded SurTemp or BotTemp but I can’t check that as the second decimal is missing in the down load. Can this down load field name be changed, so it is clear which original field it is?
• I assume Salinity in the download is the uploaded SurSal, but this is unclear at the moment, can this download field name be changed?

ELDB: update factsheet

Make changes to ELDB factsheet text, structure, and appendices suggested at WGELFADG on SharePoint.

New website for Eggs And Larvae

The current website was not supporting all the request done so a new website was developed.
The website needs to be tested before putting it on-line and replacing the current one need the green light from Hannah and Adriana.

Please test the new website here:
https://eggsandlarvae.ices.dk/beta/manage
https://eggsandlarvae.ices.dk/beta/inventory

Update Fact sheet in the Eggs and Larvae web site

This document was last updated in 2013:
http://ices.dk/marine-data/Documents/EggsandLarvae/EggsAndLarvaeDataset.pdf

Field names in the upload format are not the same in the down load format, also en lot of fields appear in the download format that cannot be uploaded

It is necessary to update the download format in line with the upload format, and keeping in mind the format of the historic data already in the E&L database

ELDB Vocab: Use of ES-PV

There might be some challenges when registering in AIMS as it is not the same category of ISO as country. Need to check with submitter if this is still required, as EDMO guarantees independent submissions from different institutions.

Publish and put in production the Fecundity and Atresia data portal

Create a new collumn called 'StatRec' to EH. This collumn will denote the ICES rectangle that the sample was taken in.

How to correct data in the E&L database?

If wrong data has been uploaded how can this be corrected? There is no information about this on the E&L webpage

Add a new description to the MIK survey

In this page: https://www.ices.dk/data/data-portals/Pages/Eggs-and-larvae.aspx

The MIK description:

Should be updated to this one:
The midwater ring-net sampling during the 1st quarter IBTS in the North Sea (MIK-survey)

The MIK-survey is carried out during night time since 1977 alongside the 1st quarter IBTS in the North Sea, eastern Channel, Kattegat and Skagerrak, and aims at large (> 18 mm standard length) herring larvae. It provides an abundance estimate of pre-recruits of the autumn spawning stock components of North Sea herring, which is used in the assessment as a recruitment index. The database contains quality checked data on length and abundance of herring larvae since 1992. In the most recent years, also data on other larval species such as sardine, sprat and lemon sole have been added. The ICES group responsible for the data is WGSINS.

FEAT: Landing Page text and Factsheet

Landing page text reviewed by members of the governance group

ELDB+FEAT: New filtering options for download

Governance Group agreed to:

• Add Country
  -Add Organization

Points on land

Coordinates on land

The Working Group on North Sea Cod and Plaice Egg Surveys presents 212 events occurring on land. While some are close to an estuary or a lagoon and these are ok, many of them seem to be completely on land.

ELDB: Change "Subsampling factor" to "Raising factor"

Requested by Governance Group:

• Create Raising factor variable in DATSU
• Change variable
• Change in template
• Change in download

Create warning when same year, country AND/OR year, vessel AND/OR year,country,vessel are screened.

If the submissions should replace the existing file in the DB, the admisnistration has to be informed.

Update resubmission message

When a survey is conducted twice a year, there is a risk that data collected in the first instance is deleted from the database.
A better "resubmission message" needs to be written.

FEAT: Format changes

Some changes in the Fecundity and Atresia format have been requested from the Governance Group:

Currently, in the format, only the latest Oocyte stage present is noted. It was discussed that they would like to also report the presence or absence of previous stages (from 2 and above, since stage 1 is always present).

Therefore, they would like to add 4 new fields (yes or no fields)

OocyteStage2 (yes/no) Presence or absence of Oocyte Stage 2
OocyteStage3 (yes/no) Presence or absence of Oocyte Stage 3
OocyteStage4 (yes/no) Presence or absence of Oocyte Stage 4
OocyteStage5 (yes/no) Presence or absence of Oocyte Stage 5

Currently, in the Fecundity and Batch Fecundity record, we have the SampleWeight and SubsampleVol fields.

During sampling, 4 samples should be taken:

Sample A,B (25 ul)
Sample C,D (100 ul)

RECORD: FM-Fish Measurement
SubsampleVolA Volume (ul) of pipette subsample A
SubsampleVolB Volume (ul) of pipette subsample B
SubsampleVolC Volume (ul) of pipette subsample C
SubsampleVolD Volume (ul) of pipette subsample D

In order to have an overview of which sample has been used, they would like to add a field
SubsampleID, using values a,b,c,d.
SubSampleID ID (a, b, c or d) of the subsample used for fecundity or batch fecundity analysis

Why is reworking of the uploaded HaulID and NationalHaulID happening and is this necessary

Apparently in the uploaded HaulID and NationalHaulID are reworked to the downloaded Haulnumber and SampleNumber that makes it difficult to check with the original uploaded data. Why does this happen and how is the uploaded data reworked and is this necessary?

Sometimes when filtering by year in the map it does not show any survey

ELDB: MEGS data still not in the database

Some submitters have issues, DK, UK and some others are still pending 2021 data. Try to help during MEGS (April 2024).

Add MIK data for herring, sardine and lemon sole to the ELDB

We need to check the data already existing in the database and then add the files with these species.

Update the EggsAndLarvae WebAPI

List of the services

• create a service to download the Historical Dataset

• create a service to download the EM data

• create a service to download the EH data

• create a service to view summary of the data

Some fields in the download are missing in EH

Netclogging, FlowmeterType and ELVolFlag appear as fields in the downloaded EH-file, but there is no data in the file.
StatRec does not appear on the downloaded EH-file.

Created a new template to be tested

Changes on this new template:

• Has the description in the downdown boxex
• Exports the data to CSV

Template can be downloaded here:
https://eggsandlarvae.ices.dk/upload/EggsAndLarvaeTemplate.xlsx

One notes field in the down load format, while both the EH and EM have a notes field

It is possible to upload notes on both the EH and EM data, however in the down loaded file only one notes filed appears. The Dutch IHLS 2019 uploaded data contained notes in both EH and EM, but only the notes of the EM seem to appear in the downloaded file

FEAT: Add option to filter by Survey Period, Month

EM fields that can be uploaded to the E&L database do not appear in the downloaded file

• Length field, Subfactor are empty in the downloaded file
• DevScale, DevStage are missing in the download
• OilGlobulesFlag will need to be linked to ‘Oil globules present’, is in the current download an empty field

Fecundity and Atresia data quality checks

New QC checks added to Datsu

ELDB: Download options

• Add multiple selection choice, mainly needed for the new Survey codes, as a single survey can have up to three codes for each quarter.
• Organize Countries, Organizations and Surveys alphabetically

What is the difference in the download format between Latitude/Longitude and HaulLat/HaulLon.

The upload format has StartLatitude and StartLongitude the downlaod format had both Latitude/Longitude and HaulLat/HaulLon. The last two fields are empty in our downloaded file when downloading the Dutch IHLS 2019 data

ELDB WEB: News and updates page

• Raising factor
• Suvey codes
• New governance group

Create page asap, with upcoming changes and send out email to all WGs for information

Not all data is downloaded from E & L db

Hi, I have tried now twice to download all data for MIK-surveys and when looking at the data, two years (1997 and 2015) are missing for Sweden (also 2011, but then we had no sampling).
https://eggsandlarvae.ices.dk/Download.aspx

I searched for all years and "Survey" MIK.

This is very strange, beacuse when I look in the Data portal I can see and download data for these years.
https://data.ices.dk/view-map?dataset=194734
@cmspinto
Do you know what could be wrong? I first thought we had not uploaded those years. But we have.

Presentation of TAF in WGSINS.

Presentation of TAF in WGSINS next December.

This is related with the EggsAndLarvae indexes calculation in TAF.

Update disclaimer

In the Eggs and Larvae disclaimer, the citation section refers to Stomach Data instead of Eggs and Larvae data. This should be updated, so that the citation refers to Eggs and Larvae data.
See below.

"Please acknowledge the following data source when using the data:

ICES Stomach Dataset 2010, ICES, Copenhagen"

Integrate the index calculation of MIK into TAF

Integrate the MIK index calculation script into TAF.

The script is here: https://github.com/ices-tools-dev/EggsAndLarvae/tree/master/Indices%20calculations/MIK

Add filter by survey to the Eggs and Larvae selection in the Data portal

Is it possible to see the last uploaded information from the data?

Is DateLastModified suppoed to be the field showing when and who uploaded the latest version of the data? In the current download this is empty.

Empty hauls (no eggs or larvae) do not appear in the down load

Dutch IHLS 2019 data were uploaded to the E&L database. When down loading the data from the database the empty hauls, containing no larvae, were missing from the data file