The data contains 4 kinds of files. Files are organized in folders for easy interpretation:

1) An Excel file. This has all the data collected from the measurement. This file can be opened using Microsoft excel.

2) Minitab project files (MPJ) . These files can be opened using the statistical software Minitab version 17. They include the data, analyses and plots used to interpret the results of the research.

3) A PDF document. This has all the plots related obtained through the research data to determine the optimal settings. This can be opened in any PDF reader.

4) Original TIF and BMP images obtained from the CT scan. Only one relevant image from each data-set is shown because it contains hundreds of images. These can be opened using most image viewing applications such as windows photo viewer.

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Internal expenditure on R & D in enterprises registered for the implementation of R & D (EUR 1 000) by activity (NACE Rev. 2) to which RiR was directed, Slovenia, annually”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

STEAD Subsample Dataset for CDiffSD Training

Overview

This dataset is a subsampled version of the STEAD dataset, specifically tailored for training our CDiffSD model (Cold Diffusion for Seismic Denoising). It consists of four HDF5 files, each saved in a format that requires Python's `h5py` method for opening.

Dataset Files

The dataset includes the following files:

• train: Used for both training and validation phases (with validation train split). Contains earthquake ground truth traces.
• noise_train: Used for both training and validation phases. Contains noise used to contaminate the traces.
• test: Used for the testing phase, structured similarly to train.
• noise_test: Used for the testing phase, contains noise data for testing.

Each file is structured to support the training and evaluation of seismic denoising models.

Data

The HDF5 files named noise contain two main datasets:

• traces: This dataset includes N number of events, with each event being 6000 in size, representing the length of the traces. Each trace is organized into three channels in the following order: E (East-West), N (North-South), Z (Vertical).
• metadata: This dataset contains the names of the traces for each event.

Similarly, the train and test files, which contain earthquake data, include the same traces and metadata datasets, but also feature two additional datasets:

• p_arrival: Contains the arrival indices of P-waves, expressed in counts.
• s_arrival: Contains the arrival indices of S-waves, also expressed in counts.

Usage

To load these files in a Python environment, use the following approach:

```python

import h5py
import numpy as np

# Open the HDF5 file in read mode
with h5py.File('train_noise.hdf5', 'r') as file:
# Print all the main keys in the file
print("Keys in the HDF5 file:", list(file.keys()))

if 'traces' in file:
# Access the dataset
data = file['traces'][:10] # Load the first 10 traces

if 'metadata' in file:
# Access the dataset
trace_name = file['metadata'][:10] # Load the first 10 metadata entries```

Ensure that the path to the file is correctly specified relative to your Python script.

Requirements

To use this dataset, ensure you have Python installed along with the Pandas library, which can be installed via pip if not already available:

```bash
pip install numpy
pip install h5py
```

This kit includes an additional revised master file, lyso009a_0087.JF07T32V01_master_rev.h5 that provides compliance with the October 2019 NXmx specification as proposed in https://github.com/HDRMX/definitions.git

To create a new NeXus master file, assuming DIALS is installed in the folder $DIALS, use this command:

libtbx.python $DIALS/modules/cctbx_project/xfel/swissfel/jf16m_cxigeom2nexus.py unassembled_file=lyso009a_0087.JF07T32V01.h5 geom_file=16M_bernina_backview_optimized_adu_quads.geom wavelength=1.368479 detector_distance=97.830 mask_file=lyso009a_0087.JF07T32V01.mask.h5

Geometry file is in CrystFEL format but has been realigned to group the modules hierarchically into quadrants.

View the data using DIALS: dials.image_viewer lyso009a_0087.JF07T32V01_master.h5

Process the data using DIALS, treating the images as stills, assuming 64 cores available on the system: dials.stills_process mp.nproc=64 lyso009a_0087.JF07T32V01_master.h5 dispersion.gain=10 known_symmetry.space_group=P43212 known_symmetry.unit_cell=77,77,37,90,90,90 refinement_protocol.d_min_start=2.5

Download DIALS at dials.github.io.

After the DIALS run, for full NXmx compliance you will need the jungfrau portions of the script that was used to generate lyso009a_0087.JF07T32V01_master_rev.h5

!/bin/bash

cp Therm_6_2.nxs Therm_6_2_rev.nxs cp Therm_6_2_master.h5 Therm_6_2_master_rev.h5 cp jungfrau/lyso009a_0087.JF07T32V01_master.h5 jungfrau/lyso009a_0087.JF07T32V01_master_rev.h5 export curdat=date +%FT%T.%3 export LD_LIBRARY_PATH=$HOME/lib export HDF5_PLUGIN_PATH=$HOME/lib export PATH=$HOME/bin:$PATH h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_master_rev.h5 -s /entry/instrument/name -d /entry/instrument/name -f ref h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_master_rev.h5 -s /entry/instrument/source -d /entry/source -f ref h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_rev.nxs -s /entry/instrument/source -d /entry/source -f ref h5copy -i jungfrau/lyso009a_0087.JF07T32V01_master.h5 -o jungfrau/lyso009a_0087.JF07T32V01_master_rev.h5 -s /entry/sample/beam -d /entry/instrument/beam -f ref export end_time=h5dump -d "/entry/end_time" Therm_6_2_master.h5 | grep ":" | sed 's/^.........//'|sed 's/.\$//' echo "end_time: $end_time" python << 'EOL' import h5py as h5 import numpy as np import os end_time=os.environ['end_time'] curdat=os.environ['curdat'] fvds = h5.File('Therm_6_2_rev.nxs','r+') fmaster = h5.File('Therm_6_2_master_rev.h5','r+') jungfrau= h5.File('jungfrau/lyso009a_0087.JF07T32V01_master_rev.h5','r+') fvds_keys=fvds.keys() fmaster_keys=fmaster.keys() jungfrau_keys=jungfrau.keys() fvds_entry=fvds['entry'] fmaster_entry=fmaster['entry'] jungfrau_entry=jungfrau['entry'] fvds_entry_keys=fvds_entry.keys() fmaster_entry_keys=fmaster_entry.keys() jungfrau_entry_keys=jungfrau_entry.keys() fvds_entry_instrument=fvds['entry']['instrument'] fmaster_entry_instrument=fmaster['entry']['instrument'] jungfrau_entry_instrument=jungfrau['entry']['instrument'] fvds_entry_instrument_keys=fvds_entry_instrument.keys() fmaster_entry_instrument_keys=fmaster_entry_instrument.keys() jungfrau_entry_instrument_keys=jungfrau_entry_instrument.keys() fvds_entry_instrument_name=(fvds['entry']['instrument']['name']) fmaster_entry_instrument_name=(fmaster['entry']['instrument']['name']) jungfrau['entry']['instrument'].create_dataset("name", data=np.string_("Paul Scherrer Institute SwissFEL Aramis 1 (Alvra)")) jungfrau_entry_instrument_name=(jungfrau['entry']['instrument']['name']) fvds_entry_instrument_short_name=fvds_entry_instrument.attrs['short_name'] fmaster_entry_instrument_short_name=fmaster_entry_instrument.attrs['short_name'] jungfrau_entry_instrument_name.attrs.modify('short_name',np.string_("Alvra")) jungfrau_entry_instrument_short_name=jungfrau_entry_instrument_name.attrs['short_name'] zero_offset=fmaster_entry_instrument['detector']['module']['fast_pixel_direction'].attrs['offset'] fmaster_det_z=fmaster_entry_instrument['transformations']['det_z'] fvds_det_z=fvds_entry_instrument['transformations']['det_z'] print('fvds_keys: ',fvds_keys) print('fmaster_keys: ',fmaster_keys) print('jungfrau_keys: ',jungfrau_keys) print('fvds_entry_keys: ',fvds_entry_keys) print('fmaster_entry_keys: ',fmaster_entry_keys) print('jungfrau_entry_keys: ',jungfrau_entry_keys) print('fvds_entry_instrument_keys: ',fvds_entry_instrument_keys) print('fmaster_entry_instrument_keys: ',fmaster_entry_instrument_keys) print('jungfrau_entry_instrument_keys: ',jungfrau_entry_instrument_keys) print('fvds_entry_instrument_name: ',fvds_entry_instrument_name) print('fmaster_entry_instrument_name: ',fmaster_entry_instrument_name) print('jungfrau_entry_instrument_name: ',jungfrau_entry_instrument_name) print('fvds_entry_instrument_short_name: ',fvds_entry_instrument_short_name) print('fmaster_entry_instrument_short_name: ',fmaster_entry_instrument_short_name) print('jungfrau_entry_instrument_short_name: ',jungfrau_entry_instrument_short_name) print('fmaster_entry_instrument_detector_module_fast_pixel_direction_offset: ',zero_offset) print('fmaster_entry_instrument_detector_detector_z_det_z: ',fmaster_det_z) print('fmaster_entry_end_time: ',end_time) fmaster.attrs.modify('file_time',np.string_(end_time)) fmaster.attrs.modify('file_name',np.string_('Therm_6_2_master_rev.h5')) fmaster.attrs.modify('HDF5_Version',np.string_('hdf5-1.8.18')) fvds.attrs.modify('file_time',np.string_(end_time)) fvds.attrs.modify('file_name',np.string_('Therm_6_2_master_rev.h5')) fvds.attrs.modify('HDF5_Version',np.string_('hdf5-1.10.5')) jungfrau.attrs.modify('file_time',np.string_(curdat)) jungfrau.attrs.modify('file_name',np.string_('lyso009a_0087.JF07T32V01_master.h5')) jungfrau.attrs.modify('HDF5_Version',np.string_('hdf5-1.10.5')) fvds_entry_instrument_name.attrs.modify('short_name',np.string_(fvds_entry_instrument.attrs['short_name'])) fmaster_entry_instrument_name.attrs.modify('short_name',np.string_(fmaster_entry_instrument.attrs['short_name'])) fmaster_entry_instrument['attenuator']['attenuator_transmission'].attrs.modify('units',np.string_("")) fmaster_entry_instrument['detector']['count_time'].attrs.modify('units',np.string_("s")) fvds_entry_instrument_name.attrs.modify('short_name',np.string_(fvds_entry_instrument.attrs['short_name'])) fvds_entry_instrument['attenuator']['attenuator_transmission'].attrs.modify('units',np.string_("")) fvds_entry_instrument['detector']['count_time'].attrs.modify('units',np.string_("s")) fmaster_det_z.attrs.modify('offset',zero_offset) fvds_det_z.attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['phi'].attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['chi'].attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['sam_x'].attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['sam_y'].attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['sam_z'].attrs.modify('offset',zero_offset) fmaster_entry['sample']['transformations']['omega'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['phi'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['chi'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['sam_x'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['sam_y'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['sam_z'].attrs.modify('offset',zero_offset) fvds_entry['sample']['transformations']['omega'].attrs.modify('offset',zero_offset) print(fmaster['entry']['instrument']['name'].attrs['short_name']) print(fmaster['entry']['instrument']['name'].attrs['short_name'].shape) print(fmaster['entry']['instrument']['name'].attrs['short_name'].dtype) print("/entry/instrument/ELE_D0/pixel_mask_applied :",jungfrau_entry_instrument['ELE_D0']['pixel_mask_applied']) del jungfrau_entry_instrument['ELE_D0']['pixel_mask_applied'] jungfrau_entry_instrument['ELE_D0'].create_dataset("pixel_mask_applied",dtype='int8', data=1) print("/entry/instrument/ELE_D0/pixel_mask_applied :",jungfrau_entry_instrument['ELE_D0']['pixel_mask_applied']) jungfrau_entry_source=jungfrau_entry.create_group('source') jungfrau_entry_source=jungfrau_entry['source'] jungfrau_entry_source.attrs.modify('NX_class',np.string_("NXsource")) jungfrau_entry_source.create_dataset("name",data=np.string_("Paul Scherrer Institute SwissFEL")) jungfrau_entry_source['name'].attrs.modify('short_name',np.string_("SwissFEL"))

jungfrau_entry_instrument.create_group['beam']

jungfrau_entry_instrument['beam']=jungfrau_entry['sample']['beam']

jungfrau_entry_instrument['beam'].create_dataset('total_flux',dtype='float64',data=1000000000000.) jungfrau_entry_instrument['beam']['total_flux'].attrs.modify('units',np.string_('/pulse')) del jungfrau_entry['sample']['beam'] del fvds_entry_instrument.attrs['short_name'] del fmaster_entry_instrument.attrs['short_name'] del fmaster_entry_instrument['source'] fvds.close() fmaster.close() jungfrau.close() quit() EOL $HOME/bin/nxvalidate -a NXmx -l /home/yaya/hdrmx_rev_29Sep19/hdrmx/definitions Therm_6_2_master_rev.h5 $HOME/bin/nxvalidate -a NXmx -l /home/yaya/hdrmx_rev_29Sep19/hdrmx/definitions Therm_6_2_rev.nxs $HOME/bin/nxvalidate -a NXmx -l /home/yaya/hdrmx_rev_29Sep19/hdrmx/definitions jungfrau/lyso009a_0087.JF07T32V01_master_rev.h5

The 4 files contain the same dataset in 4 different formats:

• Data_POMC.fits (FITS format).
• Data_POMC;json (JSON format with data as nested arrays suitable for direct import in Python).
• Data_POMC2.json (JSON format with "flattened" array).
• Data_POMC.py (a Python module containing a description and a single variable 'stack' with the data as a 3D NumPy array).

The data are POMC neuron image stack. The CCD chip size (after binning) is 60x80 and 168 fluorescence images were taken. The fluorophore used was Fura-2. Fluorescence images were acquired at 340 nm every 150 ms (exposure time: 12 ms). The imaging setup consisted of an Imago SensiCam CCD camera with a 640x480 chip (Till Photonics, Graefelfing, Germany) and a Polychromator IV (Till Photonics) that was coupled via an optical fiber into the upright microscope. Emitted fluorescence was detected through a 440 nm long-pass filter (LP440). Data were acquired as 80x60 frames using 8x8 on-chip binning. Images were recorded in analog-to-digital units (ADUs) and stored as 12-bit grayscale images. A depolarizing currrent pulse was applied just before frame 13 provoking calcium entry. The data were acquired by Andreas Pippow.
Reference: JOUCLA ET AL. (2013) CELL CALCIUM. 54(2):71-85

To read Data_POMC.fits into a Python session do:

import fitsio
import numpy as np
fits = fitsio.FITS('Data_POMC.fits','r')
fits

To read Data_POMC.py into a Python session do:

import Data_POMC
help(Data_POMC)

To read Data_POMC.json into a Python session do:

import json
import numpy as np
with open("Data_POMC.json","r") as f:
pomc = json.load(f) # pomc is a dictionary
pomc_stack = np.array(pomc['stack'])
print(pomc['metadata'])

To read Data_POMC2.json into a Python session do:

import json
import numpy as np
with open("Data_POMC2.json","r") as f:
pomc = json.load(f) # pomc is a dictionary
pomc_stack = np.reshape(pomc['stack'],(60,80,168),order='f')
print(pomc['metadata'])

Overview

3DHD CityScenes is the most comprehensive, large-scale high-definition (HD) map dataset to date, annotated in the three spatial dimensions of globally referenced, high-density LiDAR point clouds collected in urban domains. Our HD map covers 127 km of road sections of the inner city of Hamburg, Germany including 467 km of individual lanes. In total, our map comprises 266,762 individual items.

Our corresponding paper (published at ITSC 2022) is available here.
Further, we have applied 3DHD CityScenes to map deviation detection here.

Moreover, we release code to facilitate the application of our dataset and the reproducibility of our research. Specifically, our 3DHD_DevKit comprises:

• Python tools to read, generate, and visualize the dataset,
• 3DHDNet deep learning pipeline (training, inference, evaluation) for
  map deviation detection and 3D object detection.

The DevKit is available here:

https://github.com/volkswagen/3DHD_devkit.

The dataset and DevKit have been created by Christopher Plachetka as project lead during his PhD period at Volkswagen Group, Germany.

When using our dataset, you are welcome to cite:

@INPROCEEDINGS{9921866,
  author={Plachetka, Christopher and Sertolli, Benjamin and Fricke, Jenny and Klingner, Marvin and 
  Fingscheidt, Tim},
  booktitle={2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)}, 
  title={3DHD CityScenes: High-Definition Maps in High-Density Point Clouds}, 
  year={2022},
  pages={627-634}}

Acknowledgements

We thank the following interns for their exceptional contributions to our work.

• Benjamin Sertolli: Major contributions to our DevKit during his master thesis
• Niels Maier: Measurement campaign for data collection and data preparation

The European large-scale project Hi-Drive (www.Hi-Drive.eu) supports the publication of 3DHD CityScenes and encourages the general publication of information and databases facilitating the development of automated driving technologies.

The Dataset

After downloading, the 3DHD_CityScenes folder provides five subdirectories, which are explained briefly in the following.

1. Dataset

This directory contains the training, validation, and test set definition (train.json, val.json, test.json) used in our publications. Respective files contain samples that define a geolocation and the orientation of the ego vehicle in global coordinates on the map.

During dataset generation (done by our DevKit), samples are used to take crops from the larger point cloud. Also, map elements in reach of a sample are collected. Both modalities can then be used, e.g., as input to a neural network such as our 3DHDNet.

To read any JSON-encoded data provided by 3DHD CityScenes in Python, you can use the following code snipped as an example.

import json

json_path = r"E:\3DHD_CityScenes\Dataset\train.json"
with open(json_path) as jf:
  data = json.load(jf)
print(data)

2. HD_Map

Map items are stored as lists of items in JSON format. In particular, we provide:

• traffic signs,
• traffic lights,
• pole-like objects,
• construction site locations,
• construction site obstacles (point-like such as cones, and line-like such as fences),
• line-shaped markings (solid, dashed, etc.),
• polygon-shaped markings (arrows, stop lines, symbols, etc.),
• lanes (ordinary and temporary),
• relations between elements (only for construction sites, e.g., sign to lane association).

3. HD_Map_MetaData

Our high-density point cloud used as basis for annotating the HD map is split in 648 tiles. This directory contains the geolocation for each tile as polygon on the map. You can view the respective tile definition using QGIS. Alternatively, we also provide respective polygons as lists of UTM coordinates in JSON.

Files with the ending .dbf, .prj, .qpj, .shp, and .shx belong to the tile definition as “shape file” (commonly used in geodesy) that can be viewed using QGIS. The JSON file contains the same information provided in a different format used in our Python API.

4. HD_PointCloud_Tiles

The high-density point cloud tiles are provided in global UTM32N coordinates and are encoded in a proprietary binary format. The first 4 bytes (integer) encode the number of points contained in that file. Subsequently, all point cloud values are provided as arrays. First all x-values, then all y-values, and so on. Specifically, the arrays are encoded as follows.

• x-coordinates: 4 byte integer
• y-coordinates: 4 byte integer
• z-coordinates: 4 byte integer
• intensity of reflected beams: 2 byte unsigned integer
• ground classification flag: 1 byte unsigned integer

After reading, respective values have to be unnormalized. As an example, you can use the following code snipped to read the point cloud data. For visualization, you can use the pptk package, for instance.

import numpy as np
import pptk

file_path = r"E:\3DHD_CityScenes\HD_PointCloud_Tiles\HH_001.bin"
pc_dict = {}
key_list = ['x', 'y', 'z', 'intensity', 'is_ground']
type_list = ['

This dataset includes replication data for the paper: Sann, R. and Lai, P.-C. (2023), "Topic modeling of the quality of guest’s experience using latent Dirichlet allocation: western versus eastern perspectives", Consumer Behavior in Tourism and Hospitality, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/CBTH-04-2022-0084

Procedurally Generated Matrices (PGM) data from the paper Measuring Abstract Reasoning in Neural Networks, Barrett, Hill, Santoro et al. 2018. The goal is to infer the correct answer from the context panels based on abstract reasoning.

To use this data set, please download all the *.tar.gz files from the data set page and place them in ~/tensorflow_datasets/abstract_reasoning/.

\(R\) denotes the set of relation types (progression, XOR, OR, AND, consistent union), \(O\) denotes the object types (shape, line), and \(A\) denotes the attribute types (size, colour, position, number). The structure of a matrix, \(S\), is the set of triples \(S={[r, o, a]}\) that determine the challenge posed by a particular matrix.

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('abstract_reasoning', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

ImageNet-R is a set of images labelled with ImageNet labels that were obtained by collecting art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes. ImageNet-R has renditions of 200 ImageNet classes resulting in 30,000 images. by collecting new data and keeping only those images that ResNet-50 models fail to correctly classify. For more details please refer to the paper.

The label space is the same as that of ImageNet2012. Each example is represented as a dictionary with the following keys:

• 'image': The image, a (H, W, 3)-tensor.
• 'label': An integer in the range [0, 1000).
• 'file_name': A unique sting identifying the example within the dataset.

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('imagenet_r', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

https://storage.googleapis.com/tfds-data/visualization/fig/imagenet_r-0.2.0.png" alt="Visualization" width="500px">

BCCD Dataset is a small-scale dataset for blood cells detection.

Thanks the original data and annotations from cosmicad and akshaylamba. The original dataset is re-organized into VOC format. BCCD Dataset is under MIT licence.

Data preparation is important to use machine learning. In this project, the Faster R-CNN algorithm from keras-frcnn for Object Detection is used. From this dataset, nicolaschen1 developed two Python scripts to make preparation data (CSV file and images) for recognition of abnormalities in blood cells on medical images.

export.py: it creates the file "test.csv" with all data needed: filename, class_name, x1,y1,x2,y2. plot.py: it plots the boxes for each image and save it in a new directory.

Image Type : jpeg(JPEG) Width x Height : 640 x 480

To use this dataset:

import tensorflow_datasets as tfds

ds = tfds.load('bccd', split='train')
for ex in ds.take(4):
 print(ex)

See the guide for more informations on tensorflow_datasets.

https://storage.googleapis.com/tfds-data/visualization/fig/bccd-1.0.0.png" alt="Visualization" width="500px">

This database automatically captures metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Staff in R & D in the business sector by occupation, sex and size class of enterprise, Slovenia, annually”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

01_ML_phylogeny.zipThis file contains all the data necessary to re-run our ML tree using the full alignmentcontaining all 564 species. There are 5 files. 'burleigh_concat.phy' is the alignment file in phylip format. 'commandline' is the commandline we used to run RAxML version 7. 'partitions' describes how we partitioned our data (RAxML uses that file). 'RAxML_result.burleigh_concat.raxml.out' has the tree file that results from our analysis. 'raxmlHPC' is the raxml executable we used to run our analyses, on a Mac desktop computer.02_sister_pairs.zipThis file contains data and R code for the sister pairs analyses. There are 2 files. 'raw_data_sister_pairs.csv' is a comma separated values file that contains all of the branch length and life history data we used in the sister pairs analyses. There is one row per sister pair in the analysis, which has data on: the two families in the sister pair, the proportion of genera we had height data for in each family, the number of species in each family, the average height of each family (log transformed value in mm), nuclear branch length for each family (in units of substitutions/site), chloroplast dN branch length for each family (also in substitutions/site), chloroplast dS branch length for each family (also in substitutions/site), Latitude for each family (in distance from the equator), UV for each family (measured as in Davies et al 2004), and Temperature for each family (in Kelvins). 'sister_pairs_analyses.r' contains all R code used for the sister pairs analyses. To use it, you will need to download R (it's free), install the relevant packages (at the top of the .r file), and then change the line at the top which starts 'setwd' to point to the folder on your computer that contains the raw_data_sister_pairs.csv file. As you go through the R code, it will print out all of the results in the paper that used sister pairs analyses.03_R8SThis file contains input files and the results files from running R8S on the ML tree, and on 1000 bootstraps of the ML tree. There are 3 files and 2 folders. 'base_files' is just a holder for some basic R8S input files, that the python script uses (see below). This folder contains: the r8s executable we used (compiled for macs), the basic r8s.txt input file we used for each r8s analysis (which has data on our fossil calibrations), and a .txt file that contains the ML tree, and 1000 boostrapped trees estimated in RAxML as described in the paper. 'bootstrap_rates.txt' contains the results of the R8S analyses, each family is listed on its own row, and each row has 1001 associated columns. The first column contains the ML rate, subsequent columns contain bootstrap rates (in substitutions/site/myr). This is the main output file produced by the python script (see next). 'run_BS_r8s.py' is a python script that will run r8s on the ML tree and the 1000 bootstrap trees. Before running it, create an empty directory in the same folder as the script called "bootstrap_results", then change the "start_dir" and "tree_file" variables at the top of the script to point to the directory the script is in, and the tree file in the 'base_files' folder respectively. Then run the script using python. Briefly, the script takes each tree from the tree file, makes a r8s input file, runs r8s, then parses the output to extract the rates for each family. It then outputs these to the 'bootstrap_rates.txt' file. Be aware that the script stores all r8s results, which can take a lot of space (about 1GB) when the analyses are all complete.04_PGLSThis file contains 5 files, sufficient to re-do all of our PGLS analyses. 'bootstrap_rates.txt' contains the results of the ML r8s analysis and all 1000 subsequent bootstrap analyses. 'growth_forms.csv' contains information on the growth forms of species in each family. 'PGLS_analyses.r' is an R script which you can use to re-run all of our PGLS analyses. To use it you will need to change the line at the top that starts 'setwd' to point to the folder on your computer that contains all of the input files here. You'll also need to download R, and the packages listed at the top of the file. 'R8S_trees.txt' is a file of the 1001 trees from R8S. These are used in the PGLS analyses to correct for nonindepdence. The first tree is the ML tree, the rest are bootstrap trees. 'raw_data_sister_pairs.csv' is a csv file of the raw data. It's included here so that the R script will run without additional hassle. But it's identical to the file described in the '02_sister_pairs' section above. Rates of molecular evolution have a central role in our understanding of many aspects of species’ biology. However, the causes of variation in rates of molecular evolution remain poorly understood, particularly in plants. Here we show that height accounts for about one-fifth of the among-lineage rate variation in the chloroplast and nuclear genomes of plants. This relationship ho...

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Expenditure for R & D in the business sector (EUR 1 000) by source of funding and size class of enterprise, Slovenia, annually”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Employees in R & D by level of education (KLASIUS), occupation and sex, Slovenia, 2008-2016”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

This database automatically captures metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Financial resources for R & D from abroad by sectors of implementation and types of institutions (EUR 1 000), Slovenia, 2007-2016”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Staff in R & D by sector of employment, occupation and sex, Slovenia, annually”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL USE OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Employees in R & D in the business sector by occupation and activity (NACE Rev. 2), Slovenia, 2008-2016”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

Quantification of print accuracy based on comparing size of anatomical landmarks between computer model and 3D print.

This database automatically includes metadata, the source of which is the GOVERNMENT OF THE REPUBLIC OF SLOVENIA STATISTICAL OF THE REPUBLIC OF SLOVENIA and corresponding to the source database entitled “Gross domestic R & D expenditure (EUR 1 000) by sector of implementation and type of costs, Slovenia, annually”.

Actual data are available in Px-Axis format (.px). With additional links, you can access the source portal page for viewing and selecting data, as well as the PX-Win program, which can be downloaded free of charge. Both allow you to select data for display, change the format of the printout, and store it in different formats, as well as view and print tables of unlimited size, as well as some basic statistical analyses and graphics.

This dataset and software tool are for reproducing the research results related to CVE-2019-18222.

Description

enum contains the key enumeration tool.

kt_candidates contains the JSON for blinded nonce candidates, indexed by trial number. JSON fields:

kt_candidates: list of nonce candidates.

sig_data contains the JSON for ECDSA signatures, index by trial number. JSON fields:

p: the prime the curve is defined over. (P-256 here.)

Gx, Gy: Generator coordinates.

d: Ground truth ECDSA long term key.

Px, Py: Public key coordinates.

h: SHA-256 digest to sign, encoded to the finite field.

k: Ground truth ECDSA nonce.

r, s: ECDSA signature.

Build

cd enum make clean make

Run

Start with enum as the working directory.

cd enum

Pull out a kt candidate, in this example index 847.

$ jq '.kt_candidates' ../kt_candidates/kt_candidates_847.json [ "0x48ad7217d10f6c7b1a3db836d38aa3972999115f38a6b3d176fc660941aa5c882d2528ec1fc27da7610e7ee3d7dd84367c380259e0386224c2c46aa2a5eb2a0" ]

Factor that candidate.

$ time sage -c "print ecm.factor(0x48ad7217d10f6c7b1a3db836d38aa3972999115f38a6b3d176fc660941aa5c882d2528ec1fc27da7610e7ee3d7dd84367c380259e0386224c2c46aa2a5eb2a0)" [2, 2, 2, 2, 2, 3, 353, 193243, 1540830719, 9263081209, 103633959617085683, 151389566295160172521, 283135469779419532841, 572987990320782777757565685333349772719941819448953457732874126833]

real 0m5.837s user 0m5.648s sys 0m0.214s

Now pull out the r component of the ECDSA signature for that index, and convert it from hex to base 10.

$ jq '.r' ../sig_data/sig_data_847.json "0x30e2ce20a8140177a31a66763d85f431acc9790dd050ffc22ed5d454cdfbbb67" $ python -c "print 0x30e2ce20a8140177a31a66763d85f431acc9790dd050ffc22ed5d454cdfbbb67" 22111746808803128586382711090186612204136854333384650261207856620766542674791

Now run the enum tool to recover the nonce.

$ ./enum Usage: ./enum space delimited flat list of factors in base ten

The and arguments are to ease parallel execution; read the source code. But for a single core, pass them as 1 0.

$ ./enum 1 0 22111746808803128586382711090186612204136854333384650261207856620766542674791 2 2 2 2 2 3 353 193243 1540830719 9263081209 103633959617085683 151389566295160172521 2831354697794195 32841 572987990320782777757565685333349772719941819448953457732874126833 INFO:target:30E2CE20A8140177A31A66763D85F431ACC9790DD050FFC22ED5D454CDFBBB67 INFO:found:31A52C4960857E6D2F7AD82BAC7D55CE6CC9AD13B959F069002B6A949EA6A048 INFO:tests:7879

where 221..791 is the base-10 r component of the ECDSA signature, and 2 2 .. 572..833 is the full list of blinded nonce factors. In the output:

INFO:target: is the hex form of base-10 target input (ECDSA r component).

INFO:found: is the hex form of the recovered ECDSA nonce.

INFO:tests: is the number of tested nonce candidates (scalar multiplications).

We can see this successfully recovered the nonce (hence long term ECDSA private key) correctly:

$ jq '.k' ../sig_data/sig_data_847.json "0x31a52c4960857e6d2f7ad82bac7d55ce6cc9ad13b959f069002b6a949ea6a048"