Categorical scatterplots with R for biologists: a step-by-step guide

Benjamin Petre1, Aurore Coince2, Sophien Kamoun1

1 The Sainsbury Laboratory, Norwich, UK; 2 Earlham Institute, Norwich, UK

Weissgerber and colleagues (2015) recently stated that ‘as scientists, we urgently need to change our practices for presenting continuous data in small sample size studies’. They called for more scatterplot and boxplot representations in scientific papers, which ‘allow readers to critically evaluate continuous data’ (Weissgerber et al., 2015). In the Kamoun Lab at The Sainsbury Laboratory, we recently implemented a protocol to generate categorical scatterplots (Petre et al., 2016; Dagdas et al., 2016). Here we describe the three steps of this protocol: 1) formatting of the data set in a .csv file, 2) execution of the R script to generate the graph, and 3) export of the graph as a .pdf file.

Protocol

• Step 1: format the data set as a .csv file. Store the data in a three-column excel file as shown in Powerpoint slide. The first column ‘Replicate’ indicates the biological replicates. In the example, the month and year during which the replicate was performed is indicated. The second column ‘Condition’ indicates the conditions of the experiment (in the example, a wild type and two mutants called A and B). The third column ‘Value’ contains continuous values. Save the Excel file as a .csv file (File -> Save as -> in ‘File Format’, select .csv). This .csv file is the input file to import in R.

• Step 2: execute the R script (see Notes 1 and 2). Copy the script shown in Powerpoint slide and paste it in the R console. Execute the script. In the dialog box, select the input .csv file from step 1. The categorical scatterplot will appear in a separate window. Dots represent the values for each sample; colors indicate replicates. Boxplots are superimposed; black dots indicate outliers.

• Step 3: save the graph as a .pdf file. Shape the window at your convenience and save the graph as a .pdf file (File -> Save as). See Powerpoint slide for an example.

Notes

• Note 1: install the ggplot2 package. The R script requires the package ‘ggplot2’ to be installed. To install it, Packages & Data -> Package Installer -> enter ‘ggplot2’ in the Package Search space and click on ‘Get List’. Select ‘ggplot2’ in the Package column and click on ‘Install Selected’. Install all dependencies as well.

• Note 2: use a log scale for the y-axis. To use a log scale for the y-axis of the graph, use the command line below in place of command line #7 in the script.

7 Display the graph in a separate window. Dot colors indicate

replicates

graph + geom_boxplot(outlier.colour='black', colour='black') + geom_jitter(aes(col=Replicate)) + scale_y_log10() + theme_bw()

References

Dagdas YF, Belhaj K, Maqbool A, Chaparro-Garcia A, Pandey P, Petre B, et al. (2016) An effector of the Irish potato famine pathogen antagonizes a host autophagy cargo receptor. eLife 5:e10856.

Petre B, Saunders DGO, Sklenar J, Lorrain C, Krasileva KV, Win J, et al. (2016) Heterologous Expression Screens in Nicotiana benthamiana Identify a Candidate Effector of the Wheat Yellow Rust Pathogen that Associates with Processing Bodies. PLoS ONE 11(2):e0149035

Weissgerber TL, Milic NM, Winham SJ, Garovic VD (2015) Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm. PLoS Biol 13(4):e1002128

https://cran.r-project.org/

http://ggplot2.org/

Petition subject: Against railroad discrimination with focus on white passengers Original: http://nrs.harvard.edu/urn-3:FHCL:10956457 Date of creation: (unknown) Petition location: Sudbury Legislator, committee, or address that the petition was sent to: Francis R. Gourgas, Concord Selected signatures:J.H. BrownSally BrownLoring Eaton Total signatures: 76 Legal voter signatures (males not identified as non-legal): 31 Female signatures: 37 Unidentified signatures: 8 Female only signatures: No Identifications of signatories: inhabitants, [females] Prayer format was printed vs. manuscript: Printed Signatory column format: not column separated Additional non-petition or unrelated documents available at archive: no additional documents Additional archivist notes: 11057/4 written on back Location of the petition at the Massachusetts Archives of the Commonwealth: House Unpassed 1842, Docket 1153 Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

Petition subject: Execution case Original: http://nrs.harvard.edu/urn-3:FHCL:12233039 Date of creation: (unknown) Petition location: Uxbridge Selected signatures:Luther RistSusan R. UsherHarriett N. Moury Total signatures: 175 Legal voter signatures (males not identified as non-legal): 64 Female signatures: 84 Unidentified signatures: 27 Female only signatures: No Identifications of signatories: inhabitants, [females] Prayer format was printed vs. manuscript: Manuscript Signatory column format: not column separated Additional non-petition or unrelated documents available at archive: additional documents available Additional archivist notes: Leander Thompson Location of the petition at the Massachusetts Archives of the Commonwealth: Governor Council Files, April 17, 1847, Case of Leander Thompson Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

The review dataset for 3 video games - Call of Duty : Black Ops 3, Persona 5 Royal and Counter Strike: Global Offensive was taken through a web scrape of SteamDB [https://steamdb.info/] which is a large repository for game related data such as release dates, reviews, prices, and more. In the initial scrape, each individual game has two files - customer reviews (Count: 100 reviews) and price time series data.

To obtain data on the reviews of the selected video games, we performed web scraping using R software. The customer reviews dataset contains the date that the review was posted and the review text, while the price dataset contains the date that the price was changed and the price on that date. In order to clean and prepare the data we first start by sectioning the data in excel. After scraping, our csv file fits each review in one row with the date. We split the data, separating date and review, allowing them to have separate columns. Luckily scraping the price separated price and date, so after the separating we just made sure that every file had similar column names.

After, we use R to finish the cleaning. Each game has a separate file for prices and review, so each of the prices is converted into a continuous time series by extending the previously available price for each date. Then the price dataset is combined with its respective in R on the common date column using left join. The resulting dataset for each game contains four columns - game name, date, reviews and price. From there, we allow the user to select the game they would like to view.

Various water column variables, including salinity, dissolved inorganic nutrients, pH, total alkalinity, dissolved inorganic carbon, radio-carbon isotopes were measured in samples collected using a Niskin-bottle rosette at selected depths from sites offshore of California and Oregon from October to November 2019 during NOAA Ship Lasker R-19-05 (USGS field activity 2019-672-FA). CTD (Conductivity Temperature Depth) data were also collected at each depth that a Niskin-bottle sample was collected and are presented along with the water sample data. This data release supersedes version 1.0, published in August 2020 at https://doi.org/10.5066/P9ZS1JX8. Versioning details are documented in the accompanying VersionHistory_P9JKYWQU.txt file.

Petition subject: Execution case Original: http://nrs.harvard.edu/urn-3:FHCL:12232985 Date of creation: 1843-09-11 Petition location: Roxbury Selected signatures:Charles W. LillieStephen R. DoggettCaroline Williams Total signatures: 13 Legal voter signatures (males not identified as non-legal): 9 Female signatures: 4 Female only signatures: No Identifications of signatories: inhabitants, [females] Prayer format was printed vs. manuscript: Manuscript Signatory column format: not column separated Additional non-petition or unrelated documents available at archive: additional documents available Additional archivist notes: Isaac Leavitt Location of the petition at the Massachusetts Archives of the Commonwealth: Governor Council Files, September 22, 1843, Case of Isaac Leavitt Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

Petition subject: Secession from the Union Original: http://nrs.harvard.edu/urn-3:FHCL:11029681 Date of creation: (unknown) Petition location: Fitchburg Legislator, committee, or address that the petition was sent to: Charles Mason, Fitchburg; committee on the judiciary Selected signatures:Roby R. SaffordSarah S. SpauldingLevi Farnsworth Actions taken on dates: 1849-03-03 Legislative action: Received in the House on March 3, 1849 and referred to the committee on the judiciary Total signatures: 54 Legislative action summary: Received, referred Legal voter signatures (males not identified as non-legal): 38 Female signatures: 16 Female only signatures: No Identifications of signatories: inhabitants, legal voters, others, [females], ["other persons"] Prayer format was printed vs. manuscript: Printed Signatory column format: column separated Additional non-petition or unrelated documents available at archive: no additional documents Location of the petition at the Massachusetts Archives of the Commonwealth: House Unpassed 1849, Docket 2343 Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

Attached is the complete raw data from Vaughan and Dixson 2021 ‘Assessing the impact of static and fluctuating ocean acidification on the behavior of Amphiprion percula’.

Data collected from the behavioral lateralization trials has been inputted into the file ‘Vaughan_2020_Lateralization_Raw’. Column A indicate the CO₂ treatment group, where “SPD” = Static Present Day, “SFD” = Static Future Day, “FPD” = Fluctuating Present Day, and “FFD” = Fluctuating Future Day. Each individual fish used from each treatment group (n=30) is displayed in Column B. Column C shows the binary results, in order, of each fish’s turns in the T-maze, and was scored as 0 (right turn) or 1 (left turn) for a total of 10 turns. The total number of turns to the right and left are provided in Column D-E. The relative lateralization (L_R) of each fish was calculated {L_R = [(Turn to the right – Turn to the left)/(Turn to the right + Turn to the left)] ∗ 100} in Column F. Absolute lateralization (L_A) is provided in Column G.

Chemosensory response data has been inputted into the file ‘Vaughan_2020_Chemosensory_Raw’. Column A and B display treatment group and fish ID (n=20) as outlined above. The cue used in trial of either Tang (nonpredator) or Cod (predator) is provided in Column C, and the control in Column D. Numbers in these are used solely for the purpose of data analysis. The side of the cue in the flume is provided in Column E, and corresponds with the cue labelled in Column C. Buckets containing either the cue or control were placed above the flume and color coded as “BS” (blue side) and “RS” (red side), as the person scoring the trials was blinded. This also helped account for the switch (from one side of the flume to the other) that occurs halfway through each trial. Columns F-G represent results from the first 2min recording period, and Columns H-I represent results from the second 2min recording period. The total tallies from each fish are provided in Column J; the totals from each side are calculated in Columns K-L, and then sorted by either cue or control in Columns M-N. Proportions and percentages in cue and control are calculated and provided in Columns O-P and Q-R, respectively.

Carbonate chemistry data is compiled and provided in the ‘Vaughan_2020_Carbonate_Chemistry’. Measurements were taken each week (Column A) of each treatment group (as stated above, Column B). Column C reflects the time recordings were taken in the fluctuating treatments to hit the high, mid and low CO₂ points at “6:30”, “12:30” and “18:30”. Measurements of static treatment groups were taken at randomly selected times to get the reflection of the carbonate chemistry of these treatments, but for the purpose of clarity in this document they are listed as “Static”. Measurements were taken from a subset of tanks each that rotated each week (Column D). Our target pH_NBS values (i.e. what was programmed into the APEX System) are listed in Column E. Columns F-H displayed pH_NBS (taken with APEX probes), temperature °C (taken with a portable Mettler Toledo probe) and salinity (taken with a refractometer). Water samples were analyzed spectrophotometrically to provide pH_T and dissolved inorganic carbon, with values provided in Columns I-J. Using the program CO2SYS, total alkalinity and pCO₂ were calculated, with values provided in Columns K-L.

GOLD daytime disk scan (DAY) measurements are used to derive the ratio of the column abundance of thermospheric O relative to N2, conventionally referred to as O/N2 or ΣO/N2, but abbreviated to ON2 for the GOLD data product. ON2 is derived from dayside Level 1C data after binning pixels 2x2 for approximately 68 disk scan measurements performed per day by GOLD in nominal operation.

Algorithm heritage

The disk ON2 retrieval algorithm was originally developed by Computational Physics, Inc. (CPI) for use with GUVI and SSUSI radiance images (Strickland et al., 1995). The GOLD implementation of this algorithm takes advantage of GOLD's ability to transmit the full spectrum to maximize the signal-to-noise ratio and eliminate atomic emission lines that contaminate the N2 LBH bands (e.g., N I 149.3 nm). This algorithm has been extensively documented and applied over the past several decades (e.g., Evans et al. [1995]; Christensen et al. [2003]; Strickland et al. [2004]) and Correira et al. [2021] describe the implementation used for the GOLD data.

Algorithm theoretical basis

The geophysical parameter retrieved, O/N2, is the ratio of the vertical column density of O relative to N2, defined at a standard reference N2 depth of 1017 cm-2, which is chosen to minimize uncertainty in the derived O/N2. It is retrieved directly from the ratio of the O I 135.6 nm and N2 LBH band intensities measured by GOLD on the dayside disk (DAY measurement mode). The AURIC atmospheric radiance model (Strickland et al. [1999]) is used to derive this relationship as a function of solar zenith angle and to create the look-up table (LUT) used by the algorithm.

References

Christensen, A. B., et al. (2003), Initial observations with the Global Ultraviolet Imager (GUVI) in the NASA TIMED satellite mission, J. Geophys. Res., vol. 108, NO. A12, 1451, doi:10.1029/2003JA009918.

Correira, J., Evans, J. S., Lumpe, J. D., Krywonos, A., Daniell, R., Veibell, V., et al. (2021). Thermospheric composition and solar EUV flux from the Globalscale Observations of the Limb and Disk (GOLD) mission. Journal of Geophysical Research: Space Physics, 126, e2021JA029517. https://doi.org/10.1029/2021JA029517

Evans, J. S., D. J. Strickland and R. E. Huffman (1995), Satellite remote sensing of thermospheric O/N2 and solar EUV: 2. Data analysis, J. Geophys. Res., vol. 100, NO. A7, pages 12,227-12,233.

Strickland, D. J., R. R. Meier, R. L. Walterscheid, J. D. Craven, A. B. Christensen, L. J. Paxton, D. Morrison, and G. Crowley (2004), Quiet-time seasonal behavior of the thermosphere seen in the far ultraviolet dayglow, J. Geophys. Res., vol. 109, A01302, doi:10.1029/2003JA010220.

Strickland, D.J., J. Bishop, J.S. Evans, T. Majeed, P.M. Shen, R.J. Cox, R. Link, and R.E. Huffman (1999), Atmospheric Ultraviolet Radiance Integrated Code (AURIC): theory, software architecture, inputs and selected results, JQSRT, 62, 689-742.

Strickland, D. J., J. S. Evans, and L. J. Paxton (1995), Satellite remote sensing of thermospheric O/N2 and solar EUV: 1. Theory, J. Geophys. Res., 110, A7, pages 12,217-12,226.

One of the methods for the separation of stable isotopes is the thermal diffusion column. The advantages of this method include small-scale operations because of apparatus simplicity and a small inventory, especially in gas phase operations. These features attract attention to the tritium and noble gas separation system. In this research, the R cascade was used for designing and determining the number of columns. Moreover, the square cascade was adopted for the final design because of its flexibility. Calculations were performed as an example for the separation of 20Ne and 22Ne isotopes. Accordingly, all R cascades that have enriched Ne isotopes to more than 99% were investigated, and the number of columns was determined. Also, using the specified columns, the square cascade parameters were optimized. A calculation code entitled ''RSQ_CASCADE'' was developed in this regard. The unit separation factor of 3 was considered, and the number of stages was studied in the range of 10 to 20. The results showed that the column separation power, the relative total flow rate, and the required columns were linearly related to the number of stages. The separation power and relative total flow decreased with the increase of stage number while the number of columns increased. Therefore, the cascade of 85 columns was recommended to separate the Ne stable isotopes. These calculations resulted in the 17-stage square cascade, with five columns in each stage. By changing the stages cut, feed point, and cascade feed flow rate, the best parameters of square cascade were determined according to the cascade separation power and column separation power. As the column separation power had the maximum value in the cascade feed 50, it was selected for separating Ne isotopes.

Petition subject: Racial discrimination Original: http://nrs.harvard.edu/urn-3:FHCL:25500518 Date of creation: (unknown) Petition location: Massachusetts Legislator, committee, or address that the petition was sent to: J. Bernard Ferber, Boston Selected signatures:Butler R. WilsonArianna C. SparrowLillian E. ChappelleDaniel H. MinerLaura E. MinerPortia E. BirdWilliam Monroe TrotterGeraldine L. TrotterWalter E. AdamsRobert J. MorrisJ. Horatio CarterLee C. Parrish Actions taken on dates: 1906-04-06 Legislative action: Received in the House on April 6, 1906 and placed on file Total signatures: 39 Legislative action summary: Received, placed on file Legal voter signatures (males not identified as non-legal): 10 Female signatures: 15 Unidentified signatures: 14 Female only signatures: No Identifications of signatories: citizens, [females], [males of color], [females of color], ["others"] Prayer format was printed vs. manuscript: Printed Signatory column format: not column separated Additional non-petition or unrelated documents available at archive: additional documents available Additional archivist notes: Jamestown Tercentennial Exposition, Virginia, includes addresses, towns next to names including Cambridge, Somerville, Boston, Dorchester, Allston, Revere Location of the petition at the Massachusetts Archives of the Commonwealth: Resolves 1906, c.65, passed May 7, 1906 Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

This experiment contains the monthly global mean all-sky spectrally resolved outgoing longwave radiation calculated from the 1-D radiative-convective equilibrium model konrad using the radiative transfer model ARTS. The spectra are available for eight different configurations of the vertical relative humidity profile. The configurations are named as follows: ”uniform” refers to a vertically uniform profile of relative humidity R = 75%, ”cshape” refers to the global mean, C-shaped R profile. The term "base" refers to a surface temperature of 288K, all others to a surface temperature of 289K. ”Constant” means that R is the same as for "base", while ”dry” refers to a R that is 0.5% lower and "moist" refers to a R that is 0.5% higher. Furthermore, the dataset contains the atmospheric profiles of temperature, water vapour, CO2, O3, CH4, N2O and O2 used for the eight experiments.

Compared to the previous version, this dataset now contains the spectra of outgoing longwave radiation instead of spectrally resolved feedbacks. Furthermore, data of experiments in which relative increases was added, as well as the atmospheric profiles used for the experiments.

Exploration of novel alleles from ex situ collection is still limited in modern plant breeding as these alleles exist in genetic backgrounds of landraces that are not adapted to modern production environments. The practice of backcross breeding results in the preservation of the adapted background of elite parents but leaves little room for novel alleles from landraces to be incorporated. The selection of adaptation-associated linkage blocks instead of the entire adapted background may allow breeders to incorporate more of the landrace’s genetic background and to observe and evaluate novel alleles. Important adaptation-associated linkage blocks would have been selected over multiple cycles of breeding and hence are likely to exhibit signatures of positive selection or selective sweeps. We conducted a genome-wide scan for candidate selective sweeps (CSS) using Fst, Rsb, and xpEHH in state, regional, spring, winter, and market class population pairs and report 446 CSS in 19 population pairs over time and 1033 CSS in 44 population pairs across geography and class. Further validation of these candidate selective sweeps in specific breeding programs may lead to the identification of sets of loci that can be selected to restore population-specific adaptation without multiple backcrossing. Methods Folder Structure

The dataset has the following folder structure

./ or the root folder has the scripts used for analysis in R Markdown files as well as the corresponding .html output from running these scripts.

./data/ has the raw data and the intermediate data saves from the analysis

./functions/ has one file "functions_for_selection_sweep_analysis.R" that has the custom functions written for the analysis in the manuscript.

./output/ has the analysis results and figures used in the manuscript

./output/mapchart/ has the MapChart input files for drawing linkage maps of canddiate selective sweeps that were filtered for Fst, Rsb, and xpEHH thresholds of 2 standard deviations

./output/mapchart_sd2.5/ has the MapChart input files for drawing linkage maps of candidate selective sweeps that were filtered for Fst, Rsb, and xpEHH thresholds of 2.5 standard deviations.

./rehh_files/ has two subfolders /genotype and /map that store the intermediate files generated by the R package 'rehh' to calcualte Rsb and xpEHH.

Raw data files

The analysis in the manuscript uses the following raw data files. Data files not in this list are all intermediate files created by the analysis scripts.

./data/90k_SNP_type.txt

A tab-delimmited file with 4 columns as described below:

Index: serial number of genetic markers/loci on the 90K wheat SNP chip.

Name: Unique names of the genetic markers/loci on the 90K wheat SNP chip.

SNP: Alleles present in the single nucleotide polymorphism (SNP) marker/loci.

SNPTYPE: Same information as in column SNP but in a format without square brackets and /

./data/KIM_physical_positions_on_IWGSC_CS_RefSeq_v2.1.txt

A tab-delimmited filed with information on known informative markers (KIM) recorded in 8 columns described below.

Marker: Name of the marker to be used as the label in the linkage maps in Supplemental Figures.

Chromosome: Chromosome label for wheat.

Start1.0: Physical position in base pairs in the 'Chinese Spring' wheat reference genome sequence version 1.0. This information was not used in the current study.

Start: Physical position in base pairs in the 'Chinese Spring' wheat reference genome sequence version 2.1.

Prop: Proportion sequence match for the marker to the reference genome sequence version 2.1.

SNP_ID: Alternative name for the marker. This information was not used in the current study.

Gene: Name of the gene.

Function: Function of the gene.

./data/R-generated-genotype-for-analysis-imputed-AB-format.csv

Raw 90K wheat SNP chip data after quality filtering and imputation uisng LinkImpute as described in Sthapit et al. The dataset includes the 7 information column described below, followed by 753 columns with genotype information in the AB format.

Name: Unique names of the genetic markers/loci on the 90K wheat SNP chip.

SNPid: Unique IWA and IWB SNP names of the genetic markers/loci on the 90K wheat SNP chip.

Chrom: Wheat chromosome labels.

Ord: Order of the marker. This information was not used for analysis.

cM: Centimorgan position of the marker. This information was not used for analysis.

Comment: Notes on manual classification of genotype calls in GenomeStudio.

Remaining columns have variety names and their corresponding genotype calls in AB format.

./data/R-generated-genotype-for-analysis-imputed-nucleotide-format.csv

Same information as in ./data/R-generated-genotype-for-analysis-imputed-AB-format.csv but the genotype information in the last 753 columns are recorded in the nucleotide (ACGT) format.

./data/SNP_physical_positions_on_IWGSC_CS_RefSeq_v2.1.txt

Contains physical base pair positions on the 'Chinese Spring' wheat reference sequence version 2.1 for the 90K SNP chip markers. The file has 5 columns without column headers. The column descriptions are given below.

First column has unique names of the genetic markers/loci on the 90K Wheat SNP chip.

Second column has wheat chromosome labels.

Third column has the starting base pair position of the marker on the reference sequence version 2.1.

Fourth column has the ending base pair position of the marker on the reference sequence version 2.1.

Fifth column has the mid-point of the third and fourth column, which was used at the SNP position for the marker in this study.

./data/variety_details.txt

Contains information about the 753 wheat varieties used as the diversity panel for this study. The file contains 12 columns, which are described below:

GS.Sample.ID: Names of the samples/varieties as they were in the raw output from the Illumina SNP calling software Genome Studio.

Corrected.Sample.ID: Names of the samples/varieties after they were corrected for typos (for example, 'Eric' to 'Erik') and removal of the prefix "varname" for varieties for varieties that only have numbers in their names ('varname2154' to '2154').

ACNO: Accession number of the varieties from the NPGS-GRIN database.

Habit: Growth habit (spring or winter) of the varieties.

Region: U.S. wheat growing regions: EAS, Eastern; GPL, Great Plains; NOR, Northern; PAC, Pacific; PNW, Pacific Northwest. Description of how states were assigned to these regions are in the methods section of the manuscript.

State: U.S. state the varieties are from.

Year: The year the variety was released in the U.S.

MC: Market class of the wheat variety: HRS, hard red spring; HRW, hard red winter; SRW, soft red winter; SWS, soft white spring; SWW, soft white winter.

HeadType: Designates if the spike or head of the wheat is club or common.

Sector: Was the variety from the public or private sector. Information in this column is incomplete and hence was not used for any analysis in the manuscript.

Decade: Decade the variety was released.

BP: Breeding period the variety was released.

Description of Scripts

Here we describe the scripts in order along with the input data files used and the output files these scripts produced.

./00_import_RefSeqv2.1_physical_positions.Rmd ./00_import_RefSeqv2.1_physical_positions.html (R Markdown output html)

The study uses genotype data generated from our previous study (https://doi.org/10.1002/tpg2.20196) that had marker physical positions based on wheat reference sequence version 1. This script updates the marker physical positions to the wheat reference sequence version 2.1 and saves the updated genotype files for subsequent analyses.

Input files:

./data/SNP_physical_positions_on_IWGSC_CS_RefSeq_v2.1.txt ./data/R-generated-genotype-for-analysis-imputed-nucleotide-format.csv ./data/R-generated-genotype-for-analysis-imputed-AB-format.csv

Output files:

./data/genotype_AB_format_13995_loci_imputed.txt ./data/genotype_nucleotide_format_13995_loci_imputed.txt

01_define_populations.Rmd 01_define_populations.html (R Markdown output html)

The script assigns what varieties go into what sub-populations as described in the methods section of the manuscript.

Input files:

./functions/functions_for_selection_sweep_analysis.R ./data/variety_details.txt

Output files:

./data/populations.rds./output/first_last_varieties.csv 02_calculate_iHH_iES_inES.Rmd 02_calculate_iHH_iES_inES.html (R Markdown output html)

This script uses the 'rehh' package function 'scan_hh' called through the custom function 'scan_population' to calculate the integrated extended haplotype homozygosity (iHH), integrated site-specific extended haplotype homozygosity (iES), and integrated normalized site-specific extended haplotype homozygosity (inES) for all markers of all 21 chromosomes and all wheat sub-populations in the study. The intermediate files needed to run these calculations were written to the folders ./rehh_files/genotype and ./rehh_files/map. The output is saved as an RDS file to be used as input for subsequent scripts.

Input files:

./functions/functions_for_selection_sweep_analysis.R ./data/genotype_nucleotide_format_13995_loci_imputed.txt ./data/populations.rds

Output files:

./output/scan_hh_ihs_results_polFALSE_sgap2.5MB_mgapNAMB_discardBorderTRUE.rds 03_calculate_allele_freq_Fst_Rsb_xpEHH.Rmd 03_calculate_allele_freq_Fst_Rsb_xpEHH.html (R Markdown output html)

Script calculates allele frequencies for all the sub-populations and Fst, Rsb, and xpEHH statistics for defined sub-population pairs.

Input files:

./functions/functions_for_selection_sweep_analysis.R ./data/genotype_nucleotide_format_13995_loci_imputed.txt ./data/genotype_AB_format_13995_loci_imputed.txt ./output/scan_hh_ihs_results_polFALSE_sgap2.5MB_mgapNAMB_discardBorderTRUE.rds

Output files:

./output/allele_freq_Fst_Rsb_xpEHH.Rds

Introduction

Welcome to the Cyclistic bike-share analysis case study! In this case study, you will perform many real-world tasks of a junior data analyst. You will work for a fictional company, Cyclistic, and meet different characters and team members. In order to answer the key business questions, you will follow the steps of the data analysis process: ask, prepare, process, analyze, share, and act. Along the way, the Case Study Roadmap tables — including guiding questions and key tasks — will help you stay on the right path.

Scenario

You are a junior data analyst working in the marketing analyst team at Cyclistic, a bike-share company in Chicago. The director of marketing believes the company’s future success depends on maximizing the number of annual memberships. Therefore, your team wants to understand how casual riders and annual members use Cyclistic bikes differently. From these insights, your team will design a new marketing strategy to convert casual riders into annual members. But first, Cyclistic executives must approve your recommendations, so they must be backed up with compelling data insights and professional data visualizations.

Ask

How do annual members and casual riders use Cyclistic bikes differently?

Guiding Question:

What is the problem you are trying to solve?
  How do annual members and casual riders use Cyclistic bikes differently?
How can your insights drive business decisions?
  The insight will help the marketing team to make a strategy for casual riders

Prepare

Guiding Question:

Where is your data located?
  Data located in Cyclistic organization data.

How is data organized?
  Dataset are in csv format for each month wise from Financial year 22.

Are there issues with bias or credibility in this data? Does your data ROCCC? 
  It is good it is ROCCC because data collected in from Cyclistic organization.

How are you addressing licensing, privacy, security, and accessibility?
  The company has their own license over the dataset. Dataset does not have any personal information about the riders.

How did you verify the data’s integrity?
  All the files have consistent columns and each column has the correct type of data.

How does it help you answer your questions?
  Insights always hidden in the data. We have the interpret with data to find the insights.

Are there any problems with the data?
  Yes, starting station names, ending station names have null values.

Process

Guiding Question:

What tools are you choosing and why?
  I used R studio for the cleaning and transforming the data for analysis phase because of large dataset and to gather experience in the language.

Have you ensured the data’s integrity?
 Yes, the data is consistent throughout the columns.

What steps have you taken to ensure that your data is clean?
  First duplicates, null values are removed then added new columns for analysis.

How can you verify that your data is clean and ready to analyze? 
 Make sure the column names are consistent thorough out all data sets by using the “bind row” function.

Make sure column data types are consistent throughout all the dataset by using the “compare_df_col” from the “janitor” package.
Combine the all dataset into single data frame to make consistent throught the analysis.
Removed the column start_lat, start_lng, end_lat, end_lng from the dataframe because those columns not required for analysis.
Create new columns day, date, month, year, from the started_at column this will provide additional opportunities to aggregate the data
Create the “ride_length” column from the started_at and ended_at column to find the average duration of the ride by the riders.
Removed the null rows from the dataset by using the “na.omit function”
Have you documented your cleaning process so you can review and share those results? 
  Yes, the cleaning process is documented clearly.

Analyze Phase:

Guiding Questions:

How should you organize your data to perform analysis on it? The data has been organized in one single dataframe by using the read csv function in R Has your data been properly formatted? Yes, all the columns have their correct data type.

What surprises did you discover in the data?
  Casual member ride duration is higher than the annual members
  Causal member widely uses docked bike than the annual members
What trends or relationships did you find in the data?
  Annual members are used mainly for commute purpose
  Casual member are preferred the docked bikes
  Annual members are preferred the electric or classic bikes
How will these insights help answer your business questions?
  This insights helps to build a profile for members

Share

Guiding Quesions:

Were you able to answer the question of how ...

Feedback: Mukharbek Organokov organokov.m@gmail.com

Context

Sloan Digital Sky Survey current DR16 Server Data release with Galaxies, Stars and Quasars.

License: Creative Commons Attribution license (CC-BY) More datailes here. Find more here.

Content

The table results from a query which joins two tables:
- "PhotoObj" which contains photometric data
- "SpecObj" which contains spectral data.

16 variables (double) and 1 additional variable (char) 'class'. A class object can be predicted from the other 16 variables.

Variables description:
objid = Object Identifier
ra = J2000 Right Ascension (r-band)
dec = J2000 Declination (r-band)
u = better of deV/Exp magnitude fit (u-band)
g = better of deV/Exp magnitude fit (g-band)
r = better of deV/Exp magnitude fit (r-band)
i = better of deV/Exp magnitude fit (i-band)
z = better of deV/Exp magnitude fit (z-band)
run = Run Number
rerun = Rerun Number
camcol = Camera column
field = Field number
specobjid = Object Identifier
class = object class (galaxy, star or quasar object)
redshift = Final Redshift
plate = plate number
mjd = MJD of observation
fiberid = fiberID

Comments

• A four-color UVGR intermediate-band photometric system (Thuan-Gunn astronomic magnitude system) is discussed in [1]. The Sloan Digital Sky Survey (SDSS) photometric system, a new five-color (u′ g′ r′ i′ z′) wide-band CCD system is described in [2]
• The variables 'run', 'rerun', 'camcol' and 'field' features which describe a field within an image taken by the SDSS. A field is basically a part of the entire image corresponding to 2048 by 1489 pixels. A field can be identified by: - run number, which identifies the specific scan, - the camera column, or "camcol," a number from 1 to 6, identifying the scanline within the run, and the field number. The field number typically starts at 11 (after an initial rampup time), and can be as large as 800 for particularly long runs. - An additional number, rerun, specifies how the image was processed.
• The variable 'class' identifies an object to be either a galaxy (GALAXY), star (STAR) or quasar (QSO).
  ####References:
  [1] Thuan & Gunn (1976, PASP, 88,543)
  [2] Fukugita, M. et al, Astronomical J. v.111, p.1748

Data server

Data can be obtained using SkyServer SQL Search with the command below:
-- This query does a table JOIN between the imaging (PhotoObj) and spectra
-- (SpecObj) tables and includes the necessary columns in the SELECT to upload
-- the results to the SAS (Science Archive Server) for FITS file retrieval.
SELECT TOP 100000
p.objid,p.ra,p.dec,p.u,p.g,p.r,p.i,p.z,
p.run, p.rerun, p.camcol, p.field,
s.specobjid, s.class, s.z as redshift,
s.plate, s.mjd, s.fiberid
FROM PhotoObj AS p
JOIN SpecObj AS s ON s.bestobjid = p.objid
WHERE
p.u BETWEEN 0 AND 19.6
AND g BETWEEN 0 AND 20

Learn how to. Some examples. Full SQL Tutorial.

Or perform a complicated, CPU-intensive query of SDSS catalog data using CasJobs, SQL-based interface to the CAS.

Acknowledgements

SDSS collaboration.

Inspiration

The Sloan Digital Sky Survey has created the most detailed three-dimensional maps of the Universe ever made, with deep multi-color images of one-third of the sky, and spectra for more than three million astronomical objects. It allows to learn and explore all phases and surveys - past, present, and future - of the SDSS.

The collection of datasets published contain the input parameters and output simulations from a single column 1D radiative transfer simulations using the Rapid Radiative Transfer Model for General Circulation Model (GCM) applications (RRTMG).

The simulations are focused on a selected case study of low-level-stratus clouds during the PS106 research cruise conducted in 2017 in the Central Arctic. The simulations are based on remote sensing observations, which were synergistically used with the Cloudnet algorithm to derive macro and microphysical properties of clouds. The atmospheric profiles of temperature, pressure, and ozone are from ERA5 (European Centre for Medium-Range Weather Forecasts (ECMWF) Re-Analysis) and values of surface albedo from CERES (Clouds and the Earth's Radiant Energy System) SYN1deg Ed. 4.1.

This is a collection of Offenses Known and Clearances By Arrest data from 1960 to 2016. The monthly zip files contain one data file per year(57 total, 1960-2016) as well as a codebook for each year. These files have been read into R using the ASCII and setup files from ICPSR (or from the FBI for 2016 data) using the package asciiSetupReader. The end of the zip folder's name says what data type (R, SPSS, SAS, Microsoft Excel CSV, feather, Stata) the data is in. Due to file size limits on open ICPSR, not all file types were included for all the data.

The files are lightly cleaned. What this means specifically is that column names and value labels are standardized. In the original data column names were different between years (e.g. the December burglaries cleared column is "DEC_TOT_CLR_BRGLRY_TOT" in 1975 and "DEC_TOT_CLR_BURG_TOTAL" in 1977). The data here have standardized columns so you can compare between years and combine years together. The same thing is done for values inside of columns. For example, the state column gave state names in some years, abbreviations in others. For the code uses to clean and read the data, please see my GitHub file here. https://github.com/jacobkap/crime_data/blob/master/R_code/offenses_known.R

The zip files labeled "yearly" contain yearly data rather than monthly. These also contain far fewer descriptive columns about the agencies in an attempt to decrease file size. Each zip folder contains two files: a data file in whatever format you choose and a codebook. The data file is aggregated yearly and has already combined every year 1960-2016. For the code I used to do this, see here https://github.com/jacobkap/crime_data/blob/master/R_code/yearly_offenses_known.R.

If you find any mistakes in the data or have any suggestions, please email me at jkkaplan6@gmail.com

As a description of what UCR Offenses Known and Clearances By Arrest data contains, the following is copied from ICPSR's 2015 page for the data.

The Uniform Crime Reporting Program Data: Offenses Known and Clearances By Arrest dataset is a compilation of offenses reported to law enforcement agencies in the United States. Due to the vast number of categories of crime committed in the United States, the FBI has limited the type of crimes included in this compilation to those crimes which people are most likely to report to police and those crimes which occur frequently enough to be analyzed across time. Crimes included are criminal homicide, forcible rape, robbery, aggravated assault, burglary, larceny-theft, and motor vehicle theft. Much information about these crimes is provided in this dataset. The number of times an offense has been reported, the number of reported offenses that have been cleared by arrests, and the number of cleared offenses which involved offenders under the age of 18 are the major items of information collected.

Petition subject: To abolish slavery in Washington D.C. and against the admission of Florida and slave states Original: http://nrs.harvard.edu/urn-3:FHCL:11857942 Date of creation: 1838-12-19 Petition location: Boston Legislator, committee, or address that the petition was sent to: George Bradburn, Nantucket Selected signatures:Samuel E. SewallCharles R. PettengillJames C. OdiornePerez GillJoseph NoyesOrin CarpenterJohn T. HiltonRobert MorrisCatherine BarbadoesChloe A. LeeWilliam C. NellEunice R. DavisPeter GrayHenry WeedenJoel W. Lewis Total signatures: 162 Legal voter signatures (males not identified as non-legal): 133 Female signatures: 12 Unidentified signatures: 17 Female only signatures: No Identifications of signatories: citizens, [females], [males of color], [females of color] Prayer format was printed vs. manuscript: Printed Signatory column format: not column separated Additional non-petition or unrelated documents available at archive: no additional documents Additional archivist notes: Appears that only the bottom two sections include females Location of the petition at the Massachusetts Archives of the Commonwealth: Senate Unpassed 1839, Docket 10525 Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.

Petition subject: Secession from the Union Original: http://nrs.harvard.edu/urn-3:FHCL:11029642 Date of creation: (unknown) Petition location: Waltham Legislator, committee, or address that the petition was sent to: Charles W. Wilder, Leominster; committee on the judiciary Selected signatures:Jarvis LewisRebecca FarwellLucy R. Stiles Actions taken on dates: 1848-02-09 Legislative action: Received in the House on February 9, 1848 and referred to the committee on the judiciary Total signatures: 10 Legislative action summary: Received, referred Legal voter signatures (males not identified as non-legal): 5 Female signatures: 5 Female only signatures: No Identifications of signatories: inhabitants, legal voters, non voters, [females], ["other persons"] Prayer format was printed vs. manuscript: Printed Signatory column format: column separated Additional non-petition or unrelated documents available at archive: no additional documents Location of the petition at the Massachusetts Archives of the Commonwealth: House Unpassed 1848, Docket 2122 Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.