astrochun / evolution-of-galaxies

Research with Dr. Chun Ly. I am working with a sample of ~4000 galaxies from the MACT and DEEP2 surveys. The goal is to determine a robust M-Z-SFR relation for galaxies at z<1 and examine its evolution over the past 8 billion years of cosmic time.

👋. I'm Chun!

Background

🔭 I'm a research astrophysicist by trade for twenty years.

Currently I work at the Princeton University Plasma Physics Laboratory as a Research Data Developer.

These days, I build Python software for:

1. Data analysis and visualization
2. Supporting open-access initiatives such as data and article publishing

Open source development

I enjoy coming up with new coding projects in my free time. Most recently I made or am working on:

1. A streamlit app to interact with U.Arizona public salary data: https://sapp4ua.herokuapp.com
2. A GitHub Docker container action to gather GitHub traffic statistics and publish them static pages on GitHub Pages or on a web server: https://astrochun.github.io/github-stats
3. A webscraping tool to extract voting records from VoxCharta.
4. A Python tool to properly handle BibTeX records for my Hugo Academic site

U.Arizona and the ReDATA project

Previously I was the lead developer for the following software while working at U.Arizona on the ReDATA project:

1. redata-docs - Sphinx documentation on the ReDATA software and systems.
2. LD-Cool-P - Pure Python tool for data curation of figshare for Institutions submissions.
3. ldcoolp-figshare - A generalized pure Python tool for data curators/administrators to interact with their figshare for Institutions instance through the figshare API.
4. ReQUIAM - Pure Python tool for Grouper identity and access management.
5. figstats - Pure Python tool to retrieve statistics of published datasets from the figshare stats API.
6. redata-commons - Pure python libraries for all ReDATA software.
7. ReM - A FastAPI web app for the construction of README.txt for ReDATA submissions.

Astrophysics software development

As a research scientist, I built a number of software for astronomical data reduction. This includes:

1. MMTtools - Pure Python tool to automate MMT/MMIRS data reduction. You can read more here.
2. GNIRSLongSlit - Python tool to reduce Gemini/GNIRS data. You can read more here.

More information can be found on my GitHub page.

evolution-of-galaxies's Issues

Common Validation Table

Implement common validation table code from Metallicity Stack Commons to be consistent with Zcalbase_gal.

Scripts for easier automation of data analysis workflow

Commands are provided manually into an interactive ipython terminal.

Need to move toward having a folder with scripts to execute any data analysis operation.

Action items

• [ ]Create a script folder
• Create script for stacking(?)
• Create script for adaptive biinning

Error Propagation

• Implement MSC's error prop code
• Plot error bars

Adding logging tool for all print statements from analysis

Using similar codes that I have, create a LogClass object that enable easy logging.

Action items:

• Create LogClass
• Update all print statements
• Ensure all functions allow for LogClass object input

This should work with the MSC feature/48_logging enhancement

Refactor emission_line_fit.py to Incorporate Dictionaries

• Store emission line information in dictionaries to be easily saved to output files
• Reduce the number of variables for less confusion
• Reduce plot formatting code

Fix Windows compatibility issue with os.uname

As @cmccormick1 determined, os does not support Windows. However, platform does. So we will go with a simple switch

Integrate Metallicity_Stack_Commons in R_temp_calc

Import Metallicity_Stack_Commons.temp_metallicity_calc for the following functions:

1. R_calculation()
2. temp_calculation()
3. metallicity_calculation()

Refactor library.py to Incorporate Dictionaries

-Store bin information in dictionaries to be easily saved to output files
-Reduce the number of variables to make for less confusion

Integrate Metallicity_Stack_Commons in general

Changes to implement:

Refactoring:

1. Import get_user to replace getuser()
2. Import dir_date to replace get_time
3. Import fitting_lines_dict to replace lambda0, line_type, and line_name
4. Import k_dict to replace k_4363 and k_5007
5. Import attenuation to replace dust_attenuation

Integrate Metallicity_Stack_Commons in emission_line_fit

Import Metallicity_Stack_Commons.fitting to retrieve:

1. movingaverage_box1D()
2. gauss()
3. double_gauss()
4. oxy2_gauss()
5. rms_func()
6. con1

Common Table Names

Update column names in tables to common names provided in Metallicity_Stack_Commons to be consistent with MZEvolve and Zcalbase_gal.

Tables to be changed:

• binning.tbl --> implement:
  - bin_names0 (excluding "Detection")
  - bin_mzevolve_names0
  - filename_dict
• emission_line.tbl --> implement:
  - bin ID
  - gauss_lines_names0
  - RMS column
  - filename_dict
• emission_linerevised.tbl --> implement:
  - bin ID
  - gauss_lines_names0
  - RMS column
  - filename_dict
• derived_properties_metallicity.tbl --> implement:
  - bin ID
  - bin_ratios0
  - temp_metal_names0
  - filename_dict
• derived_properties_metallicityrevised.tbl --> implement:
  - bin ID
  - bin_ratios0
  - temp_metal_names0
  - filename_dict
• validation.tbl --> implement:
  - bin ID
  - detections
  - OIII4363 values
  - filename_dict
• individual_Te_emLines.tbl --> implement:
  - indv_names0
  - gauss_names0
  - filename_dict
• individual_derived_properties_metallicity.tbl --> implement:
  - composite_indv_detect.py
  - filename_dict

Tables to be created:

• individual_bin_info.tbl

Bug: Incorrect curve fitting for metallicity vs mass individual plot

After plotting functions were reorganized, the curve fit for composite measurements on the individual metallicity vs mass plot differed from past plots.

Reorganize Files into Sub-Folders

Need to:
- Create an analysis folder
- Create a plotting folder (see issue #25)
- Update import statements to reflect new file structure

Bug with saving fitting results

max0 is stored in the table and not the normalization. See:

Evolution-of-Galaxies/Analysis/emission_line_fit.py

Line 195 in 4a2c622

norm_array[rr] = max0

Implement Individual Detections

Determine composite individual galaxy detections for analysis.

Fix mistake in emission_line_fit.gauss() function with sigma parameter

@cmccormick1 found a bug. There's a factor of 2 missing in the denominator of the exponential function in gauss()

We have:

def gauss(x, xbar, s, a, c):
     return a * np.exp(-(x - xbar)**2 / s**2) + c

This should be:

def gauss(x, xbar, s, a, c):
     return a * np.exp(-(x - xbar)**2 /  (2 * s**2)) + c

Update R_temp_calcul.py

Need to:

• Implement MSC's ratios.py for line ratio calculations
• Clean up code (e.g. get rid of commented out code)
• Update documentation

Reorganize Function Call Sequence (general.py)

On feature/call_sequence:

This branch will handle updating Evolution-of-Galaxies with newly developed file naming conventions in Metallicity-Stack-Commons, input data file reorganization in Google Drive, and general improvements to codes that may cause breaks with the current version of Evolution-of-Galaxies.

To Do:

• Update naming conventions for file names (e.g. v1 added to denote original validation table)
• Update file paths to input data files on Google Drive
• In general.run_bin_analysis():
  • Add option to run result plots for each bin_derived_prop case (MC, dust, revised validation, etc)
  • Change parameter name from dust_atten to apply_dust
  • Move HbHgHd_fits() call to right after fitting call (only run once with raw data)

Archive Old/Unused Files

Create a separate folder to store the unused files to better organize the repository.

Unused Files:

• Composite Spectra Grid.ipynb
• binning_mag_mass.py
• composite_spectra.pdf
• galaxy_distribution.pdf
• hgamma_to_hbeta.py
• mass_distribution.PNG
• stack_check.py

Integrate Metallicity_Stack_Commons in library

Import Metallicity_Stack_Commons init for:

1. exclude_outliers()

library: relative indexing

Versions after 4/1/19 through 10/23/19 have incorrect relative indexing (starting in 8668ef7). The indices in those versions are the indices of the elements in the valid index array, not the actual valid index elements themselves. Versions before 4/1/19 and after 10/23/19 (fixed in 10/24/19 version: 979c84f) have indices of sources relative to passed in data table. The change that fixed it is in lines 101-103 in 979c84f.

Update indiv_gals.py

Need to:
- Get rid of unnecessary parameters to functions
- Redo documentation
- Handle reading in revised/MC/dust files
- Utilize dictionaries to reduce the number of variables needed

GitHub Action for new releases with tagging

Use actions/create_release to create releases with new tags.

See: https://github.community/t/how-to-run-github-actions-workflow-only-for-new-tags/16075/10

This can be implemented as hotfix/v0.8.1

Implement Dust Attenuation

On feature/dust_attenuation:

• Implement E(B-V) calculation [9e80b9c]

• Fix naming convention for error columns in relation_fitting.extract_error_bars (lowhigh_error --> error) [5ecd19c, 31a2f83]

• Get rid of the call to MSC's attenuation.EBV_table_update. Now handled in error_prop.fluxes_derived_prop [3a7cd22]

• Implement dust attenuation and error propagation using MSC's error_prop.fluxes_derived_prop. Test:

  • Raw data
    • Mass bins
    • Mass-LHbeta bins
  • Raw data, dust corrected
    • Mass bins
    • Mass-LHbeta bins
  • MC, original validation table
    • Mass bins
    • Mass-LHbeta bins
  • MC, revised validation table
    • Mass bins
    • Mass-LHbeta bins
  • MC, dust corrected, revised validation table
    • Mass bins
    • Mass-LHbeta bins

Reorganize Plotting Functions

Currently all plotting functions are in the plots.py file. Need to clean up and organize the plotting functions into their own files based on different types of plots:

• Make a file just for individual plots
• Make a file just for composite plots
• Create a file for unused plotting functions and move to Archives folder