ohdsi / featureextraction

An R package for generating features (covariates) for a cohort using data in the Common Data Model.

FeatureExtraction

FeatureExtraction is part of HADES.

Introduction

An R package for generating features (covariates) for a cohort using data in the Common Data Model.

Features

• Takes a cohort as input.
• Generates baseline features for that cohort.
• Default covariates include all drugs, diagnoses, procedures, as well as age, comorbidity indexes, etc.
• Support for creating custom covariates.
• Generate paper-ready summary table of select population characteristics.

Technology

FeatureExtraction is an R package, with some functions implemented in C++.

System Requirements

Requires R (version 3.2.2 or higher). Installation on Windows requires RTools. FeatureExtraction require Java.

Getting Started

1. See the instructions here for configuring your R environment, including RTools and Java.

2. In R, use the following commands to download and install FeatureExtraction:

install.packages("drat")
drat::addRepo("OHDSI")
install.packages("FeatureExtraction")

User Documentation

The documentation website can be found at https://ohdsi.github.io/FeatureExtraction/. PDF versions of the vignettes and package manual are here:

• Vignette: Using FeatureExtraction
• Vignette: Creating covariates using cohort attributes
• Vignette: Creating custom covariate builders
• Vignette: Creating covariates based on other cohorts
• Package manual: FeatureExtraction manual

These vignettes are also available in Korean:

• Vignette: Using FeatureExtraction
• Vignette: Creating custom covariate builders

Support

• Developer questions/comments/feedback: OHDSI Forum
• We use the GitHub issue tracker for all bugs/issues/enhancements

Contributing

Read here how you can contribute to this package.

License

FeatureExtraction is licensed under Apache License 2.0

Development

FeatureExtraction is being developed in R Studio.

Development status

Ready for use

Acknowledgements

• This project is supported in part through the National Science Foundation grant IIS 1251151.

featureextraction's Issues

#cov_m_count365d cleanup

#cov_m_count365d seems to be missing from the clean up at the end and this causes an error when running the covariate extraction more then once

DELETE

Same description for different arguments

In package manual for FeatureExtraction both
useDemographicsPriorObservationTime
and
useDemographicsPostObservationTime
have the same description

Distribution values for time at risk, prior observation, post observation

I wasn't able to find code in version_2 about creating distributions for time at risk (cohort_start->cohort_end), prior observation (op_start -> cohort_start) or post observation (cohort_start->op_end). We'd want to generate the quartile ranges (including min, p10, p25, median, p75, p90, max) for these continuous values.

Deadlock in case of exception in init method

Commands between lock and unlock should be wrapped into try-catch-finally so that we don't face deadlock / switch to synchronized method

issue with latest SqlRender

in line 102 in GetCovariates.R the code:
sql <- SqlRender::translateSql(sql = sql,
targetDialect = attr(connection, "dbms"),
oracleTempSchema = oracleTempSchema)$sql

is causing issues in the PLP package with the new SqlRender as it returns:
Warning: 'SqlRender::translateSql' is deprecated.
Use 'translate' instead.
See help("Deprecated")

FeatureExtraction 2 - DemographicsTime - continuous observation period

For sub_type = 'priorObservation' or sub_type = 'postObservation' in DemographicsTime.sql -- the output is days of continuous observation period in contiguous periods. i.e. if a person_id has multiple disjointed observation_periods, the disjointed periods are not all added. Only the intervals overlapping with the cohort_start_date are used in the calculation.

Could we please change label in PrespecAnalyses to say this is Time in contiguous observation-period. e.g. 'Number of continuous days of observation time preceding the index date.'

The OBSERVATION_PERIOD table contains records which uniquely define the spans of time for which a Person is at-risk to have clinical events recorded within the source systems
One Person may have one or more disjoint observation periods, during which times analyses may assume that clinical events would be captured if observed, and outside of which no clinical events may be recorded.
Each Person can have more than one valid OBSERVATION_PERIOD record, but no two observation periods can overlap in time for a given person.

What is the purpose of this code in the covariate settings builders?

As far as I can tell this code is unnecessary as the code preceding it already creates a list out of the function parameters

https://github.com/OHDSI/FeatureExtraction/blob/master/R/GetDefaultCovariates.R#L517-L521

I would also be interested in getting clarity as to why the covariateSettings builders take a long list of function parameters, stitches them into a list, and then explode them when passing them to loadRenderTranslateSql. Is the reason just to reduce the number of parameters on the getCovariateData functions?

https://github.com/OHDSI/FeatureExtraction/blob/master/R/GetDefaultCovariates.R#L90-L152

Preventing calculation of covariates on index date

I used createCovariateSettings() within Atlas to generate a cohort study. The desired behavior is to not compute covariates for propensity scoring on the index date. I assumed setting useCovariateDrugEraOverlap = FALSE would do that. However, when I run propensity scoring code, it gives errors about a high correlation between covariates(s) and treatment, listing the drugs that are my treatment and comparator drugs as the culprits. I specified a 365-day pre-treatment observation period, limiting events to the earliest event per person. My assumption is that I would have 365 days of no treatment, followed by treatment on the index date (perhaps this is wrong?). It would seem to me that the problem is that the treatment on the index date is what is picked up as the high correlation.

If I list my treatment and comparator drugs in the excluded concepts, the high correlations go away, but I'm concerned other variables such as Conditions and Observations on the index date are also getting into the propensity model.

Here is the documentation for useCovariateDrugEraOverlap:

useCovariateDrugEraOverlap: A boolean value (TRUE/FALSE) to determine if covariates will be created and used in models that look for presence/absence of drug era that overlaps the cohort index date. Only applicable if useCovariateDrugEra = TRUE.

If this is in error, I would question if useCovariateConditionEraOverlap works. Also, for symmetry, it does not appear that there is a way to exclude Observations, Procedures, or Measurements via analogous "Overlap" parameters for those covariates.

Below is the JSON code defining one of my cohorts:

{
  "ConceptSets": [
    {
      "id": 0,
      "name": "Bipolar Disorder",
      "expression": {
        "items": [
          {
            "concept": {
              "CONCEPT_ID": 436665,
              "CONCEPT_NAME": "Bipolar disorder",
              "STANDARD_CONCEPT": "S",
              "INVALID_REASON": "V",
              "CONCEPT_CODE": "13746004",
              "DOMAIN_ID": "Condition",
              "VOCABULARY_ID": "SNOMED",
              "CONCEPT_CLASS_ID": "Clinical Finding",
              "STANDARD_CONCEPT_CAPTION": "Standard",
              "INVALID_REASON_CAPTION": "Valid"
            },
            "includeDescendants": true
          }
        ]
      }
    },
    {
      "id": 1,
      "name": "Lithium",
      "expression": {
        "items": [
          {
            "concept": {
              "CONCEPT_ID": 751246,
              "CONCEPT_NAME": "Lithium Carbonate",
              "STANDARD_CONCEPT": "S",
              "INVALID_REASON": "V",
              "CONCEPT_CODE": "42351",
              "DOMAIN_ID": "Drug",
              "VOCABULARY_ID": "RxNorm",
              "CONCEPT_CLASS_ID": "Ingredient",
              "STANDARD_CONCEPT_CAPTION": "Standard",
              "INVALID_REASON_CAPTION": "Valid"
            },
            "includeDescendants": true
          }
        ]
      }
    },
    {
      "id": 2,
      "name": "Drug Lithium",
      "expression": {
        "items": [
          {
            "concept": {
              "CONCEPT_ID": 751246,
              "CONCEPT_NAME": "Lithium Carbonate",
              "STANDARD_CONCEPT": "S",
              "INVALID_REASON": "V",
              "CONCEPT_CODE": "42351",
              "DOMAIN_ID": "Drug",
              "VOCABULARY_ID": "RxNorm",
              "CONCEPT_CLASS_ID": "Ingredient",
              "STANDARD_CONCEPT_CAPTION": "Standard",
              "INVALID_REASON_CAPTION": "Valid"
            },
            "includeDescendants": true
          }
        ]
      }
    }
  ],
  "PrimaryCriteria": {
    "CriteriaList": [
      {
        "DrugEra": {
          "CodesetId": 2
        }
      }
    ],
    "ObservationWindow": {
      "PriorDays": "365",
      "PostDays": 0
    },
    "PrimaryCriteriaLimit": {
      "Type": "First"
    }
  },
  "AdditionalCriteria": {
    "Type": "ANY",
    "CriteriaList": [
      {
        "Criteria": {
          "ConditionOccurrence": {
            "CodesetId": 0
          }
        },
        "StartWindow": {
          "Start": {
            "Coeff": -1
          },
          "End": {
            "Coeff": 1
          }
        },
        "Occurrence": {
          "Type": 2,
          "Count": 2
        }
      }
    ],
    "DemographicCriteriaList": [],
    "Groups": []
  },
  "QualifiedLimit": {
    "Type": "All"
  },
  "ExpressionLimit": {
    "Type": "All"
  },
  "InclusionRules": [],
  "EndStrategy": {
    "CustomEra": {
      "DrugCodesetId": 2,
      "GapDays": "30",
      "Offset": 0
    }
  }
}

Support for observation / measurement values

I am wondering if there is a way today to create distinct covariates for Observations / Measurements based on values in the records e.g. value_as_concept_id. In other words, not a covariate per each observation_concept_id but a value per each combination of observation_concept_id and value_as_concept_id (e.g. observation_concept_id = 3022698 Stage group. Clinical cancer, value_as_concept_id = Stage 0 or Stage I or Stage II).

Delete

Parallelization of tidyCovariateData()

Would it be possible to parallelize this function?

For very large data sets (>500000 by 50000 data table), and especially for temporal covariates with small time window, the function takes relatively long to compute. We have a powerful machine (56 threads and some 700 GB of memory) and it would be great to be able to scale to that.

Add createDefaultCovariateSettings to java part

That would be great to have createDefaultCovariateSettings java method like createCovariateSettings R function. This is needed to complete OHDSI/WebAPI#975

Distinct Observation Count

Please add a setting to allow for the distinct observation count Long, Medium, Short as is provided for useDistinctConditionCountLongTerm, useDistinctIngredientCountLongTerm, etc.

useDemographicsTimeInCohort error: Invalid column name 'cohort_end_date'

When using useDemographicsTimeInCohort = TRUE, I am getting a SQL error for
Invalid column name 'cohort_end_date'.

Probably due to unavailability of the field in this select:
https://github.com/OHDSI/FeatureExtraction/blob/master/inst/sql/sql_server/DemographicsTime.sql#L43

table1.R bug

I'm having an issue with https://github.com/OHDSI/FeatureExtraction/blob/master/R/Table1.R#L140
There is only one continuous covariate, so the colnames() function fails as it is a vector. I think converting it to a data.frame should fix the issue: tempCovariates <- as.data.frame(tempCovariates)

I just tried this in debug but then I also have another issue later on: Error in data.frame(..., check.names = FALSE) :
arguments imply differing number of rows: 9, 11

Number of inpatient visits

Please add a feature to allow for Long, Medium, Short term covariates for number of inpatient visits.

Add covariates for provider specialty

Moving issue from PLP to Fe: Could have # of visits in past 365d by specialty, and binary variable for specialty on index date. Serves as interesting proxy for health behavior, but may also get at provider level effects (particularly important if interventions are services or device related)

FeatureExtraction2 - Support place_of_service_concept_id

@schuemie

place_of_service_concept_id is on the care_site table. Visit_occurrence and visit_detail may be joined to care_site using FK care_site_id. Condition_occurrence, procedure_occurrence, drug_occurrence may be joined to care_site thru visit_occurrence table.

Use-case: counts at home, emergency room, in-patient, out-patient, doctor's office. These are either concepts or concept-sets/covariates of place_of_service_concept_id.

I think using DomainConcept.sql would make it complex - e.g. in DomainConcept.sql we currently have

	FROM @cohort_table cohort
	INNER JOIN @cdm_database_schema.@domain_table
		ON cohort.subject_id = @domain_table.person_id

when @domain_table is visit_occurrence it needs to become something like

	FROM @cohort_table cohort
	INNER JOIN @cdm_database_schema.@domain_table
		ON cohort.subject_id = @domain_table.person_id
        INNER JOIN @cdm_database_schema.care_site
               ON @domain_table.care_site_id = care_site.care_site_id

when @domain_table is condition_occurrence it needs to become something like

	FROM @cohort_table cohort
	INNER JOIN @cdm_database_schema.@domain_table
		ON cohort.subject_id = @domain_table.person_id
        INNER JOIN @cdm_database_schema.visit_occurrence
               ON @domain_table.visit_occurrence_id = visit_occurrence.visit_occurrence_id
        INNER JOIN @cdm_database_schema.care_site
               ON visit_occurrence.care_site_id = care_site.care_site_id

What would be the better way of handling place_of_service_concept_id?

FeatureExtraction2: Impala subqueries in select list fails

There is an error when trying to generate cohort with feature on Impala.
Impala reports error: Subqueries are not supported in the select list

Integer out of range error MeasurementValue.sql

On PostGresql we get an integer out of range error due to this line.

FeatureExtraction/inst/sql/sql_server/MeasurementValue.sql

Line 6 in b7c8cbc

(measurement_concept_id * 1000000) + ((unit_concept_id - (FLOOR(unit_concept_id / 1000) * 1000)) * 1000) + @analysis_id AS covariate_id

Do we need to multiply with 1000000?

Excluded concepts should be removed from ancestor_concept_id AND descendant_conept_id in 'group' queries

Right now, for drug groups, we exclude 'excluded-concept_ids' from both ancestor and descendent concept-id. but for condition groups, we appear to only exclude ancestor ids (not descendant concept_id).

we should add exclusion on descendant_concept_id

https://github.com/OHDSI/FeatureExtraction/blob/master/inst/sql/sql_server/DomainConceptGroup.sql

Charlson Score Myocardial Infarction

On line #1955 the CONCEPT_ID 329847 is used to define myocardial infarction when I think it should be 4329847.

Aggregate covariates to get summary statistics

I am looking for a way to get Achilles-like summary data from the covariates produced by getDbCovariateData().
To show what I mean, I made a small script that aggregates the covariate data from two cohorts and counts the number of occurrences. See output in the table below.

My question is, can this already be done by any in-build functions of this package? Or do you have other suggestions to extract this information using OHDSI tools?

	covariateName	occurrences.cohort1	occurrences.cohort2
1	Age group: 35-39	3	1
2	Age group: 40-44	2	8
3	Age group: 45-49	9	45
4	Age group: 50-54	13	50
5	Age group: 55-59	37	55
6	Age group: 60-64	107	98
7	Age group: 65-69	486	234
8	Age group: 70-74	3112	2031
9	Age group: 80-84	4045	3451
10	Age group: 85-89	501	1234
11	Age group: 90-94	81	55
12	Age group: 95-99	14	10
13	Gender = MALE	6167	9317
14	320128-Essential hypertension	725	1743
15	4193704-Type 2 diabetes mellitus without complication	252	313

Produced for each cohort by:
agg_counts <- aggregate( covariateValue ~ covariateId, FUN=length, data = cov )

createCovariateSettings error

I get the following error when useCovariateMeasurementBelow is set to true: execute JDBC update query failed in dbSendUpdate (Invalid object name '#cov_m_below'.). createCovariateSettings runs fine when I set measurement covariates to FALSE. I haven't tested to see if this is the only problematic measurement covariate.

Schema support for sql server

I'm running into an issue where I cannot run cohort method on sql server where my ohdsi instance and vocabulary exist in a schema that is not dbo. I have tracked down the issue to two places:

1. getCovariates.sql may need to prefix the table names with @cdm_database in order to fully support schemas on sql server. The "USE" statement at the beginning of the script only puts the database in scope, not the schema.

2. Line 43 in GetDefaultCovariates.R splits the database name on periods and therefore removes the schema name. The particular line of code is on line 43 and states:

cdmDatabase <- strsplit(cdmDatabaseSchema, ".")[[1]][1]

The database name without the schema name is then passed on to sqlrender. Therefore, changing getCovariates.sql may not be enough to fully support schemas on sql server.

Improve performance on massive parallel architectures

Currently covariates are created by SELECT INTO statements creating temp tables. On massive parallel architectures this currently results in replicates of these temp tables across all nodes because no distribution key is provided.

We could add hints to distribute on row_id, but ideally we'd like to distribute on subject_id since that is the distribution key most likely used for all the CDM data. However, subject_id is currently not stored in the temp tables. Need to experiment to see if it helps to include it just to use as a distribution key.

Generate all two-way interactions

Requires a small bit of SQL code.

Add way to recreate a set of covariates in another database

Currently, if FeatureExtraction creates covariates (e.g. using the default covariate builder), it removes redundant variables in the set. For example, there might variables for female and male, but one of them would be redundant, which for some model fitting techniques would cause problem. The current behavior is to remove the most prevalent variable that will eliminate the redundancy, but in another database a different variable might be the most prevalent one, or the redundancy might not even exist. This poses a problem if we want to apply a model fitted in one database to another database.

The solution should allow the subset of covariates to be built to be specified upfront. This could also help efficiency: if my fitted model only has 100 covariates with non-zero coefficients, I will only need to build those 100 covariates to apply that model in another database.

Index year is set to incorrect domain_id

When generating features with FE2.0 and you include the feature 'Index Year', the resulting features for the index year have a domain_id value of 'domain_id'. This should be 'Demographics' like the 'Index Month' feature.

Long-term inpatient visits not found in EHR data

We had an internal discussion that I'm summarizing here for further discussion:

When using FeatureExtraction, we noticed that analysis 106 (Long-term inpatient visits) returned 0 patients on our EHR data set versus our claims data. In tracing through the code in FE v2, I noted the following:

Analysis settings for analysisId 106: https://github.com/OHDSI/FeatureExtraction/blob/master/inst/csv/PrespecAnalyses.csv#L18. Note the subType column and for analysisId = 106, the value = 'inpatient'. Furthermore, the sqlFileName setting = 'DomainConcept.sql'. These settings allow us to drill into the details of the the SQL file: https://github.com/OHDSI/FeatureExtraction/blob/master/inst/sql/sql_server/DomainConcept.sql#L38

{@sub_type == 'inpatient'} ? { AND condition_type_concept_id IN (38000183, 38000184, 38000199, 38000200)}

These condition_type_concept_ids map to:

• Inpatient detail - primary
• Inpatient detail - 1st position
• Inpatient header - primary
• Inpatient header - 1st position

However, in our EHR data, our condition types are:

• Primary Condition (44786627)
• Secondary Condition (44786629)

Some ideas that were discussed:

• The name of the analysis may need to be revised to "primary diagnosis in an inpatient setting"
• The condition type codes referenced above for the EHR data imply a primary/secondary diagnosis and not an inpatient code - to obtain this we may need to join this to the visit_occurrence where the visit_concept_id = 9201 (Inpatient Visit). However, it may be the case that a condition cannot be linked to a visit and so we need to be careful if we go with this approach.

I'd vote to attempt to continue to use the condition_type_concept_ids that are currently in the DomainConcept.sql with the change in the analysis label. Perhaps we could add a new, additional analysis that joins the visit_occurrence & condition_occurrence tables together to find the inpatient records?

Number of ER visits

Please add a feature to allow for Long, Medium, Short term covariates for number of ER visits.

Current version doesn't work for observation, device, and measurement.

Hello, @schuemie

Current version doesn't work for observation, device, measurement, and indexes such as CHADS.
When I included these terms, error occurred

r in nchar(object, type = "char"): invalid multibyte string, element 4

I'm using CDM version 5.3 with SQL Server. Do you have any idea about why it doesn't work..?

The link for vignette of 'Creating custom covariate builders' in readMe file doesn't work

@schuemie
The link for vignette of 'Creating custom covariate builders' in readMe file doesn't work, now.

Order by clause in GetCovariates.sql

Hi,
In the GetCovariates.sql file, we have some insert statements that also include "order by" clauses:

FeatureExtraction/inst/sql/sql_server/GetCovariates.sql

Lines 2245 to 2281 in 119ca34

	INSERT INTO #DCSI_scoring (
	DCSI_category,
	DCSI_ICD9_code,
	DCSI_concept_id,
	DCSI_score
	)
	SELECT 'Nephropathy' AS DCSI_category,
	source_code,
	target_concept_id,
	1 AS DCSI_score
	FROM (
	{@cdm_version == '4'} ? {
	SELECT source_code, target_concept_id
	FROM @cdm_database_schema.SOURCE_TO_CONCEPT_MAP
	WHERE source_vocabulary_id= 2
	AND target_vocabulary_id = 1
	} : {
	SELECT
	source.concept_code AS source_code,
	target.concept_id AS target_concept_id
	FROM @cdm_database_schema.concept_relationship
	INNER JOIN @cdm_database_schema.concept source
	ON source.concept_id = concept_relationship.concept_id_1
	INNER JOIN @cdm_database_schema.concept target
	ON target.concept_id = concept_relationship.concept_id_2
	WHERE LOWER(source.vocabulary_id) = 'icd9cm'
	AND LOWER(target.vocabulary_id) = 'snomed'
	AND LOWER(relationship_id) = 'maps to'
	}
	) source_to_concept_map
	WHERE
	source_code IN ('250.4', '580', '581', '581.81', '582', '583')
	OR source_code LIKE '580%'
	OR source_code LIKE '581%'
	OR source_code LIKE '582%'
	OR source_code LIKE '583%'
	ORDER BY source_code;

That's not valid in PDW (and I think in most other dialects?):
The ORDER BY clause is invalid in views, CREATE TABLE AS SELECT, INSERT SELECT, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.)

Does the ordering of these temp tables by source_code actually matter? Can the order by clauses be removed?

Thanks,
Ajit

Condition Group covariate bug

I found the following two errors in the code:

in GetCovariates.sql line 616:

AND c1.@concept_class_id = 'Clinical finding'
AND ca1.min_levels_of_separation = 1
AND c1.concept_id NOT IN (select distinct descendant_concept_id from concept_ancestor where ancestor_concept_id = 441840 and max_levels_of_separation <= 2)

Should be (see bold)

AND c1.@concept_class_id = 'Clinical Finding'
AND ca1.min_levels_of_separation = 1
AND c1.concept_id NOT IN (select distinct descendant_concept_id from @cdm_database_schema.concept_ancestor where ancestor_concept_id = 441840 and max_levels_of_separation <= 2)

The first error is probably because of a vocab update? Maybe we should always cast to uppercase when doing string comparisons in SQL??

The second error is not a problem if cdm is the default schema only..

Not all age groups captured in balance file

Hi @schuemie,

Seeing some strange behavior with my balance file when running runCmAnalyses. Although my cohorts both pre- and post-matching have many people who are age 60 to 64, this age group isn't captured in my balance file. I'm posting this here in case anyone else has seen this issue, but will follow up with you via email as troubleshooting will require examining our licensed data.

Tagging @jamieweaver

Thanks,
Ajit

FeatureExtraction 2 - Demographics - Is there a way to calculate distinct counts by subject_id?

Is there a way to calculate distinct counts by subject_id?

A cohort is defined as the set of persons satisfying one or more inclusion criteria for a duration of time. One person may qualify for one cohort multiple times during non-overlapping time intervals.

Cohort definition allows a subject_id to be present more than once in a cohort. Can we support something like COUNT(distinct subject_id) to calculate unique subjects?

Example in DemographicsGender.sql
When aggregate is chosen in FeatureExtraction for DemographicsGender, the current version performs COUNT(*). This will count the number of records, not number of unique subjects.

FeatureExtraction/inst/sql/sql_server/DemographicsGender.sql

Line 8 in 9f78454

COUNT(*) AS sum_value

One option is to change COUNT(*) to COUNT(subject_id) by default, and create a sub_type for distinct COUNT(distinct subject_id)

delete

Atlas - Generate with Reports

Expected behavior

Feature Extraction based script should calculate features for during cohort period or at-risk period

Actual behavior

Feature extraction based script only calculate for baseline period (long-term and short-term) window

Request

Please include ability to report on characteristics of the cohort between cohort_start_date and cohort_end_date.

FeatureExtraction2 - wish list

@schuemie could we add the following to the PrespecAnalyses list?

Add option to automatically include all descendants of included and excluded concepts

When done at design-time, expanding to all descendants could lead to problems because the vocab version might differ from the one used at run-time.

By default, this option should be on, including all descendants.

Allow users to specify time windows for covariate capture

This issue is related to this issue: Currently, the default covariate builder always includes events recorded on the index date, and has pre-specified lookback windows such as '30 days prior', or '365 days prior'.

We could parameterize these windows, allowing the user to define the 'short', 'medium', and 'long' lookback windows in terms of start and end dates (relative to the index date). The covariates themselves could then references these window definitions, for example 'All conditions observed in the short lookback window'.

Grouping Variables

Allow for creating group variables without requiring descendant concepts

procedure group

Error at line 1278 (exclusion of codes for procedure grouping):
selected concept_id's (76094,67368, 46042, 40949, 31332, 28263, 24955, 18791, 13449, 12571, 10678, 10592, 9878, 9727, 9652, 9451, 9192, 8975, 8930, 8786, 8370, 8161, 7763, 7059, 6923, 6752, 6690, 6611, 6336, 6264, 6204, 6003, 5783) are not procedures at all.
I expected to see there concepts like
4014829-'Consultation' or 4241075-'Injection'.
Also seems like you should collaborate with standardized vocabularies team to improve this part.
There are no proper relationships between different vocabularies for procedures. So, I'm not sure
all of procedures could be aggregated to SNOMED concepts

delete

Support branch for FeatureExtraction v1

@schuemie,

Would be grateful, if you could create a support branch for FeatureExtraction v1.2. The reason is following: we are working on bringing in Impala support into Atlas, and Atlas generates PLE R code based on FeatureExtraction v1.2.3. But there are some SQL statements inside the package, which cannot be executed by Impala in as-is state and require some SQL transformations, but these transformations are too specific to put them into SqlRender, so we would like to adopt these SQL here, inside FeatureExtraction.

Example of required transformation:

CASE WHEN v1.concept_name IS NOT NULL THEN CONCAT('Gender = ', v1.concept_name) ELSE CONCAT('Gender = ', 'Unknown invalid concept') END

into

CAST(CONCAT(CAST('Gender = ' AS VARCHAR), coalesce(v1.concept_name, 'Unknown invalid concept')) AS VARCHAR(512))

and

@a as stratum_1

into

CAST(@a AS VARCHAR(255)) as stratum_1

time_id is null in Demographic covariates for temporal covariates

I found that time_ids of demographic covariates created by temporal setting are nulls in current version of feature extraction package.
In this case, CIReNN module in PLP cannot work (because timeId is null)..

Default Measurement table

Hello! Quick question: why is the default measurement table used in GetCovariates.sql set to the Observation table rather than the Measurement table? Is this a CDM v4 artifact?

wrong concept_id

error at row 618 :
'select distinct descendant_concept_id from concept_ancestor where ancestor_concept_id = 441480...'
concept_id 441480 is id not for clinical finding. I suppose it should be changed to 441840